[WIPz] TES5Edit

[hlp]

I ain't gonna be available today, we might as well wrap it up.

 

That's the best way to find all the remaining issues anyway...

[DayDreamer]

The only viable method is to move those triangles out of play, but if they're somehow connected only on one edge, that won't be possible without ruining the rest of the navmesh. How the hell did Beth's code allow that?

Apparently, older CK allowed a lot of bad things. I've found a large number of places where triangles attached by only 1 vertex are in the same navmesh. As soon as I save, it is split to a second navmesh. (Good thing I keep a lot of backups, so I can go back and add another side before saving.)

 

Another problem is that auto-navmeshing must have in the past made them all with the same navmesh number per cell. If they have errors, it stayed that way in the released game. After I fix all the errors, they are split into separate navmeshes on the next save!

 

So far, HamvirsRestExterior01 has been the worst. There was 1 error: "Triangle 571 has a link to Triangle 661, but Triangle 661 doesn't have the matching link". Fixed it and saved -- split into over 24 navmeshes! I've found areas where some could be joined in advance, but most really are completely discontiguous, scattered up the cliff face. I've added Jump edges where they will work.

 

Some fixes are trivial. Some take hours. I've only managed to fix 40 or so cells in 4 days.

 

Bethesda developers neglecting to check all of these navmeshes must have been beyond the level of mere laziness, as the errors go into EditorWarnings.txt every time you save that cell! There's a test function!! There's a tested bit in every triangle!!! This is really despicable, releasing a game in this state.

 

When I've looked at many of these in UESP, they have the warning that a follower can get lost or disappear. I'm willing to bet this was the reason!

[Arthmoor]

Sounds like what i ran into with Shalidor's Maze then. Not quite the same mess, but that worldspace had one cell with 18 navmeshes in it. Should only have been 9 though because they had all been duplicated and were overlapping each other. NPC navigation through the cell was impossible. I ended up having to cut one set of dups down to a single triangle each, then drag those tris down behind an obstacle where they'd never be stepped on. Had to then fix the pile of errors in the remaining 9 before I could get a clean finalization on the whole mess. Took most of an entire day to win that battle.

 

Makes me wonder if these kinds of problems also exist in the Fallout 3 and NV navmeshes since those games never got the navmesh fixes that Skyrim 1.6 got.

[DayDreamer]

I ended up having to cut one set of dups down to a single triangle each, then drag those tris down behind an obstacle where they'd never be stepped on. Had to then fix the pile of errors in the remaining 9 before I could get a clean finalization on the whole mess. Took most of an entire day to win that battle.

The most in-place duplicates and embedded stray islands I've found so far have been 1 to 4 triangles. Instead of dragging them under obstacles, I've tried setting them to Z = -32767 -- but that leaves a tiny yellow smudge in the center of the screen that follows the view everywhere because of the view used for navmeshing. The distance focus must down there....

 

I've gone back and changed them all to Z = 32767, putting them high in the sky. I also set the tested flag (unknown 12) off. Then the PATHFINDING seems to ignore them. You can see them by flipping the world over, but otherwise they are above and behind the view during editing. Unlike objects (which need to be hidden under terrain), they are invisible during game play.

 

As to cutting them down to a single triangle each, although I agree that makes sense, it's probably not needed once they are out of the plain of play. Just extra icing on the cake to make them ignored by the game.

[DayDreamer]

In two places so far, the small duplicates/partial links have been on the border, and linked across the border, but not to the main navmesh -- essentially blocking the border. The error given is:

MASTERFILE: Potentially overlapping navmesh detected while saving Navmesh (001051b3) in cell Wilderness (000070db)

Since I want to preserve border links, I've kept the duplicates, pulled the main mesh away from the border instead, and used the internal jump links. The 'p' hotkey menu says ledge, but it doesn't actually have to be a ledge -- it tested fine on a flat surface.

 

[Arthmoor]

As to cutting them down to a single triangle each, although I agree that makes sense, it's probably not needed once they are out of the plain of play. Just extra icing on the cake to make them ignored by the game.

Cutting them down to one triangle each is more or less the best way to deal with any that have weird issues. One triangle not attached to anything will sit quietly and not bother the CK or the game.

For interiors I usually drag them outside the playable area. Exteriors it usually suffices to shove them underground. With Shalidor's Maze, piling them into a corner behind a giant brazier was more suitable so they wouldn't interfere with hand fixing the remaining navmeshes.

[zilav]

Uploaded r1595.

The major change is case sensitive INFO responses across all games. Also minor updates to Fallouts definitions.

[DayDreamer]

Cutting them down to one triangle each is more or less the best way to deal with any that have weird issues. One triangle not attached to anything will sit quietly and not bother the CK or the game.

For interiors I usually drag them outside the playable area. Exteriors it usually suffices to shove them underground. With Shalidor's Maze, piling them into a corner behind a giant brazier was more suitable so they wouldn't interfere with hand fixing the remaining navmeshes.

Looking at the data, my putting them high or low isn't such a good idea. It seems to confuse the newly discovered GridCell -- or at least is putting them in there. Assuming we know what it's doing, a follower or NPC could travel up there from anywhere.

 

Likewise, putting them outside the playable area might be a problem. Behind the brazier sounds good! Likewise, I'll try putting them on top of blocks and other things without an existing navmesh....

 

UPDATE: Outside the playable area must be OK. I just found an entire old navmesh side-by-side with the current one -- in Navmesh 0007406b Cell 'ThalmorEmbassy04' (0006DAA0) -- but wouldn't work for exteriors. Inside building components? Under rocks? Need a flag that says "do not use".

[DayDreamer]

BUG REPORT: When cleaning and a navmesh is removed, the corresponding entry is not removed from NAVI.

 

However, loading into CK and immediately saving (without making any changes) seems to remove them OK -- or at least the size shrank a lot and no warnings issued.

[DayDreamer]

Glad to report that I've finally fixed all the half-linked triangles in the base game.

 

Have run a slightly modified earlier SearchTriangleFlags script against the skyrim+update+fixedges -- almost all the earlier results are fixed, only two dozen cells still missing the Tested flag!

 

It's annoying that they actually had the existing flags, but the developers never bothered to check, or fix things that appeared in their logs every time they saved!

 

Well, maybe their tools just weren't as good as TES5Edit!

[zilav]

BUG REPORT: When cleaning and a navmesh is removed, the corresponding entry is not removed from NAVI.

xEdit doesn't update referencing records when deleting, only when changing FormID. For example if you delete some base object, it won't be removed from leveled lists too.

How and what to delete is very game and record specific, universal "clean up" method may work only for leveled lists, anything else will require additional coding and checks, and this is out of xEdit's scope imho. Better use official editors for that.

[Arthmoor]

@DD: Plus the NAVI updates reliably with the CK so it's not that big of a thing assuming deleting the navmesh doesn't cause the CK to crash on load - which ought to be pretty rare.

 

@zilav: Noticed 3.0.32 is now listed at Nexus. No announcement on the official forums or anything?

[zilav]

@zilav: Noticed 3.0.32 is now listed at Nexus. No announcement on the official forums or anything?

It is not my job to make announcements

[DayDreamer]

The meaning of NAVM flag Unknown 12 is "tested". In almost all of the bad edges, Unknown 12 isn't set. As soon as I fix the error, Unknown 12 is set on save. It's apparently cleared (and set) by the PATHFINDING logic used by the navmesh Find errors command. It might be used internally to mean "I've already been here" during testing.
 
I'd run the script you gave me weeks ago (SearchForNAVMTriangleFlags) trying to figure out meanings. Now I know I can use that same script to find "untested" triangles in cells. That's going to be really handy!!!

For the next version, a minor change request: the "Tested" flag could be "Found" instead.

 

I'm learning that the problem is often in the next triangle further away -- preventing this triangle from being visited by the PATHFINDING process. That triangle's problem is the blocking event, but it has the Tested flag. So "tested" is too strong a word, kinda implies it's working, even when it's the problem triangle. Since it's the FINDING process, "Found" seems somewhat better.

[DayDreamer]

It is not my job to make announcements

Or even edit the Description page, which still says .31

 

I see the NavMeshCheckEdges.pas, posted earlier in the thread, was renamed. Here's the code I'm using for checking Found.

 

Still, our thanks!

 

[old code removed]

[zilav]

Uploaded the latest version.

Changeable fonts in Options and ClearMessages function (requested on nexus).

[DayDreamer]

Glad to report that I've finally fixed all the half-linked triangles in the base game.

 

Have run a slightly modified earlier SearchTriangleFlags script against the skyrim+update+fixedges -- almost all the earlier results are fixed, only two dozen cells still missing the Tested flag!

Just reporting for the record that I'd finished the 23 cells with missing Found/Tested flags yesterday.

 

The worst was Soljund's Sinkhole. Took 2 days. Had to start over repeatedly because I couldn't fix the several "reversed normals" without creating more of the same in the field of jumbled rocks and navmesh holes. The only solution was to tediously 'F' ground vertices one at a time (the automated navmesh usually had them far above the ground, and often out beyond the ledges, which would cause an NPC to fall to their death), and rotate the view constantly to visualize the intersecting planes. Then using the Find Cover Edges (to fix the existing edges I'd undone) caused more problems. Aarrgghh!

 

Also, there's an extremely odd problem with non-island "islands". Always single triangles that are in the next cell yet still in the "world"; in one case with a Door flag. The PATHFINDING algorithm apparently won't cross cell boundaries, and isn't good about handling obviously connected islands. Ended up setting the flag myself with TES5Edit, just to be on the safe side.

 

Meanwhile, there are plenty of disconnected islands that do have the Found/Tested flag, even though there's no path to them. It makes no sense.

 

Next up, resolving conflicts with the existing USKP. Since there isn't any way to run two instances of CK side by side, I'm taking screenshots of USKP mesh, and trying to duplicate the changes. Fortunately, there aren't many conflicts. Riverwood is going to be a bear, though.

[DayDreamer]

Just tried the latest .33, looks nice.

 

For the record, the meaning of Edge Links field Unknown appears to be:

 

00 00 00 00 = border

01 00 00 00 = lower Z edge pointing to higher

02 00 00 00 = higher Z edge pointing to lower

 

So far, those are all the values I've found, but not yet written a script to examine.

 

Speaking of scripts, I've further modified the TriangleFound script above to look for other flags. No skyrim.esm + update.esm used any of the still unknown flags, although maybe something wrong with my code?

    // Flags 4-6 set
    if GetElementNativeValues(tri, 'Flags') shr 3 and 7 > 0 then begin
    // Flags 8-9 set
    if GetElementNativeValues(tri, 'Flags') shr 7 and 3 > 0 then begin
    // Flags 13-16 set
    if GetElementNativeValues(tri, 'Flags') shr 12 and 15 > 0 then begin

[tyrel68]

@Daydreamer

Out of curiosity what is the final benefit of fixing these things?

[DarthVitrial]

I am 99.9% certain that this is just the result of me doing something stupid and not a bug, but nonetheless, here is an error report from build r1597/

date/time         : 2014-06-03, 13:46:16, 658ms
computer name     : REMOVED
user name         : REMOVED
registered owner  : Windows User
operating system  : Windows 8 x64 build 9200
system language   : English
system up time    : 4 hours 8 minutes
program up time   : 8 minutes 30 seconds
processors        : 8x Intel® Core i7-2670QM CPU @ 2.20GHz
physical memory   : 3598/8028 MB (free/total)
free disk space   : (C:) 134.95 GB
display mode      : 1600x900, 32 bit
process id        : $126c
allocated memory  : 621.51 MB
executable        : FNVEdit.exe
exec. date/time   : 2014-05-31 07:54
version           : 3.0.33.0
compiled with     : Delphi XE
madExcept version : 4.0.5
callstack crc     : $4b3931b4, $36a3632c, $6c2bd099
count             : 3
exception number  : 3
exception class   : EJvInterpreterError
exception message : Error in unit 'UserScript' on line 22 : Undeclared Identifier 'GetIsEditable'.

main thread ($5e8):
0095af33 +01b FNVEdit.exe  JvInterpreter          JvInterpreterErrorN
0096466e +242 FNVEdit.exe  JvInterpreter          TJvInterpreterExpression.InternalGetValue
00963100 +134 FNVEdit.exe  JvInterpreter          Expression
00963a1d +a51 FNVEdit.exe  JvInterpreter          Expression
009636f7 +72b FNVEdit.exe  JvInterpreter          Expression
00963bdc +05c FNVEdit.exe  JvInterpreter          TJvInterpreterExpression.Expression1
00963cb5 +039 FNVEdit.exe  JvInterpreter          TJvInterpreterExpression.Expression2
00966bce +02a FNVEdit.exe  JvInterpreter          TJvInterpreterFunction.InterpretIf
00965db9 +135 FNVEdit.exe  JvInterpreter          TJvInterpreterFunction.InterpretStatement
00966b39 +0b1 FNVEdit.exe  JvInterpreter          TJvInterpreterFunction.InterpretBegin
00965a02 +052 FNVEdit.exe  JvInterpreter          TJvInterpreterFunction.InFunction
00969f47 +0c3 FNVEdit.exe  JvInterpreter          TJvInterpreterUnit.ExecFunction
0096a1f5 +145 FNVEdit.exe  JvInterpreter          TJvInterpreterUnit.CallFunctionEx
0096a076 +062 FNVEdit.exe  JvInterpreter          TJvInterpreterUnit.CallFunction
00a0aed2 +35e FNVEdit.exe  frmViewMain   5401 +62 TfrmMain.ApplyScript
00a0b614 +104 FNVEdit.exe  frmViewMain   5467 +13 TfrmMain.mniNavApplyScriptClick
00525d7f +0a7 FNVEdit.exe  Menus                  TMenuItem.Click
0052727b +013 FNVEdit.exe  Menus                  TMenu.DispatchCommand
0052845a +082 FNVEdit.exe  Menus                  TPopupList.WndProc
005283a9 +01d FNVEdit.exe  Menus                  TPopupList.MainWndProc
004c4e38 +014 FNVEdit.exe  Classes                StdWndProc
76d598ec +00b user32.dll                          DispatchMessageW
005b201b +0f3 FNVEdit.exe  Forms                  TApplication.ProcessMessage
005b205e +00a FNVEdit.exe  Forms                  TApplication.HandleMessage
005b2389 +0c9 FNVEdit.exe  Forms                  TApplication.Run
00a39bd7 +05f FNVEdit.exe  TES5Edit        73  +7 initialization
7728919d +00c KERNEL32.DLL                        BaseThreadInitThunk

thread $880 (TWorkerThread):
775e10f7 +b1 KERNELBASE.dll                      WaitForSingleObjectEx
775e1038 +0d KERNELBASE.dll                      WaitForSingleObject
005d20d1 +19 FNVEdit.exe    VirtualTrees 6308 +3 TWorkerThread.Execute
00472ca3 +2b FNVEdit.exe    madExcept            HookedTThreadExecute
004c2172 +42 FNVEdit.exe    Classes              ThreadProc
0040754c +28 FNVEdit.exe    System       2510 +0 ThreadWrapper
00472b85 +0d FNVEdit.exe    madExcept            CallThreadProcSafe
00472bef +37 FNVEdit.exe    madExcept            ThreadExceptFrame
7728919d +0c KERNEL32.DLL                        BaseThreadInitThunk
>> created by main thread ($5e8) at:
005d1fb9 +19 FNVEdit.exe    VirtualTrees 6251 +1 TWorkerThread.Create

[DayDreamer]

@Daydreamer

Out of curiosity what is the final benefit of fixing these things?

You hadn't noticed that NPCs get stuck inside and cannot get out? Or get stuck and stand in one place until you leave the room, so they can teleport to their next package location?

 

You hadn't noticed that NPCs whom you are following suddenly divert from the road and run into the wilderness, and then come back and run somewhere else? Or your followers dash away in the wrong direction and circle around to another door?

 

Basically, a lot of places are completely screwed up. You may not notice as a player, because you don't use the navmesh -- it just walks you in the direction that you point. But all the creatures use navmesh.

 

So far, I've tracked down all the cell borders with edge pointers in one direction and not the other (half links), and the same inside cells. CK doesn't have any test for the former. Although it has a test for the latter, you have to run it on each and every cell by hand.

 

Then, I've tracked all the cells where CK cannot even find some navmesh triangles, but handily has a flag to warn us!

 

Next, I'm working on the 400+ single triangle islands (NPCs cannot leave them), which the CK also shows warnings cell by cell. But I couldn't figure out a good test for multi-triangle islands though.

[DayDreamer]

I'm having some problems getting this code to work:

function Process(e: IInterface): integer;
var
  tris, tri, ore, mos, EdgeLinks: IInterface;
  TriLimit, orc, j, i, flags, errors, EdgeValue, EdgeLinkLimit, EdgeLinkFound, EdgeLinkCount: integer;
begin
  if Signature(e) <> 'NAVM' then begin
    //AddMessage(Format('Not NAVM %s', [Name(e)]));
    Exit;
  end;
  AddMessage(Name(e));

  mos := MasterOrSelf(e);
{  if mos = e then begin
    AddMessage(Format('No Override for %s', [Name(e)]));
    Exit;
  end;
}
  orc := OverrideCount(mos);
  if orc < 1 then begin
    AddMessage(Format('Override Count %d for %s', [orc, Name(e)]));
    Exit;
  end;

  ore := OverrideByIndex(mos, Pred(orc));

The "mos = e" says there's a type conflict? So I couldn't use it, and worked around it.

 

But the remainder of the code seems to always give myself, not Skyrim.esm or Update.esm. (I'm trying to get the override 1 less than myself.)

[zilav]

"Mos" and "e" are pointers stored in "variant" structure, they can't be compared directly. You need to check something unique to both of them like GetFileName(), or just rewrite the code a bit

mos := Master(e);
if not Assigned(mos) then begin
  AddMessage('no master');
  Exit;
end;

orc := ....

This will list all overriding files when applied to a record

unit UserScript;

function Process(e: IInterface): integer;
var
  i: integer;
begin
  for i := 0 to Pred(OverrideCount(MasterOrSelf(e))) do
    AddMessage(GetFileName(OverrideByIndex(MasterOrSelf(e), i)));
end;

end.

Regarding error with GetIsEditable - it is a syntax parsing exception in script, can't say anything without looking at the source code.

[zilav]

Can someone please check PLYR form usage regarding TES4 and TES5 in CS and CK

https://code.google.com/p/skyrim-plugin-decoding-project/source/detail?r=1598

[Arthmoor]

What's the PLYR form and where would it be expected to show up?