[WIPz] TES5Edit

[zilav]

Player form in REFR Activate Parent and Linked References.

[Arthmoor]

CK lets me select the player as the activate parent or as a linked ref, but Edit doesn't show it as a PLYR form in the data. Just a regular REFR as far as I can tell. It's not throwing any errors about it either. Hopefully that's what you're looking for?

 

Oblivion does not have linked refs or activate parents.

[DarthVitrial]

Had an error when using the "put worldspace references in the correct cells" script on Skyrim.

date/time         : 2014-06-04, 17:47:54, 260ms
computer name     : removed
user name         : Removed
registered owner  : removed
operating system  : Windows 8 x64 build 9200
system language   : English
system up time    : 6 days 17 hours
program up time   : 4 minutes 2 seconds
processors        : 8x Intel® Core i7-4810MQ CPU @ 2.80GHz
physical memory   : 10591/16303 MB (free/total)
free disk space   : (C:) 59.74 GB (D:) 703.66 GB
display mode      : 2880x1620, 32 bit
process id        : $2f54
allocated memory  : 1.23 GB
executable        : TES5Edit.exe
exec. date/time   : 2014-05-31 07:54
version           : 3.0.33.0
compiled with     : Delphi XE
madExcept version : 4.0.5
callstack crc     : $93b3a8eb, $94cbe6b5, $9ec12c8c
exception number  : 1
exception class   : EAssertionFailed
exception message : Assertion failure (D:\Projects\TES5Edit\wbImplementation.pas, line 5181).

main thread ($325c):
00763e1d +1a1 TES5Edit.exe wbImplementation  5182 +20 TwbMainRecord.EnsureChildGroup
0078ab4b +127 TES5Edit.exe wbImplementation 14701 +18 TwbContainedInElement.DoAfterSet
007870cd +0e9 TES5Edit.exe wbImplementation 13381 +12 TwbValue.SetEditValue
0076141c +074 TES5Edit.exe wbImplementation  4555  +9 TwbContainer.SetElementEditValue
009cfc38 +06c TES5Edit.exe wbScriptAdapter    571  +2 IwbContainer_SetElementEditValues
00960a7e +09e TES5Edit.exe JvInterpreter              GetFun
009611d7 +28b TES5Edit.exe JvInterpreter              TJvInterpreterAdapter.GetValue
00964f9c +064 TES5Edit.exe JvInterpreter              TJvInterpreterExpression.GetValue
00965bc5 +09d TES5Edit.exe JvInterpreter              TJvInterpreterFunction.GetValue
00969b50 +01c TES5Edit.exe JvInterpreter              TJvInterpreterUnit.GetValue
00964590 +164 TES5Edit.exe JvInterpreter              TJvInterpreterExpression.InternalGetValue
00966a01 +05d TES5Edit.exe JvInterpreter              TJvInterpreterFunction.InterpretIdentifier
00965d69 +0e5 TES5Edit.exe JvInterpreter              TJvInterpreterFunction.InterpretStatement
00966b39 +0b1 TES5Edit.exe JvInterpreter              TJvInterpreterFunction.InterpretBegin
00965a02 +052 TES5Edit.exe JvInterpreter              TJvInterpreterFunction.InFunction
00969f47 +0c3 TES5Edit.exe JvInterpreter              TJvInterpreterUnit.ExecFunction
0096a1f5 +145 TES5Edit.exe JvInterpreter              TJvInterpreterUnit.CallFunctionEx
0096a076 +062 TES5Edit.exe JvInterpreter              TJvInterpreterUnit.CallFunction
00a0aed2 +35e TES5Edit.exe frmViewMain       5401 +62 TfrmMain.ApplyScript
00a0b614 +104 TES5Edit.exe frmViewMain       5467 +13 TfrmMain.mniNavApplyScriptClick
00525d7f +0a7 TES5Edit.exe Menus                      TMenuItem.Click
0052727b +013 TES5Edit.exe Menus                      TMenu.DispatchCommand
0052845a +082 TES5Edit.exe Menus                      TPopupList.WndProc
005283a9 +01d TES5Edit.exe Menus                      TPopupList.MainWndProc
004c4e38 +014 TES5Edit.exe Classes                    StdWndProc
754998ec +00b user32.dll                              DispatchMessageW
005b201b +0f3 TES5Edit.exe Forms                      TApplication.ProcessMessage
005b205e +00a TES5Edit.exe Forms                      TApplication.HandleMessage
005b2389 +0c9 TES5Edit.exe Forms                      TApplication.Run
00a39bd7 +05f TES5Edit.exe TES5Edit            73  +7 initialization
76f7919d +00c KERNEL32.DLL                            BaseThreadInitThunk

thread $25d4 (TWorkerThread):
757810f7 +b1 KERNELBASE.dll                      WaitForSingleObjectEx
75781038 +0d KERNELBASE.dll                      WaitForSingleObject
005d20d1 +19 TES5Edit.exe   VirtualTrees 6308 +3 TWorkerThread.Execute
00472ca3 +2b TES5Edit.exe   madExcept            HookedTThreadExecute
004c2172 +42 TES5Edit.exe   Classes              ThreadProc
0040754c +28 TES5Edit.exe   System       2510 +0 ThreadWrapper
00472b85 +0d TES5Edit.exe   madExcept            CallThreadProcSafe
00472bef +37 TES5Edit.exe   madExcept            ThreadExceptFrame
76f7919d +0c KERNEL32.DLL                        BaseThreadInitThunk
>> created by main thread ($325c) at:
005d1fb9 +19 TES5Edit.exe   VirtualTrees 6251 +1 TWorkerThread.Create

thread $78:
76f7919d +c KERNEL32.DLL  BaseThreadInitThunk

thread $1508:
76f7919d +c KERNEL32.DLL  BaseThreadInitThunk

[zilav]

Had an error when using the "put worldspace references in the correct cells" script on Skyrim.

I need your plugin to check.

 

@Arthmoor yep, thx.

[DarthVitrial]

Odd...now I can't seem to reproduce the error. I'll keep trying though, if I find which plugin caused it I'll edit this post.

[DayDreamer]

I'm unsure about the "native" format of navmeshes. But I've managed to get the script working with Edit values instead. Here's the swap copy section:

          havePath := ElementByPath(haveLink, 'Mesh');
          saveString := GetEditValue(havePath);
          swapPath := ElementByPath(swapLink, 'Mesh');
          SetEditValue(havePath, GetEditValue(swapPath));
          SetEditValue(swapPath, saveString);

And the compare section:

    EdgeLink := ElementByIndex(EdgeLinks, k);
    if elT = GetElementNativeValues(EdgeLink, 'Triangle') then begin
      //AddMessage(Format('%d = %d', [elT, GetElementNativeValues(EdgeLink, 'Triangle')]));
      //AddMessage(elM);
      //AddMessage(GetElementEditValues(EdgeLink, 'Mesh'));
      if CompareStr(elM, GetElementEditValues(EdgeLink, 'Mesh')) = 0 then begin
        //AddMessage(Format('found %d', [k]));
        Result := k;
        Exit;
      end;
    end;

Not efficient, but won't be run very often anyway. Is there a better way for production quality?

 

[old code removed]

[DayDreamer]

I'll describe what Harmonize does in this separate message. Each Finalize makes 5 cell's worth of navmesh. Harmonize removes the redundant information from the 4 adjoining cells (and extra navmesh in the same cell, too) by comparing the data with the overriding masters. It swaps the Edge Links entries to their original override ordering. Run it before running the Filter for Cleaning, and any wholly extraneous information will be cleaned out.

It's rather verbose for problems, so that problems can be fixed. It tells you whether the CK has changed a linked edge from 0-1 to 1-2 or 2-0 or vice versa. Or that you've accidentally renumbered a triangle on your cell border -- which is a big no-no for avoiding conflicts.

Questions?

[DayDreamer]

I've been working on making the Harmonize (Simonize, Martinize) produce prettier output (such as blank lines between triangles), and re-worked it a bit to give more details on problems (and removing more of the debugging).

 

We'd spent some effort a few messages ago getting the override count details done. So Harmonize does compare any Update records overriding Skyrim....

 

But while staring at the output, discovered that Apply Filter for Cleaning does not match Update.esm, it matches only Skyrim.esm. Why?

[zilav]

But while staring at the output, discovered that Apply Filter for Cleaning does not match Update.esm, it matches only Skyrim.esm. Why?

What do you mean "does not match"? It is just an ordinary filter bypassing filter options window with predefined settings (all cleared except inherited conflicts).

[DayDreamer]

What do you mean "does not match"? It is just an ordinary filter bypassing filter options window with predefined settings (all cleared except inherited conflicts).

I meant that the conflict filter options aren't checking against Update. After nearby Finalize, I've swapped the Edge Links to be exactly the same, as shown. But the filter doesn't say no conflicts, and remove ITM doesn't remove them....

 

[zilav]

Cleaning removes only ITM records, ITM means identical to master. In this case the master is Skyrim.esm, and comparison is done with it's record when deciding conflict state. Update.esm is just an override in between and hence colored yellow (conflict), not green (ITM).

[DayDreamer]

I'm unsure about the "native" format of navmeshes. But I've managed to get the script working with Edit values instead. ...

Not efficient, but won't be run very often anyway. Is there a better way for production quality? Do you have any interest in packaging with TES5Edit?

Having no answer about "native" format, still working with EditValues. Here are my current versions of CheckEdges and CleanEdgeLinks.

 

For CheckEdges, I've added checking the Triangle Found flag, and the 1 triangle island check, rather than having 3 nearly identical scripts. It also is a bit prettier, to match CleanEdgeLinks.

 

For CleanEdgeLinks, it does all the cleanup in the Edge Link table, but only tells you where triangles might be improved. I'd like to fix the triangles automatically too, but that requires the format of the cover fields. I've discovered that setting them by hand can cause the CK to crash on load.

 

As you've renamed Checkedges to "Skyrim - Check edge links in navmeshes", the latter should probably be "Skyrim - Clean edge links in navmeshes".

[old code removed]

[DayDreamer]

Cleaning removes only ITM records, ITM means identical to master. In this case the master is Skyrim.esm, and comparison is done with it's record when deciding conflict state. Update.esm is just an override in between and hence colored yellow (conflict), not green (ITM).

I've always thought of Update as a Master. It's listed as a Master in the header.

 

Oh well, my stuff will help cleaning against Skyrim. But I did the effort to make them exactly the same as Update, so no harm (hopefully).

[DayDreamer]

Format of the cover Fields:

 

This is turning to be a bit of a nightmare. There are two bytes, but the CK only edits them per edge -- so I'd think there's missing one! But no, it simply only allows you to edit two in the CK! Examples:

CK:
edge 0-1    144, right, compute
 
TES5Edit:
#1 41  0010 1001 (1001<=>144)
#2 64  0100 0000
 
0-1 144, left, compute
89  0101 1001
10  0000 1010

0-1    96, left, compute
22  0001 0110=96
64  0100 0000

0-1 64, none, compute
 4  0000 0100=64
64  0100 0000

I'm guessing that upper nybble xx1x is right and xxx1 is left. But not all the data seems to confirm.

 

I'm unable to figure out much more than this: the lower nybble maps to these values in the menu, which are 16 times index:

 4: 0100    64

 5: 0101    80

 6: 0110    96

 7: 0111    112

 8: 1000    128

 9: 1001    144

10: 1010    160

11: 1011    176

12: 1100    192

13: 1101    208

14: 1110    224

 

There are 4 other values in the pulldown menu:

 

 0: open edge no cover

 1: wall no cover

 2: ledge cover

 3: (not in menu)

15: max

 

UPDATE! values updated above.

[DayDreamer]

TES5Edit has it wrong. This is not 2 single byte fields (that would have made more sense). This is a little-endian 16 bit field, with 2 6-bit subfields, plus some unknown bits at the top!
 
eeuu rlnn nnrl nnnn
 
One problem with listing them as two bytes is it made it harder to see the pattern, as the bottom bits of the 2nd byte pair with the top bits of the 1st -- obviously little-endian.
 
The top 2 bits 'ee' aren't settable by CK?
EDIT1: They might be data on the 3rd vertices (seem to be same among all triangles with that vertex, even those without any cover sides)?

EDIT2: Maybe near a wall? But seem to be same among neighbors without cover, and wouldn't that be wall no cover?
00:
01:
10:
11: (never seen)
 
I've got no examples of anything in the next 2 bits: uu.
 
The upper field (Edge 1-2) is rlnn nn, split across the bytes. The lower field (Edge 0-1) is rlnnnn least significant byte.
 
The Compute checkbox in the menu has to be turned off to allow manual setting of the left and right cover flags. (EDIT: These seem to mean the triangle sharing the same vertex counter-clockwise and clockwise respectively.)
 
If the fields don't match the pattern, CK will rotate the vertices. (Edge 2-0 apparently illegal.) Wow, that is a pain.
 
Worse, using the Find Cover command or the Edit Cover button actually swaps the triangle with the highest numbered triangle elsewhere in the mesh. Find Cover changing 3 triangles at once will swap it 3 times, renumbering triangles all over the place.
 
That makes a mess of things, especially for border edges. No wonder there were so many half-links!!!
 
Here's some test data, tested on the island triangle in Riverwood02:

b503c

1-2    max, none (compute)
192         11000000
3   00000011

1-2    224, none (compute)
128         10000000
3   00000011

0-1    224, none (unchanged)
1-2    max, none (compute)
206         11001110
3   00000011
^^^ tried to set 2-0, self-rotated vertices!!!

0-1    ledge, none (compute)
1-2    max, none (unchanged)
194         11000010
3   00000011

0-1    wall, none (compute)
1-2    max, none (unchanged)
193         11000001
3   00000011

0-1    wall, none (---)
1-2    max, none (unchanged)
193         11000001
3   00000011

0-1    wall, left (---)
1-2    max, none (unchanged)
193         11000001
3   00000011

0-1    ledge, left (---)
1-2    max, none (unchanged)
210         11010010
3   00000011

0-1    ledge, left & right (---)
1-2    max, none (unchanged)
242         11110010
3   00000011

0-1    ledge, left & right (unchanged)
1-2    max, left (---)
242         11110010
7   00000111

0-1    ledge, left & right (unchanged)
1-2    max, right (---)
242         11110010
11  00001011

[hlp]

Anybody using Build Reachable Info ? I want to know if looking into the stack overflow is important.

[DayDreamer]

Still not confident enough in my interpretation of cover flags to automatically swap triangles.

 

But found a minor bug in the cleaning detection routine. Missed the (requires hand fixing) case where two edges point at the same side. Obviously illegal in a universe with parallel lines, but there's one case in all of Skyrim....

 

Turned off a bunch of debugging. Still has a lot of debugging. Added more automated searching for matches, so that's less to do by hand. Twice as slow now.

 

Anywhere there's repair work to be done, you'll now see a set of messages that look like this:

Want match for triangle 87 in [NAVM:001051AE] (in GRUP Cell Temporary Children of ReachCell01 [CELL:000070BC] (in Tamriel "Skyrim" [WRLD:0000003C] at -38,7))
via old Edge 0-1 link 20
^^^ Check triangle 311 in [NAVM:001051B2] (in GRUP Cell Temporary Children of [CELL:000070DB] (in Tamriel "Skyrim" [WRLD:0000003C] at -38,6))

Triangle #343 needs repair?
Triangle #311 Edge 0-1 was found!
Have match (20) for triangle 87 in [NAVM:001051AE] (in GRUP Cell Temporary Children of ReachCell01 [CELL:000070BC] (in Tamriel "Skyrim" [WRLD:0000003C] at -38,7))
via new Edge 2-0 link 20
^^^ Check triangle 343 in [NAVM:001051B2] (in GRUP Cell Temporary Children of [CELL:000070DB] (in Tamriel "Skyrim" [WRLD:0000003C] at -38,6))

That's a tricky one. So far, I've found only 3 cases in Skyrim.... CK renumbered a border triangle that had a cover. But the edge 0-1 border crossing isn't legal any more, as that would put the cover on edge 2-0. So it rotated edge 2-0 for the border.

 

The hand solution is to give it 2-0 for the border, but reinstate the same triangle number to get rid of the half-link, so that the other cell can potentially clean its changes. The other cell doesn't care about sides in this cell; only triangle number is in the Edge Link table -- so we really really want to keep the same triangle number!!!

(I don't know why the CK keeps unnecessarily auto-renumbering triangles. Careless programming is my guess. But during testing on a single triangle island, they auto-rotate without renumbering just fine! So they wrote code for a special island case to fix their poor code elsewhere?)

[old code removed]

[zilav]

So I just wait for a finished version

And yes, navmeshes, or more precisely the way they are implemented in Skyrim, left much to desire. The good thing is, with increased RAM and CPU power of next gen consoles they will probably get rid of external edge links since those can be calculated at runtime, or maybe implement the entirey new navigation system.

[DayDreamer]

So I just wait for a finished version

And yes, navmeshes, or more precisely the way they are implemented in Skyrim, left much to desire. The good thing is, with increased RAM and CPU power of next gen consoles they will probably get rid of external edge links since those can be calculated at runtime, or maybe implement the entirey new navigation system.

I've been been posting partial results (as I'd done before) in hopes that prettier cover flags versions might show up in any upcoming beta releases. Any chance that the 2 bytes could be combined into 1 word? Or that the now identified fields could be given names, like you did with the flags word?

 

And what's the "native" version of the Edge Links Mesh field? (Those string comparisons are eating most of the CPU.)

[zilav]

I've been been posting partial results (as I'd done before) in hopes that prettier cover flags versions might show up in any upcoming beta releases. Any chance that the 2 bytes could be combined into 1 word? Or that the now identified fields could be given names, like you did with the flags word?

I was waiting for a more defined information on them, right now information is scattered across several edited posts and hard to comprehend for a person who have never dealt with navmeshes in CK. So you want to combine 2xint8 cover edge flag fields into a single int16 value and turn it into enumeration? In this case I need a list of possible enumeration values: integer values (decimal or hex) and corresponding text to display in TES5Edit.

 

And what's the "native" version of the Edge Links Mesh field? (Those string comparisons are eating most of the CPU.)

FormID 32 bit integer value with load order index (high order byte) relative to master files indexes in TES4 plugin header, they are called FileFormID. What you see in TES5Edit are LoadOrderFormID with high order byte replaced by load order index of master, or load order of plugin itself for new records.

 

[DayDreamer]

I was waiting for a more defined information on them, right now information is scattered across several edited posts and hard to comprehend for a person who have never dealt with navmeshes in CK. So you want to combine 2xint8 cover edge flag fields into a single int16 value and turn it into enumeration? In this case I need a list of possible enumeration values: integer values (decimal or hex) and corresponding text to display in TES5Edit.

Correct, 2 bytes is really 16 bits, little-endian. Yes, it would have been smarter for them to use 2 bytes, but apparently they didn't. Presumably an unaligned C bit field.

ttuu rlnn nnrl nnnn

tt: unknown, not settable by CK, OK as bits.

00 most common

x1 Unknown 15

1x Unknown 16

11 not yet found in the wild

uu: unused, OK as bits.

x1 Unknown 13

1x Unknown 14

rlnn nn: Edge 1-2 cover

r: right cover flag

l: left cover flag

nn nn:

dec bits text

0: 0000 open edge, no cover

1: 0001 wall, no cover

2: 0010 ledge, cover

3: 0011 48 (not in CK menu)

4: 0100 64

5: 0101 80

6: 0110 96

7: 0111 112

8: 1000 128

9: 1001 144

10: 1010 160

11: 1011 176

12: 1100 192

13: 1101 208

14: 1110 224

15: 1111 max

rl nnnn: Edge 0-1 cover

same values as above.

 

FormID 32 bit integer value with load order index (high order byte) relative to master files indexes in TES4 plugin header, they are called FileFormID. What you see in TES5Edit are LoadOrderFormID with high order byte replaced by load order index of master, or load order of plugin itself for new records.

I tried GetElementNativeValues(EdgeLink, 'Mesh') as an integer, it complained about mismatched types.

That's ok, GetElementEditValues(EdgeLink, 'Mesh') as a String works, it's just slow.

 

[old code removed]

[zilav]

Correct, 2 bytes is really 16 bits, little-endian. Yes, it would have been smarter for them to use 2 bytes, but apparently they didn't. Presumably an unaligned C bit field.

xEdit can display integer values in 3 ways:

1) As bit flags

2) As a list of possible values (enumeration)

3) Custom coded integer -> string and string -> integer converters

Enumeration is out of question here because there can be hundreds of possible bits combinations for cover flags.

Custom converter is doable, but only as a last resort, because it can display as string fine, but writing the opposite string -> integer converter so user can modify flags by editing this field in TES5Edit as a string is not so straightforward. It will require to provide a formatted string with predefined names for values or just an integer number.

So the best approach is bit flags, but we need to know the definite meaning of each bit as a separate value. Currently I set them as this

              'Edge 0-1 wall',
              'Edge 0-1 ledge cover',
              'Unknown 3',
              'Unknown 4',
              'Edge 0-1 left',
              'Edge 0-1 right',
              'Edge 1-2 wall',
              'Edge 1-2 ledge cover',
              'Unknown 9',
              'Unknown 10',
              'Edge 1-2 left',
              'Edge 1-2 right',
              'Unknown 13',
              'Unknown 14',
              'Unknown 15',
              'Unknown 16'

Uploaded a new version.

 

I tried GetElementNativeValues(EdgeLink, 'Mesh') as an integer, it complained about mismatched types.

That's ok, GetElementEditValues(EdgeLink, 'Mesh') as a String works, it's just slow.

GetElementNativeValues() itself won't give type mismatch, error happens somewhere later when this value is used.

[zilav]

Uploaded new version.

Small fix in BSA unpacking code.

Assets manager script can now calculate checksums of files in selected containers and later exclude them from copying. Used when vanilla BSAs are unpacked so there are no means to know what loose files are vanilla, and what are new or modified.

[Arthmoor]

Hey,

 

Those case sensitivity fixes that went in for 3.0.32? FULL subrecords in Oblivion don't seem to be acknowledging the update. Did they not get included or does this need another bug report to be filed?

[zilav]

Those case sensitivity fixes that went in for 3.0.32? FULL subrecords in Oblivion don't seem to be acknowledging the update. Did they not get included or does this need another bug report to be filed?

Nope, Oblivion was not included... maybe to preserve BOSS databse ITM counts? Will add it too. However Elminster is back in action atm so there will be other updates soon I believe.