[WIPz] TES5Edit

[hlp]

Any way to identify patterns, or at least post a decoded data block? Maybe if we're looking at the actual data it'll start to make sense.

 

My main fear here is that if this is culling data that chopping it out will tell the engine to stop culling ANYTHING in a particular cell. That would lead to performance degradation at the very least.

I honestly could not gather anything from your reference cell.

 

I can try to include it as an array when you deselect the simple records option.

 

Here what it looks like: https://www.dropbox.com/s/t3m2mft3qihihwc/TVDT-8F7A.PNG

[hlp]

It should be available as of r1510, and you can find a preview of TESDump 3.0.32 with TVDT here : https://www.dropbox.com/s/7xq6cjmv0ysdc2y/TES5Dump-3.0.32-preview20140119.zip

[Sharlikran]

Uncheck "Hide no conflct rows".

 

DOH!  

[Sharlikran]

Arthmoor, if you use Dump to look for patterns, I suggest trying these steps.  Basically you should try to use two different file names for the same plugin.  Make a change to one, and run dump on both.  Then use a Diff program to see the results.  That's how Zilav and I decoded things originally.

[Sharlikran]

Uncheck "Hide no conflct rows".

 

Thank you, now I would still like to provide feedback that there may be a potential bug.  You might have fixed it, if so let me know.  There is still the issue with the Null Fields.  I don't think I saw those when I was testing before when you were resolving the Empty list issue.

[Arthmoor]

I honestly could not gather anything from your reference cell.

 

I can try to include it as an array when you deselect the simple records option.

 

Here what it looks like: https://www.dropbox.com/s/t3m2mft3qihihwc/TVDT-8F7A.PNG

Ugh, you're sure it's an array of 32 bit integers? That's gibberish. Those numbers are all over the map

 

Arthmoor, if you use Dump to look for patterns, I suggest trying these steps.  Basically you should try to use two different file names for the same plugin.  Make a change to one, and run dump on both.  Then use a Diff program to see the results.  That's how Zilav and I decoded things originally.

This isn't a record that can be altered in the CK so making minor changes to identify patterns isn't possible.

[Sharlikran]

This isn't a record that can be altered in the CK so making minor changes to identify patterns isn't possible.

No you're right. However, it's like MHDT. Although related, and you can't edit that field directly, it has to be affected by doing something. If you know that changing a cell, world cell, or will effect that then you can still see the change once it happens. It's just a suggestion.

[Arthmoor]

I understand that, but if the prevailing theory is correct and this is culling data, there appears to be no means to try and regenerate it within the CK. So like the MHDT subrecord, there's nothing we can do about this unless we're missing something obvious.

 

That's why I hesitate to simply gut the entire subrecord from the cell. The nature of the problem means it would have to be gutted from numerous cells. This kind of performance degradation will pile up quickly if the absence of this record is interpreted as "don't cull anything".

[hlp]

Ugh, you're sure it's an array of 32 bit integers? That's gibberish. Those numbers are all over the map

 

This isn't a record that can be altered in the CK so making minor changes to identify patterns isn't possible.

Almost. On load or save they are endian swapped as 32 bit integers. Only other possibility would be floats, but I don't think FFFFFFFF is a valid float value.

 

By the way if those culling data cannot be generated with the CK, which tools generates them?

[Nico coiN]

I think the CK does generate them, but in a mysterious way that cannot be seen or handled by any regular user. I can only assume these datas are generated in a dynamic way when designing cells. But as it affects the visibility of distant elements ingame, it may also be related to LOD generation, which is for sure the most buggy feature from the CK. Especially the landscape LOD generation (as stated in my 1st post on the 2nd page from bug 55, the one with the void screenshot), which that we now know that no rendered landscape ingame = no rendered water above, too... Trees and other objects LOD seem to be a totally different subject.

 

It would be worth investigating mods with brand new huge worldspaces (such as Wyrmstooth or Falskaar), only to see if this subrecord is also filled, and if the same kind of issue happens at some various places.

[Arthmoor]

I would assume Bethesda's CK generates them somehow, and that functionality doesn't exist in ours.

 

In looking at Falskaar and Wyrmstooth, sampling cells from areas in the mods where stuff exists, Falskaar has no MHDT data to speak of, Wyrmstooth does, but neither of these mods seems to have any TVDT data defined at all. I can't say that for certain though since I obviously haven't checked every single cell.

[Nico coiN]

Maybe a custom TES5Edit script could automate a multi check ?

[zilav]

Uploaded a new version, this time it is hosted on a private server.

Hlp implemented autoupdates of counter fields (COCT, LLCT, etc.) when changing corresponding list. Please test it, we are aiming for release soon.

[Sharlikran]

I only did containers and keywords in a merge patch for Skyrim but, it worked fine for me.

[hlp]

There is also one counter in FO3 and FNV : IDLC, paired with IDLA Idle Animations.

[Jon]

Is there any reason that I can't copy the raw data for an entire subrecord, like, say "NVNM - Geometry"?  I would really like to, in order to decode the triangles.   It's all quite simple but I can't copy/paste enough since I have to do it line by line to figure out what the other data means for the tris.

 

Edit:  Guess it's already mostly decoded.  Had no idea about "Simple Records". 

 

Anyway, the obvious:

Vertex #0: 0E 00 51 46 A2 D0 24 45 C2 EE 5F 44

Is just 3 floats XYZ:  13376.01, 2637.04, 895.7306

Triangle #0:  00 00 01 00 02 00 FF FF 01 00 07 00 00 08 00 00

First 6 bytes are vert indices for points ABC of the triangle:  0, 1, 2  (ushorts)

 

The remaining bytes likely store data for the triangle and its sides AB, BC, and AC. 

[FF FF] [01 00] [07 00] [00 08] [00 00]

Somewhere in there is whether or not it's a portal triangle, water, preferred path, dropdown, and what sides can be used as cover. I assume.

 

 

Re: my first point,  It would be nice if the raw data showed up by double clicking on the header of a subrecord/group/tree (don't now the exact terminology), or some other way.  In this example it would be the "Vertices" and "Triangles" trees that I would like to copy in bulk.  Or just the entire NVNM - Geometry section.

[Sharlikran]

I recently closed the forum because a user was complaining about merging mods together and having complications with it.  I am very reluctant to have people changing NAVM with TES5Edit.  I will reopen the forum after I think about it more and change the description page.

 

I know there have been conflicts with NavMeshes and some people like yourself might be very intelligent.  However, the majority of people don't know anything about plugins to begin with.  When they use a script or make changes, they do it incorrectly and blame it on the program.  If NAVM isn't decoded completely I'm sure it can be decoded more but I still don't feel comfortable with people using TES5Edit to change it.  Just because the program allows you to change it doesn't mean that people should be changing it in anything but the CK.

 

I'm sure it's obvious to anyone that if even one byte of information is incorrect the NavMesh could be completely invalidated and NPC's won't be able to use it.  TES4/FO3/FNVEdit and Wrye Bash have never merged or changed it so why are people expecting TES5Edit to be able to do it now?  Having worked on internal and external NavMeshes myself I understand why people may not want to resolve conflicts between two mods using the CK.  However I feel it is counterproductive to try and correct the conflict for hours if not days with TES5Edit only to give up or realize that the only way to resolve the issue or problem would be to use the CK.

 

For that reason I feel that people should start getting used to making patches for two mods and then correcting the NavMesh in the CK even when doing so could mean changing 10 or more cells.  When people feel there would be too many NavMeshes to change and they don't want to do so then the mod should be labeled as incompatible for NavMesh conflicts in BOSS.

 

Since it seems to be such a big deal to resolve the NavMesh conflicts I will work With Lojack to see how feasible it would be to get Wrye Bash to merge NavMesh patches into the Bash Patch with a new tag.  When you have NavMesh conflicts between ModA and MODB you would make them masters, load them into the CK and, make a patch MODC for example that only has NavMeshes in it.  A tag would be added to the patch so that the winning NavMesh is copied, as is, no changes or bit wise operands, just dumped in as it's read from the file into the Bash patch to mitigate the conflict.  It would be added to the Bash Patch so it doesn't take away from the 255 plugin cap.  However, I won't be proposing this until most of the restructuring is done and Wrye Bash is upgraded to v3 of Python.  There is still too much to be done to be implementing any new features.

[Jon]

Edit2:  I didn't really address your post, but that's because my question had absolutely nothing to do with navmesh conflicts.    Aside from trying to decode the rest of the NVNM, I'm asking if:

 

1) There is a way to copy the raw data from a sub-record/tree/group of lines. 

 

2) If there is not a way to do so, is it possible to add this.  I assume the only alternative is to use Dump instead of Edit.

 

~~~~~~~~~~~~~~~~~~

 

OK, so some progress.

 

Take for example WhiterunJail02, it has 16 groups of triangles at the end, and accounts for the rest of the unknown data (at least in this case).   This includes the first group of triangles actually listed in Edit as "(Unknown) Triangles"

 

 

I've visualized the groups in a gallery:

 

http://imgur.com/a/3dP5T#0

 

Edit: Added animation since I figured out why the groups are grouped this way

 

These are the 16 groups in the order which they occur in the data.  There doesn't really seem to be a definitive way to say why they are in these groups.  They are not convex or an all-inclusive group interior to some bounding box.  There are a few groups which should include other triangles if it were a bounding box.    There is a definitive reason.  Pics incoming.

 

There is a clear pattern to how the groups are ordered however.  You can easily see this while clicking through all the images.

 

Edit:

 

Interesting to note that 16 = 12 + 4 which are two of the unknown values in that record.  Could be coincidence but I will look into it.   So far I don't think this is the case.

[Sharlikran]

Edit2:  I didn't really address your post, but that's because my question had absolutely nothing to do with navmesh conflicts.    Aside from trying to decode the rest of the NVNM, I'm asking if:

 

1) There is a way to copy the raw data from a sub-record/tree/group of lines. 

 

2) If there is not a way to do so, is it possible to add this.  I assume the only alternative is to use Dump instead of Edit.

That's quite alright. I go off on tangents sometimes but, normally not without a reason. At least in my mind anyway. I don't want to stop you from decoding it. Besides since decoding NAVM will inevitably lead to requests to edit, resolve, or merge something my post will still be there if it becomes relevant.

[zilav]

Thanks for taking your time into decoding the rest of the unknown values.

Just post the structure for the unknown data in any format you like, even C/C++ will suffice.

[DayDreamer]

When CK is used to delete a script that is on the underlying base object, it shows a red bar and the hover hint says "script inherited and deleted locally".

 

View by TES5Edit, VMAD has the script name and an unknown value 3. My guess is that means deleted.

 

There are some examples in USKP should you need them: 001051d5, 001051d6

[Jon]

From the "Simple Records" view, the entire block of unknown data should be:

// 010 Editor Template
// --------------------------------------

typedef struct {
    ushort length;
    ushort blank;
    if ( length > 0 )
        ushort tris[length];
} INDICES;

struct FILE { 
    INDICES covertris;
    ushort divisor;
    ushort blank;
    float floats[8];  // Max X Dist, Max Y Dist, Min X/Y/Z, Max X/Y/Z
    INDICES chunks[divisor*divisor] <optimize=false>;
} file;

Obviously I was too lazy to write out the floats separately.

 

What I am not 100% sure of is if these chunks are a 2D or 3D structure.  From what I can tell the chunks are not split into N chunks across Z (where N is the "divisor").  So for all I know the # of chunks is always N^2 and not N^3.   But I have only tried on a handful of navmeshes.   Off hand I don't even remember if one navmesh can have overlapping triangles in Z.  I would assume it can and does.  What I will do is try to find such a navmesh in order to confirm it's not split into N^3.

[hlp]

Zilav, it looks like FNV has some weather records copied over from FO3 (ie [00092BC8]), without the new period in the array. FNVDump gets confused

We may need a union based on the record size for PNAM and NAM0

[zilav]

Not only weather in FNV, but few other records have less size than expected in all of the games, even Oblivion. Probably left from the earlier development stages.

Yes it shows "Expected ... bytes", but what so bad about it? How is it confused? Do your recent updates made them worse somehow? I remember those "expected..." records from my early days of oblivion modding, but they have never been a problem.

Every added decider is a slow down factor for xEdit and quite possibly even a memory leak contributor (xEdit is leaking memory unfortunately, and has always been). Is it worth it just to hide warnings for a few legacy records? In the end it is up to you to decide really, you are the lead programmer now

I'm just being a little conservative here.

[zilav]

Uploaded a new version with VMAD and NAVM updates.

Hlp fixed an issue introduced with autocounters, so update asap please.