[WIPz] TES5Edit

[alt3rn1ty]

I dont know if I have r1630 or the newer svn at the moment, but I think it probably will be present in both :

 

Found a minor issue

 

 

 

Drag the indicated separator left and watch what happens. They start to overlap each other, so resizing those columns is not functional

[Sharlikran]

Drag the indicated separator left and watch what happens. They start to overlap each other, so resizing those columns is not functional

It has always done that, AFAIK. Which is why when I open any version of xEdit I have to resize it so I can see the columns I want. Since I know some overlap, I expand the window size to fit what I want on the left and right sides before I start doing anything. That is probably form VirtualTrees.

[Octopuss]

I downloaded the latest revision and noticed obscenely huge difference in memory usage compared to the latest version from Nexus. Previously, loading all the plugins and creating a merged patch ended with 1GB used memory. Now it's just 550. Wow.

[zilav]

I downloaded the latest revision and noticed obscenely huge difference in memory usage compared to the latest version from Nexus. Previously, loading all the plugins and creating a merged patch ended with 1GB used memory. Now it's just 550. Wow.

From the changelog

- fixed memory leaks

[Octopuss]

I know. That's why I posted it.

[Arthmoor]

Did you guys mark the Default Object Manager entry as an ITM? I just happened to note that with 3.0.33 it's marked in green, which would indicate it gets cleaned out by the cleaning process. I'm not sure that should be done since it contains entries from DLC and official patches that aren't in Skyrim.esm.

[ElminsterAU]

The DOBJ record does not operate on a "last one is winner" principle. Instead, it merges them all together, keeping only the last value for each specific "Use" entry.

 

For that reason, three changes have been made

* Redundant None/NULL entries get removed

* Entries are sorted by Use value (which makes the line up correctly)

* For each entry, the last one that sets a value should be considered as the winning entry (so if there are DOBJs later in the load order which don't have a value for a specific use value, the conflict detection shouldn't see that as overriding whatever the last value actually was).

 

Now, there is currently something wrong with the implementation of that last point. It *should* mark the first mod that sets a value for a specific use as "master". It currently always sets the left-most (always skyrim.esm) as master, even if that doesn't define a value for a specific Use at all.

 

With that fix in place (which I'll check in shortly), it now correctly marks the first mod that defines a specific use value as master. As long as there aren't any overrides or even conflicts, you'll probably still see the DOBJ show up with a green background, but a green background doesn't indicate "identical to master", it means "no conflict". "Identical to master" is signalled by a grey text colour. But after the fix, you'll see that end up being purple (meaning "master") in most cases.

 

Thanks for pointing out the problem.

[ElminsterAU]

Addition to the above: While the automatic remove ITM function isn't going to remove individual entries from the DOBJ record, authors of mods that do contain a DOBJ record (which outside of the official DLCs are exactly two in my load order of 180 module files: Immersive Armors and Weapon & Armor Fixes) should manually check their DOBJ and consider removing any unneeded entries as the "last one wins" rule applies to individual entries in the DOBJ the same way it applies normally to full records and can cause the same type of compatibility problems as ITM records.

 

Though given how rare it is that a mod actually contains a DOBJ record, this is probably a very minor issue.

[Arthmoor]

I was more concerned about the fact that, at present, 3.0.33 is removing the entire DOBJ record from each of the DLCs when that shouldn't be happening. I can see from how it's being displayed that this shouldn't be the case, but it is

[zilav]

I was more concerned about the fact that, at present, 3.0.33 is removing the entire DOBJ record from each of the DLCs when that shouldn't be happening. I can see from how it's being displayed that this shouldn't be the case, but it is

Uploaded new version, tested on DB and DOBJ is not removed.

[ElminsterAU]

On a totally different topic, anyone got a good Delphi implementation for multiple sequence alignment?

[zilav]

On a totally different topic, anyone got a good Delphi implementation for multiple sequence alignment?

When I was working on some bioinformatics software 10 years ago, I had the same problem but ended up integrating with Clustal and Blast.

Clastal has sources available http://www.clustal.org/omega/clustal-omega-1.2.1.tar.gz and it is the best afaik.

Looking back at it now, it was somewhat similar to xEdit when comparing sequences

[ElminsterAU]

I'm looking for a solution to produce the optimal alignment of CTDAs (without sorting them).

 

As an example, if you load only Skyrim.esm, Update.esm and HearthFires.esm, then have a look at:

 

[iNFO:0008BBDD] ('Constance don't care much for strangers in the orphanage. Afraid someone might hurt one of us kids. Which, I gotta admit, is kind of nice.' in GRUP Topic Children of DialogueRiftenHellos [DIAL:00040760])

 

This currently shows as a conflict. But if you look at it closely, you can see that Update.esm changed the Type for the GetDeadCount call from "Equal to" to "Greater than or equal to". And HearthFires.esm added a new, additional condition at the top.

 

With proper alignment, it would be easily visible that the last the last two conditions have remained unchanged in all 3 files, the GetDeadCount condition had its type changed in Update.esm and HearthFires.esm is correctly forwarding that change (resulting in a clean overwrite) and the new condition from HearthFires is added before the inherited conditions.

 

Basically, what I need is a function that I feed an array of stringlists (filled with SortKeys) in, and it adds additional empty lines into these stringlists to to make them all the same length and line up as well as possible (but if possible, do not require exact string matches, just similar strings, otherwise the CTDA for GetDeadCount wouldn't line up).

[Arthmoor]

SVN 1696 generates an assertion when attempting to run Update.esm through a cleaning and then save the file.

 

Actually, correction, it seems to be happening when attempting to save any file after cleaning it.

TES5Edit_bugreport.txt

[ElminsterAU]

Something to do with the Objects array in DOBJ not having the expected size after writing it into the file. I'll check it out in more detail this evening when I'm back from work.

[ElminsterAU]

Ok, the problem originated from wbDOBJObjectsAfterLoad, which called ObjectsContainer.RemoveElement(i, True);

 

The implementation of function TwbContainer.RemoveElement(aPos: Integer; aMarkModified: Boolean = False): IwbElement; called InvalidateParentStorage (when aMarkModified is true)., when it really should be calling InvalidateStorage;

 

Because of this, when the Objects array got saved, the "ExpectedSize" was the old size (with all the None/NULL entries still included), while the actual save only wrote the 5 entries that actually still existed into the stream. A sanity check afterwards that compares the before/after stream position with the expected size then raised the Assert.

 

I'll check a fix in shortly.

[zilav]

Uploaded new version.

[alt3rn1ty]

Results with latest svn 1697

 

Original esm's used

Update.esm
Apply filter for cleaning ( First Time around )
[Removing "Identical to Master" records done]  Processed Records: 1290, Removed Records: 96, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 1194, Undeleted Records: 3, Elapsed Time: 00:00
Update.esm
Apply filter for cleaning ( Second Time around )
[Removing "Identical to Master" records done]  Processed Records: 1194, Removed Records: 0, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 1194, Undeleted Records: 0, Elapsed Time: 00:00

Dawnguard.esm
Apply filter for cleaning ( First time around )
[Removing "Identical to Master" records done]  Processed Records: 98027, Removed Records: 630, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 97397, Undeleted Records: 82, Elapsed Time: 00:06
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 57 deleted references which can not be undeleted>
Dawnguard.esm
Apply filter for cleaning ( Second time around )
[Removing "Identical to Master" records done]  Processed Records: 97396, Removed Records: 6, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 97390, Undeleted Records: 0, Elapsed Time: 00:06
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 57 deleted references which can not be undeleted>
Dawnguard.esm
Apply filter for cleaning ( Third time around )
[Removing "Identical to Master" records done]  Processed Records: 97390, Removed Records: 0, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 97390, Undeleted Records: 0, Elapsed Time: 00:06
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 57 deleted references which can not be undeleted>


Hearthfires.esm
Apply filter for cleaning ( First time around )
[Removing "Identical to Master" records done]  Processed Records: 17797, Removed Records: 184, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 17613, Undeleted Records: 11, Elapsed Time: 00:01
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 5 deleted references which can not be undeleted>
Hearthfires.esm
Apply filter for cleaning ( Second time around )
[Removing "Identical to Master" records done]  Processed Records: 17613, Removed Records: 0, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 17613, Undeleted Records: 0, Elapsed Time: 00:01
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 5 deleted references which can not be undeleted>


Dragonborn.esm
Apply filter for cleaning ( First time around )
[Removing "Identical to Master" records done]  Processed Records: 214386, Removed Records: 64, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 214322, Undeleted Records: 8, Elapsed Time: 00:07
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 1 deleted references which can not be undeleted>
Dragonborn.esm
Apply filter for cleaning ( Second time around )
[Removing "Identical to Master" records done]  Processed Records: 214322, Removed Records: 0, Elapsed Time: 00:00
[Undeleting and Disabling References done]  Processed Records: 214322, Undeleted Records: 0, Elapsed Time: 00:07
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 1 deleted references which can not be undeleted>

[rickychris90]

Earlier today I applied a filter to only show ITMs on my whole load order, using the last revision.

And it showed that a lot of mods that were considered clean before, now aren't.

From what I could see, the records highlighted are completely the same, save for the "version control info" field...so I guess that it's the intended behavior.

BUT in some cases they were considered ITMs even though there was a change in the "XLRL - Location Reference" field, for example in an Unofficial Dawnguard Patch record of Abandoned Shack...is this still normal?

[Arthmoor]

You should never perform cleaning with your entire load order loaded at once. The results will not come out the way you are hoping.

 

ALWAYS clean things one file at a time, starting from the top of the load list and moving toward the bottom.

 

Yes. It's tedious. It's no fun. It's going to take a lot of time for a big load order. It's also the only way the process can work correctly.

[zilav]

Added XLRL subrecords are treated as ITM, this was discussed some time ago. If XLRL already existed and was modified, it won't be ITM.

[Arthmoor]

Hasn't that already been getting treated that way for some time now?

[rickychris90]

@Arthmoor: yeah I know, I wasn't cleaning them... just applying a filter to see which plugins were considered to have ITMs, to later clean those one by one.

@Zilav Oops, found it now in the changelog of version. Didn't know about that, sorry for the noob question!

[zilav]

New version uploaded, details in the first post.

Please keep an eye on your plugins, there were quite a lot of changes and major speed optimizations.

[Arthmoor]

<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>
<Warning: Plugin contains 1 deleted references which can not be undeleted>

 

What's the second one indicate? Never seen that one come up before. Is there a way to find it? I tried filtering for deleted items but only got the navmesh when I did.