[WIPz] TES5Edit

[Mator]

I know this may not be feasible for you and it's not what you are asking about but does it happen on another machine without MO.  MO2 for Fallout 4, even though I hear Beta 1 is working better is trying to account for the fact that xEdit is a 32-Bit app.  Programs like LOOT, Wrye Flash, and xEdit will not be built to be 64-bit apps.  So while your issue seems simple enough if you take MO out of the equation does it still happen?  If not then it's because of how MO2 handles 32 bit apps and Tannin just needs to do more work still.  It may not be anything you are doing wrong.  Skyrim ran in a 32-bit environment and MO for Skryim used a 32-bit environment.  Since TES5Edit was a 32-bit app it may not have had any issues so what I am saying is pertinent.

 

I'm using Mod Organizer 1.3.11, and TES5Edit works fine.  It's just Mod Dump that isn't working and I currently have no idea why.  It's weird.  Probably something that I'm doing but I can't for the life of me figure out what.  I can't even load Skyrim.esm when running through MO, but can load it when not running through MO.  Just all my what.

 

 

You asked, "If I want to find all errors in a particular plugin file, can I build references ONLY for that plugin file?" The answer is: yes. Perhaps you asked the wrong question?

 

I always skip building references when I load plugins. I was just helping Zilav with a Check for Errors issue when you asked your question, and Check for Errors without building references output unresolved/unexpected reference errors.

 

I can't help you with ModDump. No idea what that is.

 

 

I'm sorry if my question was unclear, I wasn't asking whether it was possible to build references for a single plugin file, I was asking if I wanted to find errors in a particular (and if some errors could only be found by building references), if building references for only that plugin would be sufficient.

 

I think you answered the question about errors not being related to building references, but it does mean then that I have something else going wrong because some errors are not appearing.

 

ModDump is a standalone command line application I'm building to run off of the command line (or as a DLL) to produce a dump of mod information.  https://github.com/matortheeternal/mod-dump

Nothing all that exciting, though I do think it's the first time someone has exposed xEdit functionality to another application through a DLL.

[Sharlikran]

ModDump is a standalone command line application I'm building to run off of the command line (or as a DLL) to produce a dump of mod information. https://github.com/matortheeternal/mod-dump

Nothing all that exciting, though I do think it's the first time someone has exposed xEdit functionality to another application through a DLL.

Silly question maybe but why do that when you have FO4Dump, or TES5Dump or whatever. It writes a text file. Are you making ModDump do something different?

[Mator]

Fixed the issue, turns out Mod Organizer messes with the CurrentDirectory variable like constantly, so I have to use fully qualified paths for everything.  >_>'

 

Sharlikran:

It does different stuff.  Mod Dump produces a JSON dump with specific information on a plugin file, sort of like this:

http://puu.sh/p2NzR.json

 

It does so extremely quickly and can generate dummy masters (empty plugins) if master files are missing.  It can be used through command line or as a DLL.

[Arthmoor]

Off the wall observation here, but is it not default behavior for xEdit to build reference data for all plugins unless you tell it not to? The only thing you'd need to go out of your way to build them for is the game's primary master file.

[Mator]

Figured out the issue with certain errors not appearing - they don't get returned through the check function like other errors do, so my code to capture them was failing.

 

Guess I'll have to write a new Check function.

 

 

Off the wall observation here, but is it not default behavior for xEdit to build reference data for all plugins unless you tell it not to? The only thing you'd need to go out of your way to build them for is the game's primary master file.

 

Yep, that's correct.

 

However, I honestly think this should not be default behavior.  It takes a long time (up to 2 minutes) to build references for a lot of large plugins, and most of that data would never be used during the xEdit session.  Kind of a waste of time if you ask me.  It'd be a little better if it skipped DLC files by default in addition to Skyrim.esm, but mneh.  ¯\_(ツ)_/¯

[Arthmoor]

Wow, I've never had it take more than about 40 seconds to load USLEEP with all the DLCs it requires behind it WITH all the reference data being built.

[Crashpilot]

You picked my curiosity so I checked; I have 218 active MODs including some rather large quest MODs, a lot of texture replacers, the unofficial patch and the bashed patch, time to process all was about 45 seconds.

[Mator]

You picked my curiosity so I checked; I have 218 active MODs including some rather large quest MODs, a lot of texture replacers, the unofficial patch and the bashed patch, time to process all was about 45 seconds.

 

Depends on your CPU Clock speed, ultimately.  My quoted time of 2 minutes is from when I was modding on my old computer where the CPU clock speed was around 2.2GHz.

 

On my current computer with loading 80 mods it takes 77 seconds.  If I skip building references it takes 10 seconds.

 

 

@Zilav: I've been trying to figure out how to capture information on Unexpected/Out of Order subrecords, but haven't figured out how to do that yet.  What would you suggest

[fireundubh]

If reference building takes less than 45 seconds for game + DLC, either you have a really fast system or you are building references when you load up xEdit.

Hold SHIFT when you load plugins to skip building references for all plugins, and then build references for all plugins manually to get your actual reference building speed.
 
Fallout 4:

[00:00] Building reference information for [03] DLCCoast.esm
[00:17] Building reference information for [01] DLCRobot.esm
[00:21] Building reference information for [02] DLCworkshop01.esm
[00:21] Building reference information for [00] Fallout4.esm
[01:49] Building reference information for [00] Fallout4.exe
[01:49] All done!

Skyrim:

[00:00] Building reference information for [02] Dawnguard.esm
[00:06] Building reference information for [03] Dragonborn.esm
[00:16] Building reference information for [04] HearthFires.esm
[00:18] Building reference information for [00] Skyrim.esm
[01:16] Building reference information for [01] Update.esm
[01:16] All done!

xEdit f416473

[Mator]

Figured out how to extract the unexpected subrecord error data.

I did it by adding a new string property to IwbMainRecord which will be populated with a list of signatures of the subrecords that are out of order in a given record.  The list will be populated in TwbMainRecord.Init at the same time as the messages are printed to the log.  Not sure if it's a change you'd want in the xEdit repo, but I'm happy to make a branch and a PR if you want.

 

Here's the commit: https://github.com/m...5a987d0a5ccb98b

[lmstearn]

Fixed the issue, turns out Mod Organizer messes with the CurrentDirectory variable like constantly, so I have to use fully qualified paths for everything.  >_>'

Tannin will want to look at that. The current working directory is global. Any thread can change it at will.

[zilav]

Figured out how to extract the unexpected subrecord error data.

Those are structural data errors along with unused and insufficient data, they are reported through wbProgressCallback() so intercept them in that handler.

You don't need to build references to check for errors, none of the current custom checking functions use them and probably never will.

[Mator]

Those are structural data errors along with unused and insufficient data, they are reported through wbProgressCallback() so intercept them in that handler.

You don't need to build references to check for errors, none of the current custom checking functions use them and probably never will.

 

The issue with intercepting wbProgressCallback() is it's a really annoying and inefficient way of doing things.  wbProgressCallback exists for the explicit purpose of logging, when what I really need is just the raw data associated with the error (in this case, the subrecord signatures that are out of order).  Using wbProgressCallback() requires me to do a lot (way too much) parsing work which wouldn't be good architecture.  I'm already doing a fair bit of parsing work for the error messages that are returned through the GetCheck function using regular expressions, which is bad enough already.  With the way that I implemented subrecord errors, I didn't have to do this at all.

 

It really doesn't matter that much and I'm sure you guys probably don't care, but I think errors should be on a file-by-file basis, where each file has a list of error objects which are themselves comprised of the data necessary to construct an error message.  There should be a single function which can be called which loops through the records/elements in a file and evaluates the errors that are present, and adds them to the list.  You can then have any kind of view logic for this data (instead of being restricted to JUST log messages), can expose the error data to external interfaces cleanly, and can potentially save/track error data between sessions (I do this for merge plugins, where the error data is tracked relative to a plugin filename + crc32 hash combination).

 

 

Tannin will want to look at that. The current working directory is global. Any thread can change it at will.

 

He's been working on MO2 for awhile now.  I don't think he'd be very interested in a (likely known) side effect of the VFS in v1.3.11.  I'd rather not bother him with it.

[zilav]

It really doesn't matter that much and I'm sure you guys probably don't care, but I think errors should be on a file-by-file basis, where each file has a list of error objects which are themselves comprised of the data necessary to construct an error message.  There should be a single function which can be called which loops through the records/elements in a file and evaluates the errors that are present, and adds them to the list.  You can then have any kind of view logic for this data (instead of being restricted to JUST log messages), can expose the error data to external interfaces cleanly, and can potentially save/track error data between sessions (I do this for merge plugins, where the error data is tracked relative to a plugin filename + crc32 hash combination).

xEdit parses records on demand, so keeping track of additional errors objects per file is like another form of reference building with all the update code overhead on every operation and increased memory usage. Doesn't worth it for the purposes of xEdit since showing errors to the user is more than enough. I can see it being useful for your task, but you already solved that

[Mator]

xEdit parses records on demand, so keeping track of additional errors objects per file is like another form of reference building with all the update code overhead on every operation and increased memory usage. Doesn't worth it for the purposes of xEdit since showing errors to the user is more than enough. I can see it being useful for your task, but you already solved that

 

That's true, if you wanted it to remain accurate as the user updates a file.  I think the way I'd do it is if the file's CRC32 changed I would just clear the error information and require it to be re-determined.  But you're right that my use case(s) differ a bit from xEdit's.  My main point is just in making errors accessible to external interfaces.  It's "good enough" with what I built, but could be better I feel.  But maybe I'll end up making an xEdit DLL some day and will handle it then.

[Kesta]

Changing the subject (but keep going, an xEdit .dll is definitely an interesting project ).

 

Will the "Localize Plugin" feature work properly for FO4 ? (since stuffs changed regarding string files encoding for some languages ?)

Or... is there a way to localize a plugin from the CK ? I actually never did this, and was pretty surprised that I could not find an option to do so.

[Nico coiN]

@Kesta : to be honest, localization has technically no use at all. It can even cause an instant CTD if the appropriate localized strings files are missing (an english-only localized plugin would not run on any non-english game version). However, one of the unexpected advantage of strings files is that it enables almost any typo correction without using an esp/esm, and so avoids conflicts with large mods that edit many records. I guess you're aware that the french USLEEP port is far more than a simple translation...  We also now have Skyrim and FO4 master esm that are absolutely identical all over the different countries.

[Mator]

I think an xEdit DLL is completely possible, the only difficulty is in regards to how elements (IwbElements, and types that inherit from it) should be handled.  In order to be useful the DLL should never pass these elements out, but doing that means the DLL needs to pass out pointers.  Ultimately the programmer will want to be able to work with "element-like" objects, but only needs them as arguments to pass back to the DLL.  That means the content of what is passed out of the DLL is irrelevant so long as the DLL can know what the user is talking about when they pass an element back into the DLL as an argument.  Possibly the best way to do this is just pass pointers back and forth.  Beyond that it's literally just a matter of wrapping the xEdit framework functionality in functions in a manner similar to what was done for xEdit scripting.  It's really fairly trivial to do (I already did it to an extent in ModDump), and it'd allow any language which can use DLLs to use xEdit as a framework for handling plugin files, including (but not limited to) Delphi, Python, C++, C#, Visual Basic, Java, and many more.

 

---

 

On another topic:

Zilav, I had someone who had merged a plugin a bunch of times in a row and because of how Merge Plugins renumbers FormIDs, ran out of FormID space in their plugin (the guy from reddit here).  I wrote a little algorithm which can compact FormIDs in a plugin file, and will likely use some of the logic in Merge Plugins in the future.  However, I was thinking for now it might make sense for xEdit to be able to compact FormIDs for a given plugin file (I have a script that does it, but it could be added natively).

[Kesta]

@Mator: As far as an xEdit .dll goes, I'd just need one class per group, each with a set of property representing fields with getter/setter and enumerators for array, and constructors that initialize the pre-req values. I know, not that simple since several fields can support a variety of forms but not all of them (typically refr's base object), but that's the general idea. Eventually other xEdit base functions in a secondary .dll (check for errors, remove ITM, ...).

 

@Nico: Yeah, I know they crash the game if the string files for the language used is missing, I intended to use dummy for non-translated languages. (foreign-language string files, but with english text in). The advantage isn't only that you can do it without the .esp, but also that you can pack all of them in a single archive... pretty convenient with a modding platform where you can only upload one plugin/ba2 couple per mod (Hi Bethesda.net ). But this should probably be talked directly with the Beth guys since they currently seems to listen (at least, partially) to suggestions.

[Mator]

Kesta: But that's the thing, you'd have to define those classes in the language you wanted to use the DLL in, which is basically repeating the decoding work of xEdit.  The whole point in making a DLL is to avoid having to do that work, so you can just include the DLL and then execute some code using the existing xEdit definitions encapsulated within it.

[zilav]

Will the "Localize Plugin" feature work properly for FO4 ? (since stuffs changed regarding string files encoding for some languages ?)

I don't see why it wouldn't. There are no changes between Skyrim and FO4 regarding how the localization works.

 

 

Zilav, I had someone who had merged a plugin a bunch of times in a row and because of how Merge Plugins renumbers FormIDs, ran out of FormID space in their plugin (the guy from reddit here).  I wrote a little algorithm which can compact FormIDs in a plugin file, and will likely use some of the logic in Merge Plugins in the future.  However, I was thinking for now it might make sense for xEdit to be able to compact FormIDs for a given plugin file (I have a script that does it, but it could be added natively).

That is a very specific task. In 8 years of xEdit history this is the first time I hear that someone ran out of FormIDs space, probably because one of plugins used very high values for it's records for some reason which leaved huge empty unused range(s). I prefer that you'll implement that in your merger directly since it is relative to merging task. And/or if you can write a script for that, I'lll include it with xEdit.

[Mator]

That is a very specific task. In 8 years of xEdit history this is the first time I hear that someone ran out of FormIDs space, probably because one of plugins used very high values for it's records for some reason which leaved huge empty unused range(s). I prefer that you'll implement that in your merger directly since it is relative to merging task. And/or if you can write a script for that, I'lll include it with xEdit.

 

I agree, I've never had anyone run into this before either.  It's pretty weird.  And yes, it was a matter of there being huge unused ranges.  I expand on the problem and solution in this PSA on reddit.

 

I'm happy to integrate it directly into merge plugins (I actually already have).  But I still think it could be a useful function in xEdit.  (similar to renumber Form IDs, but more intelligent)

[Arthmoor]

Having some minor trouble with Skyrim - Tree LOD files patcher.pas. I've got a cell where some of the trees are being incorrectly positioned too high up. You can see an example of this in this guy's video here:

Obviously if I do not run the tree LOD script, the cell in question simply never displays the LOD trees.

000ac1e2, 000ac1e4, and 000ac1b7 are the 3 I can readily identify as being wrong, but I think there's more. The z-height is not getting handled correctly on them.

meshes\Terrain\tamriel\trees\Tamriel.4.36.-16.BTT and meshes\Terrain\tamriel\trees\Tamriel.4.40.-16.BTT are the two affected files. If you need them, they can be downloaded here: http://www.iguanadons.net/trees.7z

 

Please don't suggest that I just use DynDOLOD. That's way too much of a hassle to go through for a couple of trees, but I'd like to have them fixed so I can put the proper tree LOD into the mod

[zilav]

Those are not vanilla trees, so he MUST use DynDoLOD/TES5LODGen with proper billboards. The issue is not a wrong Z height, but wrong LODs. The patcher script itself doesn't even touch Z values, only X and Y.

[Arthmoor]

Except they are vanilla trees because I was able to validate them as such in my own game and when I remove the tree files that xEdit patched, they return to their normal positions at those spots.

 

What do you mean by wrong LODs?