Jump to content

[RELz] LOOT - Load Order Optimisation Tool


WrinklyNinja

Recommended Posts

2 hours ago, WrinklyNinja said:

I've been working on improving how LOOT serialises data when it's sent between the C++ backend and the JS frontend, as currently LOOT is re-using the YAML serialisation that the LOOT API uses, which is messy and fragile to do. What I wanted to do instead was define a schema for the JSON that gets transmitted, and generate code from that, rather than having to write the serialisation code myself all over again.

After trying a bunch of different libraries, I've gotten it working with Protocol Buffers, and you can try it out with this build. One thing that is untested but probably not working is validation of metadata added in the UI - I've left every field optional in the schema, which isn't actually always the case (Bash Tags need names, for example), and condition syntax isn't checked, so I probably need to add that back in, but I'm still exploring how best to do that.

Using Protocol Buffers only to work with JSON is overkill, but unfortunately the other libraries didn't fit my requirements. There is a lighter version of Protocol Buffers I could use though, if I used its binary encoding instead of the JSON, which I may do in future.

Since this schema-first approach is working for LOOT, I'm now thinking of adopting it for the API, so that the interfaces give and take the serialised Protobuf binary format instead of the C++ classes they currently do. It's less efficient, but more flexible, and probably a better fit overall.

Found three things with this build. First, the SOC filter only shows the plugin card of the plugin you are filtering for, not the plugin cards of the plugins that are in conflict with the target plugin. Second, this tooltip get's cut off. Third, if a cyclic interaction is detected, LOOT will recognize this and display this in the general information card, but the window that pops up and informs the user about this will not be displayed.

Link to comment
Share on other sites

2 hours ago, pStyl3 said:

Found three things with this build. First, the SOC filter only shows the plugin card of the plugin you are filtering for, not the plugin cards of the plugins that are in conflict with the target plugin. Second, this tooltip get's cut off. Third, if a cyclic interaction is detected, LOOT will recognize this and display this in the general information card, but the window that pops up and informs the user about this will not be displayed.

Thanks, I've fixed all those and re-implemented the metadata editor validation in this build.

Link to comment
Share on other sites

I'm thinking of changing LOOT settings file format from YAML to TOML to swap a large, complex dependency for a much simpler, smaller one. However, LOOT wouldn't be able to automatically upgrade a YAML settings file to a TOML settings file itself, so do people have any preference on how the upgrade should be handled?

My current best idea is to add a page to LOOT's website that people can copy/paste their settings.yaml into and get a settings.toml out, and if LOOT is launched and finds a settings.yaml but no settings.toml, it will automatically send the settings.yaml to the page to be converted, and use the result. I don't think it should remove the settings.yaml afterwards, in case people have comments or extra bits in there that don't get converted.

EDIT: Actually, the automatic bit wouldn't work since it would be client-side JS, but LOOT could run that itself 'in-browser', and js-yaml is still a lot smaller than yaml-cpp... Though there are probably initialisation order issues there, with running JS to read the settings that might be necessary for initialise a browser window.

Link to comment
Share on other sites

1 hour ago, WrinklyNinja said:

Thanks, I've fixed all those and re-implemented the metadata editor validation in this build.

I can confirm them being fixed. That said, I've found that LOOT crashes if one tries to change the current game LOOT handles. I experienced the crash for both Skyrim-->SSE and Skyrim-->FO4 (without MO being involved). Here is my LOOTDebugLog.txt.

Edit: LOOT does not crash if I switch from Skyrim to FNV, only if I switch to either SSE or FO4.

Link to comment
Share on other sites

9 hours ago, pStyl3 said:

I can confirm them being fixed. That said, I've found that LOOT crashes if one tries to change the current game LOOT handles. I experienced the crash for both Skyrim-->SSE and Skyrim-->FO4 (without MO being involved). Here is my LOOTDebugLog.txt.

Edit: LOOT does not crash if I switch from Skyrim to FNV, only if I switch to either SSE or FO4.

Damn, I can't replicate the crash. Does this happen with the latest update-api build too? If so, can you try a few older builds to see when it started happening? Does it happen if you set SSE or FO4 to be the game on startup and switch to Skyrim or FNV?

EDIT: Also, if you set the RUST_BACKTRACE environmental variable to have a value of 1, that might give more info (if the crash is happening in libloadorder).

Link to comment
Share on other sites

It happens with "loot_0.11.0-84-g4581a9c_protobuf.7z", "loot_0.11.0-67-gb2ef365_update-api.7z" and "loot_0.11.0-41-gf3d3bce_update-api.7z" but not with "loot_0.11.0-40-gf1e4c4f_dev.7z". The crash also occurs if I set LOOT to start with SSE or FO4 (and not if it is FNV). If I set the starting game to be SSE/FO4 the general information card will include the following:

"Error: Settings parsing failed. bad conversion"

And CEFDebugLog.txt will include:

"[1030/093605.530:WARNING:dns_config_service_win.cc(669)] Failed to read DnsConfig."

I don't know where to find the RUST_BACKTRACE variable.

Link to comment
Share on other sites

9 hours ago, pStyl3 said:

It happens with "loot_0.11.0-84-g4581a9c_protobuf.7z", "loot_0.11.0-67-gb2ef365_update-api.7z" and "loot_0.11.0-41-gf3d3bce_update-api.7z" but not with "loot_0.11.0-40-gf1e4c4f_dev.7z". The crash also occurs if I set LOOT to start with SSE or FO4 (and not if it is FNV). If I set the starting game to be SSE/FO4 the general information card will include the following:

"Error: Settings parsing failed. bad conversion"

And CEFDebugLog.txt will include:

"[1030/093605.530:WARNING:dns_config_service_win.cc(669)] Failed to read DnsConfig."

I don't know where to find the RUST_BACKTRACE variable.

Weird, I can't see why settings parsing would be at all related, since it seems related to the new libloadorder... My guess is that there's an out-of-bounds index happening somewhere.

See here or here on how to set environmental variables on Windows.

Link to comment
Share on other sites

So I'll created a shortcut pointing to LOOT.exe, opened it's properties and entered the following for the path (following the example here):

C:\Windows\System32\cmd.exe /c "SET RUST_BACKTRACE=1 && START /D ^"C:\Spiele\Modding\0_Programme\LOOT^" LOOT.exe"

No idea if that is even close to what you want me to do wrinkly. Even if that's correct .. and now? If I run LOOT that way, it still crashes .. I see no difference, not in LOOT nor in the Debug Logs.

Link to comment
Share on other sites

28 minutes ago, pStyl3 said:

So I'll created a shortcut pointing to LOOT.exe, opened it's properties and entered the following for the path (following the example here):

C:\Windows\System32\cmd.exe /c "SET RUST_BACKTRACE=1 && START /D ^"C:\Spiele\Modding\0_Programme\LOOT^" LOOT.exe"

No idea if that is even close to what you want me to do wrinkly. Even if that's correct .. and now? If I run LOOT that way, it still crashes .. I see no difference, not in LOOT nor in the Debug Logs.

Hmm, I need to try it out myself... I think the backtrace would be printed to the console, so maybe RUST_BACKTRACE needs to be set and LOOT needs to be run from a command line prompt so it doesn't get lost...

Link to comment
Share on other sites

I can reproduce the crash when switching between games:

Unhandled exception at 0x012DB138 in LOOT.exe: Fatal program exit requested

Here is a zip of the logs and all of my metadata. This is with RUST_BACKTRACE = 1 set in my user account's environment variables.

EDIT:  I did have debug logging turned on after all.

 

Link to comment
Share on other sites

I installed FNV, but can't replicate the crash. However, I did test out how to get the backtrace, and the easiest way is:

  1. Open a PowerShell command line prompt
  2. cd <LOOT's install directory>
  3. $env:RUST_BACKTRACE=1
  4. ./LOOT.exe

That will run LOOT, and when you get it to crash, the backtrace will be printed in the PowerShell window.

I'll be adding panic catching to libloadorder and esplugin as they should have it, but in the meantime the above may be useful.

Link to comment
Share on other sites

As I use Windows 7, I have Windows PowerShell 2.0. Running PowerShell the normal way and as administrator results in this, so LOOT started, I switch to either SSE or FO4, it crashes and I close the window that it stopped working .. and nothing get's written to PowerShell. Do I need to update PowerShell or am I still doing something wrong?

Edit: I'll update to Windows Management Framework 5.1 anyway.

Link to comment
Share on other sites

24 minutes ago, pStyl3 said:

As I use Windows 7, I have Windows PowerShell 2.0. Running PowerShell the normal way and as administrator results in this, so LOOT started, I switch to either SSE or FO4, it crashes and I close the window that it stopped working .. and nothing get's written to PowerShell. Do I need to update PowerShell or am I still doing something wrong?

Edit: I'll update to Windows Management Framework 5.1 anyway.

No, I don't think you're doing anything wrong, it's just not behaving how I expected...

I'll keep trying to investigate. I've discovered that if I change the LOOT API to load load order state whenever it creates a load order handle (as opposed to when loading plugins), LOOT will crash on startup, despite all the tests passing in LOOT and the LOOT API... This may be related to your crashes, or maybe not, but I'll try figuring it out and we'll see if fixing it helps.

Link to comment
Share on other sites

2 hours ago, pStyl3 said:

Could we add some blank (non-CC) .esl plugins to the testing-plugin repository (for SSE as well as FO4)?

I've just read How to Batch Rename Multiple Files in Windows and, after being a bit mindblown on how easy this actually is, would love to test LOOT with thousands of .esl plugins. :teehee:

I've just been renaming the existing files there for my tests, since there's no significant content differences. I haven't tested LOOT with thousands of plugins, but I wouldn't be surprised if it was a little slow. :P

In other news, I think I've identified why the crash is happening, and I now know how to avoid such things causing crashes in future. It was actually a combination of things:

  1. Libloadorder has some logic for determining where a plugin should get inserted into the load order when loading plugins, e.g. a master should go before non-masters, the game master should load first, that kind of thing. However, to put implicitly-active plugins in their correct order it loops through the list of such known plugins and checks which are already loaded or installed, incrementing a counter for each one until the correct implicitly-active plugin is found. This counter value is then used as the insert position, but this can be out-of-bounds if there are either no plugins loaded yet or a plugin that is installed but not loaded was counted, causing a panic.
  2. The FFI wrapper for libloadorder doesn't currently catch panics, so the whole process aborts (it's actually undefined behaviour to propagate panics across a language boundary, the abort just happens to be what happens).
  3. It looks like even if I do catch the panic and error nicely, the exception that the LOOT API throws isn't handled correctly, again causing the whole process to abort.

I've fixed the first issue, catching panics is tedious but not difficult, and I've yet to look into why the exception handling isn't great. @pStyl3, @Beermotor, can you try with this LOOT API DLL?

Link to comment
Share on other sites

Tested the new .dll in conjunction with "loot_0.11.0-84-g4581a9c_protobuf.7z" - no more crashes.  :)

Edit: Same with "loot_0.11.0-85-g4f53992_protobuf.7z", the issue seems to be fixed for me.

Link to comment
Share on other sites

While NMM is no longer in official development and thus it's last version is 0.63.14, DuskDweller has released an unofficial build that supports .esl files. I haven't tested that build just yet, but thought I'd mention it anyway.

Edit: Link

I've also searched for mods on Nexus, that already converted their plugins to the .esl plugin format, and found these four for Fallout 4:

I'm not aware of any SSE mods on Nexus that use .esl plugins, yet.

Link to comment
Share on other sites

Interesting, thanks for sharing that.

I'm currently working on replacing the LOOT API's logging library with one that's a lot simpler. I should have that done by the weekend, at which point I think I'll release v0.12.0 of the API. I can then merge LOOT's update-api branch, though I think I want to include the settings file format change before releasing LOOT v0.12.0, but maybe not.

Link to comment
Share on other sites

So. Yeah.

  1. Fallout 4, all official DLCs, no other mods
  2. Download No Blur while Crafting and install it (it's a small .esl plugin)
  3. Multiply it 8250 times.
  4. Communicate that to FO4's plugins.txt
  5. Start LOOT.

*Drum roll* .. LOOT did it! No crash. To mention is though, it took around 168 seconds for LOOT to start. Sorting on the other hand .. I closed LOOT after 30 minutes of sorting .. it probably would've finished, but 30 min waiting for this test were enough.

Also, regarding point 4. I guess the hardest part, after all .esl's are properly named and in place is to add all the entries to plugins.txt. I did that with Windows PowerShell, I created a textfile containing all the plugin names, then I could copy&paste their names to plugins.txt. Here are the commands:

Task:    Change directory
Command: Set-Location -Path *Path*
Example: Set-Location -Path C:\Spiele\Modding\"Test 1"


Task:    Get, Sort and Print the content of the folder to a textfile
Command: Get-ChildItem *.esl | Sort-Object {$_.BaseName -replace "\D+" -as [Int]} | Select-Object -ExpandProperty Name > !files.txt

Note: All the 8280 plugins were temporarily within my Test 1 folder for this.

So as a conclusion. While it's absolutely nuts, LOOT can handle absurdely high number of plugins. :teehee:

Link to comment
Share on other sites

Well, that is a bit slow, though I'm not really surprised that sorting takes so long, because it's very unoptimised for larger load orders (the complexity is something horrible) - something I want to improve sometime. Was the UI laggy with all those plugins?

Link to comment
Share on other sites

The UI does not lag, at all in fact. However, *sometimes* the UI sort of freezes when either the left or right pane is dragged up or down. I noticed this on two occassions, the first time the freeze lasted for only say 10 seconds and then LOOT was back to normal, the second freeze however was permanent and I needed to close LOOT. Also, during these freezes either the sidebar or the card list would go blank. Given that this happens with well over 8000 plugins however .. not really a problem!

Link to comment
Share on other sites

Get-ChildItem *.esl | Sort-Object {$_.BaseName -replace "\D+" -as [Int]} | Select-Object -ExpandProperty Name | ForEach-Object {Write-Output "*$_" | Out-File '!files.txt' -Append -NoClobber}

This PowerShell command will now add an asterisk * in front of each entry of the plugin names within the to-be-created file "!files.txt". I've actually asked in a powershell forum for help for this, and they greatly did help me out. Anyway, now all the plugins will be displayed as active within LOOT.

Image 1 - Image 2 - Image 3 - Image 4

Currently the decimal numbers for the .esl plugins start with 0. That isn't strictly needed, isn't it? Therefore wouldn't it be better, if that would start with 1?

Another thing is also, that with # plus 3 decimal numbers, so e.g. #123, the numbers get already ankwardly close to the plugin names. Take # plus 4 decimal numbers, so e.g. #1234, the numbers basically touch the plugin names in the sidebar. Maybe display the decimal numbers without the # after all?

Image 4 also shows, that the load order indices won't be displayed if greater than 4095 (so the 4096th .esl, as we currently start with 0). This is the max. value of .esl plugins the game can handle after all, right? At least that's my understanding reading this and this. Maybe LOOT should prevent people from loading more than 4096 .esl's.

Not that this will ever happen realistically, but hey, if we've got an upper bound, why not make sure of it.

Link to comment
Share on other sites

The decimal indices start with 0 for consistency, though I'm actually wondering if hex indices wouldn't be more useful as they could be used to work out in-game FormIDs, and would be consistent with the other indices too.

LOOT (actually libloadorder) already cuts off at 4096 active light masters, that's why indices aren't displayed past the 4096th, because they're not active.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...