Jump to content

[RELz] LOOT - Load Order Optimisation Tool


WrinklyNinja

Recommended Posts

Intel Core i7 4790K processor, 16gb ram, 1tb SSD. Steam/Skyrim/MO/LOOT are all installed on C:\, which is the SSD. No slowdown when I start LOOT.

Link to comment
Share on other sites

47 minutes ago, WrinklyNinja said:

Any chance of numbers vs. the old build for your system (and what's your CPU, are you using an SSD)?

I can run debug on both and give you the stats. Stand by and I'll update in a moment.

Hardware wise I'm running:

CPU: Intel i7-4790 @ 4Ghz 

SSD: Samsung 840 Pro 256GB SSD mounted as C: - SSE is installed on this drive.

Loot is running from my D: drive which is a Seagate 2TB SSHD Hybrid drive. The OS TMP/TEMP environment variables are set to D:\temp\

 

Link to comment
Share on other sites

2 hours ago, pStyl3 said:

Intel Core i7 4790K processor, 16gb ram, 1tb SSD. Steam/Skyrim/MO/LOOT are all installed on C:\, which is the SSD. No slowdown when I start LOOT.

Hmm, given Beermotor's similar specs and the timings he gets, I'm not sure what's going on.

1 hour ago, Beermotor said:

@WrinklyNinja Here are debug logs for comparison.   https://drive.google.com/drive/folders/0B8gJeDB67uPfTzNMNWFSeXR2ZG8?usp=sharing

LOOT-Github-Beermotor is the 0.11 version from GitHub.

loot_0.11.0-41-gf3d3bce-Debug-Beermotor is the new build.

 

Some interesting numbers:

  • Header-only load time went from 1.2s to 0.3s
  • Full load time went from 4.7s to 0.7s
  • Adding overlap edges (i.e. comparing the FormIDs of your plugins' records looking for overrides) went from 23.5s to 0.7s
  • Getting the load order (which happens a lot during sorting, it's not very efficient...) went from ~ 6ms to 0us
  • Setting the load order went from ~ 10 ms to ~ 200 ms

I'm surprised that load times improved so much, esplugin does pretty much the same thing as libespm. Some of it could be down to libloadorder improvements, because loading also checks if a plugin is active, but that takes the same time for header-only and full loads. It's the kind of difference that would make me think I've missed something major, but all LOOT's tests still pass.

The overlap edges speedup is probably overwhelmingly due to the libespm version operating on a copy of the FormIDs (which I only just noticed...), while esplugin uses references to them. I'd expect the two to be much closer in performance if it weren't for that.

The speedup when getting the load order will be due to no longer loading the current load order state before doing so (it's not instantaneous, it's just taking nanoseconds, and the LOOT API logging only goes down to microseconds), and the setting slowdown I've already covered earlier.

Link to comment
Share on other sites

From a post by CDCooley https://forums.nexusmods.com/index.php?/topic/6043958-unofficial-skyrim-creation-club-content-patches/page-15#entry54334023

Charlie breaks it down to 4 groups ..

Quote

There's absolutely nothing magical happening here, but people have gotten too comfortable with the idea that those first two digits of a formId you see in the game's console represent load order. They used to but now they don't thanks to ESL files. It's time to adapt folks. I'm seeing various people testing how things load by looking that the formID numbers in-game. 

Load order is controlled by the game's hard-coded list of official content and then things listed in the plugins.txt file. The numbers assigned as the first two digits of formID values in-game have nothing to do with load order when ESL files are involved. 

From my testing there appear to be only four groups of files that the game considers when loading files. 

First are the official files that are hard-coded into the game itself. Those used to be just Skyrim.esm, Update.esm, Dawnguard.esm, HearthFires.esm, and Dragonborn.esm but now include all of the new Creation Club ESL files too. That means the survival ESL will always load before any traditional mods including USSEP. If they exist in your Data folder they will be loaded and any entries in the plugins.txt file for them are ignored. 

Second are files listed in plugins.txt that are internally tagged as masters and marked as active. It doesn't matter what file extension they use (ESM, ESP, or ESL). They get loaded in the order they appear in the plugins.txt file (but only their order in relation to each other matters). 

Third are any other mods as listed in plugins.txt that are marked as active. The are the regular mods that we used to call ESPs but can technically include files with the extensions ESM and ESL too. 

And finally, the fourth group consists of only one file. It's the save file (ESS) being loaded (or a blank file if you're starting a new game).

Hopefully various mod managers (and LOOT) can be updated soon before too much chaos results from people thinking there's something more complicated and confusing happening. 

 

Edit : My bold in yellow

Link to comment
Share on other sites

My findings, as posted elsewhere, differ slightly from CDCooley's, in that the master flag of ESL files doesn't seem to matter, I've seen ESLs with the master flag load after ESLs without it. This is with Fallout 4 v1.10.26.0.0, I wouldn't be very surprised if Bethesda had Skyrim SE behaving slightly differently...

Anyway, I've added preliminary support for ESLs to libloadorder, based on what I know, and will be going on to adding support to esplugin, the LOOT API, then finally LOOT itself.

Link to comment
Share on other sites

I'm not sure exactly how he's testing all that but it differs from my own checks based on in-game form IDs from clicking on things in the console. Every standard ESM and ESP file is exactly where the load order indexes in Wrye Bash say they will be. Only the load index on ESL files is different, and the behavior of how those are overridden or how they override other things is what seems to be varying so wildly since it doesn't seem to matter where they show up in plugins.txt. I think that's why we're all getting such differing results from all of this, because there literally is no pattern other than the blocks they get written into on the save headers.

I should make it clear that everything I've tested has been on SSE, not FO4. I have yet to do any kind of serious checking with how that behaves with ESL files. I seriously doubt things would be different but I guess you never know.

Link to comment
Share on other sites

So crazy question: How likely is Bethesda to give us a technical answer about these remaining ESL behavior mysteries? 

EDIT: I just saw the post about the ESL thread. I'll pose the question over there.

Link to comment
Share on other sites

New LOOT build, this adds support for ESL plugins. They don't yet have any special UI, but Copy Load Order has been updated to output ESL-specific data, e.g.

  0  0      Fallout4.esm
255 FE    0 ccbgsfo4004-pipboy(camo02).esl
            master.esm
255 FE    1 non_master.esl
255 FE    2 master.esl
  1  1      non_master.esp

The new third column is the decimal load order index amongst active light masters.

Link to comment
Share on other sites

Another build, giving ESL plugins a new icon on their card. Also, ESL plugins won't have the "Master File" icon displayed, even if they're flagged as a master, as the flag has no effect for them.

Link to comment
Share on other sites

Awesome work @WrinklyNinja:)

I've been up testing the first build for the past hour or two (I just downloaded the new version) and I haven't had any trouble whatsoever other than once I sort the mods, the CC Survival ESL is deactivated. My other ESLs are not deactivated.  The only difference between the Survival Mode ESL and the others is the others do not have the master flag set. 

I just downloaded the new version and I will test to see if this is still happening and update shortly with a debug log if it is.

Link to comment
Share on other sites

Update: I was able to replicate it with the new version.

I've uploaded my debug logs to my Google Drive:  https://drive.google.com/drive/folders/0B8gJeDB67uPfTzNMNWFSeXR2ZG8?usp=sharing

This is the info from Wrye Bash on the plugin:

Spoiler

ccqdrsse001-survivalmode.esl
Load Order: -
Installer: -
Modified: 9/28/2017 3:59:44 PM
Size: 233 KB
Author: QDR
Mod Status: Non-Active

And here is the top of my plugins.txt after a sort:

Spoiler

*Unofficial Skyrim Special Edition Patch.esp
*Campfire.esm
*Apachii_DivineEleganceStore.esm
*EFFCore.esm
*AyleidCitadel.esp
*SGHairPackBase.esm
ccqdrsse001-survivalmode.esl
*Road to Morthal.esl
*Mertz_Rustic_TexturePack.esl
*Cutting Room Floor.esp
*Bring Out Your Dead.esp

 

Link to comment
Share on other sites

25 minutes ago, Beermotor said:

Update: I was able to replicate it with the new version.

I've uploaded my debug logs to my Google Drive:  https://drive.google.com/drive/folders/0B8gJeDB67uPfTzNMNWFSeXR2ZG8?usp=sharing

This is the info from Wrye Bash on the plugin:

  Reveal hidden contents

ccqdrsse001-survivalmode.esl
Load Order: -
Installer: -
Modified: 9/28/2017 3:59:44 PM
Size: 233 KB
Author: QDR
Mod Status: Non-Active

And here is the top of my plugins.txt after a sort:

  Reveal hidden contents

*Unofficial Skyrim Special Edition Patch.esp
*Campfire.esm
*Apachii_DivineEleganceStore.esm
*EFFCore.esm
*AyleidCitadel.esp
*SGHairPackBase.esm
ccqdrsse001-survivalmode.esl
*Road to Morthal.esl
*Mertz_Rustic_TexturePack.esl
*Cutting Room Floor.esp
*Bring Out Your Dead.esp

 

AFAIK Wrye Bash hasn't been updated to handle CC plugins correctly. You can't deactivate them. Does LOOT show it as active?

EDIT: Utumno's apparently just updated Bash with the CC plugins, so maybe its next build will work.

Link to comment
Share on other sites

1 hour ago, WrinklyNinja said:

AFAIK Wrye Bash hasn't been updated to handle CC plugins correctly. You can't deactivate them. Does LOOT show it as active?

EDIT: Utumno's apparently just updated Bash with the CC plugins, so maybe its next build will work.

Yes, it is rather strange what I'm seeing and it is entirely possible there is some work left to be done on the WB side. This may help both projects.  

What I'm seeing now will be easier to explain if I show a before and after sorting comparison.

Before Sorting:

Plugins.txt:

Spoiler

*Lanterns Of Skyrim - All In One - Main.esm
*Unofficial Skyrim Special Edition Patch.esp
*Campfire.esm
*Apachii_DivineEleganceStore.esm
*EFFCore.esm
*AyleidCitadel.esp
*SGHairPackBase.esm
*ccqdrsse001-survivalmode.esl
*Road to Morthal.esl
*Mertz_Rustic_TexturePack.esl

Wrye Bash  - Note the load slot 0C

Spoiler

59e20f0c276b7_WBBefore.PNG.ef0a54fe54abf8713284f97ea67a90e0.PNG

LOOT: Load slot 0A

Spoiler

Loot-before.PNG.87f78f9f8d3c9e65c855d9b865f2fb8a.PNG

 

After sorting

Plugins.txt:

Spoiler

*Lanterns Of Skyrim - All In One - Main.esm
*Unofficial Skyrim Special Edition Patch.esp
*Campfire.esm
*Apachii_DivineEleganceStore.esm
*EFFCore.esm
*AyleidCitadel.esp
*SGHairPackBase.esm
ccqdrsse001-survivalmode.esl
*Road to Morthal.esl
*Mertz_Rustic_TexturePack.esl

Wrye Bash

Spoiler

59e210adb684b_WBafter.PNG.864ef0f30d4b80cc4bcfc196081a53dc.PNG

LOOT  - Note load slot has been changed to 07.

Spoiler

loot-after.thumb.PNG.222caf52d3ab6988178f26bcf2d0168e.PNG

 

Just for reference I'm using Utumno's latest WB build from yesterday (13 October).  I thought it may be possible that WB is disabling the plugin after the LOOT sort due to some condition that made Bash think there was a problem.  So I ran LOOT without Wrye Bash running.  The Survival Mode plugin `ccqdrse001-survivalmode.esl` plugin is no longer in plugins.txt at all.

Plugins.txt:

Spoiler

*Lanterns Of Skyrim - All In One - Main.esm
*Unofficial Skyrim Special Edition Patch.esp
*Campfire.esm
*Apachii_DivineEleganceStore.esm
*EFFCore.esm
*Road to Morthal.esl
*AyleidCitadel.esp
*SGHairPackBase.esm
*Mertz_Rustic_TexturePack.esl
*MLPSoulGems.esl
*Mertz_Texture_Collection.esl
*Cutting Room Floor.esp

I hope this is helpful. If you would like me to test anything else or try something different I will be glad to help.

EDIT: @Utumno released a new build this morning (14 Oct) so I'm going to see if that changes anything. 

EDIT2: Yep it fixed it. I removed the images to save attachment space for BAIN Wizards. 

Link to comment
Share on other sites

UPDATE: I am enlightened now. After seeing what @Utumno posted in the WB thread I realize why the plugin disappeared from the plugins.txt. Since it is a hard-coded ESL (similar to one of the default master files) it will not appear in the load order list. Apparently the previous build of WB was adding it back in. 

So the latest WB build fixed it. I wish I had seen the update before I went to all of that effort documenting everything. :)

Link to comment
Share on other sites

I currently can't test any snapshot, as I do not have access to my main computer. I will do on Monday, but then have no internet access until next week (aside from internet access via smartphone).

Regarding the Copy Load Order feature, shouldn't it be 254 instead of 255 for FE?

Also, I would like to upload the latest snapshot as an optional file to Nexus. I would mark it as experimental and write a post regarding it (and pin that post), that way we could gather more data, if people run into problems or not. You're okay with that, wrinkly?

Link to comment
Share on other sites

:redface: I had forgotten about that, it is now unstuck, and has warped off to adopt whatever date position it should hold in the sequence of postings.

Link to comment
Share on other sites

1 hour ago, pStyl3 said:

Regarding the Copy Load Order feature, shouldn't it be 254 instead of 255 for FE?

:facepalm:Fixed.

1 hour ago, pStyl3 said:

Also, I would like to upload the latest snapshot as an optional file to Nexus. I would mark it as experimental and write a post regarding it (and pin that post), that way we could gather more data, if people run into problems or not. You're okay with that, wrinkly?

Could you write a post but link to the Bintray page instead? Nexus doesn't give a breakdown of downloads by version, AFAIK, whereas Bintray does, and it was useful to see how many people were downloading snapshots when we last did wider testing.

Link to comment
Share on other sites

Another day, another LOOT build. This time it's nothing ESL-related (but on the same branch so it has ESL support): the API logging now uses a callback provided by LOOT, so everything gets logged to LOOTDebugLog.txt like it used to. It was really bugging me that the API logged to LOOTAPIDebugLog.txt, so I'm glad I finally got around to improving that.

I've also gone ahead and released libloadorder v10.0.0 as though there are a few questions remaining about ESL load order, they're to do with edge cases and if I need to make changes they can be done as bug-fix releases.

I think I'm now ready to prepare a LOOT API v0.12.0 release, but I'll hold off until LOOT's ESL support has been merged into its main branch in case I need to make further changes.

Link to comment
Share on other sites

Everything but the docs should now be done for ESL support as of this build. Sidebar indices now count light masters separately from other plugins, and give the index for light masters as FE followed by the index amongst other light masters. LOOT will also display warning messages if you've got 255 plugins and a light master active, as the 255th active plugin and active light masters can have overlapping records which could be bad news, and it will also warn if you've got a light master that depends on a non-master plugin, as that leads to wacky behaviour.

Link to comment
Share on other sites

Out of curiosity, I ran a code line counter on all the repositories that make up the bits of LOOT (not including third-party stuff), and these are the results:

> tokei esplugin jed-gettext-parser libloadorder LOOT\loot-api LOOT\loot pseudosem
-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Autoconf                2           74           54            2           18
 C                       1          250          196            0           54
 C Header              131        17246         9949         3593         3704
 CMake                   6          940          713           86          141
 C++                    41         7524         4976          991         1557
 CSS                     3          456          371           52           33
 Dockerfile              1            5            4            0            1
 HTML                   21         2809         2471           54          284
 JavaScript             33         6692         5222          301         1169
 JSON                    5          154          154            0            0
 Markdown               18         1057         1057            0            0
 Python                  3          752          114          472          166
 ReStructuredText       34         2468         2468            0            0
 Rust                   35         8761         6489          787         1485
 Shell                   2           31           17            7            7
 Plain Text             17         3233         3233            0            0
 TOML                    7          203          169            8           26
 YAML                    2          204          167            3           34
-------------------------------------------------------------------------------
 Total                 362        52859        37824         6356         8679
-------------------------------------------------------------------------------

I then did the same for the masterlists:

> tokei oblivion fallout3 falloutnv skyrim fallout4 skyrimse
-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Markdown               12          177          177            0            0
 YAML                    6        63867        63365          439           63
-------------------------------------------------------------------------------
 Total                  18        64044        63542          439           63
-------------------------------------------------------------------------------

Which just goes to show how much metadata there is!

I also ran counts for Wrye Bash and xEdit, Bash has > 100,000 code lines of Python, and xEdit has > 260,000 code lines of Pascal, but both projects include some third-party code in their repositories, so even ignoring variation in language verbosity, they're not really comparable (at least beyond 'wow, that's a much bigger number').

Lines of code is generally a pretty useless metric, of course, but IMHO the reasons for its uselessness (how closely or not it couples to clarity, functionality, uniqueness, etc.) make it interesting all the same.

Link to comment
Share on other sites

2 hours ago, alt3rn1ty said:

@WrinklyNinja Off topic /

Do you have a forum topic for Buddah?, I seem to recall one being made somewhere but cant recall where.

A few tools to add - Skyrim Performance Monitor 64 (SSE version), SSEEdit, SSE NIF Optimizer.

Edit : SSELODGen

Edit 2 : BodySlide and Outfit Studio SSE

Edit 3 : Wrye Bash short description says its for Oblivion and Skyrim, it is also now for Fallout 4 and Skyrim SE

Edit 4 : TesModManager - "TesModManager is a new version of OBMM Extended that supports Skyrim SE, Skyrim, Oblivion and Morrowind." OBMMex programmed by Scent Tree, was a rework of Timeslips original, and made to be more reliable with additional features. TMM is based on that.

// Off topic

I think I had a thread on the old official forums, but I can't find it now. I've created a new one here.

Link to comment
Share on other sites

  • 2 weeks later...

I've been working on improving how LOOT serialises data when it's sent between the C++ backend and the JS frontend, as currently LOOT is re-using the YAML serialisation that the LOOT API uses, which is messy and fragile to do. What I wanted to do instead was define a schema for the JSON that gets transmitted, and generate code from that, rather than having to write the serialisation code myself all over again.

After trying a bunch of different libraries, I've gotten it working with Protocol Buffers, and you can try it out with this build. One thing that is untested but probably not working is validation of metadata added in the UI - I've left every field optional in the schema, which isn't actually always the case (Bash Tags need names, for example), and condition syntax isn't checked, so I probably need to add that back in, but I'm still exploring how best to do that.

Using Protocol Buffers only to work with JSON is overkill, but unfortunately the other libraries didn't fit my requirements. There is a lighter version of Protocol Buffers I could use though, if I used its binary encoding instead of the JSON, which I may do in future.

Since this schema-first approach is working for LOOT, I'm now thinking of adopting it for the API, so that the interfaces give and take the serialised Protobuf binary format instead of the C++ classes they currently do. It's less efficient, but more flexible, and probably a better fit overall.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...