Jump to content

Plugin Files And You (ESM/ESL/ESP)


Arthmoor

Recommended Posts

Just now, WrinklyNinja said:

Yes, yes and yes.

My outstanding questions are:

  • Can a normal plugin and a light master share the FE slot? (They could in Fallout 4 when I tested in early September, but that might have changed.)
  • What's the precise relationship between number of records in ESLs and how many can be active?

I can test the first question myself, but the latter would be painful to answer with black-box testing, and could really do with some clarification from a dev or someone who can poke around the runtime innards to figure it out.

Hopefully

Re: the precise relationship between number of records in ESLs and how many can be active - this however does not impact load order handling ? If yes we're good to go patching our lo implementations

Link to comment
Share on other sites

Just now, Utumno said:

Re: the precise relationship between number of records in ESLs and how many can be active - this however does not impact load order handling ? If yes we're good to go patching our lo implementations

Not load order, no. It just affects not letting the user activate more plugins than they can actually activate (which is harmless, but misleading).

In terms of implementation, in libloadorderĀ I'm hardcoding the CC plugin names because resolving their relative load order otherwise is a little tricky and will require more thought if it's to be done with good performance, and I'm using a fixed value 4096 as the active light master limit (once the real equation is discovered, it may not be performant to implement anyway).

Link to comment
Share on other sites

I don't believe it would be wise to let a normal ESP and a bunch of ESLs share the FE slot. That seems like the kind of thing that Bethesda would not have checked for since they don't need to worry about it and the kinds of record conflicts such an arrangement could cause in-game if they overlap incorrectly would be a mess to diagnose.

Link to comment
Share on other sites

7 minutes ago, Arthmoor said:

I don't believe it would be wise to let a normal ESP and a bunch of ESLs share the FE slot. That seems like the kind of thing that Bethesda would not have checked for since they don't need to worry about it and the kinds of record conflicts such an arrangement could cause in-game if they overlap incorrectly would be a mess to diagnose.

Yeah, it seems sensible to err on the side of caution.

Link to comment
Share on other sites

I've done some more testing after being pointed here and seeing what you guys have found. The tool we need to be using to look at this issue is the game's built-in mod Load Order screen.

It seems that the ESM flag on ESL files really is ignored by the game. I had been testing my load order changes using Bash which was forcing files flagged as masters to load earlier. Without Bash running the master flag on an ESL does absolutely nothing to load order as far as I can determine.

I now also confirm reports that all ESLs are loaded with the masters in the order they appear in your plugins.txt (assuming its not one of the hard-coded ESL files of course).

But the overriding of ESPs is really explained by something I would have never expected.

If an ESL file has an ESP listed as a master that ESP file will be forced to load immediately before the ESL even if that places the ESP before other masters!

I created a simple ESL and tagged it to use Inigo.esp as master then put the ESL in my load order just after the unofficial patch and before my other ESMs and that's exactly where it loaded both in terms of record overwrites and the formid slot number.

The only way an ESL can override an ESP is by making it a master but by doing that it will also force that ESP to load with the masters! If you try to place the ESL later in the order it gets moved to just after the last of the other masters.

This is going to make load orders really interesting if people start creating ESL files with ESP masters (other than things like the unofficial patch which are already sorted with the masters).

Ā 

Ā 

Link to comment
Share on other sites

For load order handling (which is pretty much my current selfish interest here), is it important to try and mirror the order in which CC ESLs load?Ā  I mean, conflict is conflict in any direction.Ā  In theory the CC content shouldn't have any conflict amongst itself, but in practice, Bethesda.

That said, for a nitpicking perfectionist such as myself, sure, I guess it'd be nice toĀ see exactly which Stendarr's Hammer prevails in a crazy world in which all CC content is Stendarr's Hammer variations.

Any solution that can readily accommodate new CC ESLs before any hardcoded "master CC ESL list" can be updated for a given tool would be nice.Ā  Again, I presume a crazy world in which I keep spending 150 BethesdaBuxā„¢ onĀ variations of Stendarr's Hammer churned out daily that keep conflicting with each other.Ā  Because I will.Ā Ā :geek:Ā  And thus they should.

Link to comment
Share on other sites

5 hours ago, cdcooley said:

If an ESL file has an ESP listed as a master that ESP file will be forced to load immediately before the ESL even if that places the ESP before other masters!

Yeah, that behaviour is going to be a PITA to deal with. IMHO it should be a hard crash, like having a master dependent on a non-master (IIRC).

libloadorder doesn't check plugins' masters, it will accept a plugin loading before those it depends on, for simplicity. If I need to start moving plugins around depending on their masters, it'll slow things down, and just after I got > 16 ms load times... :(

As for LOOT, it'll complicate the sorting algorithm with another preprocessing pass to flag non-masters that ESLs depend on as masters so they get sorted correctly.

For the time being, I'm in favour of ignoring the special case of a light master depending on a non-master in the hopes that Bethesda will 'fix' the current behaviour.

2 hours ago, Nukenin said:

For load order handling (which is pretty much my current selfish interest here), is it important to try and mirror the order in which CC ESLs load?Ā  I mean, conflict is conflict in any direction.Ā  In theory the CC content shouldn't have any conflict amongst itself, but in practice, Bethesda.

That said, for a nitpicking perfectionist such as myself, sure, I guess it'd be nice toĀ see exactly which Stendarr's Hammer prevails in a crazy world in which all CC content is Stendarr's Hammer variations.

Any solution that can readily accommodate new CC ESLs before any hardcoded "master CC ESL list" can be updated for a given tool would be nice.Ā  Again, I presume a crazy world in which I keep spending 150 BethesdaBuxā„¢ onĀ variations of Stendarr's Hammer churned out daily that keep conflicting with each other.Ā  Because I will.Ā Ā :geek:Ā  And thus they should.

I would assume it is important to match the order, and it's what introduces the most complexity. For libloadorder's case, I don't think I can support detecting non-hardcoded plugins, because a false positive would mean the client is unable to set a valid load order, while not knowing about a CC plugin just means the client can put it in a location it doesn't really load at, or deactivate it when it can't really be deactivated, which is less of an issue.

Link to comment
Share on other sites

6 hours ago, cdcooley said:

If an ESL file has an ESP listed as a master that ESP file will be forced to load immediately before the ESL even if that places the ESP before other masters!

Would that mean the ESP has automatcially got the ESM flag appllied, as soon a modder has tagged that ESP as a master for an ESL?

From what I understand only SSEEdit can make an ESP to load as an ESM by flipping the ESM flag.

Link to comment
Share on other sites

2 minutes ago, Leonardo said:

Would that mean the ESP has automatcially got the ESM flag appllied, as soon a modder has tagged that ESP as a master for an ESL?

From what I understand only SSEEdit can make an ESP to load as an ESM by flipping the ESM flag.

It happens at runtime, the master flag in the ESP file doesn't actually change (if I've understood correctly).

Link to comment
Share on other sites

I uploaded a SKSE64 plugin that provides the DumpModInfoList console command. Results go intoĀ  My Documents\my games\Fallout4\SKSE\skse64_ModInfoDump_plugin.log

This will dump the order the plugins were loaded into memory and then the two arrays present into the save.

Ā 

The plugin is here : https://github.com/TES5Edit/TES5Edit/tree/sharlikran-fo4dump/skse64_modInfoDump

Ā 

Note: I am not going to comment the results at the moment :)Ā (It definitively wasn't what I expected on my very first try).

Ā 

Link to comment
Share on other sites

6 hours ago, WrinklyNinja said:

It happens at runtime, the master flag in the ESP file doesn't actually change (if I've understood correctly).

How does it happen, I mean suppose you have a mod (esp) and you want to have that mod to act as a master for another mod (ESL) without altering it in SSEEdit then how can the ESP be loaded before any master except for the official masters?

Is a master ESP for an ESL hardcoded?

Link to comment
Share on other sites

21 minutes ago, Leonardo said:

How does it happen, I mean suppose you have a mod (esp) and you want to have that mod to act as a master for another mod (ESL) without altering it in SSEEdit then how can the ESP be loaded before any master except for the official masters?

Is a master ESP for an ESL hardcoded?

I think there's a bit of a misunderstanding here, I can't make sense of the above, so I'll try answering your original question again:

7 hours ago, Leonardo said:

Would that mean the ESP has automatcially got the ESM flag appllied, as soon a modder has tagged that ESP as a master for an ESL?

From what I understand only SSEEdit can make an ESP to load as an ESM by flipping the ESM flag.

Creating an ESL that depends on an ESP doesn't modify the ESP file, it's still a perfectly normal ESP file. However, my understanding is that when the game loads, it sees that there's an ESL that depends on the ESP, and before it loads the ESL it loads the ESP it depends on, ignoring the load order position it has in plugins.txt andĀ the fact that the ESP doesn't have the master flag set.

Though that raises a question, actually: are there second-order effects? That is, if and ESL depends on an ESP that depends on an ESP, do both ESPs get loaded early?

Link to comment
Share on other sites

I am resisting the temptation to do silly things like have modA.esl list modB.esl as a master, which in turn lists modA.esl as a master.Ā  I'm trying to actually play the game this weekend, darn it!Ā Ā :)

Link to comment
Share on other sites

1 hour ago, WrinklyNinja said:

I think there's a bit of a misunderstanding here, I can't make sense of the above, so I'll try answering your original question again:

Creating an ESL that depends on an ESP doesn't modify the ESP file, it's still a perfectly normal ESP file. However, my understanding is that when the game loads, it sees that there's an ESL that depends on the ESP, and before it loads the ESL it loads the ESP it depends on, ignoring the load order position it has in plugins.txt andĀ the fact that the ESP doesn't have the master flag set.

Though that raises a question, actually: are there second-order effects? That is, if and ESL depends on an ESP that depends on an ESP, do both ESPs get loaded early?

Are we sure about that?

Link to comment
Share on other sites

28 minutes ago, Leonardo said:

Are we sure about that?

Well, I don't know about you, but I haven't personally verified it, I'm just going off what cdcooley wrote, and I think I read it somewhere else too.

Link to comment
Share on other sites

That was one thing I've checked and seen happen. I tried turning a patch for Open Cities into an ESL and it forced the ESP to load into the master slots despite the ESL itself getting marked with index FE and being listed in the save header where it was expected to end up. Yet another source of confusion.

Link to comment
Share on other sites

2 hours ago, Arthmoor said:

That was one thing I've checked and seen happen. I tried turning a patch for Open Cities into an ESL and it forced the ESP to load into the master slots despite the ESL itself getting marked with index FE and being listed in the save header where it was expected to end up. Yet another source of confusion.

You have?Ā  Interesting, god knows what *crazy* stuff Bethesda had created this time and it looks more like a Pandora's box being opened by Bethesda to me.

So my question here is valid and needs to be tested, just to either disprove or confirm what is happening when an ESP is a master to an ESL without having the ESM flag.

Ā 

Link to comment
Share on other sites

10 hours ago, Leonardo said:

How does it happen, I mean suppose you have a mod (esp) and you want to have that mod to act as a master for another mod (ESL) without altering it in SSEEdit then how can the ESP be loaded before any master except for the official masters?

Is a master ESP for an ESL hardcoded?

How? The game just does it. And I'm not sure what you mean by hardcoded in that second question.

The term master means two different but somewhat related things.

There's a master flag bit in the header. The ESP involved in my example doesn't have that set and it doesn't matter whether the ESL does or not. In the past this master flag was what separated the load order into two distinct groups with everything having the master bit getting loaded before anything that didn't have it. The unofficial patch gets classified as a master because of that flag even though it has an ESP extension. Now that master flag bit and the new ESL flag bit seem to be used together to grab what we would call "masters" that get loaded before regular ESPs. Other than treating the new ESL flag bit as the equivalent of the original master bit there's not much of interest for this definition of master.

Individual files can also be masters for specific plugins. That information is recorded in the header of the dependent mod by listing the masters it requires. requires them.

When I make my ESL dependent on Inigo.esp the game is loading Inigo.esp right before the ESL to satisfy the dependency even if that means loading Inigo before other ESM files. Again no change was made to Inigo.esp but the fact that it was listed as a master for the ESL forced it to be loaded early.

NEW DATA:

I've just tested making an ESL dependent on my personal patch which itself has many files listed as masters. The patch itself gets forced to load early but none of the files it lists as masters are being loaded. So all of the changes my patch was trying to make are getting ignored if I put it first in my load order. If I put the ESL after the unofficial patch then my personal patch is still winning those conflicts but not affecting anything later. Turning that patch into an ESL gets really ugly fast because it's doing a major reshuffle of my load order.

So basically if people start creating patches using the ESL format we're in for a world of trouble. It think it's probably worth coding at least a warning into the various tools if an ESL is found to list a regular ESP as a master.

  • Like 2
Link to comment
Share on other sites

12 hours ago, WrinklyNinja said:

Though that raises a question, actually: are there second-order effects? That is, if and ESL depends on an ESP that depends on an ESP, do both ESPs get loaded early?

Another candidate for race conditions when the ESL under question itself is a master for goodness knows what!

Link to comment
Share on other sites

4 hours ago, cdcooley said:

How? The game just does it. And I'm not sure what you mean by hardcoded in that second question.

The term master means two different but somewhat related things.

There's a master flag bit in the header. The ESP involved in my example doesn't have that set and it doesn't matter whether the ESL does or not. In the past this master flag was what separated the load order into two distinct groups with everything having the master bit getting loaded before anything that didn't have it. The unofficial patch gets classified as a master because of that flag even though it has an ESP extension. Now that master flag bit and the new ESL flag bit seem to be used together to grab what we would call "masters" that get loaded before regular ESPs. Other than treating the new ESL flag bit as the equivalent of the original master bit there's not much of interest for this definition of master.

Individual files can also be masters for specific plugins. That information is recorded in the header of the dependent mod by listing the masters it requires. requires them.

When I make my ESL dependent on Inigo.esp the game is loading Inigo.esp right before the ESL to satisfy the dependency even if that means loading Inigo before other ESM files. Again no change was made to Inigo.esp but the fact that it was listed as a master for the ESL forced it to be loaded early.

NEW DATA:

I've just tested making an ESL dependent on my personal patch which itself has many files listed as masters. The patch itself gets forced to load early but none of the files it lists as masters are being loaded. So all of the changes my patch was trying to make are getting ignored if I put it first in my load order. If I put the ESL after the unofficial patch then my personal patch is still winning those conflicts but not affecting anything later. Turning that patch into an ESL gets really ugly fast because it's doing a major reshuffle of my load order.

So basically if people start creating patches using the ESL format we're in for a world of trouble. It think it's probably worth coding at least a warning into the various tools if an ESL is found to list a regular ESP as a master.

Highlighting mine, the 0x200 (ESL)Ā flag in the TES4 record doesn't affect load order, it's purely the file extension. See here for the relevant test (specifically, Order 4).

Thanks for doing that test, and I agree with treating ESLs with ESPs as masters as invalid, it seems too broken to let it happen. Maybe someone who has contact with people at Bethesda can report it as a bug? From what I've heard they fixed allowingĀ > 4096 active ESLs (though I haven't verified that), maybe this can get fixed too.

For libloadorder, I'll continue the current approach of ignoring what a plugin's masters are entirely. For LOOT, trying to sort a load order involving an ESL dependent on an ESP will continue to cause a cyclic interaction error, and I'll add a specific error message for any ESLs that do depend on ESPs.

EDIT: I've updated the Creation Kit wiki page with the latest info, but is it just me that hits Cloudflare reCAPTCHA and loses edits there all the time?

Link to comment
Share on other sites

5 hours ago, cdcooley said:

How? The game just does it. And I'm not sure what you mean by hardcoded in that second question.

My second question is just a response of my first "How does it happen?" question and your anwser is "The game just does it".

Which means that somehow the ESL makes a master ESP for an ESL without the ESM flag a fake ESM at the same time the user tag an ESP as a master for an ESL.Ā  That's what I mean by my " Is a master ESP for an ESL hardcoded?" question.

Link to comment
Share on other sites

18 hours ago, hlp said:

I uploaded a SKSE64 plugin that provides the DumpModInfoList console command. Results go intoĀ  My Documents\my games\Fallout4\SKSE\skse64_ModInfoDump_plugin.log

This will dump the order the plugins were loaded into memory and then the two arrays present into the save.

Ā 

The plugin is here : https://github.com/TES5Edit/TES5Edit/tree/sharlikran-fo4dump/skse64_modInfoDump

Ā 

Note: I am not going to comment the results at the moment :)Ā (It definitively wasn't what I expected on my very first try).

Ā 

Running DumpModInfoList with the game master, 254 normal plugins and two ESLs active causes Fallout 4 to lock up, I had to kill the game process in Task Manager.

However, I did manage to get an interesting result... I created a plugin that added a new copy ofĀ Armor_Railroad01 with "Non-Master ESL" appended to its name. The new record had a FormIDĀ of xx000800. I created an ESL from this, then made another copy of the record with a FormID of xx001800 (so that it would have the same runtime FormID as the ESL record, I have a CC ESL too), and edited both records'Ā names to end in "Non-Master ESP". I then created ~260 copies of this and listed all of them and my new ESL as active in plugins.txt.

I then launched Fallout 4, loaded a clean save, and ran "player.additem fe000800 1", and the item added had a name ending in "Non-Master ESP". I then ranĀ "player.additem fe001800 1", and the item that was added also had a name ending in "Non-Master ESP". So it looks like it is still possible to have a normal plugin and an ESL share the FE slot, and when records conflict the ESP record replaces the ESL record.

This seems like another case where we should probably just ignore what the game allows and only allowĀ 254 normal plugins active if an ESL is active.

Just now, Leonardo said:

My second question is just a response of my first "How does it happen?" question and your anwser is "The game just does it".

Which means that somehow the ESL makes a master ESP for an ESL without the ESM flag a fake ESM at the same time the user tag an ESP as a master for an ESL.Ā  That's what I mean by my " Is a master ESP for an ESL hardcoded?" question.

No, that's not what happens at all. Making an ESP a master for an ESLĀ does not change the ESP. There is no "at the same time" because no change occurs. The ESP getting loaded as if it were master-flagged is purely a runtime effect and does not persist any changes to the ESP file. If you were to edit the ESL to make it no longer depend on the ESP, the ESP would go back to loading like a normal plugin, again without any changes being made to the ESP.

Link to comment
Share on other sites

On 10/13/2017 at 2:57 PM, WrinklyNinja said:
  • What's the precise relationship between number of records in ESLs and how many can be active?

Why would there be any difference?

From my testing it seems that the first ESL is getting loaded as FE000xxx, the second is FE001xxx, the third is FE002xxx even when the three files only have a single record.

If the 2048 records per ESL file is based on the final three hex digits being in the 0x800-0xFFF range for the items and all ESLs are loading into slot FE that still leaves the remaining 3 hex digits for the ESL number giving exactly 4096 ESL plugins.

Is there yet another quirky game behavior I've missed?

Link to comment
Share on other sites

4 minutes ago, cdcooley said:

Why would there be any difference?

From my testing it seems that the first ESL is getting loaded as FE000xxx, the second is FE001xxx, the third is FE002xxx even when the three files only have a single record.

If the 2048 records per ESL file is based on the final three hex digits being in the 0x800-0xFFF range for the items and all ESLs are loading into slot FE that still leaves the remaining 3 hex digits for the ESL number giving exactly 4096 ESL plugins.

Is there yet another quirky game behavior I've missed?

The creation kit wiki page says there's a variable limit, but doesn't really give details, and I don't recognise the author's name, so I assume they're a Bethsoft dev.

Link to comment
Share on other sites

2 hours ago, WrinklyNinja said:

Maybe someone who has contact with people at Bethesda can report it as a bug? From what I've heard they fixed allowingĀ > 4096 active ESLs (though I haven't verified that), maybe this can get fixed too.

To be fair, the CK won't allow you to make the ESLs dependent on an ESP (unless you set the ESM flag bit) so using the official tools the problem can never occur, so they might not be interested in fixing it.

If we're going to ask for something being fixed it would be better to see if they can allow user created ESLs to be loaded anywhere (and honor the ESM and ESL flag bits) so that they could be used like light weight ESPs and light weight ESMs. That's more of an enhancement than a bug fix, but the very fact that the current ESL support is based on file extension rather than the internal flags (that were created) means someone didn't follow spec or two independent teams didn't coordinate. That's likely to cause problems for them long-term so they might be willing to change it now.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...