Jump to content

Plugin Files And You (ESM/ESL/ESP)


Arthmoor

Recommended Posts

Beginning with Patch 1.5.3 of Skyrim Special Edition, the game supports a new type of mod file known as a "light plugin" with the .esl extension. Elminster has spent a considerable amount of time studying how this works and has had a number of community members aid in verifying this information. Since questions about new plugin types come up all the time, I felt it was best to make sure our source here is accurate.

Flag Types

Plugins can have 3 different flag types: ESP, ESM, and ESL. This causes them to load in different ways, and in some cases handle their form IDs different from the old ways.
ESP files are standard every day mods. There will be no type flag set on these and they have the .esp file extension. Mod managers all know how these work and various modding tools consider these to be the basics of modding.
Files flagged with ESM are commonly referred to as "masters". They load early, immediately following the 5 hardcoded official files and any Creation Club content you may have.
Files flagged with ESL are known as "light plugins" or "light masters". How these load is determined by their file extension. The exception being any Creation Club content you may have downloaded. Those files load immediately following the 5 official files, according to the order they're listed in the Skyrim.ccc file.

File Extensions

The engine treats files in a certain way based on their file extensions:

A .esm file extension will always be treated as a master file. The ESM flag will automatically set in memory.
A .esl file extension will always be treated as a light master. The ESM and ESL flags are automatically set in memory.
A .esp file can function as either type of file depending on which flags are set in its file header. If the ESM flag is set, then the .esp is treated as a master file. If the ESL flag is set, it will be treated as a light plugin. If BOTH flags are set, then it is treated as a light master.
It is important to note that as far as the game is concerned, a .esp file with no ESM flag will load as a normal .esp file would. All standard load order procedures apply to them, and any properly functioning mod manager should handle them the same way it does regular .esp files.

The Golden Goose

Files with the ESL flag are given special treatment with their form IDs. Upon loading into the game, the form IDs are remapped into the FE mod index. This is now considered a special slot in SSE and should not be occupied by any mods in your load order. Not even the Bashed Patch or any other form of merged data.

As a result of this new support, the engine can allow you to bypass the previously hard limit of 255 plugin files. The theoretical limit now should be 4096 ESL flagged mods. Individual files flagged as ESL can in theory hold up to 4096 form ID records. In practice this ends up actually being 2048 form ID records because the engine reserves everything from $0x0000 to $0x07FF. All files using ESL flags must therefore contain their internal form IDs between xx000800 and xx000FFF. Anything exceeding this range is invalid and the game will either crash or you'll have severely corrupted data due to overruns.

The reason for this is how the mapping works. We're all familiar with the normal 8 digit form ID. The first 2 digits are only used to handle load order, the other 6 are internal to their plugin file. What the ESL flag system does is it break this down even further. The First 3 digits get mapped into an internal table for the FE slot. The remaining 3 belong to the actual records in the plugin. You'll end up with something like this:

FE000800 - FE000FFF ModFile1
FE001800 - FE001FFF ModFile2
FE002800 - FE002FFF ModFile3

FExxxyyy and so forth up until you reach FEFFFFFF and run out of IDs and mappings. This is where the 4096 files and 2048 form IDs per file comes from.

The benefits of all of this should be fairly clear. No more need to merge the 20 weapon mods with 5 records each. Just flag them with ESL and you're good to go. This is much safer than merging and is fully supported by the game engine so it works the same way for everyone.

Other Information

Using ESL flagged files come with some important issues to consider:

  • You must be playing on a copy of SSE version 1.5.3 or newer in order to use ESL flagged files.
  • You must be using a copy of the SSE CK version 1.5.3 or higher in order to properly recognize these files for editing.
  • The file cannot contain more than 2048 records in it or the form IDs will overflow into another FE slot and corrupt the save.
  • Form IDs need to be compacted for the file to be considered valid. The CK has an option called "Compact Active File Form IDs" in the file menu. xEdit also has a function for this. When using either of these, PAY ATTENTION TO THE WARNINGS.
  • You cannot compact IDs on a file you intend to continue using in an existing save or you WILL corrupt it due to the form IDs contained within it being rewritten. Only do this on files you have not yet begun using, and which do not yet exist as public releases.
  • Other mods which reference form IDs in the mod you are compacting (compatiblity patches, script calls, etc) will break if you compact the form IDs on a mod that's been released to the public.
  • NMM does not work with files that have the .esl extension or carry an ESL flag. It has been reported to completely lock up if even one of these is present in your Data folder. Use a functional mod manager such as Wrye Bash to manage these types of mods.
  • If by some wierd twist of fate you end up with a ESL flagged file that's also ESM flagged and someone uses a standard .esp file as a master to it for some reason, that file will be forced to load into the wrong part of your setup. Normally a standard .esp file will never load before anything ESM flagged, this is the one exception to that now and it should be considered a bug to report to any mod's author who has done this.
  • DO NOT attempt to patch anything inside of a CELL record that originates with a file flagged as ESL. Doing so will cause the cell to break and not load most of its contents when you enter it. This includes any official Creation Club content you may have. It doesn't matter who made the file, it will break.
  • Mods that use the GetFormFromFile() function in their scripts to reference things within them cannot be compacted unless the scripts are also recompiled with the new form ID.
  • Mods using recorded voice lines or that include NPC facegen data cannot be compacted because all of those asset files will need to be manually renamed to match the new dialogue and NPC form IDs.

This post will be updated again as needed to reflect new information that may be uncovered.

Link to comment
Share on other sites

  • Arthmoor pinned this topic

I've tested having a normal ESP in the FE slot with ESL plugins present, and if the normal plugin and an ESL have a record with the same Object ID, the ESL record is used. The other record is replaced, it doesn't get remapped to a different FormID. Otherwise, non-conflicting records share the FE index.

Also, if you have over 4096 ESLs, their records overflow into FF.

Link to comment
Share on other sites

User-created ESLs flagged with ESM can load above or below user-created ESLs not flagged with ESM; Plugins.txt dictates this.

ESLs can also load before USSEP.  The in-game mod load order menu will default ESLs to right below USSEP and will not allow ESLs to be moved below normal ESPs.  It will allow them to be moved above USSEP.   USSEP wins conflicts with ESLs above it, and loses conflicts with ESLs below it.  (USSEP is the only ESM-flagged ESP mod I have, but I assume this would be true for other such mods; not sure about user-created ESMs as I do not run with any currently.)

Is there any reason at all user-created ESLs be flagged ESM?  This could be reserved to distinguish official CC ESLs, as they will always be flagged ESM.

 

Link to comment
Share on other sites

To make sure I understand the load-order implications of ESL files correctly: What happens if there is a conflict between an ESL flagged as a master and a normal esp?

Will the game detect the conflict and leave out the conflicting record when merging all the ESL files into FE (meaning the ESP wins) or will it write the ESPs version of the conflicting record into FE or will the ESL file win the conflict, despite being in a higher block according to the save file headers?

Link to comment
Share on other sites

4 hours ago, Daidalos said:

To make sure I understand the load-order implications of ESL files correctly: What happens if there is a conflict between an ESL flagged as a master and a normal esp?

Will the game detect the conflict and leave out the conflicting record when merging all the ESL files into FE (meaning the ESP wins) or will it write the ESPs version of the conflicting record into FE or will the ESL file win the conflict, despite being in a higher block according to the save file headers?

The ESP's version gets used as the FE record.

Link to comment
Share on other sites

6 hours ago, Nukenin said:

User-created ESLs flagged with ESM can load above or below user-created ESLs not flagged with ESM; Plugins.txt dictates this.

ESLs can also load before USSEP.  The in-game mod load order menu will default ESLs to right below USSEP and will not allow ESLs to be moved below normal ESPs.  It will allow them to be moved above USSEP.   USSEP wins conflicts with ESLs above it, and loses conflicts with ESLs below it.  (USSEP is the only ESM-flagged ESP mod I have, but I assume this would be true for other such mods; not sure about user-created ESMs as I do not run with any currently.)

Is there any reason at all user-created ESLs be flagged ESM?  This could be reserved to distinguish official CC ESLs, as they will always be flagged ESM.

 

The problem here is that it doesn't matter where you tell it to put them, the results will be as listed in the OP. The save headers will come out with them listed in block order according to the plugin type. I've already tried numerous combinations like you specify and they always came out listed in the same groupings and things were not being overridden in the manner you are describing when examining the form IDs in the console for objects in the world. So it very much looks like a case where ESLs mapping to FE means that where they show up in the load order list is absolutely meaningless.

This means they won't be suitable for use as anything more than simple stuff, such as mods that alter vanilla records exclusively, or mods that only add their own new records and don't conflict with where they get placed into the game. Trying to account for ESLs in a bashed patch would be meaningless too and likely cause problems.

Link to comment
Share on other sites

38 minutes ago, Arthmoor said:

This means they won't be suitable for use as anything more than simple stuff, such as mods that alter vanilla records exclusively, or mods that only add their own new records and don't conflict with where they get placed into the game.

So both ours map marker mods can be or should be created as ESL, at least your Dawnguard Map Markers mod.  And it is understable why you would convert it to ESL due to the loadorder structure Bethesda introduced in the 1.5.3 patch.

Link to comment
Share on other sites

Yes, USSEP has been confirmed by form IDs in-game to load right where it's expected in the 05 slot. All of the ESL content shows up as FE, so the ESLs will override everything above that when there's a conflict.

Link to comment
Share on other sites

I was able to get USSEP to override ESLs placed above it in load order.   I was testing with two dead-simple ESLs that modified Dragonbane—one zeroed its enchantment cost (and enchantment, to remove the bar), while the other just changed its name.

Placing both above USSEP resulted in USSEP's Dragonbane (original name, charges/enchantment) winning the conflict.

I'm not doing any deep investigative work, just anecdotal goofin' 'round.  At varying points one or the other of the aforementioned ESLs was also flagged with ESM, but I never really did a thorough breakdown of what was happening when as I was already borrowing time I needed to spend on the paying work. :)

My interest is purely load order and not where things end up in data structures*.  Things win or lose based on when they load, not where they end up.  Is that too simplistic?  (And if it's wrong, let me have it!)  [EDIT: Well, of course I'm intrigued by data structures; I just have the (possibly mis-?)perception that the where is just a consequence here, and not the arbiter.]

I apologize for coming at this sideways; I'm selfish, I want to be able to use SSEEdit and LOOT without freakouts or wacky error messages just because I might want to patch one of the CC ESLs with an ESP.  I've avoided modding Skyrim for six years (as a mod creator, that is) but every time I stray into the periphery of these technical discussions it threatens to stir my interest.

Also, I suspect that the CC ESLs loading alphabetically might just be coincidental fluke because they are hardcoded in the EXE alphabetically.  (Something governs their automagic loading, and it's not just having a filename beginning with "cc" nor is it having the ESM flag set.)  I guess we'll have a better insight when the next round of CC content hits.

 

Edited by Nukenin
conflict resolution
Link to comment
Share on other sites

Nukenin, my interest is stirred as well.  However, I have no idea why Bethesda would create CreationClub and the .esl data structure.  Isn't the modding community working on the current framework?  I just watched a video on Creation Club - new weapons, armor, blah, blah, blah.  If ESL's are easier for new modder's to create new content, that makes sense.  Of course, I find the current modding paradigm working just fine to add new content to Skyrim and Skyrim SE.

Is Creation Club what jquery is to web development?  Curious.  Edit: I just watched GamerPoets  "I can't sleep, so I'll talk about Creation Club and paid mods...sort of." video on YouTube, and it seems that Creation Club and the ESL paradigm is not the same as jquery for web development. 

Jquery doesn't cost anything, and enhances web development.  Creation Club is more of a "stable" mods that you can purchase using credits and the mod is compatible with current saves.   Gopher's youtube video is hilarious but he did give some credit to this new format.  Creation Club is similar to jquery where you can select mods you want, and they will work with current saves.  Instead of searching Nexus for mods, the player just selects through the in-game menu for what they want and install it.   Thoughts?

I'm still wondering why change the paradigm because I think there are some really talented modders - hello Arthmoor! - and I'm not sold on adding another data structure (ESL) to a working mod paradigm.

Strat.

Link to comment
Share on other sites

@Arthmoor: The wiki page contradicts your posted info on where CC ESLs load: the wiki says they load before modder ESMs, you say after. I haven't checked which is right myself, but the wiki page makes CC ESLs consistent with other Bethesda-supplied plugins.

*Edit:* I pressed enter trying to @mention and it submitted my post... Also, does anyone know how the variable limit on the number of active ESLs arises? The wiki page says it varies with the number of records each ESL contains, but I don't see why that matters, and no further details are given.

The wiki page also contradicts the OP by saying there's no difference between ESLs with and without the master flag, they all load before ESP files.

Link to comment
Share on other sites

Yeah, sorry, I was the one who edited the wiki to say that before Bash was reading the save headers and cleared things up. I guess I should go fix that.

Link to comment
Share on other sites

23 hours ago, Arthmoor said:

Yeah, sorry, I was the one who edited the wiki to say that before Bash was reading the save headers and cleared things up. I guess I should go fix that.

The save headers are misleading though, they place all ESLs after all other plugins - I was trying to replicate the OP's statements using the save headers to see what was happening, and the save header contains:

Skyrim.esm
Update.esm
Dawnguard.esm
HearthFires.esm
Dragonborn.esm
Blank.esm
Blank.esp
ccqdrsse001-survivalmode.esl
Test Master.esl
test.esl

Where Blank.esm/Blank.esp are plugins with no masters and the appropriate master flag values, Test Master.esl is a copy of  ccqdrsse001-survivalmode.esl with the localisation flag removed, and test.esl is a copy of ccqdrsse001-survivalmode.esl with the localisation and master flags removed.

The above doesn't make any sense except as the load order after all ESLs have been mapped to FE, so doesn't describe the pre-mapping load order as I'm assuming the OP is trying to. It also doesn't match the results of the testing I did when ESLs initially hit FO4, but I haven't retested them against Skyrim SE 1.5.3.

@Nukenin's tests reported in the TES5Edit thread check using record overriding as mine did, and his findings are that:

  • ESLs load in their plugins.txt order amongst themselves
  • ESLs load before plugins without the master flag set
  • Non-CC ESLs can load before or after (non-harcoded) plugins with the master flag set, depending on plugins.txt order
  • CC ESLs always load before other ESLs and (non-hardcoded) plugins with the master flag set

However, it's unclear if the ESL plugins he made had the master flag or not. Please correct me if I've misunderstood anything, Nukenin.

Now, I previously found but have not re-verified that:

  • CC ESLs load immediately after the other hardcoded plugins, as it's both more sensible and easier to implement (and why would the devs make their own lives harder for no obvious benefit?).
  • Whether or not an ESL had the master flag set had no effect on where it loaded relative to non-master-flagged plugins.

It seems to me that for the purposes of tools interacting with load order, ESLs are just like master-flagged plugins, to be treated no differently - they always go before non-master-flagged plugins, and can intermingle with master-flagged plugins (excluding hardcoded plugins). So using the blocks defined in the OP, I think it goes:

  1. Block 1
  2. Block 3
  3. Blocks 2, 4 and 6 possibly all intermingled
  4. Block 5

It would be good if someone could independently verify or disprove this.

Link to comment
Share on other sites

I've been traveling so apologies for getting back a bit late on this.  I know at some point I fiddled with setting ESM on the ESLs I was testing with, but I don't recall it making a distinction in the load order.  But I don't trust my memory enough to say that with certainty. :)

Link to comment
Share on other sites

23 hours ago, Nukenin said:

I've been traveling so apologies for getting back a bit late on this.  I know at some point I fiddled with setting ESM on the ESLs I was testing with, but I don't recall it making a distinction in the load order.  But I don't trust my memory enough to say that with certainty. :)

That's OK, I did some more testing and answered most of my questions.

Link to comment
Share on other sites

From a post by CDCooley https://forums.nexusmods.com/index.php?/topic/6043958-unofficial-skyrim-creation-club-content-patches/page-15#entry54334023

Charlie breaks it down to 4 groups ..

Quote

There's absolutely nothing magical happening here, but people have gotten too comfortable with the idea that those first two digits of a formId you see in the game's console represent load order. They used to but now they don't thanks to ESL files. It's time to adapt folks. I'm seeing various people testing how things load by looking that the formID numbers in-game. 

Load order is controlled by the game's hard-coded list of official content and then things listed in the plugins.txt file. The numbers assigned as the first two digits of formID values in-game have nothing to do with load order when ESL files are involved. 

From my testing there appear to be only four groups of files that the game considers when loading files. 

First are the official files that are hard-coded into the game itself. Those used to be just Skyrim.esm, Update.esm, Dawnguard.esm, HearthFires.esm, and Dragonborn.esm but now include all of the new Creation Club ESL files too. That means the survival ESL will always load before any traditional mods including USSEP. If they exist in your Data folder they will be loaded and any entries in the plugins.txt file for them are ignored. 

Second are files listed in plugins.txt that are internally tagged as masters and marked as active. It doesn't matter what file extension they use (ESM, ESP, or ESL). They get loaded in the order they appear in the plugins.txt file (but only their order in relation to each other matters). 

Third are any other mods as listed in plugins.txt that are marked as active. The are the regular mods that we used to call ESPs but can technically include files with the extensions ESM and ESL too. 

And finally, the fourth group consists of only one file. It's the save file (ESS) being loaded (or a blank file if you're starting a new game).

Hopefully various mod managers (and LOOT) can be updated soon before too much chaos results from people thinking there's something more complicated and confusing happening. 

 

Edit : My bold in yellow

Link to comment
Share on other sites

This is starting to get weird. I think what needs to happen at this point is someone needs to come up with a small series of tests to ferret out just how this all works. The stuff I did was fairly basic. Official CC files, a master flagged ESL, and one regular ESL. It seems clear we need a few more cases to try out and gather results from. Like, say, a master flagged ESL using a CC ESL as a parent, a standard ESL using CC files as parents, normal ESLs using each other as parents, and so forth.

Either that or someone needs to go bug Bethesda for a proper official explanation of how all this works because despite various people testing things, one thing still seems clear to me - load order index numbers as shown in Wrye Bash are where the standard ESM and ESP files get mapped to in-game because clicking on their stuff in the console demonstrates that much.

Link to comment
Share on other sites

A test matrix to track what has been tested and what needs to be tested would be an excellent thing. If it needs to be done I can create one in Google Docs and share it out.

Also regarding the more technical specifics of this, back in the DOOM/Quake days we used to just email John Carmack (or Romero or John Cash) to get technical details when we couldn't figure something out.

Is there anyone at Bethesda that could help or at least direct us to someone that could answer the question?

Link to comment
Share on other sites

1 hour ago, Beermotor said:

Is there anyone at Bethesda that could help or at least direct us to someone that could answer the question?

I could be wrong. If something similiar happen on the old BSF then SmkViper (not sure if he is still he working for Bethesda or not) is the one the community could try to contact, because he was probably the only one who was fairly active in the forum.

Link to comment
Share on other sites

I ran a few tests to check stuff that Arthmoor, CDCooley and I disagree on:

Quote

Edited an item's name so that it had the following suffixes:
    Master ESL
    Non-Master ESL
    Master ESM
    Non-Master ESP

Then spawned the coat using player.additem and checked its name. A clean save was used each time.

The items were:
    Fallout 4: 001BBCB5 (Armor_Railroad01)
    Skyrim SE: 000CC842 (AncientNordAmulet)

Order 1:

    *non_master.esl
    *master.esl
    *master.esm
    non_master.esp

    The coat was called Master ESM, so ESLs respect plugins.txt relative to ESMs, ignoring master flag.

Order 2:

    *non_master.esl
    *master.esl
    master.esm
    non_master.esp

    The coat was Master ESL, so ESLs respect plugins.txt relative to each other, ignoring master flag.

Order 3:

    master.esm
    *non_master.esp
    *non_master.esl
    *master.esl

    The coat was Non-Master ESP, so ESPs always load after ESLs.

Order 4:

    As Order 3, but I edited the non_master.esl to remove its 0x200 flag.

    The coat was Non-Master ESP, so the 0x200 flag does not determine whether or not the game treats a plugin as a light master.

 

Link to comment
Share on other sites

11 minutes ago, WrinklyNinja said:

I ran a few tests to check stuff that Arthmoor, CDCooley and I disagree on:

Edited an item's name so that it had the following suffixes:
    Master ESL
    Non-Master ESL
    Master ESM
    Non-Master ESP

Then spawned the coat using player.additem and checked its name. A clean save was used each time.

The items were:
    Fallout 4: 001BBCB5 (Armor_Railroad01)
    Skyrim SE: 000CC842 (AncientNordAmulet)

Order 1:

    *non_master.esl
    *master.esl
    *master.esm
    non_master.esp

    The coat was called Master ESM, so ESLs respect plugins.txt relative to ESMs, ignoring master flag.

Order 2:

    *non_master.esl
    *master.esl
    master.esm
    non_master.esp

    The coat was Master ESL, so ESLs respect plugins.txt relative to each other, ignoring master flag.

Order 3:

    master.esm
    *non_master.esp
    *non_master.esl
    *master.esl

    The coat was Non-Master ESP, so ESPs always load after ESLs.

Order 4:

    As Order 3, but I edited the non_master.esl to remove its 0x200 flag.

    The coat was Non-Master ESP, so the 0x200 flag does not determine whether or not the game treats a plugin as a light master.

Good job! Seems like good news - I guess the Orders are the plugin.txt entries ?

1. games behave the same

2.  The rest basically means we have to force load the esls (based solely on extension)  with the esms and let plugins.txt decide amongst them ?

Link to comment
Share on other sites

14 minutes ago, Utumno said:

Good job! Seems like good news - I guess the Orders are the plugin.txt entries ?

1. games behave the same

2.  The rest basically means we have to force load the esls (based solely on extension)  with the esms and let plugins.txt decide amongst them ?

Yes, yes and yes.

My outstanding questions are:

  • Can a normal plugin and a light master share the FE slot? (They could in Fallout 4 when I tested in early September, but that might have changed.)
  • What's the precise relationship between number of records in ESLs and how many can be active?

I can test the first question myself, but the latter would be painful to answer with black-box testing, and could really do with some clarification from a dev or someone who can poke around the runtime innards to figure it out.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...