Jump to content

Plugin Files And You (ESM/ESL/ESP)


Arthmoor

Recommended Posts

On 10/6/2017 at 5:16 AM, Arthmoor said:

A .esm file extension will always be treated as a master file. The ESM flag will automatically set in memory.

So is this true for all games (Oblivion, Fallout3, Oldrim etc)?

On 10/6/2017 at 5:16 AM, Arthmoor said:

Files with the ESL flag are given special treatment with their form IDs. Upon loading into the game, the form IDs are remapped into the FE mod index. This is now considered a special slot in SSE and should not be occupied by any mods in your load order. Not even the Bashed Patch or any other form of merged data.

 

This should mean that Bash should disallow activating more than 254 non esl plugins?

On 10/6/2017 at 5:16 AM, Arthmoor said:

In practice this ends up actually being 2048 form ID records because the engine reserves everything from $0x0000 to $0x07FF.

Is this true for non esl mods too? (just for clarity) What are the dollar signs?

 

On 10/6/2017 at 5:16 AM, Arthmoor said:

Files flagged with ESL are known as "light plugins" or "light masters". How these load is determined by their file extension. The exception being any Creation Club content you may have downloaded. Those files load immediately following the 5 official files, according to the order they're listed in the Skyrim.ccc file.

1

Ok should this be corrected to determined by their the esm flag set implicitly (for .esl files) or explicitly (for .esp files).

What happens with esm files - can those have the esl flag set?

Also (and that's my main question):

Given that the formIds of the esl flagged files end up in slot FE don't they override all the rest (apart from the save file FF formids)? So what difference does it make .esl files load with esms? What does the load order mean really? Maybe the save file displaying its esl masters at the end is more correctly? Where would such a file display an esl flagged esp?

I probably don 't understand load order meaning - I thought it was there to actually sort conflicts in form ids from mods (last loading wins) - and that's how the Bashed Patch thinks IIUC. Does this mean we should exclude all esl flagged mods from the patch?

What is the function of load order if not this? Is the way we display load order correct or should we display esl flagged files at the bottom? Or does the load order has to do with master/children relationships - but how would then an esp override a record from an esl master, since that record would end up in FE?I need the answer to these questions however elementary they may seem to get things right in bash. I do acknowledge that the questions may be answered again but then there is no central place I can find the answers (also looked https://tes5edit.github.io/docs/9-appendix.html#s_9-2 and https://www.creationkit.com/fallout4/index.php?title=Data_File ) but did not find a clear answer and I got some contradictions on top.

 

Better have all the info and links on OP here

Link to comment
Share on other sites

I don't recall if anyone tested the .esm handling with anything other than SSE and FO4. It would probably be safe to assume that the older games handle the .esm extension the same way SSE and FO4 do though.

Yes, Wrye Bash should not allow enabling 254 non-ESL mods because occupying the FE slot will cause corruption during gameplay. That slot is now reserved for the use of ESL flagged files in SSE and FO4.

Yes, form IDs in all files begin at $0x0800. Anything below that value is a hardcoded engine ID.

An .esm file can have the ESL flag set. It's forms will be mapped into the FE slot just like any other.

And yes, I realize it's confusing, but the ESL flag does not have any bearing on load order. Everything is still ordered according to the same rules we had up until these were introduced. Load order still boils down to this basic setup:

The official base master files and DLCs. The order of these is still hardcoded.
Creation Club files. The order of these is specified by the *.ccc file for each game. If that file is missing then it falls back on a hardcoded list.
ESM flagged files, regardless of how that flag was acquired. Order specified by plugins.txt.
Everything else that does not have an ESM flag. Order specified by plugins.txt.

As far as the list of files in the save game, there are apparently 2 lists. One for all the regular plugins and another for all of the ESL flagged files. I don't know the details of how those are handled internally. Someone who knows how to reverse engineer the game data would have to comment on that. The game somehow knows how to keep it straight though because load order has been working just fine with all the ESL flagging everyone has been doing.

Bottom line is that Wrye Bash currently does NOT need to change anything in its load order handling. All of that is working. Only the list of files in the save is out of order. Which isn't going to be an easy thing to work out how to fix.

Bashed patches can also process ESL flagged files just fine, but that should not be allowed to include any CELL records that originate with an ESL flagged files in SSE or the cells will not load correctly in the game. So far this issue doesn't seem to exist with FO4.

Link to comment
Share on other sites

Very interesting @Arthmoor thanks indeed. Please edit the OP (it would be a good idea to allow the last edit date to be seen so we know when it's updated)

I answered and asked some more questions on github: https://github.com/wrye-bash/wrye-bash/issues/426#issuecomment-448718981

Please reply to me there - I would need some testing with the esls on saves (rearranging esl masters and including esp/esm ESL flagged plugins)

Where do people discuss those these days ? Is SKSE discord? Can't seem to find a link for that - I would like to keep technical info on github so it's easy to find but I need to contact the more technical fellows like @ElminsterEU

Link to comment
Share on other sites

The ESL stuff was discussed extensively on the xEdit Discord. So that's probably the best way to get to the folks who know this stuff best.

Link to comment
Share on other sites

.esm extensions sets ESM flag in SSE and FO4 only, they are the only games that look at the extension at all. Older games don't care about extensions and check the header flags only.

Link to comment
Share on other sites

That's actually not the case though. Oblivion loaded a mod I intentionally renamed to .esm into the ESM block and its form IDs confirmed this to be the case in-game. So it would seem to be valid for all of the games and not just SSE and FO4.

Link to comment
Share on other sites

17 hours ago, zilav said:

.esm extensions sets ESM flag in SSE and FO4 only, they are the only games that look at the extension at all. Older games don't care about extensions and check the header flags only.

 

4 hours ago, Arthmoor said:

That's actually not the case though. Oblivion loaded a mod I intentionally renamed to .esm into the ESM block and its form IDs confirmed this to be the case in-game. So it would seem to be valid for all of the games and not just SSE and FO4.

Ahem :P

Link to comment
Share on other sites

On 12/21/2018 at 1:39 AM, zilav said:

Then need to recheck older games because the last time I heard extension doesn't matter in FO3 and FNV.

Please do I need a definite answer if I am to ever finalize support for esls in bash

Link to comment
Share on other sites

  • 3 weeks later...
  • 1 month later...
On 10/6/2017 at 11:16 AM, Arthmoor said:

engine reserves everything from $0x0000 to $0x07FF. All files using ESL flags must therefore contain their internal form IDs between xx000800 and xx000FFF. Anything exceeding this range is invalid and the game will either crash or you'll have severely corrupted data due to overruns.

I don't understand this part.. 
Can somebody explain in like more detail and in noob friendly terms? :P How do I know if my plugin is valid to be converted to .esl or to be flagged as an esl?
But in any case, should not compacting the form ID's trhough CK and xEdit automatically fix that by itself? Cause it will change its original form ids, and it should put the correct range of form id's that can be used so the engine doesn't shit itself?

On 10/6/2017 at 11:16 AM, Arthmoor said:

DO NOT attempt to patch anything inside of a CELL record that originates with a file flagged as ESL. Doing so will cause the cell to break and not load most of its contents when you enter it. This includes any official Creation Club content you may have. It doesn't matter who made the file, it will break.

Also.. So you need to de-convert it and patch it?


I also noticed in the CK, you cannot use .esl files as active plugins. Does this mean I have to remove it's esl extension and re-compact and re eslify after I am done?

 

Link to comment
Share on other sites

  • 1 year later...

What is the benefit of removing the ESL flag on ESPs (in VR, no ESL support)

I've got a lot of ESL flagged ESPs. Because I'm running VR the ESL flag doesn't save me a plugin slot. I don't know how an ESP flagged as an ESL affects load order. I read they are loaded in ESL space. I went through and removed the ESL flags in xEdit. I imagine this makes them be able to be ordered in the load order as a normal ESP. Is that a benefit? Is it true that VR won't even load an actual ESL until it's renamed as an ESP?

I've read this but possibly only pertains to the plugin's master file(?), "ESL-Flagged ESP's.. do not shuffle their masters to a different Block, loading everything in order on runtime as intended except for taking up the FE slot by force. So an ESP with an ESL flag should be treated as a regular ESP."


Thanks

PS. Do you think it would ever be possible for someone to mod in support for ESL files for VR?

Link to comment
Share on other sites

Removing the flag will simply cause the game to treat it as a normal .esp or .esm file.

If the file was originally named  with the .esl extension, then changing that to .esm would be preferred to keep it from disrupting its intended load ordering.

That said, you need to be aware that VR is a different beast and there won't be alot of mods that can be safely run under those conditions, no matter what folks on reddit may claim.

Link to comment
Share on other sites

I've been doing a bit of testing of my own, converting a couple mods with scripts from esp to esp-fe. I found that there might be another issue with converting to esp-fe. Which is persistency of some registration functions. Specifically RegisterForModEvent and RegisterForKey. When they are placed only in the Init() function, they will run fine on starting a new game or from a clean save. However saving and reloading and the registrations will no longer be there. To bypass this I've been placing the registrations inside of OnPlayerGameLoad. I've only converted a couple mods to esls, so I don't know if there are any more functions that have this behavior. It is also possible that this is due to my setup. The testing I've done is for a compacted esp that works fine, but when the esl flag is ticked the registration issue suddenly occurs.    

Link to comment
Share on other sites

Those are both SKSE functions so you may want to bring that to the attention of the development team for that.

Link to comment
Share on other sites

I might do that, this seems to have happened to other before, see link below. In the meantime, until this gets fixed, its probably a good idea to keep in mind that converting a mod to esl also requires recompiling scripts that have skse registrations in the Init or the MCM, and not in the OnPlayerGameLoad. 

 

Link to comment
Share on other sites

  • 1 year later...
  • 1 month later...

Can SkyUi be given a ESL flag safely? Vortex labs it as "could be light" and it seems to fit the criteria above (It doesn't even require in compacting). but i've heard some people say its a bad idea.

Link to comment
Share on other sites

  • 2 months later...
On 2/15/2022 at 7:23 PM, Jason Parker said:

Can SkyUi be given a ESL flag safely? Vortex labs it as "could be light" and it seems to fit the criteria above (It doesn't even require in compacting). but i've heard some people say its a bad idea.

I too would like a definitive answer to this- I have been running SkyUI_SE.esp as an ESPFE through 3 playthroughs, each with 400 or so mods. As far as I can see, this has not caused any issues.

Because of this success (and similar anecdotal successes on the part of a couple of other users) I uploaded a copy of the ESPFE SkyUI plugin to my SkyUI Added Master patch page, but now someone is citing Ruddy88's admonition never to eslify SkyUI, Racemenu or FNIS and I'm concerned that I may be inflicting future grief upon unsuspecting players (the ones at a level of knowledge such that they are not comfortable adding Skyrim.esm as a Master or ESLifying the plugin for themselves.)

If eslifying SkyUI_SE.esp is a bad idea despite Vortex and xEdit showing it as "Could be light," I would appreciate knowing about it so I can remove the ESLified plugin from the patch page.

Thanks.

Link to comment
Share on other sites

There is no harm in ESLifying the SkyUI plugin. Whether that's going to do anything for you or not is a different story.

Mine has been ESLified for a very long time and has caused me no issues in my game.

  • Like 2
Link to comment
Share on other sites

Its harmless in itself but some mods may complain (read: chesko's) they're mostly in need of an update/rewrite at this point though

Link to comment
Share on other sites

16 hours ago, Arthmoor said:

There is no harm in ESLifying the SkyUI plugin. Whether that's going to do anything for you or not is a different story.

Mine has been ESLified for a very long time and has caused me no issues in my game.

Thank you, Arthmoor, for this response and for your stunningly-prolific sharing of your expertise on Bethesda modding.

Link to comment
Share on other sites

  • 3 months later...

I have come here to share with you folks the method for creating a Light Master without using tools to turn flags on/off, or renaming file extensions or some combination thereof.

In the Creation Kit, under the File Dropdown Menu is the option "Convert Active File to Light Master". 


image.png.ca26b60d653b914dc4e624083c50572f.png

 

Using this option does some things automagically.  It compacts the active plugin and then verifies that the compacted plugin is within the size restrictions for a Light Master and if not, prompts the user to compact the plugin.  Once verified, the ESL bit in the header is set on and the file is saved with the ESL extension.

No fuss, no muss.  Just a one click verified Light Master.

I know this is a Skyrim forum and the screen shot is from Fallout 4, but the option is common to both versions of the CK.

Being human, I misunderstood the write up on the Convert Active File to Light Master function.  I have corrected my writings accordingly.  I apologize for any confusion my error may have caused. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...