Search the Community

Cleaning the Official Master ESMs NOTE : I do not have any Bethesda games installed anymore, and xEdit has changed greatly since I last had it installed. Anyone wishes to re-do this topic in a new thread bringing it up to date .. Crack on This guide assumes using TES5Edit on Skyrim Nexus, Or SSEEdit on Skyrim SE Nexus Due to this guide being dual purpose ( For Skyrim and Skyrim SE ) for the rest of this guide I will refer to both tools as xEdit. Screenshots of tools used may be one or the other, or older versions, which does not matter, the images are only to illustrate the method / options used. Why Clean the Master Files ? Firstly because the masters have entries that are identical to the same records in Skyrim.esm or other DLC esms'. They exist because Bethesda may have looked at something in the CK and an unneeded entry was auto included in the plugin even though the item was not altered in any way. The Official Creation Kits are notoriously buggy and randomly create dirty / wild edits, often when the author of the plugin is completely unaware. Wherever that plugin is placed in your load order its records overwrite all the conflicting records from plugins loaded before it ( the rule of one ) resetting the settings back to the values contained in the Official Bethesda DLC. It won't cause crashes, it just changes the values of plugins loaded before it. Which can alter mods that you have for Weapon Damage, Armor, Lighting, Food Effects and so on. The masters are very early in your load order but there is potential for a mod to be made as a fake.esm, and placed among them, and so ITMs in a later loading master may cause problems for that mods esm. Chance is remote that a master will affect another master, and this procedure is best used on all of your mods plugins, but cleaning everything of ITMs ( Identical to Master records ) causes no harm, is more optimal giving the game less to process in your load order, and so it is best to get rid of these completely unnecessary dirty edits. The Second reason is that Bethesda chose to delete some things that are in the Official DLC. Any mods loaded with references to deleted records from the Official Bethesda DLC will cause your game to crash. This problem particularly affects older mods ( especially mods that were made before newer official patches were released, with more deleted references the old mod did not anticipate - It will also become problematic for the Skyrim Special Edition community where old Original Skyrim mods are being converted to SSE, and Bethesda have deleted even more records from the plugins before they released the newer plugins for that version of the game ). xEdit can restore and properly assign values to these records that will disable them and still allow mods to access them. This is done using the "Undelete and Disable References" option. For further explanations of why it is still recommended to clean the games masters .. Read on from this post, Zilav and Arthmoor, most valued technical and vastly experienced modding authors, weigh in on the subject. The following mostly apply to mod authors, but worth knowing about for mod users too : xEdit will also report when a mod has Deleted NavMeshes as part of the report from Automatic cleaning. Like deleted references, any mod that references a deleted NavMesh will cause Skyrim to Crash. Properly optimizing your mods NavMeshes and checking your mod for Deleted Vanilla NavMeshes ( which can also be caused by a CK wild edit even if you did not do it yourself ) is important. Mods altering the same cell and the same NavMeshes when your mod is not optimized will cause Skyrim to Crash. Poorly optimized NavMeshes with errors reported by the CK will make Skyrim unstable. Instabilities like fast travelling to a location and Skyrim crashes. Note the ones found to be deleted in the games masters, cannot be undeleted. To fix deleted Navmeshes in your mods, Arthmoor has provided a walkthrough in Skyrim - Fixing Navmesh Deletion in TES5Edit Manually cleaning your mods is also important to remove wild edits. This is mostly down to the experience of Mod Authors to solve such problems, but there are a few noted later in this guide which are in the DLCs which everyone can easily Manually clean. Some mods can have accidental Wild Edits in them caused by the author looking at how Bethesda did something they wish their mod to do as well. These Wild Edits often prevent Skyrim from doing things like advancing quests, spawning NPCs, assigning dialogue to NPCs, preventing NPCs from patrol areas they are assigned to. They can also alter Vanilla Lighting and Triggers that the author wished to use. All of these things affect any plugin with conflicting records loaded before a mod with Wild Edits. Mod authors - Learn to use xEdit, and ensure the only records in your mod plugins are what you would expect to be in there, its the most important tool the community can make use of when used properly. Mod Users - Follow this guide... Before moving on to the Manual cleaning, something everyone should do prior to Manual Cleaning : Automatic cleaning of Bethesda's ESMs with xEdit With the games Original esm's installed ( You can use Steam to Verify Integrity of Game Cache of Skyrims files to ensure you have good error free copies of the original master files ), and in accordance with the following wiki article http://www.creationk...ty_Plugins_List : Load up xEdit. 1. Right click the plugin selection screen and select "none" 2. Tick the relevant esm to edit, and click okay ( If you have not cleaned any of your Master files yet, the first one to tick will be Skyrims "Update.esm" ), then click Okay After each of the following actions, wait for a message in the message window that the previous operation has finished / Done : 3. Right click the plugin after you get the "Background Loader : Finished" message,and choose "Apply Filter for Cleaning" Wait until Filtering is finished then .. 4. Right click the plugin and choose Remove Identical to Master Records Wait until it finishes then .. 5. Right click the plugin and choose Undelete and Disable references Wait until it finishes then .. 6. Close xEdit, and it should check with you that you wish to save the plugin ( this only happens if you have made any changes to the plugin to save, if it just closes .. Then you have not cleaned anything ) Rinse and repeat the Automatic cleaning ( steps 1 - 6 above ) for each of the master files. Working from first to load, to last, not including Skyrim.esm or any unofficial patches ( No point doing Skyrim.esm, and the unofficial patches are already done and should not be cleaned ) So clean in this order Update.esm Dawnguard.esm Dawnguard.esm ( Yes it needs to be done twice ) Hearthfire.esm Dragonborn.esm Dawnguard.esm needs to be cleaned twice ( as of xEdit 3.1 onwards - After doing the automatic cleaning routine once on Dawnguard.esm, and saving it, load it up again in xEdit and you will be able to clean a few more ITMs ) : ------------------ Dawnguard.esm needs manual cleaning aswell as automatic cleaning After the automated cleaning is done, you can also now manually clean a few more Wild edits xEdit will not have touched ... Recently Arthmoor has brought to the attention of the community additional information regarding manual cleaning of Dawnguard.esm, which everyone needs to do for their own setup same as automatic cleaning ( because nobody can legally upload official master files anywhere, everyone needs to do their own ) First load up xEdit When the plugin selection comes up, right click and select None Then put a tick in the box just for Dawnguard.esm, click Okay After its finished loading, right click Dawnguard.esm and choose "Filter for Cleaning" 1. For "CELL 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm." .... Expand the records as in the following screenshot, and right click the indicated sub-record, and choose Remove 2. For "CELL 0001FA4C: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove 3. For "CELL 0006C3B6: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove NOTE : This guide used to include cleaning instructions for "CELL 00039F67: Wild edit. Remove this record. It's a testing cell" ( The WICourier edit ) - But since the new version of TES5Edit 3.1+ now cleans that as part of the automated cleaning ( which you should have done prior to manual cleaning ), you no longer need to clean it manually afterwards. ----------------------------------------- Now that the Master files are cleaned, you could put them in a zip, and get your mod manager to install them - Maybe at a future date you want to do a refresh of steam cache and it redownloads the masters which are not the same as the originals anymore (because you cleaned them), so then you would need to reclean them again. But beware, Bethesda have started redoing some masters due to Creation Club mods compatability, so make sure any redownloaded masters are not newer than your previously cleaned ones, because in that case you will need to reclean and rezip them again anyway. You can go through the rest of your Load Order using Automatic cleaning of ITMs and UDRs on all your mods plugins. The sequence of cleaning mods plugins should be after you have your Load Order correct, masters are cleaned, then clean them with the last to load being the last to clean. Mod authors should have done them already, so most will probably not need cleaning. Also look out for any mod specific cleaning instructions in the mods description. Prime example = The Unofficial Patches will not need any cleaning, they are already done, and any remaining ITMs in those plugins should be left alone because they do have a purpose .. ( its a very rare occasion when this is true ). The xEdit Work In Progress Development topic is at the following link https://www.afkmods.com/index.php?/topic/3750-wipz-tes5edit/ Development project is at GitHub https://github.com/TES5Edit/TES5Edit And newer versions of xEdit (3.2.7 +) have a link to Discord top right of xEdit window.

Beginning with Patch 1.5.3 of Skyrim Special Edition, the game supports a new type of mod file known as a "light plugin" with the .esl extension. Elminster has spent a considerable amount of time studying how this works and has had a number of community members aid in verifying this information. Since questions about new plugin types come up all the time, I felt it was best to make sure our source here is accurate. Flag Types Plugins can have 3 different flag types: ESP, ESM, and ESL. This causes them to load in different ways, and in some cases handle their form IDs different from the old ways. ESP files are standard every day mods. There will be no type flag set on these and they have the .esp file extension. Mod managers all know how these work and various modding tools consider these to be the basics of modding. Files flagged with ESM are commonly referred to as "masters". They load early, immediately following the 5 hardcoded official files and any Creation Club content you may have. Files flagged with ESL are known as "light plugins" or "light masters". How these load is determined by their file extension. The exception being any Creation Club content you may have downloaded. Those files load immediately following the 5 official files, according to the order they're listed in the Skyrim.ccc file. File Extensions The engine treats files in a certain way based on their file extensions: A .esm file extension will always be treated as a master file. The ESM flag will automatically set in memory. A .esl file extension will always be treated as a light master. The ESM and ESL flags are automatically set in memory. A .esp file can function as either type of file depending on which flags are set in its file header. If the ESM flag is set, then the .esp is treated as a master file. If the ESL flag is set, it will be treated as a light plugin. If BOTH flags are set, then it is treated as a light master. It is important to note that as far as the game is concerned, a .esp file with no ESM flag will load as a normal .esp file would. All standard load order procedures apply to them, and any properly functioning mod manager should handle them the same way it does regular .esp files. The Golden Goose Files with the ESL flag are given special treatment with their form IDs. Upon loading into the game, the form IDs are remapped into the FE mod index. This is now considered a special slot in SSE and should not be occupied by any mods in your load order. Not even the Bashed Patch or any other form of merged data. As a result of this new support, the engine can allow you to bypass the previously hard limit of 255 plugin files. The theoretical limit now should be 4096 ESL flagged mods. Individual files flagged as ESL can in theory hold up to 4096 form ID records. In practice this ends up actually being 2048 form ID records because the engine reserves everything from $0x0000 to $0x07FF. All files using ESL flags must therefore contain their internal form IDs between xx000800 and xx000FFF. Anything exceeding this range is invalid and the game will either crash or you'll have severely corrupted data due to overruns. The reason for this is how the mapping works. We're all familiar with the normal 8 digit form ID. The first 2 digits are only used to handle load order, the other 6 are internal to their plugin file. What the ESL flag system does is it break this down even further. The First 3 digits get mapped into an internal table for the FE slot. The remaining 3 belong to the actual records in the plugin. You'll end up with something like this: FE000800 - FE000FFF ModFile1 FE001800 - FE001FFF ModFile2 FE002800 - FE002FFF ModFile3 FExxxyyy and so forth up until you reach FEFFFFFF and run out of IDs and mappings. This is where the 4096 files and 2048 form IDs per file comes from. The benefits of all of this should be fairly clear. No more need to merge the 20 weapon mods with 5 records each. Just flag them with ESL and you're good to go. This is much safer than merging and is fully supported by the game engine so it works the same way for everyone. Other Information Using ESL flagged files come with some important issues to consider: You must be playing on a copy of SSE version 1.5.3 or newer in order to use ESL flagged files. You must be using a copy of the SSE CK version 1.5.3 or higher in order to properly recognize these files for editing. The file cannot contain more than 2048 records in it or the form IDs will overflow into another FE slot and corrupt the save. Form IDs need to be compacted for the file to be considered valid. The CK has an option called "Compact Active File Form IDs" in the file menu. xEdit also has a function for this. When using either of these, PAY ATTENTION TO THE WARNINGS. You cannot compact IDs on a file you intend to continue using in an existing save or you WILL corrupt it due to the form IDs contained within it being rewritten. Only do this on files you have not yet begun using, and which do not yet exist as public releases. Other mods which reference form IDs in the mod you are compacting (compatiblity patches, script calls, etc) will break if you compact the form IDs on a mod that's been released to the public. NMM does not work with files that have the .esl extension or carry an ESL flag. It has been reported to completely lock up if even one of these is present in your Data folder. Use a functional mod manager such as Wrye Bash to manage these types of mods. If by some wierd twist of fate you end up with a ESL flagged file that's also ESM flagged and someone uses a standard .esp file as a master to it for some reason, that file will be forced to load into the wrong part of your setup. Normally a standard .esp file will never load before anything ESM flagged, this is the one exception to that now and it should be considered a bug to report to any mod's author who has done this. DO NOT attempt to patch anything inside of a CELL record that originates with a file flagged as ESL. Doing so will cause the cell to break and not load most of its contents when you enter it. This includes any official Creation Club content you may have. It doesn't matter who made the file, it will break. Mods that use the GetFormFromFile() function in their scripts to reference things within them cannot be compacted unless the scripts are also recompiled with the new form ID. Mods using recorded voice lines or that include NPC facegen data cannot be compacted because all of those asset files will need to be manually renamed to match the new dialogue and NPC form IDs. This post will be updated again as needed to reflect new information that may be uncovered.

Now that this kalpa for the USKP is complete (for you non-lorehound types, kalpa = cycle) it's probably time we make public our plans for the next one. Due to an increasing number of odd engine related bugs it's been decided among the team that we will need to convert the USKP (and therefore all the DLC patches too) into what's called a "False ESM". That is, and ESP file which is flagged as an ESM by way of changing the flag in TES5Edit. So why would we want this? There are a few classes of bugs this has already been shown to help with in internal testing: Seemingly random crashes in and around major cities, and other areas where a lot of data is edited. Especially navmeshes. The "landscape foliage" bug in which a child worldspace begins displaying landscape foliage from its parent worldspace because an object in the parent worldspace was edited. This is readily apparent in Darkwater Pass. The "LOD trees show up at close range" bug. This is where you will sometimes see trees that are clearly intended to be displayed as LOD sitting within the currently loaded cells, sometimes even merged into the real trees. As described here. There may be others, but these 3 classes of bugs are confirmed to be correctable by converting to a false ESM. So you might be asking then. Why not a proper ESM file? Two reasons mainly: 1. Steam cannot support ESM files, we'd have to cease distribution there all together and not just for the USKP. 2. Other mods are relying on the filenames being ESP files and not all of those mods are currently being maintained anymore. It's already been verified that Wrye Bash is OK with changing the flag. It will not complain when the files get bumped up into the ESM section. With proper CRC checks, BOSS won't complain either. Though I'm hoping some other method can be used to detect it ongoing because having a growing mountain of CRC checks won't be helpful. Perhaps version number parsing from the description field. I cannot vouch for how NMM will react since I don't use it, and I'm not sure anyone else on the team has tried it either. Who knows what Mod Organizer will think. We will bump all the version numbers on all the patches up to 2.0 when this happens. We don't have a time frame yet, because there's also a giant mountain of vanilla bugs piled up in the tracker to address too. But when it happens, all of the patches will be updated at the same time. Our proposed changes in the load order for the game will look like this: Skyrim.esm Update.esm Unofficial Skyrim Patch.esp Dawnguard.esm Unofficial Dawnguard Patch.esp Hearthfires.esm Unofficial Hearthfire Patch.esp Dragonborn.esm Unofficial Dragonborn.esp [The rest of your stuff] The main reason for this should be obvious. It will generate much less interference with things following on in the ESM group, and at the same time allow us to undo some of the changes in each DLC patch that are only there because the USKP canceled them out due to its own edits. Best of all, thus far in testing, it poses no risk for converting them in the middle of an active playthrough, even with a lot of other mods loaded. So anyway, that's where we're headed now. We've already kind of leaked hints at it, so now it can be considered confirmed rather than rumor The ESM Conversion Checklist Wrye Bash Support - Supported as of release 304.2. NMM Support - Works as expected, NMM flags the files with a warning but allows them to move into the ESM group. Mod Organizer Support - v1.0.0rc1 and up has been confirmed to work. TES5Edit Support - Supported as of release 3.0.31. BOSS Support - Working in 2.1.1. (was some confusion here, but it *IS* working) Checklist items need to be completed before we move forward on this. BETA Scheduling We're planning to go into beta on all 4 patches Saturday (October 5). I expect we'll have a minimum of 2 weeks testing, perhaps longer, to see how all this affects things and to validate that none of the new fixes coming in with this are broken. With that in mind, I figured this is as good a time as any to say so. Once this round of beta testing is complete, the USKP will no longer be distributed on Steam. Leaving the 1.3.2c version up there is already causing us problems and we can't in good conscience leave it up there after 2.0 has gone live. So anyone still using Steam to get the USKP needs to transition to one of the other available download sites. We're also going to have to check to see if Steam will accept a falsely flagged ESP file and deliver it back to the user intact as such. I mention this now because if it doesn't, then ALL of the unofficial patches will have to be taken down from Steam and everyone using those too will need to make provisions to change to another download site. So keep that in mind for those of you using Steam.