Jump to content
alt3rn1ty

Manual Cleaning Skyrim and Skyrim SE Master files

Recommended Posts


Cleaning the Official Master ESMs

 

NOTE : I do not have any Bethesda games installed anymore, and xEdit has changed greatly since I last had it installed.

Anyone wishes to re-do this topic in a new thread bringing it up to date .. Crack on :D

 

This guide assumes using TES5Edit on Skyrim Nexus,
Or SSEEdit on Skyrim SE Nexus

Due to this guide being dual purpose ( For Skyrim and Skyrim SE ) for the rest of this guide I will refer to both tools as xEdit.
Screenshots of tools used may be one or the other, or older versions, which does not matter, the images are only to illustrate the method / options used.

Why Clean the Master Files ?

 

Firstly because the masters have entries that are identical to the same records in Skyrim.esm or other DLC esms'. They exist because Bethesda may have looked at something in the CK and an unneeded entry was auto included in the plugin even though the item was not altered in any way. The Official Creation Kits are notoriously buggy and randomly create dirty / wild edits, often when the author of the plugin is completely unaware. Wherever that plugin is placed in your load order its records overwrite all the conflicting records from plugins loaded before it ( the rule of one ) resetting the settings back to the values contained in the Official Bethesda DLC. It won't cause crashes, it just changes the values of plugins loaded before it. Which can alter mods that you have for Weapon Damage, Armor, Lighting, Food Effects and so on. The masters are very early in your load order but there is potential for a mod to be made as a fake.esm, and placed among them, and so ITMs in a later loading master may cause problems for that mods esm. Chance is remote that a master will affect another master, and this procedure is best used on all of your mods plugins, but cleaning everything of ITMs ( Identical to Master records ) causes no harm, is more optimal giving the game less to process in your load order, and so it is best to get rid of these completely unnecessary dirty edits.

The Second reason is that Bethesda chose to delete some things that are in the Official DLC. Any mods loaded with references to deleted records from the Official Bethesda DLC will cause your game to crash. This problem particularly affects older mods ( especially mods that were made before newer official patches were released, with more deleted references the old mod did not anticipate - It will also become problematic for the Skyrim Special Edition community where old Original Skyrim mods are being converted to SSE, and Bethesda have deleted even more records from the plugins before they released the newer plugins for that version of the game ). xEdit can restore and properly assign values to these records that will disable them and still allow mods to access them. This is done using the "Undelete and Disable References" option.

For further explanations of why it is still recommended to clean the games masters .. Read on from this post,

Zilav and Arthmoor, most valued technical and vastly experienced modding authors, weigh in on the subject.

 

The following mostly apply to mod authors, but worth knowing about for mod users too :

xEdit will also report when a mod has Deleted NavMeshes as part of the report from Automatic cleaning. Like deleted references, any mod that references a deleted NavMesh will cause Skyrim to Crash. Properly optimizing your mods NavMeshes and checking your mod for Deleted Vanilla NavMeshes ( which can also be caused by a CK wild edit even if you did not do it yourself ) is important. Mods altering the same cell and the same NavMeshes when your mod is not optimized will cause Skyrim to Crash. Poorly optimized NavMeshes with errors reported by the CK will make Skyrim unstable. Instabilities like fast travelling to a location and Skyrim crashes. Note the ones found to be deleted in the games masters, cannot be undeleted. To fix deleted Navmeshes in your mods, Arthmoor has provided a walkthrough in Skyrim - Fixing Navmesh Deletion in TES5Edit

Manually cleaning your mods is also important to remove wild edits. This is mostly down to the experience of Mod Authors to solve such problems, but there are a few noted later in this guide which are in the DLCs which everyone can easily Manually clean. Some mods can have accidental Wild Edits in them caused by the author looking at how Bethesda did something they wish their mod to do as well. These Wild Edits often prevent Skyrim from doing things like advancing quests, spawning NPCs, assigning dialogue to NPCs, preventing NPCs from patrol areas they are assigned to. They can also alter Vanilla Lighting and Triggers that the author wished to use. All of these things affect any plugin with conflicting records loaded before a mod with Wild Edits.

Mod authors - Learn to use xEdit, and ensure the only records in your mod plugins are what you would expect to be in there, its the most important tool the community can make use of when used properly.

Mod Users - Follow this guide...

 

 

 

Before moving on to the Manual cleaning, something everyone should do prior to Manual Cleaning :


Automatic cleaning of Bethesda's ESMs with xEdit

With the games Original esm's installed ( You can use Steam to Verify Integrity of Game Cache of Skyrims files to ensure you have good error free copies of the original master files ), and in accordance with the following wiki article http://www.creationk...ty_Plugins_List :


Load up xEdit.

1. Right click the plugin selection screen and select "none"


large.xEdit1.png.4e327d2849d996c61b870d6


2. Tick the relevant esm to edit, and click okay ( If you have not cleaned any of your Master files yet, the first one to tick will be Skyrims "Update.esm" ), then click Okay


large.xEdit2.png.a538b216ea95a79a5e15ef5


After each of the following actions, wait for a message in the message window that the previous operation has finished / Done :

3. Right click the plugin after you get the "Background Loader : Finished" message,and choose "Apply Filter for Cleaning"



large.xEdit3.png.8392a40f0139e2199b57b4e


Wait until Filtering is finished then ..

4. Right click the plugin and choose Remove Identical to Master Records



large.xEdit4.png.43f1c1457ae7c6d949de243


Wait until it finishes then ..

5. Right click the plugin and choose Undelete and Disable references



large.xEdit5.png.8c600acaeb8e116f182f03b


Wait until it finishes then ..

6. Close xEdit, and it should check with you that you wish to save the plugin ( this only happens if you have made any changes to the plugin to save, if it just closes .. Then you have not cleaned anything )


Rinse and repeat the Automatic cleaning ( steps 1 - 6 above ) for each of the master files.

Working from first to load, to last, not including Skyrim.esm or any unofficial patches
( No point doing Skyrim.esm, and the unofficial patches are already done and should not be cleaned )

So clean in this order

Update.esm

Dawnguard.esm

Dawnguard.esm ( Yes it needs to be done twice )

Hearthfire.esm

Dragonborn.esm


Dawnguard.esm needs to be cleaned twice ( as of xEdit 3.1 onwards - After doing the automatic cleaning routine once on Dawnguard.esm, and saving it, load it up again in xEdit and you will be able to clean a few more ITMs ) :






------------------



Dawnguard.esm needs manual cleaning aswell as automatic cleaning


After the automated cleaning is done, you can also now manually clean a few more Wild edits xEdit will not have touched ...

Recently Arthmoor has brought to the attention of the community additional information regarding manual cleaning of Dawnguard.esm, which everyone needs to do for their own setup same as automatic cleaning ( because nobody can legally upload official master files anywhere, everyone needs to do their own )



First load up xEdit

When the plugin selection comes up, right click and select None
Then put a tick in the box just for Dawnguard.esm, click Okay
After its finished loading, right click Dawnguard.esm and choose "Filter for Cleaning"


1. For "CELL 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm." .... Expand the records as in the following screenshot, and right click the indicated sub-record, and choose Remove



large.58af75b0a7fef_Cleaning1.png.4d9fd9


2. For "CELL 0001FA4C: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af75e78d857_Cleaning2.png.cf5c54


3. For "CELL 0006C3B6: Wild edit. Remove this record. It's a testing cell." .... Expand the records as in the following screenshot, and right click the indicated record, and choose Remove


large.58af761722959_Cleaning3.png.e4213b
 

 


NOTE : This guide used to include cleaning instructions for "CELL 00039F67: Wild edit. Remove this record. It's a testing cell" ( The WICourier edit ) - But since the new version of TES5Edit 3.1+ now cleans that as part of the automated cleaning ( which you should have done prior to manual cleaning ), you no longer need to clean it manually afterwards.


-----------------------------------------


 

Now that the Master files are cleaned, you could put them in a zip, and get your mod manager to install them - Maybe at a future date you want to do a refresh of steam cache and it redownloads the masters which are not the same as the originals anymore (because you cleaned them), so then you would need to reclean them again. But beware, Bethesda have started redoing some masters due to Creation Club mods compatability, so make sure any redownloaded masters are not newer than your previously cleaned ones, because in that case you will need to reclean and rezip them again anyway.

 

You can go through the rest of your Load Order using Automatic cleaning of ITMs and UDRs on all your mods plugins. The sequence of cleaning mods plugins should be after you have your Load Order correct, masters are cleaned, then clean them with the last to load being the last to clean. Mod authors should have done them already, so most will probably not need cleaning. Also look out for any mod specific cleaning instructions in the mods description.

 

Prime example = The Unofficial Patches will not need any cleaning, they are already done, and any remaining ITMs in those plugins should be left alone because they do have a purpose .. ( its a very rare occasion when this is true ).



The xEdit Work In Progress Development topic is at the following link https://www.afkmods.com/index.php?/topic/3750-wipz-tes5edit/

 

Development project is at GitHub https://github.com/TES5Edit/TES5Edit

 

And newer versions of xEdit (3.2.7 +) have a link to Discord top right of xEdit window.

Share this post


Link to post
Share on other sites

Actually I posted the wiki info. Didn't think anyone was even paying attention :P

 

Your guide looks good. Shouldn't be a problem.

Share this post


Link to post
Share on other sites

Excellent .. will tidy it up a bit, and also link the images in my gallery here on AFK Mods instead of at Imageur.

 

Thank you for confirming. And with that confirmation .. I can confidently apply the same to my setup :D

Share this post


Link to post
Share on other sites

I recently modified Compressed flag in record header to be ignored for conflicts, so xEdit should clean more records now for all games if they were just compressed/uncompressed.

Share this post


Link to post
Share on other sites

Ah thanks Zilav - I have just this morning installed the latest svn of TES5Edit .. After doing the above screens  :facepalm:

 

No problem anyway, will redo everything just to make sure its accurate and as up to date as possible

( I have original un-touched official plugins in a BAIN ready for re-installs and recleans as necessary )

Edit : With latest svn - ITMs and UDRs tallied with previous results, updated screenshots for manual editing

Share this post


Link to post
Share on other sites

Curious question, because I've never read this before so obviously have never done it. Are the "Testing Cells" doing any harm if they aren't cleaned out?

Share this post


Link to post
Share on other sites

Personally I dont think so. If you were to have a fake esm from a n other mods coming before dawnguard.esm .. and for some bizarre reason it was referenced, and they had a normal esp following the esm's which depended on its fake esm changes ( maybe using one of these testing areas for placing a chest with custom armour in it .. seen that trick before a few times in Oblivion, not so much in Skyrim )

 

I think it would have to be pretty unusual circumstances for it to matter, but no point leaving them in either really as a potential hiccup. Same as ITMs.

 

Arthmoor may have a better answer with other considerations.

 

 

Edit : Request : This topic stickied ? - Although, on this site its hard to say which topics are more of a priority for being seen over others, a lot of the guides here could be stickied

Share this post


Link to post
Share on other sites

Curious question, because I've never read this before so obviously have never done it. Are the "Testing Cells" doing any harm if they aren't cleaned out?

No, they do no harm, but with the new Bash tag detection script they can cause a mod to get tag recommendations that don't fit.

Share this post


Link to post
Share on other sites

Guess this means I need to get off my lazy ass and add this to my Skyrim setup guide. Thanks for the find Arth, and for the writeup alt3rn1ty!

 

EDIT: Just ran a test and with the latest version of TES5Edit the automated cleaning catches the WICourierCell on its own now.

Share this post


Link to post
Share on other sites

Guess this means I need to get off my lazy ass and add this to my Skyrim setup guide. Thanks for the find Arth, and for the writeup alt3rn1ty!

 

EDIT: Just ran a test and with the latest version of TES5Edit the automated cleaning catches the WICourierCell on its own now.

 

3. For "CELL 00039F67: Wild edit. Remove this record. It's a testing cell." ..

 

.. Expand the records as in the following screenshot, and right click the blue circled record, and choose Remove

 

NOTE : If you have already automatically cleaned the Dawnguard.esm prior to doing this mini guide ( Right Click - Filter for cleaning / Right Click - Remove Identical to Master records / Right Click - Undelete and Disable references ) :

Then you will not be able to find the following record, auto cleaning also gets rid of this wild edit.

 

Its getting cleaned with any of the last three svn to my knowledge .. I wondered if it ought to be included if automatic cleaning will remove it anyway ( although I dont know if it is the first or second clean, but going forward everyone should be cleaning dawnguard twice anyway )

Share this post


Link to post
Share on other sites

Zilav just posted the version that deals with that. It's part of ignoring CELL records where the only change is the compressed flag.

 

The wiki data is accurate for what's available for download on Nexus.

Share this post


Link to post
Share on other sites

Thank you for this great guide, much appreciated !

 

I just came back to Skyrim after many months of absence and did a fresh installation. I am a bit rusty, it's a timely guide !

Share this post


Link to post
Share on other sites

Just found this page with regard to testing cells. I need to see which ones TESEdit cleans up on its own vs which ones would need to be manually cleaned.

Share this post


Link to post
Share on other sites

If you are referring to just the manual cleaning - See posts 6 / 7 / 8

 

If you mean all cleaning including automatic cleaning of the DLCs the reasons are discussed in the cleaning guide here ..

 

http://www.creationkit.com/TES5Edit_Cleaning_Guide_-_TES5Edit#Section_1:_Understanding_Mod_Cleaning

 

 

If you were to run the game with no mods at all, ITMs and UDRs would not be a problem ( thats why bethesda dont bother their arse about them so long as the game works and sells its out the door, but also they cannot use a community made tool like TES5Edit in their profession to fix these problems for modders before they release the DLCs )

 

If you are really not experiencing any problems on your setup then no you dont need to clean, but without cleaning .. how do you know you have not got ITMs in later loading mods upsetting any changes earlier loading mods want to win ?

 

UDRs would let you know there was a problem if another mod referenced them because they would crash your game - But it may just be a case of you have not been to the area where the deleted object is actually referenced, so its just waiting for your random adventure to take you there.

 

 

What you gain is :

 

No potential for CTDs because of deleted references

 

Your mods working properly because they are not being reverted to vanilla settings with a later loading mod which has Identical to master records

 

And yes this applies early to the DLCs too because as per the Unofficial patches you can have mods which are fake esm and potentially cause hiccups in that area of your load order.

Share this post


Link to post
Share on other sites

Now that TES5Edit 3.1.0 is released - Removed the WICourier edit ( which is cleaned by normal auto cleaning using the newer version of TES5Edit - so not necessary in this manual cleaning guide anymore ) - Previous screenshot 4 and instructions have become number 3 in the OP in place of the removed instructions.

Share this post


Link to post
Share on other sites

Slight update to the description wording, and redid the screenshots hopefully for more clarity and prevent any misunderstanding of exactly what should be edited.

Share this post


Link to post
Share on other sites

Thanks. Reloaded all my Skyrim and tried it myself.

 

Following the wording exactly for those cells, I'd have deleted the Skyrim.esm version. Need some verbiage to control-click on the Editor ID under Dawnguard. Then you have the correct line on the left.

Share this post


Link to post
Share on other sites

I dont follow ..

 

What part of the OP are you referring to ?, parts of the :

 

"Dawnguard.esm needs manual cleaning aswell as automatic cleaning"

 

Descriptive text for what needs to be manual edited in this esm ..

 

or Images, which show the Dawnguard.esm records being expanded ?

 

Or maybe the wiki linked which describes the Dawnguard.esm needing manual cleaning

 

 

I would like to know very much how this is being misinterpreted, so that I can make the guide better for all who may read it the same way you have.

 

No matter how I look at it I cannot see how anyone would think they were supposed to be editing Skyrim.esm which would be a very bad idea.

 

Do you mean the lines like ..

 

1. For "CELL 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm."

 

Because I can see maybe describing that as ..

 

1. Dawnguard.esm \ Cell \ Block 5 \ Sub-Block 3 \ 00016BCF: Remove XEZN subrecord referring to RiftenRatwayZone [ECZN:0009FBB9]. Otherwise it blocks the official fix in Update.esm.

 

But those are quoting the linked wiki entries ( thats why I precede them with For "quotation" .. do this ), and the image shows people what to do with the information given there

 

 

Maybe better for anyone who does not read any of the rest of the description ( to be expected I suppose but not really me at fault there ), and / or not looking at the images ( maybe reading in a foreign language via Google translate so the images in those cases would be worthless ) it also needs the wiki editing ??

 

 

Dont get me wrong, I want this to be as bullet proof as can be reasonably expected to be, no matter who reads it .. So I am anticipating your feedback, and not being sarcastic

 

 

Edit : In case Daydreamer does not come back - Anyone else seeing the same ?, is the OP ambiguous in any way that can be open to interpretation so much that you would end up editing the Skyrim.esm instead of Dawnguard.esm ?

 

Edit 2 : Note also that I have never used Ctrl Click in TES5Edit, so do not understand where you would want that done or even why

Share this post


Link to post
Share on other sites

Hi, thank you again alt3rn1ty for this great guide that I always read again before cleaning the master files.
 
I noticed today that cleaning Dragonborn.esm removed 4 more ITM :

[Removing "Identical to Master" records done]  Processed Records: 214386, Removed Records: 68, Elapsed Time: 00:00


I'm using the latest TES5Edit build (10 july).

Share this post


Link to post
Share on other sites

Thanks Elgar, I do have the latest WIP but have not got around to retrying cleaning with it ( too many machines needing Windows 10 bound and gagged )

 

Will have a look over the next week and see what needs amending in the OP - Will probably add a note too that results reflect the use of the latest work in progress, and not just the results achieved with the official release. Or something along those lines, will have a think on it. Really I should just stick with what the official release version can achieve, but if anyone reads the OP here they are probably using the WIP anyway :)

 

Edit : Amended the OP - Dragonborn.esm 68 ITMs

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Support us on Patreon!

×
×
  • Create New...