Jump to content

Manual Cleaning Skyrim and Skyrim SE Master files


alt3rn1ty

Recommended Posts

@Alt: Thanks. As per, SSEEdit is yet another rename of the same Xedit executable.

Wonder if we can get a script going that does 'em all in one hit.

Link to comment
Share on other sites

Just use shortcuts with command line? they remain identical between versions, and all you need to do is rename the exe to Xedit each new install

 

e.g.

​"C:\path\XEdit.exe -tes5 -edit" in the properties of the shortcut

Link to comment
Share on other sites

For anyone wondering about cleaning Skyrim SE ..

 

Zilav has already done a SSE version of xEdit - http://www.nexusmods.com/skyrimspecialedition/mods/164/?

( Apparently there was not much in terms of records to update to make it compatible )

 

So I have done the usual and cleaned the masters for SE .. And its pretty much the same as for Skyrim Original

 

 

I will make another guide specifically over the coming weeks among other things I will be up to, but for now if you follow this guide and apply it to SE ( just dont pay too close attention to getting the same ITMs and UDRs ) you will be fine.

 

Dawnguard still needs cleaned a couple of times, and the manual cleaning for that plugin remains the same too.

 

Amongst all the plugins you will probably notice Bethesda has managed to introduce a few more ITMs and UDRs along the way.

 

 

Bethesda, start using xEdit ffs :P, you know it makes sense :).

 

If anyone followed this and has started cleaning masters of Skyrim SE ..

 

.. I would take the advice of someone who really knows instead and wait for word from the man

 

Think I jumped the gun a bit on that post.

 

 

But since then there has been an update to SSEEdit which solved a concern after I made that post.

 

At time of writing this post, Zilav is not warning against cleaning with the latest version of SSEEdit ( see post #1409 in the same topic )

 

Just keep a mint backup copy of the original masters for re-cleaning if any more surprises are found with Skyrim SE in the near future

( and refresh your mint backup with any official updates which are imminent )

 

 

For now I am holding off doing another manual cleaning master files topic for SE, until we are 100% certain SSEEdit has it all nailed.

Link to comment
Share on other sites

The problem is - I don't know myself. Cleaning is a common knowledge gathered by people. If 100 players cleaned their files and only a few of them reported to have problems after that, then it is probably safe. The more people try to clean and play, the more data we have to make a decision. Bethesda'e engine is too flaky and unreliable to know something beforehand.

That's why I never say "yes" or "no".

Never "Yes" because I don't want to be responsible for breaking someone's game and fun.

Never "No" because someone would still need to test that sooner or later.

 

Technically there are no critical problems left in the current SSEEdit build. Some records like SPGD still need investigation, but that's it.

Link to comment
Share on other sites

Thanks Zilav, I personally am still cleaning them and keeping up with updates and re-cleaning as necessary

Im happy to be testing stuff like this because no doubt I will need to start a new game or two as Bethesda's Official patches come along, and there is nothing that cant be replaced with a refresh game cache

I think the ideal time for me to make a new guide will be when I am confident there have been no changes to results between versions, and Bethesda have stopped updating their files, but I will probably be bored before then and make a new topic anyway which will then no doubt get amended as required :)

Link to comment
Share on other sites

My opinion is: if the record definition are correct and the "behind the scene" editing is also correct, cleaning should be safe. If people report cleaning does affect their game then:

For anything before FO4:

  My first reaction is: Which other mod are they using that interfere with what was cleaning?

  If no mod can be identified, as in the issue can be reproduced WITHOUT any other mods, then the most likely issue is a bug in xEdit form definitions.

 

Unfortunately there are people who constantly claim that cleaning is not safe but they never provide a simple to test set of files to track the issue.

 

WIth FO4 ITM obviously helps to get around the issues brought by previsualisation data, so we know "some ITM may be required". I still hope to track down the source in the engine, but I haven't enough to go on at the moment.

Link to comment
Share on other sites

  • 4 weeks later...

I think you need to update the information about what result the mod cleaning has for each DLC and I got these results when cleaning the official DLC in the latest SSEEdit 3.1.3 build (cc08ca7).

 

Update.esm

[Removing "Identical to Master" records done]  Processed Records: 17216, Removed Records: 233, Elapsed Time: 00:00

[undeleting and Disabling References done]  Processed Records: 16983, Undeleted Records: 91, Elapsed Time: 00:00

 

Dawnguard.esm - 1st mod cleaning

[Removing "Identical to Master" records done]  Processed Records: 98676, Removed Records: 628, Elapsed Time: 00:00

[undeleting and Disabling References done]  Processed Records: 98048, Undeleted Records: 82, Elapsed Time: 00:02
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>

 

Dawnguard.esm - 2nd mod cleaning

[Removing "Identical to Master" records done]  Processed Records: 98047, Removed Records: 6, Elapsed Time: 00:00

[undeleting and Disabling References done]  Processed Records: 98041, Undeleted Records: 0, Elapsed Time: 00:02
<Warning: Plugin contains 57 deleted NavMeshes which can not be undeleted>

 

Hearthfires.esm

[Removing "Identical to Master" records done]  Processed Records: 17830, Removed Records: 178, Elapsed Time: 00:00

[undeleting and Disabling References done]  Processed Records: 17652, Undeleted Records: 11, Elapsed Time: 00:00
<Warning: Plugin contains 5 deleted NavMeshes which can not be undeleted>

 

Dragonborn.esm

[Removing "Identical to Master" records done]  Processed Records: 214498, Removed Records: 69, Elapsed Time: 00:00

[undeleting and Disabling References done]  Processed Records: 214429, Undeleted Records: 8, Elapsed Time: 00:03
<Warning: Plugin contains 1 deleted NavMeshes which can not be undeleted>

 

 

 

Also, this guide should be pinned due for being important to a lot of people.

Link to comment
Share on other sites

Leo .. I wish you would think before posting

 

This guide is for Skyrim ( For people still just have the original none 64 bit variety )

 

 

On the previous page if you had been following the conversation you may have noted that I intend to make a new guide for Skyrim SE at some point ..

 

 

.. Or, the new guide may take the form of a new mod page with a heap of similar information in the Description on Nexus ( if I get certain permission - You dig ? )

And have an associated Skyrim SE guide here on AFK Mods .. Edit : Could also have a download here too if its given the green light

Link to comment
Share on other sites

  • 1 month later...
On 11/26/2016 at 10:41 PM, Leonardo said:

 

 

Edit : Included new results in the OP now it is a dual purpose guide .. See following posts

Link to comment
Share on other sites

Just downloaded and re-installed Old Skyrim - updated the first post to reflect correct CRCs and MD5 Checksums, using the newer TES5Edit version 3.2

( There were different results on the auto cleaning and manual cleaning of the Dawnguard.esm with the new TES5Edit )

Fingers crossed I can uninstall Oldrim for good this time :)

Link to comment
Share on other sites

Link to comment
Share on other sites

@Leonardo

See the last link in the previous post - There is a file on Nexus now, which is just a folder template to make the BAIN

 

Still needs DIY and put the Original and Cleaned masters into the BAIN yourself ..

But until Bethesda give permission its all I can do is to show people how. :)

Link to comment
Share on other sites

  • 2 weeks later...

@Arthmoor @zilav @hlp @Sharlikran and @anyone who wishes to give two pence worth :

I was contacted privately by @Mator, I think to dissuade me from having these guides up ( I havent seen mention of just changing certain parts ).

 

Apparently someone on Reddit thought the Guide was a good idea ..

.. And Mator is currently laying into every post made with counter arguments to gather support from the Reddit community and downplay the practice of cleaning Masters

https://www.reddit.com/r/skyrimmods/comments/5xh5b6/guide_manual_cleaning_master_esms_for_skyrim_and/

 

Does he have good points, is cleaning ITMs and UDRs from the Masters really detrimental to modding ( see a later post there where he mentions this is so .. though no details have been given ) ?

And if so .. Should I remove part of the guide if not all of it .. or, all of it ?

Personally I see no harm in cleaning, it certainly does not cause problems in my experience.

So I call upon the community to give a good opinion on whether or not I should keep this guide up / change it in some way / or get them brought down.

 

See the conversation linked above on Reddit for Mators arguments against the guide and particularly against cleaning Masters, before giving a response.

I am not a member at Reddit so cannot participate there ( and after seeing the kind of people giving death threats to mod authors during the pay wall episode and the typical cesspit of awful comments you see there quite often, I do not wish to join either ) .. Hence why I am consulting the more knowledgeable people I know and trust here for good opinions.

Its just a guide and game anyway so this is not any kind of serious matter, if it has to go for good reason then down it comes.

 

The floor is yours .. Cast your own honest summary appraisals for or against parts or whole. 

Link to comment
Share on other sites

I commented on the Reddit thread. Mator makes good points, but he also mentioned in the thread that there is nothing WRONG with cleaning masters. He just doesn't feel it is necessary, and that it is a waste of time.

I honestly don't know enough either way...IMO there is no real downside to cleaning masters (besides a possible waste of time), and your guide is the best on the web. Unless it can be proven to be detrimental, which Mator even says it is not, then please DO NOT remove your guides. The info that is particularly useful, IMO (that no other guide I have seen mentions) is the manual removal of the dirty edits.

Link to comment
Share on other sites

I would suggest ignoring him because he's given absolutely no evidence based argument that cleaning the DLCs is harmful, and done nothing to counter the fact that cleaning them DOES have benefits.

The most obvious benefit is that if someone is not using the unofficial patch, Dawnguard has dirty edits that break house upgrades for Hearthfire. Failing to run the ITM removal will result in people complaining they can't buy Breezehome upgrades and that the steward is stealing their money. So there's no truth to the claim that ITMs in official DLC are harmless.

This also applies with SSE btw, since those dirty edits remained when Bethesda updated. So that problem WILL still exist for anyone who doesn't clean them and isn't using the USSEP.

UDRs are well known to cause CTDs whether the references are in scripts or not so that advice clearly still holds.

His claim to know more about esp related errors than anyone else is frankly laughable. The only thing he might have a valid point on is manual editing, but he bases that on "official DLC should never be edited", which is kind of a silly thing to argue if you're seriously trying to claim ITM/UDR remove isn't also editing.

And I'm not about to get into it with him on Reddit since he all too frequently just hurls insults at anyone who disagrees with him. Bottom line is, there's legit reasons to clean the DLCs, he's provided nothing to prove otherwise.

Link to comment
Share on other sites

22 minutes ago, Arthmoor said:

The most obvious benefit is that if someone is not using the unofficial patch, Dawnguard has dirty edits that break house upgrades for Hearthfire. Failing to run the ITM removal will result in people complaining they can't buy Breezehome upgrades and that the steward is stealing their money. So there's no truth to the claim that ITMs in official DLC are harmless.

UDRs are well known to cause CTDs whether the references are in scripts or not so that advice clearly still holds.

So they should be using the unofficial patch or another plugin which fixes the issue.  Simply put, I don't think cleaning the Bethesda ESMs is the correct solution to that problem.  Under what circumstances would a mod author reference a deleted reference?  I can't imagine it happening unless the mod author is doing so knowingly (intentionally causing a crash) OR the plugin is so old the reference was since marked as deleted in Update.esm or a DLC.  The chances of this are miniscule, and if this occurs the problem ISN'T the Bethesda Master Files, it's the mod referencing something it shouldn't be (the mod plugin can undelete it if the reference is absolutely needed).

 

22 minutes ago, Arthmoor said:

I would suggest ignoring him because he's given absolutely no evidence based argument that cleaning the DLCs is harmful, and done nothing to counter the fact that cleaning them DOES have benefits.

I think I was actually very specific about the potential issues with cleaning these files.  Here's a few quotes:

Quote
  • Con: The time/effort required to clean the records.
  • Con: If you ever re-validate the game files through steam, you will have to re-clean the plugins.
  • Con: Your ESMs are now different from everyone else's. For the sake of simplicity everyone should be working with the same Bethesda Master Files.
Quote

If you're a mod author cleaning may lead you to reference an undeleted REFR in a script which would cause a CTD for any user who didn't clean the Bethesda ESMs. If you're a mod author and are making a plugin that needs to be loaded between Bethesda ESMs cleaning ITMs will make it so you would not know when a change would be overwritten by an ITM.

I don't think it's a worthwhile tradeoff.  Users can of course make their own decisions, but my main concern is that cleaning the Bethesda Master Files becomes an expectation and users start attributing cleaning them to fixing problems.  If there's an issue with a mod plugin that gets resolved by cleaning the Bethesda Master Files, the problem is in the mod plugin, not the Bethesda Master Files.  To me it almost seems like peddling snake-oil to recommend cleaning them.


-Mator

EDIT: Also, to be clear, I'm not trying to force you to take the guide down or anything.  Whether or not you make it available is entirely your prerogative.  I'm just trying to keep people informed and deal with misinformation prior to it being spread.  Whether or not the community resolves cleaning masters is a good/acceptable idea is important to me because I'm currently working on a utility to aid with cleaning plugins.  My current thought is that I'll have Bethesda Master Files skipped by default but allow the user to clean them with a disclaimer alerting them that it's unnecessary.

  • Like 1
Link to comment
Share on other sites

We have literally 15 years of precedent on the issue of cleaning official game master files from Morrowind onward. I will grant that the ITMs aren't as big of a problem, even with considering the Breezehome issue in Dawnguard. It's more the UDRs. They could easily be referenced by something a DLC deletes that's valid in the Skyrim.esm file. You only find out once things start breaking.

As I'm sure you know, there are patch-ophobics out there who flatly refuse to use USLEEP/USSEP or any other mod geared toward fixing bugs "because they aren't official". Those people will benefit on PC from cleaning the DLC files because it will in fact fix some things, and will reduce the CTD footprint of anything in a mod referencing an object the DLCs deleted. Yes, this is definitely still an issue in SSE too. Console users are SOL if they refuse to use these things, and I can assure you the crash reports are numerous from people on those platforms whether they're using USSEP or not, because there's lots of badly made mods running around out there and many were improperly ported on top of having loads of dirty edits.

Time and effort is an argument for the lazy btw. It takes less than 15 minutes to do, and it's ludicrously easy to then keep the cleaned copies in a backup location in case they need to revalidate files. The reason we all push for cleaning is because it's known to resolve issues - simple as that.

Link to comment
Share on other sites

17 minutes ago, Arthmoor said:

We have literally 15 years of precedent on the issue of cleaning official game master files from Morrowind onward. I will grant that the ITMs aren't as big of a problem, even with considering the Breezehome issue in Dawnguard. It's more the UDRs. They could easily be referenced by something a DLC deletes that's valid in the Skyrim.esm file. You only find out once things start breaking.

So, when a mod references something that's been deleted in a DLC, you consider that the fault of the DLC and not the mod?  That's where I disagree with you.

By my perspective a mod which references something that gets deleted in an official Bethesda Master File is broken.  The Bethesda Master Files are the basis for all of our mods.  I'm not saying they're infallible but that if a mod causes a crash because it didn't properly take the Bethesda Master Files into account that's the mod's fault, and the problem should be resolved in the mod, NOT the Bethesda Master Files.

EDIT: You know what would be okay with me?  A patch plugin which undeletes records from all the Bethesda Master Files.  That would avoid all the "cons" I outlined above and would fix any issues created by stupid mod authors who reference REFRs that get deleted in the DLC ESMs in their plugins.

Link to comment
Share on other sites

If the mod predates the DLC, then what? Still the modder's fault? No. The error is in the DLC deleting the object improperly. It may not cause any problems with other DLCs, but that doesn't make the DLC blameless for causing it.

Even so, to this day, for LE there are still mods out there that don't include every DLC as their masters. In fact I'd say that's still a small minority of mods that explicitly include them. If the reference is totally valid in Skyrim.esm or Update.esm but a DLC deletes it, and the mod author doesn't even have that DLC, how would they ever know?

This is precisely the same logic we use to justify cleaning UDRs from every mod, not just from the DLCs. If the DLCs are not clean, then the state of those files should be considered unstable, and the mod author is then justified in asking if the user cleaned them.

As far as your edit, that's not possible. It would induce precisely the problem cleaning the UDRs is mean to avoid, because that patch esp is now the one responsible for referencing deleted data in a second record. People have tried this approach before and found it to cause exactly the kind of CTDs cleaning prevents.

Link to comment
Share on other sites

2 hours ago, Arthmoor said:

If the mod predates the DLC, then what? Still the modder's fault? No. The error is in the DLC deleting the object improperly. It may not cause any problems with other DLCs, but that doesn't make the DLC blameless for causing it.

Yes.  The mod is incompatible with the DLC because the modder did something that causes an error.

 

2 hours ago, Arthmoor said:

Even so, to this day, for LE there are still mods out there that don't include every DLC as their masters. In fact I'd say that's still a small minority of mods that explicitly include them. If the reference is totally valid in Skyrim.esm or Update.esm but a DLC deletes it, and the mod author doesn't even have that DLC, how would they ever know?

A plugin doesn't need to include the DLC in its masters (and shouldn't unless it explicitly references/requires something from the DLCs) in order to be valid and compatible with them.  If the mod author has the DLC then they just need to open their plugin in TES5Edit with the DLC to make sure the DLC didn't delete a reference used in their mod.  If the mod author doesn't have the DLC they should either get the DLC or get someone else's help to make their mod compatible with the DLC.

 

2 hours ago, Arthmoor said:

As far as your edit, that's not possible. It would induce precisely the problem cleaning the UDRs is mean to avoid, because that patch esp is now the one responsible for referencing deleted data in a second record. People have tried this approach before and found it to cause exactly the kind of CTDs cleaning prevents.

This sounded like bullshit to me, so I made a plugin which undeletes deleted references from all the DLC master files, started a new game with Random Alternate Start, and coc'd to the cells where I had undeleted things in my patch plugin.  I thoroughly tested TolvaldsCave02 and Mzulft04z, and had absolutely no CTDs.

An override record doesn't "reference" the deleted record, it replaces it.

Link to comment
Share on other sites

It may have sounded like bullshit, but your test is also inconclusive. I've done tests on it several times. Sometimes it generates CTDs, sometimes it will simply hang, sometimes the references show up in the 0,0,0 cell west of Whiterun at the giant camp, about 14,000 units up in the air.

You seem to just be too quick to dismiss the fact that there's 15 years of knowledge built up around this stuff and that NOTHING about how it all behaves has changed with Skyrim at all. You're basing your entire argument on nothing but not wanting some people to spend a little time on simple cleanup that has no downside and would clear up several potential pitfalls. Your opposition to this really doesn't make a lot of sense.

 I have a suggestion for you since this appears to be opposition born of the desire to not touch Bethesda's official files. Write your tool so it won't. Let people do the cleaning the way it's always been done. Then you won't be the one they come to should something ever go wrong (hint: it won't) and you can direct them elsewhere. It serves no purpose to attack the process just because people use it for dirty DLC, because the logic then follows "why clean anything" and people WILL ask and you'll end up with Giskards everywhere.

Link to comment
Share on other sites

1 hour ago, Arthmoor said:

It may have sounded like bullshit, but your test is also inconclusive. I've done tests on it several times. Sometimes it generates CTDs, sometimes it will simply hang, sometimes the references show up in the 0,0,0 cell west of Whiterun at the giant camp, about 14,000 units up in the air.

Then provide me with a test case.  I can't verify your findings without any specifics.  I'll take a look at the giant camp on my game, where should I look next if I don't find anything there?

EDIT: Went to the Giant's Camp outside of Whiterun.  Saw nothing unusual.  Went ~14,000 units up in the air above it, still saw nothing unusual.  No crashes either.

 

1 hour ago, Arthmoor said:

You seem to just be too quick to dismiss the fact that there's 15 years of knowledge built up around this stuff and that NOTHING about how it all behaves has changed with Skyrim at all. You're basing your entire argument on nothing but not wanting some people to spend a little time on simple cleanup that has no downside and would clear up several potential pitfalls. Your opposition to this really doesn't make a lot of sense.

This "knowledge" you refer to is largely undocumented.  So yeah, I'm dismissive of knowledge which is lacks evidence.  I don't believe things just because people say them - intellectual rigor is important.

My opposition may not make a lot of sense to you because I have different values.  I put a lot of value on the time invested by users and on the complexities associated with heavy modding.  By how I weigh the situation it seems that cleaning them is more costly than anything else.  I have never cleaned the Bethesda Master Files, and hundreds of thousands of other mod users haven't either.  What I'm trying to establish is whether or not we should clean them.  So far based on my understanding of plugin errors it's not worth it.  Unless someone convinces me otherwise that will continue to be my position, and I will promote that position in the community as supported by the evidence/reasoning I have provided.

If you want to change my position you should provide explicit examples supporting your position.  Thus far it's mostly been generalizations with the exception of the dirty edit, which I replied to.

 

1 hour ago, Arthmoor said:

I have a suggestion for you since this appears to be opposition born of the desire to not touch Bethesda's official files. Write your tool so it won't. Let people do the cleaning the way it's always been done. Then you won't be the one they come to should something ever go wrong (hint: it won't) and you can direct them elsewhere. It serves no purpose to attack the process just because people use it for dirty DLC, because the logic then follows "why clean anything" and people WILL ask and you'll end up with Giskards everywhere.

It's more than that, it's about what I should tell people.  It's in everyone's best interest to have a well-informed community, which means providing advice that is backed up by evidence and rational arguments.  The evidence and rational in support of cleaning Bethesda Master Files is paltry, and as such I recommend people not do so.

The notion that telling people not to clean the DLC leads to people thinking "why clean anything" and getting Giskards everywhere is a slippery slope fallacy.  It's completely ludicrous to think that way.  By that logic, my saying "don't merge Bethesda Master Files" should lead to people asking "why merge anything?".  (clearly that's not the case.)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...