Jump to content
alt3rn1ty

Manual Cleaning Skyrim and Skyrim SE Master files

Recommended Posts

Largely undocumented? So I suppose the years worth of information stacked up on the CK wikis back to Oblivion and even on the Reddit sub don't count?

No, it honestly sounds to me like you're promoting an unsupported position based on absolutely nothing, that contradicts the collective knowledge of a community you don't seem to know that much about, let alone actual processes involved in it. That you have not personally seen these things happen over the years means nothing. Nothing at all. You don't get to just dismiss everything we all know because it doesn't meet your unrealistic academic standards or something.

And it absolutely will lead to Giskardism. You clearly have not been around long enough to know that's what ALWAYS happens when something like this comes up and the disinformation that comes from it is allowed to spread by someone who asserts knowledge authority they don't have. You've been given the evidence. The rational arguments in favor of this position have been available for 15 years. You simply haven't done the research, which IMO is incompatible with your claim of being intellectually rigorous.

I don't see any logical distinction between "don't clean master files" and "don't clean anything" because Bethesda's master files are quite simply giant mod files and nothing more than that. The exact same position applies to merging things too. "Don't merge the DLCs" is logically equivalent to "don't merge anything" because they should all be mergeable if the process behind that is sound.

Share this post


Link to post
Share on other sites

I already said some time ago that I never tell someone to clean or not to clean, it is always up to a user to decide. My only concern is that the cleaning routine itself in xEdit must work properly - don't remove what it should not, and remove what it should. As of right now there are no reported bugs of cleaning to cause any issues in Oblivion, Skyrim, Fallout 4 and Skyrim SE. All former known issues are fixed in the current available xEdit 3.2 version. There ARE for Fallout 3 and New Vegas however due to some strange Havok behaviour in interior cells, apparently Beth fixed that when they updated engine (and used Havok version) for Skyrim back in 2011.

Having said that, if cleaning is worth the effort? I have to agree with Arthmoor here and accumulated knowledge from commnunity. In 2012 when Hearthfire was released, people who used USKP or some Breezehome mods started to crash when entering that cell, which was caused by deleted navmesh modified in USKP. And reports were numerous, it was not some single user doing stupid things and claiming everything and everyone for no reason. As soon as offending navmesh override was removed in USKP, everything went back to normal. But at the same time I remember some mod author(s) on nexus forum, or reddit, or somewhere else saying they don't crash. They even want on deleting navmeshes and overriding them on purpose in other places trying to reproduce the issue and could not. Why it always crashes for someone and never for the other? Nobody knows, but the general accumulated community knowledge settled on the fact that it is bad, it crashes and must be dealt with.

Again, having said that, we can conclude that (apart from FO3/FNV) the cleaning process is ALWAYS beneficial. Maybe the chance for it to solve issues are pretty low and depends on the particular mods setup for each user and if custom mods actually change the same references as DLCs, but still why not do something that can potentially benefit you and at the same time never hurts. The only cons argument is "wasted" time spent on cleaning.

Share this post


Link to post
Share on other sites
4 minutes ago, zilav said:

I already said some time ago that I never tell someone to clean or not to clean, it is always up to a user to decide. My only concern is that the cleaning routine itself in xEdit must work properly - don't remove what it should not, and remove what it should. As of right now there are no reported bugs of cleaning to cause any issues in Oblivion, Skyrim, Fallout 4 and Skyrim SE. All former known issues are fixed in the current available xEdit 3.2 version. There ARE for Fallout 3 and New Vegas however due to some strange Havok behaviour in interior cells, apparently Beth fixed that when they updated engine (and used Havok version) for Skyrim back in 2011.

Having said that, if cleaning is worth the effort? I have to agree with Arthmoor here and accumulated knowledge from commnunity. In 2012 when Hearthfire was released, people who used USKP or some Breezehome mods started to crash when entering that cell, which was caused by deleted navmesh modified in USKP. And reports were numerous, it was not some single user doing stupid things and claiming everything and everyone for no reason. As soon as offending navmesh override was removed in USKP, everything went back to normal. But at the same time I remember some mod author(s) on nexus forum, or reddit, or somewhere else saying they don't crash. They even want on deleting navmeshes and overriding them on purpose in other places trying to reproduce the issue and could not. Why it always crashes for someone and never for the other? Nobody knows, but the general accumulated community knowledge settled on the fact that it is bad, it crashes and must be dealt with.

Again, having said that, we can conclude that (apart from FO3/FNV) the cleaning process is ALWAYS beneficial. Maybe the chance for it to solve issues are pretty low and depends on the particular mods setup for each users and if custom mods actually change the same references as DLCs, but still why not do something that can potentially benefit you and at the same time never hurts. The only cons argument is "wasted" time spent on cleaning.

Some good thoughts here and a nice explicit example, thanks.

The fact that it was specifically related to a Navmesh is interesting to me, that wasn't part of my tests.  I'll do some poking around and see what I can learn.

Still, it sounds like in that situation that the solution was to fix the USKP, not to having all users clean Hearthfires.esm, right?  So in some ways that is support for my position that cleaning the Bethesda Master Files is not something that should be promoted.  Because, at the end of the day, not everyone will do it (nor can we realistically expect everyone to do so), so any issues where a mod will cause a crash unless the Bethesda Master Files have been cleaned will ultimately need to be resolved in the mod itself, else the mod will become unusable to the majority of the userbase.

Share this post


Link to post
Share on other sites
11 minutes ago, Mator said:

Because, at the end of the day, not everyone will do it (nor can we realistically expect everyone to do so), so any issues where a mod will cause a crash unless the Bethesda Master Files have been cleaned will ultimately need to be resolved in the mod itself, else the mod will become unusable to the majority of the userbase.

Time proved that it is actually much easier to resolve issues on the user's side than on the author's. Users usually listen to others, follow guides and use BOSS/LOOT which give hints on cleaning, while some mod authors are stubborn persons doing things "their" way. Arthmoor already mentioned one. Also there is case of mods when author is not available for various reasons, and with the current established owner rights system fixing such mods if they have issues with DLCs is impossible. In the end it is easier to clean DLCs once and forget about it, than to expect perfectly made mods coming out from mod authors owning every DLC. So there are always pros to clean, and no cons apart from spent time.

Share this post


Link to post
Share on other sites
16 minutes ago, zilav said:

Time proved that it is actually much easier to resolve issues on the user's side than on the author's. Users usually listen to others, follow guides and use BOSS/LOOT which give hints on cleaning, while some mod authors are stubborn persons doing things "their" way. Arthmoor already mentioned one. Also there is case of mods when author is not available for various reasons, and with the current established owner rights system fixing such mods if they have issues with DLCs is impossible. In the end it is easier to clean DLCs once and forget about it, than to expect perfectly made mods coming out from mod authors owning every DLC. So there are always pros to clean, and no cons apart from spent time.

The easier path isn't always the right one.  It's totally possible for users to fix the mod plugins which reference records deleted in Bethesda Master Files, it's even possible to make a script to do that for them.  You may think the spent time is insignificant, but if we expect all users to clean the Bethesda Master Files that's a time investment of at least 15 minutes per user.  At 4 million users (the number of unique downloads on SkyUI) that is an investment of 1 million man hours, or 115 years.

I think the correct solution to this problem is either doing nothing, creating and distributing a patch plugin fixing UDRs and ITMs, or creating and distributing a script which fixes references to records deleted in the Bethesda Master Files.

Share this post


Link to post
Share on other sites
6 minutes ago, Mator said:

I think the correct solution to this problem is either doing nothing, creating and distributing a patch plugin fixing UDRs and ITMs, or creating and distributing a script which fixes references to records deleted in the Bethesda Master Files.

Users always cleaned DLCs in Oblivion and Skyrim, this has never been a problem. Can't understand why it suddenly changed right now. Have Bethesda released or announced anything recently? I don't follow modding news, maybe missed something.

Patching UDRs is impossible and you should know that. Patching ITMs will require extra esp slot. Distributing a script and asking users to run it is no different than asking them to clean. Doing nothing is also not a solution. People spend much more time on reading forums, watching cats, installing mods and playing games. I'm sure they can afford extra 15 minutes of their life if they are going to play a modded Bethesda game and want the best experience.

We have a working, time proved solution without any detrimental effects called cleaning mods. I don't see any reason to change it. Do they actually clean mods or not - I don't care personally. It is their game and their choice only.

PS: You might want to check this project http://www.nexusmods.com/fallout4/mods/22442/? and create a separate executable tool to make plugins out of jsons because you've already done a similar thing for Skyrim. Then it would be possible to load esp in CK and generate precombined meshes and previs data for settlement, along with LODs. And then upload on consoles and play without performance loss, opens up pretty revolutionary and awesome possibilities. I have a feeling that you got some extra free time and energy to spend on modding lately but poured them into wrong direction :)

Share this post


Link to post
Share on other sites

Regarding the navmesh issue, I didn't mention that because it's also not something that can be cleaned and restored to a working state by xEdit. So the 57 deleted navmeshes in Dawnguard alone are still there, still lurking, still waiting on someone to edit and end up causing problems with.

We were specifically discussing cleaning mods of ITMs and UDRs - which is why I kept to those and provided you specific ways to check for it. Ways everyone in the past has tried and verified to cause issues. So the only sane solution is still to clean the DLC files and eliminate them as causes for disruption.

Share this post


Link to post
Share on other sites

Yep, cleaning the masters is certainly the recommended cause of action, there's no hard right or left to it for anyone's particular config.

Guess common pains for a gamer include Steam file verify when the cleans are replaced by dirties, or the occasion of Beth releasing frequent updates during play, when if the updates are configured to run in the background, it may be missed, and then after manual clean at a later stage they might  be asking the question: When could this all be automated by some kind of listener anyway? Perhaps a plugin script using the XEdit API for one of the mod managers a possibility?

Share this post


Link to post
Share on other sites

Well now that Zilav and Arthmoor have discussed undocumented reasons for cleaning I am convinced - The guides stay put.

I was playing it safe with my description "Why Clean the Master Files" .. But knew there were other good reasons for cleaning which have appeared many times in forum troubleshooting topics, I am just not experienced and technical enough to articulate any further reasons.

Now that our vastly experienced and knowledgeable community members, with many years handling a wide diversity of problems thrown at them from the community have weighed in, I have no doubt that the guide is not misleading anyone into placebo and unecessary actions, and is still a recommended and needed practice.

For anyone further curious about the description and in need of more reason to clean .. I will just link back to this conversation commencing with this post.

 

Well thats most of the guide pages descriptions amended, with the exception of Nexus guides and Nexus Forum guides which are down for the planned maintenance at the moment .. Will do those later.

But it now just includes a couple more lines in the Why Clean Master Files section ..

"

For further explanations of why it is still recommended to clean the games masters .. Read on from this post,

Zilav and Arthmoor, most valued technical and vastly experienced modding authors, weigh in on the subject.

"

I think that should do for anyone needing more detail, maybe Mator does not entirely trust the word of more experienced mod authors when it comes to modding advice but I certainly do, kept me straight for many years with their knowledge, and I greatly appreciate they took time out from the many things they do for this community to participate here. Thank you.

Share this post


Link to post
Share on other sites
10 hours ago, zilav said:

Users always cleaned DLCs in Oblivion and Skyrim, this has never been a problem. Can't understand why it suddenly changed right now. Have Bethesda released or announced anything recently? I don't follow modding news, maybe missed something.

Patching UDRs is impossible and you should know that. Patching ITMs will require extra esp slot. Distributing a script and asking users to run it is no different than asking them to clean. Doing nothing is also not a solution. People spend much more time on reading forums, watching cats, installing mods and playing games. I'm sure they can afford extra 15 minutes of their life if they are going to play a modded Bethesda game and want the best experience.

We have a working, time proved solution without any detrimental effects called cleaning mods. I don't see any reason to change it. Do they actually clean mods or not - I don't care personally. It is their game and their choice only.

PS: You might want to check this project http://www.nexusmods.com/fallout4/mods/22442/? and create a separate executable tool to make plugins out of jsons because you've already done a similar thing for Skyrim. Then it would be possible to load esp in CK and generate precombined meshes and previs data for settlement, along with LODs. And then upload on consoles and play without performance loss, opens up pretty revolutionary and awesome possibilities. I have a feeling that you got some extra free time and energy to spend on modding lately but poured them into wrong direction :)

Nothing changed, it just came to my attention, I thought through it, and I reasoned it wasn't a good practice.

Why is patching UDRs impossible?  I JUST did it and it worked totally fine.  An extra ESP slot is meaningless in this day and age where merging is easy and available.  Distributing a script and asking users to run it is completely different from asking users to clean because it requires less actions by the users and does not ultimately edit the Bethesda Master Files.  I'm also sure they can "afford it", but that doesn't mean it's insignificant.

Again, I'm fine with people cleaning mods, I just don't think it's right to clean the Bethesda Master Files.  My working, time proved solution has been to not clean them, and I know many other users haven't done so either.  I'm aware it's a user's choice, and I also don't care what an individual user chooses.  My concern is with what practices are promoted by the community and the consequences of promoting those practices.

That project is very cool, thanks for the link.  I had a friend who really wanted something like that for awhile, I'm glad to see someone made it (F4SE must have advanced to have the scripting functions needed, I had been asking Expired for that for awhile).  I don't actually have much extra free time at all, Zilav, I'm pretty heavily booked right now working on Mod Picker.

Share this post


Link to post
Share on other sites
10 hours ago, Arthmoor said:

Regarding the navmesh issue, I didn't mention that because it's also not something that can be cleaned and restored to a working state by xEdit. So the 57 deleted navmeshes in Dawnguard alone are still there, still lurking, still waiting on someone to edit and end up causing problems with.

We were specifically discussing cleaning mods of ITMs and UDRs - which is why I kept to those and provided you specific ways to check for it. Ways everyone in the past has tried and verified to cause issues. So the only sane solution is still to clean the DLC files and eliminate them as causes for disruption.

Can I not do fix deleted navmeshes in xEdit as well as the CK?  Frankly speaking, I've never cleaned a Deleted NavMesh before.  I know that xEdit's automatic cleaning option for UDRs doesn't cover them, but the whole "reduce to one triangle and move under the map" step should be manually doable in xEdit.

Share this post


Link to post
Share on other sites

You can' fix a deleted navmesh in the CK since the form will no longer be visible in the editor. That's what this tutorial is for:

Even if one is as careful as I am about it, this can still easily happen. I just had to do this process the other night for my next town expansion.

Share this post


Link to post
Share on other sites
59 minutes ago, Arthmoor said:

You can' fix a deleted navmesh in the CK since the form will no longer be visible in the editor. That's what this tutorial is for:

Even if one is as careful as I am about it, this can still easily happen. I just had to do this process the other night for my next town expansion.

Ah d'oh, of course.  Your phrasing made me forget for a moment that deleted records don't appear in the CK.  X_X

Share this post


Link to post
Share on other sites

@Arthmoor I linked that Fixing Navmesh Deletion guide of yours in this Cleaning guide a few days ago, Hana gave me a poke that it would be a good idea for Mod Authors initially reading this guide if the Automatic cleaning of mods gave them warnings about Navmesh deletions, then they should be pointed towards that guide to solve those problems.

So your guide has now been given a bit more of a wider audience via this guide being plastered everywhere.

Share this post


Link to post
Share on other sites

Thanks alt, more visibility is always good. xEdit will spit out a warning for deleted navmeshes when the UDR cleaning is run, so people will be made aware of it that way if they're paying attention to the results.

Share this post


Link to post
Share on other sites

Alt3rn1ty, thanks for the guide I am sure many people find it useful and helpful. Just do what you want and don't worry about what other people think.

To everyone else:

People are entitled to their opinion. I don't have the time or patience to deal with others confirmation biases[1]. Besides if it gets out of hand what are my options? Argue with people about it? What would I gain by doing that?  For Oblivion there used to be a page on manual cleaning and a list of mods not to clean.  There is a TES5Edit page for that but I don't see anyone adding to it. If it's needed for other games then maybe someone could start a wiki page.

Back in 2012 I was talking to Elminster and this was his response to one of my questions about dealing with user comments, "As for how I dealt with assholes and idiots, I simply ignored them and went ahead working on the code. To quote the title of Richard Feynman’s second autobiography: “What do *you* care what other people think?” There have probably been 100’s of PMs and comments that I simply haven’t bothered to answer since I started work on TES4Edit."  Also I like this quote from TheModernStoryteller that appeared in a recent nexus Sunday discussion, "Never argue with an idiot. They will only bring you down to their level and beat you with experience."

I use mods for Skyrim whatever seems interesting.  I have even found mods that CTD my game. One was by Elianora back when she just started making house mods.  I saved her mod with the CK, cleaned the plugin, looked for deleted vanilla Navmeshes and resolved them. By resolving the deleted Navmeshes the plugin didn't CTD anymore. We even became good friends. I bet she still laughs about the personal video I made for her on how I resolve the deleted Navmeshes because I coughed really loud and didn't edit it out.  I remember her saying, "I'm paying attention now!" on Steam.

I was cleaning Sjel Blad Castle by soulbladex because for some reason I noticed certain quests were not working correctly. He was just looking at things in the CK and ended up with some wild edits and ITMs.  He is an incredible mod author and just pointing out those things and helping with his mod has made that one of my favorite castles of all time. Elvenwood and Helgen Reborn were not working together and I ended up doing some cleaning and Navmesh work to resolve a CTD so they would work together. Both authors used my submissions as new versions for their mod.

I think I would much rather discuss with my daughter about how to play the guitar or how well she did in choir. I have attended all of her choir events. She has so much energy when she sings. She doesn't just stand there with her mouth open trying her best to sing as her teacher has instructed, she smiles and moves around. She performs not just sings. There is a school nearby that just finished being built. She wants to try and work there as she is interested in pursuing a degree in Early Childhood development.

So debate all you want whether or not you should clean plugins. I respect everyone that has commented so far.  However, I really don't give a crap.  I know why I clean plugins and master files and why it works. Come to think of it, I have never had a time when it didn't work.  I suppose if I did I would restore the original file and manually clean it or not clean it at all.

[1]: Confirmation bias, also called confirmatory bias or myside bias, is the tendency to search for, interpret, favor, and recall information in a way that confirms one's preexisting beliefs or hypotheses, while giving disproportionately less consideration to alternative possibilities.

Share this post


Link to post
Share on other sites
10 hours ago, Sharlikran said:

For oblivion there used to be a page on manual cleaning and a list of mods not to clean.

Yep for Oblivion Mods Thomas Kaira put quite a bit of effort into making lists ..

http://forums.bethsoft.com/topic/1200914-list-tes4edit-cleaning-list/?&hl=tes4edit

 

And I think all of that was also included on the old Construction Set Wiki by Brumbek

( Warning : Repetitive Strain Injury possible due to scrolling ) ..

http://cs.elderscrolls.com/index.php?title=TES4Edit_Cleaning_Guide

 

And further Unclean mods were then all submitted to the old BOSS masterlist thereafter for that game

To my knowledge there has never been any similar community effort put into Skyrim / Fallout by new bloods, and new mod authors.

Nor has there been any great uptake contributing to the LOOT masterlist with Load After / Prioritisation metadata which would make LOOT just as comprehensive as BOSS used to be in its detailled collective knowledge, which is a shame because its the ideal tool for the job and most people just use it as it comes for its automatic Load Ordering when it is capable of being refined so much more just from community feedback if they would just spend a bit of effort building upon what Wrinkly has provided.

I dont think the community in general gives a rats arris as much as the Morrowind and Oblivion communities did to help each other help themselves.

Share this post


Link to post
Share on other sites

Oh yes I know Sharlikran, you have done a lot toward the communities, your documentation is linked all over the communities and is very good

My last post was just talking generally to anyone listening really, and not meant to be at yourself.

Share this post


Link to post
Share on other sites

I wasn't upset with it or thinking it was pointed at me.  I honestly wasn't sure if it was known or not. I suppose you and most of the people here know but you never know really.

As for documentation, I have to give credit where credit is due. I haven't done anything, so no credit to me.  Those pages are altered copies of the oblivion pages and the documentation is Miax and JustinOther training manual, just in text format. I added some images but not all of them. I haven't updated the training manual ever, I'd like to but usually don't have the time or Acrobat and I can't get LaTeX working for pStyl3's version of the training manual.

Share this post


Link to post
Share on other sites

Had just finish this cleaning guide and notice that the version of TES5Edit I have is newer than in this guide.  The latest version is 3.2.1 and not 3.2.0.  Thanks for maintaining it. :)

Share this post


Link to post
Share on other sites
23 minutes ago, Leonardo said:

Had just finish this cleaning guide and notice that the version of TES5Edit I have is newer than in this guide.  The latest version is 3.2.1 and not 3.2.0.  Thanks for maintaining it. :)

The official releases on Nexus are v 3.2 for TES5Edit and SSEEdit (as linked in the first post in this topic) - And the guide is still correct using those versions.

Development / experimental may be 3.2.1 on AFK Mods - But I dont think there are any more ITMs / UDRs to concern this guide, so until the next official upload to Nexus the version number can remain as it is.

The experimental version is the one I am using aswell, but I see no need to update the guide yet.

Share this post


Link to post
Share on other sites
11 hours ago, Leonardo said:

I see.  Anyway, I downloaded this TES5Edit_f382a67 archive and I think that's what you also use.  Correct?

I dont know for certain, I'm not at home, but I would imagine Zilav would not produce different downloads with the same version number of 3.2.1, that would be confusing for downloaders, and would not be good for any feedback he wanted from any particular version ..

".. wait, are you using 3.2.1 with CRC of blah or 3.2.1 with CRC of bleh", surely he would just make a 3.2.2 instead :).

Share this post


Link to post
Share on other sites

Yep, it looks like I am using 3.2.1 of both <TES5/SSE>Edit and <TES5/SSE>LODGen, at least what the name bar says.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Support us on Patreon!

Patreon
×
×
  • Create New...