Jump to content

Unofficial Patch ESM Conversion


Arthmoor

Recommended Posts

Now that this kalpa for the USKP is complete (for you non-lorehound types, kalpa = cycle) it's probably time we make public our plans for the next one.

Due to an increasing number of odd engine related bugs it's been decided among the team that we will need to convert the USKP (and therefore all the DLC patches too) into what's called a "False ESM". That is, and ESP file which is flagged as an ESM by way of changing the flag in TES5Edit.

So why would we want this? There are a few classes of bugs this has already been shown to help with in internal testing:
 

  • Seemingly random crashes in and around major cities, and other areas where a lot of data is edited. Especially navmeshes.
  • The "landscape foliage" bug in which a child worldspace begins displaying landscape foliage from its parent worldspace because an object in the parent worldspace was edited. This is readily apparent in Darkwater Pass.
  • The "LOD trees show up at close range" bug. This is where you will sometimes see trees that are clearly intended to be displayed as LOD sitting within the currently loaded cells, sometimes even merged into the real trees. As described here.


There may be others, but these 3 classes of bugs are confirmed to be correctable by converting to a false ESM.

So you might be asking then. Why not a proper ESM file? Two reasons mainly:

1. Steam cannot support ESM files, we'd have to cease distribution there all together and not just for the USKP.
2. Other mods are relying on the filenames being ESP files and not all of those mods are currently being maintained anymore.

It's already been verified that Wrye Bash is OK with changing the flag. It will not complain when the files get bumped up into the ESM section.

With proper CRC checks, BOSS won't complain either. Though I'm hoping some other method can be used to detect it ongoing because having a growing mountain of CRC checks won't be helpful. Perhaps version number parsing from the description field.

I cannot vouch for how NMM will react since I don't use it, and I'm not sure anyone else on the team has tried it either.

Who knows what Mod Organizer will think.

We will bump all the version numbers on all the patches up to 2.0 when this happens. We don't have a time frame yet, because there's also a giant mountain of vanilla bugs piled up in the tracker to address too. But when it happens, all of the patches will be updated at the same time.

Our proposed changes in the load order for the game will look like this:

Skyrim.esm
Update.esm
Unofficial Skyrim Patch.esp
Dawnguard.esm
Unofficial Dawnguard Patch.esp
Hearthfires.esm
Unofficial Hearthfire Patch.esp
Dragonborn.esm
Unofficial Dragonborn.esp
[The rest of your stuff]

The main reason for this should be obvious. It will generate much less interference with things following on in the ESM group, and at the same time allow us to undo some of the changes in each DLC patch that are only there because the USKP canceled them out due to its own edits.

Best of all, thus far in testing, it poses no risk for converting them in the middle of an active playthrough, even with a lot of other mods loaded.

So anyway, that's where we're headed now. We've already kind of leaked hints at it, so now it can be considered confirmed rather than rumor :)

 

The ESM Conversion Checklist

 

Wrye Bash Support - Supported as of release 304.2.

NMM Support - Works as expected, NMM flags the files with a warning but allows them to move into the ESM group.

Mod Organizer Support - v1.0.0rc1 and up has been confirmed to work.

TES5Edit Support - Supported as of release 3.0.31.

BOSS Support - Working in 2.1.1. (was some confusion here, but it *IS* working)

 

Checklist items need to be completed before we move forward on this.

 

BETA Scheduling

 

We're planning to go into beta on all 4 patches Saturday (October 5). I expect we'll have a minimum of 2 weeks testing, perhaps longer, to see how all this affects things and to validate that none of the new fixes coming in with this are broken.

 

With that in mind, I figured this is as good a time as any to say so. Once this round of beta testing is complete, the USKP will no longer be distributed on Steam. Leaving the 1.3.2c version up there is already causing us problems and we can't in good conscience leave it up there after 2.0 has gone live. So anyone still using Steam to get the USKP needs to transition to one of the other available download sites.

 

We're also going to have to check to see if Steam will accept a falsely flagged ESP file and deliver it back to the user intact as such. I mention this now because if it doesn't, then ALL of the unofficial patches will have to be taken down from Steam and everyone using those too will need to make provisions to change to another download site.

 

So keep that in mind for those of you using Steam.

Link to comment
Share on other sites

I last used Wrye Bash a very long time ago... Does it process proper ONAMs for Skyrim when converting to esm ?

Link to comment
Share on other sites

No, Wrye Bash does not do that. TES5Edit is the only tool capable of generating the necessary ONAM subrecords.

Link to comment
Share on other sites

I don't think that was caused by the same sort of issue, but we can certainly test it to find out.

Link to comment
Share on other sites

If esmifaking really helps in getting rid of all sorts of obscure problems, everybody will esmifake their mods in the not too long run. What does this mean for game stability ?

Link to comment
Share on other sites

Does this mean all the compatibility bug patchs for the patches overwritging previous stuff need to be undone?

Link to comment
Share on other sites

If esmifaking really helps in getting rid of all sorts of obscure problems, everybody will esmifake their mods in the not too long run. What does this mean for game stability ?

technically, afaik esm is always > then esp so doubt it would hurt

Link to comment
Share on other sites

If esmifaking really helps in getting rid of all sorts of obscure problems, everybody will esmifake their mods in the not too long run. What does this mean for game stability ?

 

Well, it led to a mess with Fallout 3. Probably with New Vegas too, but that was largely because the engine was in much worse shape than it is now and couldn't adequately deal with it. Bethesda's continuing refinements to things have made it somewhat clear that they favor the ESM. If they'd just make the CK play along it wouldn't even be that big of a deal anymore.

 

So stability shouldn't be affected, but I'd still caution against doing this until there's a good reason to do so in other mods.

 

Does this mean all the compatibility bug patchs for the patches overwritging previous stuff need to be undone?

Just the records in the DLC patches that exist to override a USKP edit that broke a DLC change.

Link to comment
Share on other sites

Turning a basic esp to a fake esm has more about hacking than an official and approved feature. We could say the same thing about the note/scroll/book flag that can only be altered with TES5Edit or about creating an esp dependancy, for example. But if it works and is safe, why not ?

Link to comment
Share on other sites

Just the records in the DLC patches that exist to override a USKP edit that broke a DLC change.

sure?

but that would still mean the DLCchange will overwrite what fix was in the USKP.

before the USKP overwrite the change from the DLC, but in both cases, the U DLC P should provide a merge, or is there some clever new option to work around this by esmifying?

Link to comment
Share on other sites

The cases I'm thinking of are of the type where the USKP fixed something and the DLC mad an obviously intentional change that the USKP fix genuinely breaks. There's not very many, but I'd have to go digging for them to be able to tell you.

Link to comment
Share on other sites

If I understand right, the intended 'dirty edits' in UDGP, UHFP and UDBP will all be removed because they'll have no use anymore ? All patches will finally be truly '100 % cleaned' according to TES5Edit's standards and definitions ?

Link to comment
Share on other sites

  • 3 weeks later...

Added a checklist to the bottom of the OP. Bash used to work but doesn't now, that's a setback, but hopefully fixing it isn't too difficult. All that stuff needs to be confirmed as working before we can go forward with the plan.

Link to comment
Share on other sites

Some quick turn around on Bash support, Tes5edit, and Mod Organizer puts this on good footing. BOSS ought to work fine as is, but needs to be verified. The hard part comes from getting the USKP's piled up bug fixes done now.

Link to comment
Share on other sites

Turning the unofficial patches into false esm's is mostly about squashing bugs, but it will also avoid some engine issues. Most notably it should reduce the number of CTDs on some areas where a lot of game data has been edited by mods, so it could be a stability improvement as well. However, no sure answer can be given about that until it is more widely tested.

Link to comment
Share on other sites

  • 3 weeks later...

We're planning to go into beta on all 4 patches Saturday (October 5). I expect we'll have a minimum of 2 weeks testing, perhaps longer, to see how all this affects things and to validate that none of the new fixes coming in with this are broken.

 

With that in mind, I figured this is as good a time as any to say so. Once this round of beta testing is complete, the USKP will no longer be distributed on Steam. Leaving the 1.3.2c version up there is already causing us problems and we can't in good conscience leave it up there after 2.0 has gone live. So anyone still using Steam to get the USKP needs to transition to one of the other available download sites.

 

We're also going to have to check to see if Steam will accept a falsely flagged ESP file and deliver it back to the user intact as such. I mention this now because if it doesn't, then ALL of the unofficial patches will have to be taken down from Steam and everyone using those too will need to make provisions to change to another download site.

 

So keep that in mind for those of you using Steam.

 

Eh. Well. That was answered rather quickly:

post-4-0-49195300-1380495989_thumb.jpg

 

CK won't even bother to try. Looks like all 4 will be coming down.

Link to comment
Share on other sites

since the patches won't be uploaded to steam now anyway, whill they still be distributed as fals esms (with esp file ending) or does this open up the possibility of simply renaming them to esm?

might confuse a lot less people i suppose.

Link to comment
Share on other sites

No, we still need to stick with the esp filename because changing that would cause the game to see them all as new files while at the same time kicking the old ones out. It would result in Papyrus havoc.

Link to comment
Share on other sites

Bit of a monkey wrench in things.

 

Obviously we've now gone public with the beta files. Unfortunately we've now found out that BOSS cannot sort things in the proper order, even with the masterlist being doctored to use CRCs for the current batch. It simply lumps them at the top of the ESP list, which isn't going to work.

 

So heads up there - manually sort the beta files into the correct order for now.

Link to comment
Share on other sites

Bit of a monkey wrench in things.

 

Obviously we've now gone public with the beta files. Unfortunately we've now found out that BOSS cannot sort things in the proper order, even with the masterlist being doctored to use CRCs for the current batch. It simply lumps them at the top of the ESP list, which isn't going to work.

 

So heads up there - manually sort the beta files into the correct order for now.

 

Arthmoor I just ran BOSS GUI and then loaded the new Wrye Bash afterwards ..

 

3131183Untitled0.png

 

No problem here

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...