Wrye Bash - All Games

[Utumno]

Re: BP issue - it's serious. I need to find the time and bisect the dev branch to see where it is introduced and this won't be anytime soon. Provided I can reproduce it - hey haven't launched skyrim in aeons. Meanwhile could you test the bashed patch in xEdit ? Although I really can't see why loading a wrong BP would freeze windows (wtf ?!). Maybe a red herring and the issue is something else ?

[Beermotor]

1 hour ago, Utumno said:

Re: BP issue - it's serious. I need to find the time and bisect the dev branch to see where it is introduced and this won't be anytime soon. Provided I can reproduce it - hey haven't launched skyrim in aeons. Meanwhile could you test the bashed patch in xEdit ? Although I really can't see why loading a wrong BP would freeze windows (wtf ?!). Maybe a red herring and the issue is something else ?

I've got a terrible cold so I spent the last day or so in bed, and just now getting vertical again.  I'll do some experimentation with the BP to see if I can make something bad happen.  

I also need to devote some time to #400 as planned.

EDIT: Paydirt. I'm on to something with Size Does Matter. The mod was created with TES5Edit, not the CK, so the records are all fv40 just like the records in the master. Need to test something else before I can make a complete report, but yeah it doesn't look like it was created in the CK.

The reason this is relevant is this is the same reason mods like "Insignificant Object Remover" were causing issues in SSE: they were started as plugins in xEdit then records were copied in out of the masters into the new plugin. I'm going to see if I can make Skyrim crash, then I'm going to resave "Size Does Matter" in the CK and see if it still crashes.

EDIT2: Yep, locked up tight as a drum.. 

EDIT3: I forgot what a glorious piece of shit the Oldrim CK was. Extracted strings, edited INIs... Ugh..

[alt3rn1ty]

@Beermotor Can you draw the same conclusions with Enhanced Vanilla Trees

If thats the case then its an underlying problem with the plugins (rather than Wrye Bash) nobody was aware of prior to beta2

[Beermotor]

5 minutes ago, alt3rn1ty said:

@Beermotor Can you draw the same conclusions with Enhanced Vanilla Trees

Yes indeed. That's next. It's a bit more complicated because it isn't a proper BAIN package but I can get a basic install built.

I just finished saving "Size Does Matter" in the oldrim CK and it converted all of those fv40 records to 43. It also added some NavMesh records to the plugin which was odd. Now it won't merge. I'm going to see if the game will launch with the NAVI present then yank it out and see if it will still work when merged.

EDIT: yep once SDM was saved in the CK -  Oldrim runs just fine. 

I'll repeat the same process with EVT.

[alt3rn1ty]

The navmesh additions could just be dirty edits the CK is famous for adding randomly, ITMs maybe.

Edit : Just saw your edit .. Looks like I need to add another line or two to the sticky posts on the matter of "If the GAME will not start" (will wait for your further testing and see if the consensus agrees before making it a part of the sticky ... )

Quote

If the game will not start after merging a mod in the bashed patch, and it will start if you do not merge that mod - Take it back to the author and get them to at least load the plugin into the Creation Kit and resave the mod properly so that record versions are saved correctly (and not just copies of record versions from masters)

 

I didn't even realise this could be a thing for Oldrim

Edit 2 : @Utumno Could a warning be added for Oldrim similar to SSE mentioning any plugins which have this trait ?. I swear that warning has been a godsend for SSE Wrye Bash, and suppressing bad reports at WB

[Beermotor]

21 minutes ago, alt3rn1ty said:

The navmesh additions could just be dirty edits the CK is famous for adding randomly, ITMs maybe.

Thank you. I forgot how rickety and janky Oldrim was.

I yanked them out with zero issues.  Once I did that it merged and the new re-saved "Size Does Matter" plugin merged and the game launched as expected.

Screenshots for review:

Spoiler

SDM prior to saving in the CK. Note the records are the same FV as the master, which leads me to believe the plugin was created by simply copying the records out of the masters into the plugin using TES5Edit.

Once the plugin was re-saved in the CK note the form versions on the records are now v43. This version of the plugin works fine both unmerged and merged into the BP.

 

Now on to EVT. I'm on to something here. Will report shortly.

EDIT: Yep same thing. Copies of Skyrim.esm fv40 records.. 

More info shortly.

[alt3rn1ty]

Beermotor, used to work with rockets .. nuff said. So glad you popped in here one day 

[Utumno]

Pheeew - beta 3 out shortly - well we had not a public beta out in one year I expected it to blow somehow, here was it. I will add warnings once you spell them out to me but meanwhile how on earth windows freezes because of some formVersion mismatch ? Also (easier to answer) what on earth do we write in the patch ?

[Beermotor]

Findings with Enhanced Vanilla Trees:

The original EVT plugin (SRG Enhanced Trees Activator.esp) looks to be a TES5Edit created plugin that was never re-saved in the CK.  In this screenshot, you can see the tree records in the plugin are basically copies of the records in Skyrim.esm. Later on in this post I'll demonstrate how this works.

Spoiler

To cut to the chase. the EVT plugin is mergeable as @alt3rn1ty said, and those v40 records make Oldrim shit the bed. Re-saving the mod in the CK redoes the records as v43 so all is well.  Oldrim will launch fine after this has been done.

Spoiler

So how does this happen? It's actually super-easy to do in TES5Edit. This functionality is useful in xEdit but I don't think it was ever intended to be used to create mods from scratch. Here's how the process works.

First you find the records you want to use in the master. You right click and select "Copy as override into.." then select a new plugin.

Spoiler

In this case I named it "Mertz-buggy-assed-plugin.esp". Then I started copying more records into it.

Herein lies the problem. If you just check the TES4 Header in the plugin everything looks legit.

Spoiler

However the records are still v40 from the master. This is bad.

Spoiler

I tested with the plugin I just created above, merged it into my bashed patch. SSE behaved EXACTLY as Oldrim did (showed logo, no menu).

@Utumno My hypothesis for what is happening is that the Bashed Patcher is writing all of the records included in the BP as the appropriate form version for the target game, but since the original records from these "problem" plugins were not in that form version to begin with, the records written in the patch are not actually really the form version claimed in the record. This causes the record to be corrupt perhaps.

I don't know enough about what determines the actual form version at that low of a level. You would need @Arthmoor, @fireundubh,  @zilav, or  @Sharlikran to elaborate on that.   

But TLDR: the BP is writing munged records if they are not the proper version to begin with. Beta 1 did not force the record versions so that is why we're just now seeing this.

What I would recommend as a work-around (if possible) is to check not only the TES4 header of the plugins, but also the actual records themselves. This seems like a hellish performance drag, so it may have to happen during BP generation rather than on Wrye Bash launch or Mods Tab selection.

If you need me to look at anything else or test any other scenarios, please let me know.

 

 

 

[alt3rn1ty]

If you can produce a warning, then just let the corrupted Bashed Patch happen, it makes it obvious to the user any merged plugins need to be looked at by the Mod Author, which will be expedited by the masses if its locking up the game starting.

Or if thats not palatable, how about after the warning, finish off by telling them "<name of plugin> will be marked not mergeable, otherwise your game will not start - This plugin needs to be correctly saved from the Creation Kit by its author" and auto bash tag the offending mod not merge-able?

 

Edit : Meanwhile I am going to edit the sticky for oldrim, fo4 and sse, and also go visit a few mod authors hopefully getting positive results from a bit of diplomacy (well, I think I may have to be just straight to the point on this one really without being offensive).

[Utumno]

10 hours ago, alt3rn1ty said:

Having had a look at both Bashed Patches ( 1 and 2 ) to compare them in TES5Edit, the only difference I see is the old beta1 bashed patch has a version of 0, and the new beta2 bashed patch has a version of 43 - Do we need to enforce 43 on Old Skyrim ?

Here are the two bashed patches if you want to have a look at them https://www.dropbox.com/s/1z31b9o1xngnln3/Old Skyrim Beta1 and Beta2 Bashed Patches for comparison.7z?dl=0

Well missed that - so if bashed patch had version 0 the game somehow didn't care ?

The issue is that game.esms have formVersions that are different than 43. I am not sure what should I check for (plus it would make a hell of a complication to try and tell records based on whether they are beth mod records or not). Omitting the records on creating the patch would be a nono I think - not clear about the patcher process but they should be omitted on loading - @Sharlikran do you have any ideas on where could we  add a check for those mods ?

Then, if it's just that those mods should not be marked mergeable, maybe simpler to edit mergeability checks - maybe

@zilav, @Arthmoor technical advice appreciated

[Utumno]

39 minutes ago, Beermotor said:

My hypothesis for what is happening is that the Bashed Patcher is writing all of the records included in the BP as the appropriate form version for the target game, but since the original records from these "problem" plugins were not in that form version to begin with, the records written in the patch are not actually really the form version claimed in the record. This causes the record to be corrupt perhaps.

Hmmm - could we check this out ?

 

[Utumno]

On the other hand - if the game loads those alright then maybe the BP should just respect the form version it encounters ? And we should only write the correct (current) formVersion forcibly in the TES4 record header ?

[Beermotor]

20 minutes ago, Utumno said:

Well missed that - so if bashed patch had version 0 the game somehow didn't care ?

I think it was that the previous version of the bashed patch just copied whatever the records were into the BP with no form version alteration at all.. so if a plugin record was v40 it was written into the Beta 1 BP as v40.  The TES4 header being 0 was unrelated to this specific issue. 

So to clarify (using SSE as an example) the new Beta 2 bashed patcher is taking v40 records, copying them in, and saying "hey this record is now v44" despite it not really being v44. The record in the BP is still really v40 at the data (binary) level, so the game sees it as a corrupt v44 record.

17 minutes ago, Utumno said:

Hmmm - could we check this out ?

Possibly. I think we need folks more knowledgeable than me to expound on how the record formats are handled at the actual data level.

 

15 minutes ago, Utumno said:

On the other hand - if the game loads those alright then maybe the BP should just respect the form version it encounters ? And we should only write the correct (current) formVersion forcibly in the TES4 record header ?

That could be a monkey patch work-around until more information is available. Perhaps also throw an error like the existing exception.py error handler when a record is encountered that is lower than the expected version for the game:

Spoiler

class ModSizeError(ModError):
    """Mod Error: Record/subrecord has wrong size."""
    def __init__(self, in_name, rec_type, read_size, max_size,
                 exact_size=True, old_skyrim=False):
        ## type: (Path, basestring, int, int, bool, bool) -> None
        self.rec_type = rec_type
        self.read_size = read_size
        self.max_size = max_size
        self.exact_size = exact_size
        if old_skyrim:
            message_form = (u'\nWrye Bash SSE expects a newer format for {} '
                            u'than found.\nLoad and save {} with the Skyrim '
                            u'SE CK\n'.format(rec_type, in_name))
        else: message_form = u''
        op = '==' if exact_size else '<='
        message_form += (u'{}: Expected size {} {}, but got: {}'
                         u''.format(rec_type, op, read_size, max_size))
        super(ModSizeError, self).__init__(in_name.s, message_form)

 

If you need me to create an issue on github for us to track it there I can fill in all the information I've gathered so far.

[alt3rn1ty]

@Beermotor 44 for oldrim ?

-----------------

Updated the sticky posts, replied the original Oldrim thread of posts linking the conversation here and Beermotor detective work, and also gave the authors of the mods a heads up what needs to be done.

[Utumno]

Pushed a commit in utumno-wip that tries to respect existing form version - completely untested (wink)

[Beermotor]

1 hour ago, alt3rn1ty said:

@Beermotor 44 for oldrim ?

Probably a derp typo. I'm pretty sick right now and messed up on cold meds.

 

10 minutes ago, Utumno said:

Pushed a commit in utumno-wip that tries to respect existing form version - completely untested (wink)

Will test and report.

[Utumno]

Haha: "sorry you can not add more reactions today". Well I need a holiday anyway - let's hope this will be an easy fix. All the best with your cold @Beermotor - I cough too as I write this

[Beermotor]

Thank you @Utumno 

Initial testing on the new build - records are being passed from the bad plugin unaltered.  Note this is my SSE bashed patch which should have v44 records.

Spoiler

This will certainly allow the game to start as a work-around, but I'm not sure if this is the right long-term solution to the problem because now it's back to having down-level record versions in the Bashed Patch which is why the patcher was changed in the first place.

Perhaps @Arthmoor could weigh in on it.  I'll be out of pocket for the rest of the night but I'll be able to test again in the AM.

[zilav]

The only records that are binary different when their version is 44 are AMMO, MATO, STAT, WATR and WEAP. Also LTEX and WTHR have new subrecods, but I don't consider this a binary difference since subrecords are optional and the game ignores unknown ones.

Your example with GRAS is very strange. This record is the same binary wise across all form versions. So you are saying that when this record having version 39 is merged into the bashed patch having TES4 version 44 and WB updates the record version itself to 44, the game locks up? But merging the same record already having form version 44 (resaved in CK) works fine? This simply can't be true, something else is going on here.

 

PS: Actually there is a binary difference - MODT in newer version inside of Update.esm is zeroed. But this field is not marked for conflicts resolution and hence it is green. If you still have bashed patch that locks up the game, try to drag&drop MODT from Update.esm into the patch in all grass records.

[alt3rn1ty]

Ehm @zilav I think you have missed a heap of the conversation, need to start back here to catch up on the issue

[Daidalos]

Hey @Utumno since I have now (hopefully) finished updating the loot api integration, what issue should I work on next? Before you said that #390 Restore Settings is the next focus. Is that still the case? I seem to recall skimming over some posts of you working on something related to that.

[Beermotor]

 

8 hours ago, zilav said:

So you are saying that when this record having version 39 is merged into the bashed patch having TES4 version 44 and WB updates the record version itself to 44, the game locks up? But merging the same record already having form version 44 (resaved in CK) works fine? This simply can't be true, something else is going on here.

Yes that's what appears to be happening, at least on the surface. After reading your post I agree there must be something else going on.

It is interesting that you mentioned the MODT (model texture) records because when I looked at the records in a binary diff, that's part of what the CK changed in the resaved plugins.

Here's the setup for my experiment:

Plugin was "Size Does Matter". This plugin modifies a bunch of furniture records. I compared the "Size Does Matter" plugin to see the difference before and after saving it in the CK. Left side is the old original plugin, right is the resaved plugin. Here are the changes:

Update.esm was added as a master:

Spoiler

and MODT records were added to the NIF records:

Spoiler

I also checked Enhanced Vanilla Trees plugin. The CK also did MODT modifications on it when resaved. These already had the MODT records present but they were modified by the CK on save. Update.esm was not added as a master.

Spoiler

So the records with MOTDs appear to be what is locking up Skyrim after they are merged into the Bashed Patch. I noticed that in @Lojacks work on CBash for Skyrim, he mentioned these were just hashes and were read and written as raw data in his prototype for Skyrim CBash. Perhaps that's a lead into what's going on here.

Here are the plugins in before/after versions if anyone would like to look at them. Plugins for testing 20171230.7z

EDIT: Here are the actual diffs as HTML file. I've also included a diff for Insignificant Object Remover.esp. Open these in a new tab.

diff_Insignificant Object Remover_Insignificant Object Remover fixed_Dec-30-2017 09-28.htm

diff_SRG Enhanced Trees Activator_SRG Enhanced Trees Activator fixed_Dec-30-2017 09-49.htm

(this one is zipped because it is huge) diff_Size Does Matter_Size Does Matter fixed_Dec-30-2017 09-50.zip

 

[Arthmoor]

You will not be able to let the patch retain anything in SSE that's not form 44 and in proper form 44 format. You'll just negate the entire point of having fixed all of those form version checks and put things right back to where they were before beta 2.

So what I think needs to happen if it's not already is that any record added to the patch MUST be properly updated to the correct form version. Not just tagged with that number because obviously that was never going to work.

[Utumno]

3 hours ago, Daidalos said:

Hey @Utumno since I have now (hopefully) finished updating the loot api integration, what issue should I work on next? Before you said that #390 Restore Settings is the next focus. Is that still the case? I seem to recall skimming over some posts of you working on something related to that.

Yep we need to sort that out - will post in 390 ASAP - thanks!

3 minutes ago, Arthmoor said:

You will not be able to let the patch retain anything in SSE that's not form 44 and in proper form 44 format. You'll just negate the entire point of having fixed all of those form version checks and put things right back to where they were before beta 2.

So what I think needs to happen if it's not already is that any record added to the patch MUST be properly updated to the correct form version. Not just tagged with that number because obviously that was never going to work.

Can you give a whirl at latest utumno-wip ? What I do is just add the formVersion as I find it. Patching the code to examine form versions of each record seems not trivial at all - in fact I have no clue as of now how to go about it. But probably since the BP TES4 record has correct form version the rest will be ok

Anyway we need to get to the bottom of this