Jump to content

Wrye Bash - All Games


Utumno

Recommended Posts

1 hour ago, Tannin said:

 

Sorry to raise an old topic but I just came across this and wanted to drop to clear this up:

What's breaking MO isn't actually the VRAM fix MS did, they also changed some other stuff in filesystem-related APIs and that's breaking MO. It's not too hard to fix so I wouldn't bet on MO being dead. ;) The fix isn't coming from me though.

MO2 isn't affected because it's hooking into lower level APIs that weren't changed. Vortex isn't affected because it's not hooking anything but uses OS functionality.

That's good to know, thanks for clarifying that. I didn't even know anything was wrong until I saw this. I thought to myself, fiddlesticks because so many people swear by it. They won't use anything except MO/MO2, ever. Even GrantSP didn't know whether or not something could be disabled that would address the issue.

Link to comment
Share on other sites

26 minutes ago, Utumno said:

Hi @Tannin :)

Guys/girls I just pushed a bunch of new stuff in my utumno-wip:

- support for esl (not sure about load order handling yet)

- support for new save formats - read write (do test)

- fixups in Save Details refreshes and duplicate save file

- small BAIN refactorings

- scandir support

Please test while I smooth things for stable branch

Is this also going to be nightly build ?

 

Edit : Also @Utumno I noticed something recently with the 201708021813 nightly build, have been doing a lot of testing on Beermotors WizBAINs, and was pleasantly suprised to see that when executing a wizard, after making lots of selections, using the Back button now actually remembers all previous choices, whereas it never used to but recovered the default selection for the page you went back to. Its a nice little detail I was not aware had been amended.

Link to comment
Share on other sites

50 minutes ago, Utumno said:

Guys/girls I just pushed a bunch of new stuff in my utumno-wip:

- support for esl (not sure about load order handling yet)

- support for new save formats - read write (do test)

- fixups in Save Details refreshes and duplicate save file

Please test while I smooth things for stable branch

This is a screenshot of two errors. One is the Save Game errors, it errors on every single Fallout 4 Save Game now both new and old formats.  Here is a pastebin of the Traceback. Your changes gave me some inspiration, thanks for taking the time to do that I appreciate it.

The other is the missing Strings files for the Creation Club content. I purchased their content so I could test things more thoroughly. One thing to note. I remember @zilav saying something about Fallout 4 ignoring case. I recommend you do so as well.  Use str.lower(ESP/ESL name) [compared to] str.lower(BA2/BSA Name) because right now the purchased content has an esl file of ccbgsfo4004-pipboy(camo02).esl but the file with the strings in it is named ccBGSFO4004-PipBoy(Camo02) - Main.ba2. Those are files from Bethesda and in the case that Bethesda put them in.

Link to comment
Share on other sites

@Utumno I'm still testing and trying to draw inspiration from your changes. For Fallout 4 __init__.py I have added doc strings to explain what is expected to be happening.

Spoiler



        #  Skip old ESL masters for Fallout 4
        if ((header.version > 12) and (header.formVersion == 68)):
            """If new FO4 Format, One ushort 'H' of header.numEslMasters
            with no length byte, which is the count of ESL files,
            needs to be added to the data stream immediatly after the last
            NON ESL file, to be written out to the save file.

            Old numESLMasters needs to be read from the file and used
            as it is found or discarded. """
            numESLMasters, = unpack('H',2)
            print u"numESLMasters:", numESLMasters
            pack('H',header.numEslMasters)
            """Skip old ESL masters and add them to oldMasters Array??"""
            for x in xrange(numESLMasters):
                # size, = unpack('H',2)
                # oldMasters.append(ins.read(size))
                oldMasters.append(unpack_str16(ins))
            """Write new ESL masters, header.numEslMasters should already be
            added to data stream and in queue to be written, only add
            file names to the data stream to be writtento the save file."""
            for master in header.masters:
                pack('H',len(master))
                out.write(master.s)

The above code fails because of header.mMasters somehow. I feel like the first block skips the file names of the first block, and then packs the change into what it's going to write to the file. However, I'm writing master.s twice, I think.

EDIT: Even better I found "Working with Binary Data in Python", I need to experiment.

 

 

Link to comment
Share on other sites

Here is a screen shot of a somewhat short Plugin List. It has one length of 318, count of 12 (byte), 12x Files Names, Count of 3 (ushort), 3x File Names, Offsets. My code block with doc strings.

Spoiler

 

EDIT: I made some print statements when you read the File Names to see what is going on here:

            header.numMasters, = struct.unpack('B',ins.read(1))
            for count in xrange(header.numMasters):
                header.masters.append(unpack_str16(ins))
            print u""
            print u"Header Masters: ", header.masters
            if ((header.version > 12) and (header.formVersion == 68)):
                eslMasters = []
                header.numEslMasters, = struct.unpack('H',ins.read(2))
                for count in xrange(header.numEslMasters):
                    eslMasters.append(unpack_str16(ins))
                print u"Header ESL Masters: ", eslMasters
            if ins.tell() != header.mastersStart + mastersSize: raise Exception(
                u'Save game masters size (%i) not as expected (%i).' % (
                    ins.tell() - header.mastersStart, mastersSize))

Header Masters:  ['Fallout4.esm', 'DLCRobot.esm', 'DLCworkshop01.esm', 'DLCCoast.esm', 'DLCworkshop02.esm', 'DLCworkshop03.esm', 'DLCNukaWorld.esm', 'ArmorKeywords.esm', 'Homemaker.esm', 'Armorsmith Extended.esp', 'Cannabis Commonwealth.esp', 'SettleObjExpandPack.esp']
Header ESL Masters:  ['ccbgsfo4001-pipboy(black).esl', 'ccbgsfo4004-pipboy(camo02).esl', 'SettlementKeywords.esl']

It's probably better to have onyl header.masters , but I wanted to observe what was going on.

}

uint: size of masters block

ubyte: count of ESP/ESM Files

Array of [count] : string_16

ushort: eslCount of ESL Files  <<-- Added with new FO4 Save File Format

Array of [ eslCount ] : string_16   <<-- Added with new FO4 Save File Format

}

I'm going to look at your save_files.py code and see if you take the count, like 12 which like most arrays starts at 0 to it's really 11. 

for 11 to 11 + header.numEslMasters:

for ( header.numMasters - 1) to ( header.numMasters - 1) + header.numEslMasters:

Write out count, then file names. Maybe something like that.


 

 

Link to comment
Share on other sites

Peeps on FO4 Nexus have started making save file problem reports, all I have done is point them at the work in progress conversation here and gave a bit of an explanation.

Link to comment
Share on other sites

3 hours ago, alt3rn1ty said:

Peeps on FO4 Nexus have started making save file problem reports, all I have done is point them at the work in progress conversation here and gave a bit of an explanation.

I have finally gotten my routines to work. I don't think I can understand what Utumno is doing for the Fallout 4 save files with his code.

@Utumno I think I am done.  I will be testing other game modes.  Again it is set up so it could be one class for all three games, FO4, TES5, SSE. All you need to add is the ability to read the SSE save file header.

I will make an attempt to get your code working from utumno-wip, but can't promise anything.

Link to comment
Share on other sites

I don't think I can do this. Because your whole routine inherits things in blocks that don't fit what I am used to.

Spoiler

 

For FO4, SSE, TES5 Load header would look something like this:


    def load_header(self, ins):
        if ins.read(12) != 'FO4_SAVEGAME':
            raise Exception(u'Save file is not a Fallout 4 save game.')
        headerSize, = struct.unpack('I',ins.read(4))

        #-- Header Elements
        header.version, = struct.unpack('I',ins.read(4)) # saveVersion
        saveNumber, = struct.unpack('I',ins.read(4)) # saveNumber
        header.pcName = unpack_str16(ins) # playerName
        header.pcLevel, = struct.unpack('I',ins.read(4)) # playerLevel
        header.pcLocation = unpack_str16(ins) # playerLocation
        header.gameDate = unpack_str16(ins)
        header.pcRace = unpack_str16(ins) # playerRace
        header.pcSex, = struct.unpack('H',ins.read(2)) # playerGender
        header.pcExp, = struct.unpack('f',ins.read(4)) # playerExp
        header.pcLvlExp, = struct.unpack('f',ins.read(4)) # levelExp
        header.filetime, = struct.unpack('8s',ins.read(8)) # FILETIME
        #-- ssWidth and ssHeight
        ssWidth, = struct.unpack('I',ins.read(4))
        ssHeight, = struct.unpack('I',ins.read(4))
        if (header.version > 9) and (header.version < 15):
          compressionType, = struct.unpack('H',ins.read(2))

The above code grabs everything but the screen shot data.

For FO4, SSE, TES5 Load Screen Shot Data would look something like this:


    def load_image_data(self, ins):
        if header.version > 9:
            # Skyrim SE is in 32bit RGB, Bash is expecting 24bit RGB
            ssData = ins.read(4*ssWidth*ssHeight)
            # pick out only every 3 bytes, drop the 4th (alpha channel)
            ## TODO: Setup Bash to use the alpha data
            #ssAlpha = ''.join(itertools.islice(ssData, 0, None, 4))
            ssData = ''.join(itertools.compress(ssData, itertools.cycle(reversed(range(4)))))
            header.image = (ssWidth,ssHeight,ssData)
        else:
            ssData = ins.read(3*ssWidth*ssHeight)
            header.image = (ssWidth,ssHeight,ssData)

For FO4, SSE, TES5 to get the FormVersion (Required) and the GameVersion (unused) would look like this:


    def get_FormVersion_GameVersion(self, ins):
        if (( header.version < 12 ) or ( header.version > 12 )):
            header.formVersion, = struct.unpack('B',ins.read(1))
        if ( header.version > 12 ):
            gameVersion = unpack_str16(ins) # gameVersion

For FO4, SSE, TES5 Load Masters would look like this:


    def load_masters(self, ins):
        # Skyrim SE files can be listed, displayed, but not renamed
        # without lz4 support.
        if self.version == 12:
            self._load_masters_16(self._decompress_masters_sse(ins))
        else:
            mastersSize, = struct.unpack('I',ins.read(4))
            header.mastersStart = ins.tell()
            del header.masters[:]
            header.numMasters, = struct.unpack('B',ins.read(1))
            for count in xrange(header.numMasters):
                header.masters.append(unpack_str16(ins))
            if ((header.version > 12) and (header.formVersion == 68)):
                header.numEslMasters, = struct.unpack('H',ins.read(2))
                for count in xrange(header.numEslMasters):
                    header.masters.append(unpack_str16(ins))
            if ins.tell() != header.mastersStart + mastersSize: raise Exception(
                u'Save game masters size (%i) not as expected (%i).' % (
                    ins.tell() - header.mastersStart, mastersSize))

 

Spoiler

 

For FO4, SSE, TES5 Seek to Plugin List and Write New Size would look like this:


    def get_startNewPluginList(self, ins):
        # set two vars global to def writeMasters, required
        numMasters = 0
        numESLMasters = 0

        """Rewrites masters of existing save file."""
        def unpack(fmt, size): return struct.unpack(fmt, ins.read(size))
        def pack(fmt, *args): out.write(struct.pack(fmt, *args))
        out.write(ins.read(header.mastersStart-4))
        #--plugin info
        oldSize, = unpack('I',4)
        newSize = (3 if ((header.version > 12) and (header.formVersion == 68)) else 1) + sum( len(x) + 2 for x in header.masters )
        pack('I',newSize)
        """ The header size is a uint so at this point the file
        has been read in, and written out with the change in size
        of the plugin list. No information has shifted position yet."""

For FO4, SSE, TES5 Assign Old List of Masters would look like this:


    def _store_old_masters(self, ins, numMasters, out, pack):
        oldMasters = []
        numMasters, = unpack('B',1) # Get the plugin count
        """Append the NON ESL files to oldMasters"""
        for x in xrange(numMasters):
            # size, = unpack('H',2)
            # oldMasters.append(ins.read(size))
            oldFile = unpack_str16(ins)
            oldMasters.append(oldFile)
        """We still need any ESL files if present to be added the list
        of old master file names."""
        if ((header.version > 12) and (header.formVersion == 68)):
            """New FO4 Format, One ushort 'H' for the count of NON
            ESL files."""
            numESLMasters, = unpack('H',2) # Get the ESL count
            """Read ESL filenames and add them to oldMasters"""
            for x in xrange(numESLMasters):
                # size, = unpack('H',2)
                # oldMasters.append(ins.read(size))
                oldFile = unpack_str16(ins)
                oldMasters.append(oldFile)
        return oldMasters
        """At this point all the files names for the save game have been
        read and added to oldMaster. Now we need to write out the new
        list of files names since it changed."""

For FO4, SSE, TES5 Write The New Masters would look like this:


    def _write_new_masters(self, ins, numMasters, out, pack):
    """ Global variables are required at this point. They were 
        obtained durring _store_old_masters."""
        pack('B',numMasters) # This is why global is required
        for count in xrange(numMasters):
            pack('H',len(header.masters[count].s))
            out.write(header.masters[count].s)
        if ((header.version > 12) and (header.formVersion == 68)):
            pack('H',numESLMasters) # This is why global is required
            for count in xrange(numESLMasters):
                pack('H',len(header.masters[count+numMasters].s))
                out.write(header.masters[count+numMasters].s)

For All games, Write the Offsets would look like this:


    def _write_offsets(self, ins, out, pack):
        #--Offsets
        offset = out.tell() - ins.tell()
        #--File Location Table
        for i in xrange(6):
            # formIdArrayCount offset, unkownTable3Offset,
            # globalDataTable1Offset, globalDataTable2Offset,
            # changeFormsOffset, globalDataTable3Offset
            oldOffset, = unpack('I',4)
            pack('I',oldOffset+offset)
        #--Copy the rest
        while True:
            buff = ins.read(0x5000000)
            if not buff: break
            out.write(buff)

 

 

 

One thing about reading the header, is now that I know what everything is, I don't have to read one byte, read two bytes, read 16 bytes for one game and 17 for the other, and all that random stuff. Now it is very straight forward.

Link to comment
Share on other sites

21 hours ago, alt3rn1ty said:

Is this also going to be nightly build ?

 

Edit : Also @Utumno I noticed something recently with the 201708021813 nightly build, have been doing a lot of testing on Beermotors WizBAINs, and was pleasantly suprised to see that when executing a wizard, after making lots of selections, using the Back button now actually remembers all previous choices, whereas it never used to but recovered the default selection for the page you went back to. Its a nice little detail I was not aware had been amended.

Yep once more stable -> @Sharlikran, @Supierce - works for me all right - can you link me to a couple saves ? I can't open the image links Sharlikran

So @alt3rn1ty do you think I can close https://github.com/wrye-bash/wrye-bash/issues/225 ?

 

EDIT: @Sharlikran - you use header.formVersion but don't set it anywhere in your code - where's that ?

Link to comment
Share on other sites

Yep I think Issue 225 is history - I do not have the same mod used to repro the issue anymore (SparrowPrince has once again taken his mod down, the duplicate which TechAngel had is also down, and due to the authors wishes of not wanting it published I had to remove the linked dropbox WIP for the same just in case). However I have been testing more complex mods with many selection pages, going backwards and forwards with them, and have not noticed anything which should not be happening.

Link to comment
Share on other sites

45 minutes ago, alt3rn1ty said:

Yep I think Issue 225 is history - I do not have the same mod used to repro the issue anymore (SparrowPrince has once again taken his mod down, the duplicate which TechAngel had is also down, and due to the authors wishes of not wanting it published I had to remove the linked dropbox WIP for the same just in case). However I have been testing more complex mods with many selection pages, going backwards and forwards with them, and have not noticed anything which should not be happening.

Seconded. I've done quite a bit of testing recently :D and by all indications it is fixed.

Link to comment
Share on other sites

58 minutes ago, Beermotor said:

Seconded. I've done quite a bit of testing recently :D and by all indications it is fixed.

Thanks - out of curiosity can you test if fixed in dev branch ?

Link to comment
Share on other sites

@Utumno If I figure out the save game header for utumno-wip (I think I'm slowly getting it) would you be willing to update the class for altering the Save Game names.  For Skyrim you can edit the Save Game header to change the filenames. For Fallout 4 you can change them. For Skyrim SE when it's the compressed header you shouldn't be able to at this time, but when you have an old save game you ported over from old Skyrim you should be able to edit those files since the header isn't compressed.  Would you be willing to do that please?

Maybe use some type of flag or variable such as _compressType.  When compressType = 2 then it's an Skyrim SE file, and you could block editing the save header.

Link to comment
Share on other sites

@Supierce if you want to test/have time to test, see if this works. I only tested it with Fallout 4 and the old save games. I have to do something with my daughter or I would work on it more.  I need to test with the newer style save games. What I was doing is trying to see if I could rename the file names for both new and old style Fallout 4 save games. Then I was going to try with old Skyrim since when you are in Skyrim SE mode, even if it's the old style of file (from TES5), you can't rename anything. If I can rename the header for old Skyrim (while in oldrim mode) and rename the files for Fallout 4, then the changes should be good.

Also Utumno said he may have some save games with 11 for the Save Version. So I don't know how to get a hold of them because I would like to see what it different.

Link to comment
Share on other sites

1 hour ago, Sharlikran said:

@Supierce if you want to test/have time to test, see if this works. I only tested it with Fallout 4 and the old save games. I have to do something with my daughter or I would work on it more.  I need to test with the newer style save games. What I was doing is trying to see if I could rename the file names for both new and old style Fallout 4 save games. Then I was going to try with old Skyrim since when you are in Skyrim SE mode, even if it's the old style of file (from TES5), you can't rename anything. If I can rename the header for old Skyrim (while in oldrim mode) and rename the files for Fallout 4, then the changes should be good.

Also Utumno said he may have some save games with 11 for the Save Version. So I don't know how to get a hold of them because I would like to see what it different.

It works perfectly! No F4 error messages and all saves visible now. Thank you! :)

Link to comment
Share on other sites

More testing with Fallout Save Games. Believe it or not, I have a user provided (gave me with SKE testing) that is Save Version 12, Form Version 62 from 1.4.132.  So it's the same as Skyrim SE.  Also, so far, going through all my save games, for Fallout 4 I have found versions 11, 13, and 15.

EDIT: Current changes should account for Skyrim versions 8, 9, and 12 and Fallout 4 versions 11, 12, 13, and 15.

Link to comment
Share on other sites

21 hours ago, Utumno said:

Thanks - out of curiosity can you test if fixed in dev branch ?

Confirmed it is fixed with the dev branch. Tested with 10 different wizards to make absolutely certain. :D

Link to comment
Share on other sites

16 hours ago, Sharlikran said:

@Supierce if you want to test/have time to test, see if this works. I only tested it with Fallout 4 and the old save games. I have to do something with my daughter or I would work on it more.  I need to test with the newer style save games. What I was doing is trying to see if I could rename the file names for both new and old style Fallout 4 save games. Then I was going to try with old Skyrim since when you are in Skyrim SE mode, even if it's the old style of file (from TES5), you can't rename anything. If I can rename the header for old Skyrim (while in oldrim mode) and rename the files for Fallout 4, then the changes should be good.

Also Utumno said he may have some save games with 11 for the Save Version. So I don't know how to get a hold of them because I would like to see what it different.

I've tested this build and confirmed it to be working with my new FO4 saves.  I rolled a new character and saved, then purchased one of the Pip boy skins (got an .esl), reloaded and saved again. Both saves were diff-able, etc  and I was able to rename the exitsave.

Link to comment
Share on other sites

Also: heads up I've got a new Windows Insider build ready to be installed on my next reboot. I expect (I haven't checked yet this morning) this to be either the last RC or the RTM of the Fall Creator's Update.

 

I don't expect any major changes from the last revision, but if anyone wants me to do a battery of specific tests I'll be glad to.

EDIT: The upgrader is being a total dick. Apparently there's a known issue upgrading from the version I was on to the latest build. So I guess I'm sticking around on this version until MS unfucks it.

Link to comment
Share on other sites

@Beermotor thanks ! I would be curious to see when this was fixed but needs alot of work I guess (bisect)

@Sharlikran - the _formVersion and _compressType attributes are only for FO4 and SSE - or TESV too ?

I used some of Sharlikran's changes so now I am able to read the test saves by @Supierce (the quicksave I could read before too) - please test utumno-wip (read and write masters)

What I can't understand is how come we have FO4 save with version 12. We use that test (version == 12) to distinguish between Oldrim and SSE - should we use the formVersion here ? Is that defined for SSE ? (actually the code works just fine cause I use different classes fo Skyrim and Fallout)

 

Link to comment
Share on other sites

22 minutes ago, Utumno said:

@Beermotor thanks ! I would be curious to see when this was fixed but needs alot of work I guess (bisect)
 

You may not need to do that.. :)
 

Spoiler



beermotorsrealname@Legion:/mnt/d/Dump$ find . -type f -name "Wrye*.7z"
./Wrye Bash 306 - Python Source-6837-307.7z
./Wrye Bash 307.201704280551 - Standalone Executable.7z
./Wrye Bash 307.201705010951 - Standalone Executable.7z
./Wrye Bash 307.201705070428 - Python Source.7z
./Wrye Bash 307.201705070428 - Standalone Executable.7z
./Wrye Bash 307.201705101231 - Python Source.7z
./Wrye Bash 307.201705101231 - Standalone Executable.7z
./Wrye Bash 307.201705181249 - Python Source.7z
./Wrye Bash 307.201705262232 - Python Source.7z
./Wrye Bash 307.201707052007 - Python Source.7z
./Wrye Bash 307.201707052007 - Standalone Executable.7z
./Wrye Bash 307.201707081049 - Python Source.7z
./Wrye Bash 307.201707081049 - Standalone Executable.7z
./Wrye Bash 307.201707092223 - Python Source.7z
./Wrye Bash 307.201707150002 - Python Source.7z
./Wrye Bash 307.201707160231 - Python Source.7z
./Wrye Bash 307.201707201839 - Python Source.7z
./Wrye Bash 307.201707270418 - Python Source.7z
./Wrye Bash 307.201707270418 - Standalone Executable.7z
./Wrye Bash 307.201707311453 - Python Source.7z
./Wrye Bash 307.201708021813 - Python Source.7z

also

./wrye-bash-dev.zip
./wrye-bash-dev_001.zip
./wrye-bash-utumno-wip.zip
./wrye-bash-utumno-wip_001.zip
./wrye-bash-utumno-wip_002.zip
./wrye-bash-utumno-wip_003.zip
./wrye-bash-utumno-wip_004.zip
./wrye-bash-utumno-wip_005.zip
./wrye-bash-wip-save-changes.zip

 

If you can narrow it down to when you *think* it was fixed I can back up my settings then test with any of those versions.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...