Wrye Bash - All Games

[alt3rn1ty]

7 hours ago, Utumno said:

@all -> roll up your sleeves (wait for upload!):

307.201709071619

alpha esl, scandir, new saves etc support - basically we need more info on load order handlin g of esls and wait for SSE update - then beta

It seems to be handling the new fallout 4 saves ok here too, I do not have any esl's though.

The new scandir is a vast improvement

Testing on Oblivion / Fallout 4 / Skyrim SE

[Sharlikran]

I have to do more testing and unfortunately at the moment, I have to do family/real life stuff now.  Here are my initial results. I'll work on this more this evening.

Spoiler

 

Wrye Bash starting
Using Wrye Bash Version 307
OS info: Windows-7-6.1.7601-SP1
Python version: 2.7.12
wxPython version: 2.8.12.1 (msw-unicode)
input encoding: UTF8; output encoding: None; locale: ('en_US', 'cp1252')
filesystem encoding: mbcs 
bash.py  324 main: Searching for game to manage:
bush.py   76 _supportedGames: Detected the following supported games via Windows Registry:
bush.py   78 _supportedGames:  Fallout3: C:\Games\steamapps\common\Fallout 3 goty
bush.py   78 _supportedGames:  Fallout4: C:\Games\steamapps\common\Fallout 4
bush.py   78 _supportedGames:  FalloutNV: C:\Games\steamapps\common\Fallout New Vegas
bush.py   78 _supportedGames:  Oblivion: G:\Games\steamapps\common\Oblivion
bush.py   78 _supportedGames:  Skyrim Special Edition: F:\Games\steamapps\common\Skyrim Special Edition
bush.py   78 _supportedGames:  Skyrim: F:\Games\steamapps\common\Skyrim
bush.py  136 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.py  142 _detectGames: Set game mode to Oblivion found in parent directory of Mopy:  G:\Games\steamapps\common\Oblivion
bush.py  156 __setGame:  Using Oblivion game: G:\Games\steamapps\common\Oblivion
testing UAC
mods_metadata.py  228 __init__: Using LOOT API version: 0.10.2
obse path: bolt.Path(u"C:\\Users\\DanoPDX\\Documents\\My Games\\Oblivion\\Saves\\Save 5 - Lilly - Jensine's 'Good as New' Merchandise, Level 1, Playing Time 00.50.24.obse")
self.signature - cosave header: OBSE
self.formatVersion: 1
self.obseVersion: 21
self.obseMinorVersion: 4
self.oblivionVersion: 16908704
numPlugins: 1
opcodeBase: 5120
numChunks: 6
pluginLength: 292
chunkType: SDOM
chunkVersion: 0
chunkLength: 220
numMasters: 12
size: 12
pluginFile: Oblivion.esm
size: 21
pluginFile: DLCShiveringIsles.esp
size: 17
pluginFile: DLCHorseArmor.esp
size: 13
pluginFile: DLCOrrery.esp
size: 15
pluginFile: DLCVileLair.esp
size: 20
pluginFile: DLCMehrunesRazor.esp
size: 17
pluginFile: DLCSpellTomes.esp
size: 17
pluginFile: DLCThievesDen.esp
size: 23
pluginFile: DLCBattlehornCastle.esp
size: 16
pluginFile: DLCFrostcrag.esp
size: 11
pluginFile: Knights.esp
size: 13
pluginFile: TEST_WEAP.esp
chunkBuff: [12, 12, 'Oblivion.esm', 21, 'DLCShiveringIsles.esp', 17, 'DLCHorseArmor.esp', 13, 'DLCOrrery.esp', 15, 'DLCVileLair.esp', 20, 'DLCMehrunesRazor.esp', 17, 'DLCSpellTomes.esp', 17, 'DLCThievesDen.esp', 23, 'DLCBattlehornCastle.esp', 16, 'DLCFrostcrag.esp', 11, 'Knights.esp', 13, 'TEST_WEAP.esp']
chunkType: SVTS
chunkVersion: 0
chunkLength: 0
chunkBuff: 
chunkType: EVTS
chunkVersion: 0
chunkLength: 0
chunkBuff: 
chunkType: SVRA
chunkVersion: 1
chunkLength: 0
chunkBuff: 
chunkType: EVRA
chunkVersion: 1
chunkLength: 0
chunkBuff: 
chunkType: BOLG
chunkVersion: 0
chunkLength: 0
chunkBuff: 

 

 

[Beermotor]

6 hours ago, Utumno said:

> Quick question regarding scandir: for those of us on python installs, is it necessary to do a 'pip install scandir' or is this built-in now?

Yep, python.exe -m pip install --upgrade scandir

It's installed, new WB is in place, and holy shit is this thing fast.

Edit: ok the initial shock has passed.  I have a FO4 ESL for one of the Pip Boy re-skins, so I was able to record the behavior on start up. I have screenshots which I can post on request. I do not know if this behavior is expected or unexpected, I'm just relaying observations.

• The first thing on launch was I received an error for the ESL: "Missing String Localization" for the pipboy esl. I can provide a name of the esl if needed, but it is the black pip-boy recolor available for 50 credits.
• Once the error is cleared, the esl is in the middle of the load order and pink, but will turn red (deactivate me) f selected. 
• No bugchecks in the BashBugDump.
• Also checked saves, and the ESLs are showing in the saves as normal plugins.

[Sharlikran]

@Utumno Tell me if this makes is condensed, simple and to the point.

OBSE, NVSE, FOSE CoSave Format:

typedef struct {
    char  chunkType[4];
    uint  chunkVersion;
    uint  chunkLength;
    if (chunkLength != 0)
        char    chunkData[chunkLength];
} CHUNK <name="Chunk">;

struct FILE {
    struct HEADER {
        char signature[4] <name="Signature">;
        uint formatVersion <name="Format Version">;
        ushort obseVersion <name="OBSE Version">;
        ushort obseMinorVersion <name="OBSE Minor Version">; 
        uint oblivionVersion <name="Oblivion Version">;
        uint numPlugins <name="Number of Plugins">;
    } header <open=true, name="Header">;


    struct CHUNKBUFF {
        uint opcodeBase;
        uint numChunks;
        uint pluginLength;
        CHUNK chunkBuff[numChunks] <open=true, optimize=false, name="chunkBuff">;
    } chunkBuff[header.numPlugins] <open=false, optimize=false, name="Chunk">;

} file <open=true, name="Script Extender Save Game">;

List of espm Masters:

if (opcodeBase = 0x1400) and (chunkType == 'SDOM'):
    [Read Script Extender list of Masters espm files]
if (self.signature = 'F4SE') and (chunkType == 'DOML'):
    [Read Script Extender list of Masters esl files for Fallout 4]

Does that explain it?

If not this is for Python 3 and, should read any cosave file from any game:

Spoiler

 

# Imports ---------------------------------------------------------------------
import struct
import io

path = 'Save70_A3AA8A36M4C696C6C79_Commonwealth_010659_20170908034503_16_2.f4se'

class FileReader(io.FileIO):
    """File-object with convenience reading functions."""

    def unpack(self, fmt):
        return struct.unpack(fmt, self.read(struct.calcsize(fmt)))

    def readUByte(self): return struct.unpack('B', self.read(1))[0]
    def readUInt16(self): return struct.unpack('H', self.read(2))[0]
    def readUInt32(self): return struct.unpack('I', self.read(4))[0]

    def readByte(self): return struct.unpack('b', self.read(1))[0]
    def readInt16(self): return struct.unpack('h', self.read(2))[0]
    def readInt32(self): return struct.unpack('i', self.read(4))[0]

    def readSig(self): return struct.unpack('4s', self.read(4))[0]
    def readString8(self): return self.read(struct.unpack('B', self.read(1))[0])
    def readString16(self): return self.read(struct.unpack('H', self.read(2))[0])
    def readString32(self): return self.read(struct.unpack('I', self.read(4))[0])

def unpack_names(raw_data):
    offset = 1
    name_list = []
    num_names, = struct.unpack('B', raw_data[:1])

    for i in range(num_names):
        name_length, = struct.unpack('H', raw_data[offset : offset + 2])
        name_data = raw_data[offset + 2 : offset + 2 + name_length]
        offset = offset + 2 + name_length
        name_list.append(name_data.decode())

    return name_list

def pack_names(name_list):
    raw_data = b''
    num_names = len(name_list)
    raw_data += struct.pack('B', num_names)

    for name in name_list:
        name_length = len(name)
        raw_data += struct.pack('H', name_length)
        raw_data += name.encode()

    return raw_data

def readChunk(file):
    chunkType = FileReader.readSig(file).decode("utf-8")
    chunkVersion = FileReader.readInt32(file)
    chunkLength = FileReader.readInt32(file)
    chunkData = file.read(chunkLength)
    chunkValue = []
    chunkValue.append(chunkType)
    chunkValue.append(chunkVersion)
    chunkValue.append(chunkLength)
    chunkValue.append(chunkData)
    if chunkType == 'SDOM':
        print (unpack_names(chunkData))
    elif chunkType == 'DOML':
        print (unpack_names(chunkData))
    else:
        print (chunkValue)
    return chunkValue

def readChunks(file):
    opcodeBase = FileReader.readInt32(file)
    numChunks = FileReader.readInt32(file)
    pluginLength = FileReader.readInt32(file)
    chunkBlock = []
    for x in range(numChunks):
        chunkBlock.append(readChunk(file))
    return chunkBlock


def main():
    """Main function, fires everything off."""
    cosave_file = open(path,'rb')
    signature = FileReader.readSig(cosave_file).decode("utf-8")
    formatVersion = FileReader.readInt32(cosave_file)
    obseVersion = FileReader.readInt16(cosave_file)
    obseMinorVersion = FileReader.readInt16(cosave_file)
    oblivionVersion = FileReader.readInt32(cosave_file)
    numPlugins = FileReader.readInt32(cosave_file)

    print ("Signature: ", signature)
    print ("formatVersion: ", formatVersion)
    print ("obseVersion : ", obseVersion)
    print ("obseMinorVersion: ", obseMinorVersion)
    print ("oblivionVersion: ", oblivionVersion)
    print ("numPlugins: ", numPlugins)
    for x in range(numPlugins):
        readChunks(cosave_file)
    cosave_file.close()

if __name__=='__main__':
    main()

 

This is what it printed to the console:

Spoiler

 

C:\Python35\python.exe G:/Sharlikran/TESDumpStats/DumpCosave.py
Running on Python 3.5.2
Signature:  b'F4SE'
formatVersion:  1
obseVersion :  0
obseMinorVersion:  5
oblivionVersion:  17432896
numPlugins:  1
['SDOM', 0, 2528, b'j\x0c\x00Fallout4.esm\x0c\x00DLCRobot.esm\x11\x00DLCworkshop01.esm\x0c\x00DLCCoast.esm\x11\x00DLCworkshop02.esm\x11\x00DLCworkshop03.esm\x10\x00DLCNukaWorld.esm\x11\x00ArmorKeywords.esm\r\x00Homemaker.esm\x16\x00SettlementKeywords.esm\x12\x00BuildingBlocks.esm\x16\x00Robot Home Defence.esm\x10\x00Snap\'n Build.esm\x0e\x00ACS Master.esm\x15\x00EthreonMasterPlan.esp\x17\x00Armorsmith Extended.esp\x1d\x00Armorsmith All DLCs Patch.esp\x07\x00BNK.esp\x0b\x00BNK_SKE.esp\x17\x00SettleObjExpandPack.esp\x16\x00AdvBubbleTurretSet.esp\x1f\x00AdvBubbleTurretSet-RHDPatch.esp\x16\x00AdvSettleTurretSet.esp\x16\x00AlternateFurniture.esp\x18\x00AlternateSettlements.esp\x18\x00Better Vendor Stalls.esp\x11\x00ACS Paintings.esp\x11\x00ABP Paintings.esp\x15\x00VanillaExtensions.esp\x10\x00BetterStores.esp\x1b\x00TD_DLC01Workshop_Addons.esp\x18\x00Business Settlements.esp\x1f\x00BusinessSettlements-SKPatch.esp\x19\x00Cannabis Commonwealth.esp\x1a\x00Eli_Crafting Shiz 9000.esp\r\x00Test_WEAP.esp\x0f\x00EasyHacking.esp\x19\x00Friffy_Fixed Curtains.esp\x15\x00Minutemenoverhaul.esp\x16\x00FunctionalDisplays.esp#\x00FunctionalDisplays-Collectibles.esp$\x00FunctionalDisplays-Patch-DLC-ALL.esp#\x00FunctionalDisplays-MISC-Vanilla.esp"\x00FunctionalDisplays-AID-Vanilla.esp\x14\x00ImmersiveVendors.esp\x17\x00MinutemenPropaganda.esp\x16\x00NX Pro - Farm Core.esp\x1d\x00NX Pro - Harvest Overhaul.esp"\x00NX Pro - Farm Core - Homemaker.esp\x0f\x00OCDispenser.esp\x0f\x00OCDecorator.esp\x07\x00OWR.esp\x10\x00OWR_SKEPatch.esp\x16\x00OWR_CraftableDecor.esp\x19\x00OWR_CraftableDecor_CW.esp\x1f\x00OWR_CraftableDecor_SKEPatch.esp"\x00OWR_CraftableDecor_CW_SKEPatch.esp.\x00Homemaker - Streetlights Use Passive Power.esp\x0f\x00NPC Spawner.esp\x1e\x00SignsOfTheTimesCategorized.esp\x1c\x00GruffyddsSignsAndPosters.esp\x13\x00RedRocketReborn.esp\x08\x00SSEX.esp\x1a\x00Flags of the Old World.esp\t\x00Boats.esp$\x00Stackable Brick Wall Foundations.esp\x12\x00Wall Oil Lamps.esp\x15\x00Workshop_Planters.esp\x0e\x00PipeGalore.esp.\x00Conduits and Pylons Various Wireless Power.esp\x13\x00ScrapDeadThings.esp\x11\x00BB_Buildable1.esp/\x00DD_ROBO-TRUMP_2016_MAKE_MODDING_GREAT_AGAIN.esp\x10\x00Summer Roses.esp\x16\x00ExoticHarvestables.esp\r\x00NewBridge.esp\x17\x00BS-ExtraCustomProps.esp\x1b\x00AES_Renovated Furniture.esp\x12\x00FarHarborDecor.esp.\x00Rebsy\'s Greebles 1.5 SKE PC - Boat Release.esp\x1f\x00ProvisionerSupplyContainers.esp\x19\x00Northland Diggers New.esp\x1b\x00StaticHarvestCollection.esp\x12\x00OCDecoratorDLC.esp\x19\x00BS-FarHarborExpansion.esp\x14\x00RainbowBeds_Kids.esp\x14\x00WildHarvestables.esp\x12\x00CraftableBirds.esp\x1b\x00Craftable_Blossom_Trees.esp\x19\x00Craftable_Birch_Trees.esp\x14\x00SchoolDeskObject.esp\x12\x00SanctuaryParts.esp\x07\x00SLR.esp\x17\x00WastelandBillboards.esp\x13\x00CabinInTheWoods.esp\r\x00BoxHouses.esp\x12\x00BobIguanaDiner.esp\x11\x00AtomGreatness.esp\x0b\x00PlantMe.esp#\x00Crimsomrider\'s Unique Furniture.esp\x1d\x00Graffity2ElectricBoogaloo.esp\x18\x00SettlementGarageSale.esp\r\x00AutoDoors.esp\x0e\x00Desk Lamps.esp\x12\x00CastleInTheSky.esp\x13\x00Bashed Patch, 0.esp']
['DOML', 0, 64, b'\x02\x1d\x00ccbgsfo4001-pipboy(black).esl\x1e\x00ccbgsfo4004-pipboy(camo02).esl']
['RYEK', 1, 0, b'']
['EGER', 1, 0, b'']
['RLTC', 1, 0, b'']
['EGER', 1, 0, b'']
['VEXE', 1, 0, b'']
['EGER', 1, 0, b'']
['RMAC', 1, 4, b'\x00\x00\x00\x00']
['RNRF', 1, 4, b'\x00\x00\x00\x00']
['GMBO', 1, 8, b'\x00\x00\x00\x00\x00\x00\x00\x00']
['GMFD', 1, 0, b'']
['QNUF', 1, 4, b'\x00\x00\x00\x00']
['TSLW', 1, 12, b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
['____', 1, 0, b'']

Process finished with exit code 0

 

 

[Sharlikran]

Quote

# _compressType prevent Skyrim SE saves from editing

Maybe it should be "# _compressType used to specify Skyrim SE compressed save"

Then as I mentioned in this post, would you please allow editing the old Skyrim save games file header while in Skyrim SE mode, please.  Currently you can't edit either. Porting an old save game it would be helpfull to assign the plugin that the author renamed from SomeModName.esp to SomeModName-SE.esp because he made an SE version.

[hlp]

@Utumno   The master table should be, for all games, in chunk SDOM (as MODS in reverse ) owned by the first plugin which is the script extender itself.

All chunk names are acronym spelled in reverse so they read correctly in the source.

WARNING: F4SE just introduced the LMOD master table which is the list of ESL files used by that save. The DOML chunk has the same format as the SDOM chunk and immediately follows it. Of course it should be considered optional.

opCodeBase for OBSE, FOSE and NVSE is 0x01400 = 5120, Any plugin will use a larger opCodeBase (greater than 0x02000). A mod using the 0x02000 opCodeBase is not allowed to use serialization.

"opCodeBase" for SKSE and F4SE is 0 (see below).

 

@Sharlikran

Since Papyrus, opcode are no longer necessary nor attributed centrally by the xSE team. So rather than using opCodeBase, plugin comes up with their own unique "signature" as a 32 bit integer. That number is usually a 4 character string as can be seen in the plugin_example project provided with SKSE sources which uses 'TEST' as a signature.

The script extender (SKSE and F4SE) use 0 as their signature.

You may want to use the term signature rather than opCodeBase in your source as this will be more generic going forward.

Also, as you found out, 2330 identifies pluggy which has been treated as "part of the standard" by some oblivion tools. Elminster also made code specific for pluggy in TES4Edit.

The number of plugins should be the ones that have registered to the serialization interface. Plugin that do not use the cosave should not be counted IF I remember correctly.

Each plugin save and load in turn, so you cannot have the same opCodeBase twice. The way they split their data in single or multiple chunk is up to them.

[Sharlikran]

I read back to the first page so if this was mentioned in a previous thread on Bethesda.net or the old Beth threads, I'm sorry I missed that.

EDIT: I missed it...

Spoiler

I am testing a mod that uses Pluggy to see what happens. The mod I chose didn't seem like it would intrude too much. The first thing I noticed is that when I went to install the mod, Windows told me that the folder the mod would be installing into was potentially missing. I do see a checkbox so that I can set a default operation for that, but is that expected now? If so users are going to notice and mention that right away.

 

 

The mod I chose was Map Marker Overhaul and I noticed that when I went to install it that Wrye couldn't read the wizard file. It told me to check and make sure it's in UTF-8 format first. I'm used to remaking installers for Skyrim so I can use them with BAIN, but is this also expected? Will users have to use Notepad++ to change the encoding with the newer versions of Wrye Bash? The wizard file was encoded in ANSI.

[Sharlikran]

@hlp I finally found something that uses Pluggy so I could see a pluggy chunk. It's odd, there is no list of masters esp or esm files in the chunk at all. I was looking at the Wrye Bash source code and when the opcodeBase = 0x2330 then Wrye looks for a 1 and it's a uint32, which makes sense because I see 0x01000000 in the .obse file. At L413-L419 it's going to read in a list of file names. That routine makes sense, although two ubyte seems weird, but not seeing the pluggy source code I don't know what it should and shouldn't do.

However, the code is saying that while the position you are at is less then the length of the chunk of data, assign that to a ubyte,ubyte,uint. Which then says that the length of the string for the filename will be the uint, which also makes sense. Yet my save game doesn't have any plugin names like found in SDOM. What is serilization? You mentioned the serilization interface, is seriliation the 4 char signature of the chunk like SDOM or is it something else?

Do all 3rd party plugins like Pluggy save a chunk with file names? Do you remember if that changed at some point with Pluggy? I am about to downgrade Pluggy to 128 because MMO says that's the required version and one way or another I want a mod installed that causes esp and esm files to appear in the cosave in the pluggy chunk. So far I'm not seeing that.

 

Spoiler

 

 

 

[alt3rn1ty]

28 minutes ago, Sharlikran said:

I read back to the first page so if this was mentioned in a previous

Second from last post on previous page

[Sharlikran]

Just now, alt3rn1ty said:

Second from last post on previous page

Awww nuts, it was in the spoiler tag. Fiddlesticks, I though I clicked all of them.  Thanks for pointing that out.

[hlp]

@Sharlikran Only the first plugin (the extender itself) saves the save's masters. Other plugins only save whatever they need to keep track off. For Wrye Bash you only need to decode the first plugin.

[Sharlikran]

27 minutes ago, hlp said:

@Sharlikran Only the first plugin (the extender itself) saves the save's masters. Other plugins only save whatever they need to keep rack off. For Wrye Bash you only need to decode the first plugin.

And there are no offsets like in normal save games.  Once the name is changed, and the susequent data shifts because I added or removed characters, there shouldn't be an exception somewhere because I didn't update an offest pointing to a specific block of data. Is that correct?

class PluggyFile:
"""Represents a .pluggy cofile for saves. Used for editing masters list."""

This is why I need to read more docstrings. It says it's a .pluggy cofile for saves. So when there is a .pluggy cosave, if it even has that extension, then you would read in that file IIUC and change the masters. It might not be talking about the .obse file at all.

Also helps to read the documentation that comes with the download, specifically PluggySaveFile_Format.html. The pluggy cosave starts with "PluggySave" just like an OBSE save starts with OBSE and a TES4 Save File starts with TES4SAVEGAME. So the routines to remap the esp and esm files appear (needs more research) to be only for the Pluggy save. I am suspecting that nobody ever took the time to decode the normal TES4SAVEGAME and change the file names.  I don't see why though.

[Sharlikran]

@Utumno I'm still not done investigating everything, but there is or at least can be a seperate save file for the 3rd party OBSE plugin named Pluggy. (unless you know what that is already) And I don't yet have a mod that makes that kind of file. Under class ObseFile:, def mapMasters(self,masterMap): it looks for the opcodeBase of 0x2330 and if it is true it then begins to process the information. At L416-L422 I can see it parsing thought the information. This isn't (and I still need to find a mod that makes one) any part of the .obse cosave file itself, it's a separate file.

The link above that I gave to the Pluggy download comes with source and documentation.  I can see that the code in Wrye Bash is correct for reading the masters for this separate file.
Just clarifying in case you didn't know, since I presumed that def mapMasters(self,masterMap): was updating the plugin names for the .obse file.

 

Spoiler

 

 

 

[Utumno]

3 hours ago, hlp said:

@Utumno   The master table should be, for all games, in chunk SDOM (as MODS in reverse ) owned by the first plugin which is the script extender itself.

All chunk names are acronym spelled in reverse so they read correctly in the source.

WARNING: F4SE just introduced the LMOD master table which is the list of ESL files used by that save. The DOML chunk has the same format as the SDOM chunk and immediately follows it. Of course it should be considered optional.

opCodeBase for OBSE, FOSE and NVSE is 0x01400 = 5120, Any plugin will use a larger opCodeBase (greater than 0x02000). A mod using the 0x02000 opCodeBase is not allowed to use serialization.

"opCodeBase" for SKSE and F4SE is 0 (see below).

 

@Sharlikran

Since Papyrus, opcode are no longer necessary nor attributed centrally by the xSE team. So rather than using opCodeBase, plugin comes up with their own unique "signature" as a 32 bit integer. That number is usually a 4 character string as can be seen in the plugin_example project provided with SKSE sources which uses 'TEST' as a signature.

The script extender (SKSE and F4SE) use 0 as their signature.

You may want to use the term signature rather than opCodeBase in your source as this will be more generic going forward.

Also, as you found out, 2330 identifies pluggy which has been treated as "part of the standard" by some oblivion tools. Elminster also made code specific for pluggy in TES4Edit.

The number of plugins should be the ones that have registered to the serialization interface. Plugin that do not use the cosave should not be counted IF I remember correctly.

Each plugin save and load in turn, so you cannot have the same opCodeBase twice. The way they split their data in single or multiple chunk is up to them.

Excellent - thank you very much for the hlp I start groking this mess in _saves.py.  I have some questions I posted in https://github.com/wrye-bash/wrye-bash/issues/236#issuecomment-328143490

Please reply to me there
 

11 hours ago, Sharlikran said:

Maybe it should be "# _compressType used to specify Skyrim SE compressed save"

---

Then as I mentioned in this post, would you please allow editing the old Skyrim save games file header while in Skyrim SE mode, please.  Currently you can't edit either. Porting an old save game it would be helpfull to assign the plugin that the author renamed from SomeModName.esp to SomeModName-SE.esp because he made an SE version.

Thanks - how does it specify SSE compressed save ? What values can it take ?

---

Not trivial - to begin with how can I differentiate the SSE saves from Skyrim ones ? pairs of  (version, formVersion) ? What are the valid ones for those games ? Is all this documented someplace (even in source code) ?

[Utumno]

1 hour ago, hlp said:

@Sharlikran Only the first plugin (the extender itself) saves the save's masters. Other plugins only save whatever they need to keep track off. For Wrye Bash you only need to decode the first plugin.

Pluggy is an OBSE plugin that keeps the espms names of the save masters - so when we remaped masters we remaped those:

https://github.com/wrye-bash/wrye-bash/blob/dev/Mopy/bash/bosh/_saves.py#L403-L422

Apparently back then OBSE itself didn't keep the save masters ? https://github.com/wrye-bash/wrye-bash/commit/3ac0142aa0096dc9f301e9a69c9b729e865315ee

But let's talk about it at https://github.com/wrye-bash/wrye-bash/issues/236#issuecomment-328143490

1 hour ago, Sharlikran said:

@Utumno I'm still not done investigating everything, but there is or at least can be a seperate save file for the 3rd party OBSE plugin named Pluggy. (unless you know what that is already) And I don't yet have a mod that makes that kind of file. Under class ObseFile:, def mapMasters(self,masterMap): it looks for the opcodeBase of 0x2330 and if it is true it then begins to process the information. At L416-L422 I can see it parsing thought the information. This isn't (and I still need to find a mod that makes one) any part of the .obse cosave file itself, it's a separate file.

The link above that I gave to the Pluggy download comes with source and documentation.  I can see that the code in Wrye Bash is correct for reading the masters for this separate file.
Just clarifying in case you didn't know, since I presumed that def mapMasters(self,masterMap): was updating the plugin names for the .obse file.

 

  Hide contents

 

 

 

Of course is the cosave file what do you mean a separate file ? I know what pluggy is (I have been browsing through this for a decade) but now thanks to hlp I start groking what's in the source. Anyway saves handling is 308 although now you started me off I will probably be able to fix #236 for 307 proper. Anyway we have more pressing stuff (esl saves etc) so be sure to answer to my questions above or let me know if you don't know the answers

 

EDIT:

if (opcodeBase = 0x1400) and (chunkType == 'SDOM') or (opcodeBase = 0x0) and (chunkType == 'SDOM'): [Read Script Extender list of Masters espm files]

if (self.signature = 'F4SE') and (chunkType == 'DOML'): [Read Script Extender list of Masters esl files for Fallout 4]

Right ? Time for some nice classes

The python3 code is yours ? Needs to actually read the plugin names - you run into decoding errors ?

[hlp]

1 hour ago, Sharlikran said:

And there are no offsets like in normal save games.  Once the name is changed, and the susequent data shifts because I added or removed characters, there shouldn't be an exception somewhere because I didn't update an offest pointing to a specific block of data. Is that correct?

class PluggyFile:
"""Represents a .pluggy cofile for saves. Used for editing masters list."""

This is why I need to read more docstrings. It says it's a .pluggy cofile for saves. So when there is a .pluggy cosave, if it even has that extension, then you would read in that file IIUC and change the masters. It might not be talking about the .obse file at all.

Also helps to read the documentation that comes with the download, specifically PluggySaveFile_Format.html. The pluggy cosave starts with "PluggySave" just like an OBSE save starts with OBSE and a TES4 Save File starts with TES4SAVEGAME. So the routines to remap the esp and esm files appear (needs more research) to be only for the Pluggy save. I am suspecting that nobody ever took the time to decode the normal TES4SAVEGAME and change the file names.  I don't see why though.

No offsets, no but you have to keep the plugin and the main header size field correct.

[Sharlikran]

2 minutes ago, hlp said:

No offsets, no but you have to keep the plugin and the main header size field correct.

Yeah that makes sense. Thanks for clarifying about the offsets. I think I have some things I could change for xEdit I'll do that as soon as I can. I noticed a few games don't have saves capability and with all the information I found, If I can help update something I'd like to do so.  Thanks again.

[Sharlikran]

Quote

Of course is the cosave file what do you mean a separate file ?

The format for reading a pluggy cosave file name from the cosave is "espId,modId,modNameLen, = unpack('=BBI',6)" which is not how it is in the SDOM block at all. I posted a screen shot to AFK rather then a link to imgur. You mentioned you can't see screen shots, you can't see any at all?? Not from AFK or external links?  Do you use DOS, Linux, Wine?

That screen shot is from the Pluggy documentation included in the download. It is telling you all the information that is stored in the pluggy cosave.  It's clearly a separate file. I just haven't found a mod yet that causes Oblivion to make a pluggy cosave. Even some of the docstrings refer to a .pluggy file implying that the extension is somefilename.plyggy. In the screen shot is shows that the cosave header is named "PluggySave" not OBSE.

Quote

 

Right ? Time for some nice classes

The python3 code is yours ? Needs to actually read the plugin names - you run into decoding errors ?

 

 

I can't write in Python. I presume you have been using Python for many many years, I don't have that kind of experience.  When I look for tutorials they tell me to print "foo" or in Python3 print ("foo").  They explain how to read 202 bytes where the first byte is a count and the rest are strings with a 16 bit length byte from a while until you reach the end of the data.  They just show basic stuff.  I'm just not finding documentation to do the complex things you are asking for. The fact that I could even write some code that uses Python3 to read that cosave was a miracle in itself.

@Utumno I wrote the file definition above, under OBSE, NVSE, FOSE CoSave Format. I have been decoding and researching this since August 28. Decoding the ESL format for the Fallout 4 saves, and now the CoSaves.  Last night it took me a while to piece together enough Python code to print some information to the screen using PyCharm. My code is meant to run with python. All it does it take a file name in the folder where the .py file is located, read it in, and print that to the screen in Pycharm. It's just s simple demonstration of how to read the CoSave file because you keep asking, what is the format of the cosave, where is the data for the espm masters, and can you show me some source code.

I explained how the Fallo0ut 4 save game worked using the same information. Originally it was one post, just one.  You wanted more information so after the simple post of how the Fallout 4 file is formatted, I linked a template used with a 3rd party hex editor, and linked old source code from an older commit of Wrye Bash because I understand that syntax better.  Only after I updated the code for the save games in save_files.py, your current code, did you proceed. However, instead of using my commit and adding your changes in a commit after, you just made the appropriate updates. Which were much better I will admit.

I am trying to be helpful. I thought it would be nice to decode the Fallout 4 save game files since I'm pretty good at that. Better then most people. I am unsure if you could have done it so thoroughly and easily, or whether or not you would still be working on other things.

[Utumno]

The code is for reading the masters of the pluggy plugin _inside the cosave_ - the .pluggy cosave is indeed another file (and another class) - that is https://github.com/wrye-bash/wrye-bash/blob/dev/Mopy/bash/bosh/_saves.py#L403-L422

pluggy keeps also the masters that is what confuses you

I will use your code to set up a skeleton for reading cosave files and notify you if you can handle that - but this will be come next week at least, we indeed have more pressing masters too - please answer to rest of the questions re: saves or let me know if you don't know

[Sharlikran]

8 hours ago, Utumno said:

The code is for reading the masters of the pluggy plugin _inside the cosave_ - the .pluggy cosave is indeed another file (and another class) - that is https://github.com/wrye-bash/wrye-bash/blob/dev/Mopy/bash/bosh/_saves.py#L403-L422

pluggy keeps also the masters that is what confuses you

I will use your code to set up a skeleton for reading cosave files and notify you if you can handle that - but this will be come next week at least, we indeed have more pressing masters too - please answer to rest of the questions re: saves or let me know if you don't know

Next week is fine, enjoy your weekend.  I am happy to help.  It's been a struggle to learn how to use the 010 editor and come up with any kind of sample code that even works.  It's been difficult and a challenge.  I know the information was helpfull but it didn't have the same feel it should have after acomplishing my goals.  Mostly due to all the craziness of what the python code is doing.  I am a very devoted supporter of Wrye bash. I also like to improve in every aspect of my life, both personally and with my hobbies like Wrye Bash and xEdit. It's a constant struggle.

Take care and thanks for the work so far.

[Sharlikran]

I had already e-mailed about this, and both the author of Pluggy and Ian have gotten back to me now so I'm just passing it along.

 

[godescalcus]

Hey guys, I've been using the python version of wrye bash available at nexus (Wrye Bash 307 beta1 - Python Source-1840-307) but have just come across the github page and this topic. Can you advise me on whether it's worth it to get one of the later development versions, and which?

[alt3rn1ty]

52 minutes ago, godescalcus said:

Hey guys, I've been using the python version of wrye bash available at nexus (Wrye Bash 307 beta1 - Python Source-1840-307) but have just come across the github page and this topic. Can you advise me on whether it's worth it to get one of the later development versions, and which?

It very much depends on your experience with Wrye Bash with all of its files and expected behaviour, rolling back to previous versions and sometimes little system problems or having to delete settings files to solve some odd behaviour.

The bleeding edge (read strange unexpected things can happen occasionally) = Utumno WIP branch

A bit more stable but still close enough to bleeding edge to be current with the conversation exchanges in this topic = Go to the second post of this topic, at the very end of that second post is a link for "307 WB wip standalone" - At the dropbox link there are three files, Installer (can install Standalone or Python, my preferred package to test install new standalones and the whole installation routine which everyone eventually will be subjected to), then theres Source (Python version as loose files to overwrite your mopy installation), and lastly theres Standalone (as per source this is a loose manual installation of the standalone files)

The latest nightly build currently has a minor issue of asking you to create folders when installing from BAINs (which Utumno will be fixing for the next nightly build), but otherwise so far its pretty good.

If you use the nightly build, watch out for posts by Utumno when he mentions a new long version number, that means he has updated the nightly build on dropbox (last one was here, see just after "@All")

Note though all of the above suggestions are not guaranteed stable .. But then again beta testers are valuable, and everyone has to start somewhere, so if you think you can handle the occasional hiccups - Welcome to testing developments :), any good feedback and BashBugDump logs of errors experienced very much appreciated by Utumno posted here in a code box.

[godescalcus]

2 minutes ago, alt3rn1ty said:

It very much depends on your experience with Wrye Bash with all of its files and expected behaviour, rolling back to previous versions and sometimes little system problems or having to delete settings files to solve some odd behaviour.

The bleeding edge (read strange unexpected things can happen occasionally) = Utumno WIP branch

A bit more stable but still close enough to bleeding edge to be current with the conversation exchanges in this topic = Go to the second post of this topic, at the very end of that second post is a link for "307 WB wip standalone" - At the dropbox link there are three files, Installer (can install Standalone or Python, my preferred package to test install new standalones and the whole installation routine which everyone eventually will be subjected to), then theres Source (Python version as loose files to overwrite your mopy installation), and lastly theres Standalone (as per source this is a loose manual installation of the standalone files)

The latest nightly build currently has a minor issue of asking you to create folders when installing from BAINs (which Utumno will be fixing for the next nightly build), but otherwise so far its pretty good.

My experience is basic - I used to install the standalone executable loose files, then I replaced it with the python source. Right now I run wb through Wrye Bash Launcher.pyw. I wanted to ask you if it's possible/adviseable to just replace my mopy folder with the latest nightly build or bleeding edge version or whatever, keeping my personal preference files (folder locations and such). My knowledge of wb goes no further than trying to fix issues with plugins in xEdit when an error occurs, before blaming wb  But I'm not afraid of some learning and I do have a backup of my last stable skyrim.

[alt3rn1ty]

10 minutes ago, godescalcus said:

~ to just replace my mopy folder with the latest nightly build or bleeding edge version or whatever, keeping my personal preference files (folder locations and such). My knowledge of wb goes no further than trying to fix issues with plugins in xEdit when an error occurs, before blaming wb  But I'm not afraid of some learning and I do have a backup of my last stable skyrim.

Yeah from the nightly build when its updated just grab the source files and overwrite mopy folder with the new (I think, its been a long while since I had Python installed .. Pretty sure thats the case though), keep a copy of your last known good mopy zipped as a backup to roll back to if you experience problems which make using it more of a problem than you can manage, then just overwrite with the older mopy files. On a rare occasion I have experienced issues where after doing a roll back, the newer saved settings files are then no longer compatible with the older setup, and so you experience random bizarre problems after the roll back, but then you just delete all settings files like bashsettings.dat, and installers.dat, and then the next time Wrye Bash loads up, you have to set your settings again and overwrite install all your BAINs, and do a Full Refresh on the Installers Tab .. and that pretty much puts you back in a good place again.