Jump to content

Wrye Bash - All Games


Utumno

Recommended Posts

No, Vortex will have a VFS, but it uses fully supported OS junctions and links instead of an external VFS like MO.

Link to comment
Share on other sites

4 hours ago, Sharlikran said:

~ who works on MO, same or different person?

Nobody as far as I know. Tannin dropped MO dev in preference for MO2, then he handed over MO2 to LePresidente when he was hired to do Vortex.

NMM as we all know is now dropped from being supported by nexus, in favour of Vortex (whenever that will be released as an early beta is still not known)

So the choices going forwards are Wrye Bash, MO2 and eventually Vortex

Link to comment
Share on other sites

Since MO2 still uses the VFS as far as I know, it will still be crippled by the Windows 10 update. That will effectively remove it as a selection for about half of SSE's playerbase. They can't use the original MO at all anyway because that only works on 32bit Skyrim.

Link to comment
Share on other sites

Somehow I had the thought that MO and MO2 vfs may be different in some way, but you might be right and it affects both.

Either way it looks like LePresidente is running out of steam on the MO2 project.

So Wrye Bash and eventually Vortex will be the remaining dependables.

And so allowing for FOMod support will still be a thing in packages (for Vortex and legacy mod support).

Link to comment
Share on other sites

13 hours ago, Arthmoor said:

It's got something to do with the virtual file system no longer working once that update goes live. People on Reddit have been up in arms about it because Microsoft has no intention of reversing whatever did it. So they got their VRAM "bug" fixed and traded it for killing MO instead. Tannin has already said he has no plans to try and work around it somehow for Vortex either.

What is breaking the MO overlay stuff with Windows 10 CU it is something in the filesystem driver(s) (I haven't kept up with their driver subsystem) added for security (anti-malware) reasons. I'm not sure of all of the details, but the gist of it I've gathered from a contact of mine is unregistered/unsigned applications are no longer allowed to nail up a union filesystem. The goal of this is to prevent malware and ransomware which uses the same techniques. MO seems to be a casualty of this change. 

Knowing MS there will probably be a way to override this but it is NOT something the average user would want to do because it would expose them to too much risk. 

 

Link to comment
Share on other sites

That sounds about like the explanation I read elsewhere. Tannin doesn't seem too bothered by it though because even he's said his VFS was a solution looking for a problem to fix when he wrote it.

Link to comment
Share on other sites

16 hours ago, Arthmoor said:

That sounds about like the explanation I read elsewhere. Tannin doesn't seem too bothered by it though because even he's said his VFS was a solution looking for a problem to fix when he wrote it.

IMO it was never a problem to begin with, and the VFS trick is just cosmetic. 

When MO runs it nails up its VFS and immediately your game's Data folder is just as populated as it would be if you had installed all of the mods directly into the data folder. That's by definition what a union filesystem does: it overlays files from other locations into another directory. From the OS's point of view, it is a contiguous directory as if everything had always been there.  If you have a bunch of garbage mods installed you will still have garbage mods installed, you just can't see them when MO isn't running. 

Consider this analogy:

Bill: My car produces zero emissions!

Joe: But Bill your car is turned off

Bill: Yes but it produces zero emissions!

Bill cranks his car

Joe: Bill, your car is producing emissions dude

Bill turns his car off.

Bill: My car produces zero emissions! 

Link to comment
Share on other sites

True, but MO is correct in the assertion that once you shut it off, the Data folder is indeed devoid of mods and will be as you last had it before using MO. Some people seem to think this is somehow advantageous but the reality is it requires more work on the part of the user to keep things straight. With Windows 10 killing it off soon, those people are gonna be lost. Vortex won't be ready to take its place before that. It would be nice to be able to gain some new loyal users for Wrye Bash.

Link to comment
Share on other sites

Mod Organizer works fine on Windows 10 Production Ring (non developmental) does it not ? This issue with the VFS might just be in the Fast/Slow Rings because of extra hooks in place for debugging. Leaving Mod Organizer open while playing and/or disabling Windows Defender completely via local group policy or regedit and turning off UAC might be workarounds.

Link to comment
Share on other sites

8 hours ago, Sladen2019 said:

Mod Organizer works fine on Windows 10 Production Ring (non developmental) does it not ? This issue with the VFS might just be in the Fast/Slow Rings because of extra hooks in place for debugging. Leaving Mod Organizer open while playing and/or disabling Windows Defender completely via local group policy or regedit and turning off UAC might be workarounds.

MO works fine on the current Windows 10, MO2 works "ok" depending on your definition of "fine". ;)

I do hope that if it is possible to do a work around that somebody finds it because it still provides utility for a lot of people.  

 

 

Link to comment
Share on other sites

So to un-derail the thread since I so unceremoniously derailed it.. (my apologies)

@Utumno I saw your message on GitHub. I apologize for being slow.. I've been stupendously busy lately as I'm sure you have. I'm sitting on my Mac right now with the updated docs so I'll see what I can do in a bit.

As for the thread:  I'm going to be working on documentation for a bit because I think docs are needed right now, particularly if we have a bunch of MO refugees coming over.  We're going to need to focus on MO to WB migration documentation. Basically "How do I do this in Wrye Bash?" answers.  I've brainstormed a list but I need input because I don't use MO so I don't know exactly what most of the questions will be. 

So here's my brainstorm list of docs that I want to write:

  • MO to WB FAQ  (tree style, with answers in the docs below)
  • Explaining the CRC-based tracking system in simplified layman's terms.
  • Mod grouping (using markers)
  • Mass-installation of mods (hold down shift or ctrl then click the mods, then right click and pick "install")
  • Save profiles and how to diff plugins and load plugin masters for a particular save.
  • How to unpack a mod.
  • What a BAIN package should look like, and how to politely complain to mod authors that don't use agreed-upon community standards. 
  • How monitor BodySlide to capture all of the output it creates.
  • How to monitor the CK to capture FaceGen or anything else it spits out (plugins, BSAs)

There are also some existing documents that exist (using WB for DynDOLOD, Cleaning masters + WB) that we could add to the FAQ. 

Does anyone else have any suggestions or ideas?

 

  • Like 2
Link to comment
Share on other sites

Greetings to all after a larger period of absence.

I was successful in compiling a test release-version of Wrye Bash based on Utumnos scandir branch. It should offer significant performance improvements when reading directories.

We need help with testing if it works on different system configurations and to see if scandir is correctly recognized everywhere. To see if Wrye Bash is using scandir a new startup message was added to the debug log:

Wrye Bash starting
Using Wrye Bash Version 307 (Standalone)
OS info: Windows-7-6.1.7601-SP1
Python version: 2.7.13
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5

(For how to generate a debug log see the readme)

Link to the Standalone Zip, Link to the Installer

Thanks to all who find the time to test this.

Link to comment
Share on other sites

@Daidalos I'm willing to give it a shot .. If I know where this is coming from in the development timeline ..

Not familiar with Utumnos Scandir branch, so how does that relate to the Nightly builds ( 307 WB Wip Standalone ) at the end of the second post ( I use the Installer from Dropbox to install Standalone separately for each game Oblivion / Fallout 4 / Skyrim SE, and do not have any Python setup these days )

Is not testing this also going to hold up the next beta we were expecting to be possibly the next thing?, and is that still going to be relz and we would then have to sort of revert back to it when it does get released, away again from (your fork?, or is this a combined effort with Utumno using the latest of his bleeding edge code that has been discussed in another conversation somewhere)

Link to comment
Share on other sites

The utumno-scandir branch is a little bit behind dev, the last update was commited on 2017-04-23.

As for holding up the next beta, scandir implementation is listed for the 307 Milestone as Issue 371. I was out of the loop for a little while, so I do not know what was planned for the next beta release. I just saw the issue, tried my hand at compiling the branch and had more success than Utumno had had, so he asked me to share this version to see if it also worked for other people.

At this point I think it's not about doing some detailed testing but rather seeing if it runs at all. In the version Utumno compiled he recieved the following exception:

Unhandled exception at 0x740B8FAC (msvcr90.dll) in Wrye Bash.exe: 0xC0000005: Access violation reading location 0x0023FFFF.

My compiled version works for me locally and on a virtual machine. The question now is whether this is a mater of the configuration when building the exe or if it depends on the system running the exe.

Link to comment
Share on other sites

Brrr scary territory :), but here we go : Windows 10 Creators update x64 (but not the insiders ring which will be messing up other Mod Managers VFS .. yet)

Open Spoiler for results :

Skyrim Special Edition Wrye Bash Standalone runs fine (with a little error on LOOT API)

Wrye Bash starting
Using Wrye Bash Version 307 (Standalone)
OS info: Windows-10-10.0.15063
Python version: 2.7.13
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5
input encoding: None; output encoding: None; locale: ('en_GB', 'cp1252')
filesystem encoding: mbcs 
Searching for game to manage:
bush.pyo   70 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   72 _supportedGames:  oblivion: e:\oblivion
bush.pyo   72 _supportedGames:  skyrim special edition: D:\Steam\steamapps\common\Skyrim Special Edition
bush.pyo   72 _supportedGames:  fallout4: D:\Steam\steamapps\common\Fallout 4
bush.pyo  130 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  136 _detectGames: Set game mode to skyrim special edition found in parent directory of Mopy:  D:\Steam\steamapps\Common\Skyrim Special Edition
bush.pyo  171 setGame: No preferred game specified.
bush.pyo  151 __setGame:  Using skyrim special edition game: D:\Steam\steamapps\Common\Skyrim Special Edition
mods_metadata.pyo   40 <module>: Failed to import the loot_api module: (No module named loot_api)
testing UAC

 

Fallout 4 Wrye Bash Standalone runs fine (again has an error on LOOT API)

Wrye Bash starting
Using Wrye Bash Version 307 (Standalone)
OS info: Windows-10-10.0.15063
Python version: 2.7.13
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5
input encoding: None; output encoding: None; locale: ('en_GB', 'cp1252')
filesystem encoding: mbcs 
Searching for game to manage:
bush.pyo   70 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   72 _supportedGames:  oblivion: e:\oblivion
bush.pyo   72 _supportedGames:  skyrim special edition: D:\Steam\steamapps\common\Skyrim Special Edition
bush.pyo   72 _supportedGames:  fallout4: D:\Steam\steamapps\common\Fallout 4
bush.pyo  130 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  136 _detectGames: Set game mode to fallout4 found in parent directory of Mopy:  D:\Steam\steamapps\Common\Fallout 4
bush.pyo  171 setGame: No preferred game specified.
bush.pyo  151 __setGame:  Using fallout4 game: D:\Steam\steamapps\Common\Fallout 4
mods_metadata.pyo   40 <module>: Failed to import the loot_api module: (No module named loot_api)
testing UAC

 

Oblivion (GOG Manual Install Version) Wrye Bash Standalone - CTD on first run

Wrye Bash starting
Using Wrye Bash Version 307 (Standalone)
OS info: Windows-10-10.0.15063
Python version: 2.7.13
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5
input encoding: None; output encoding: None; locale: ('en_GB', 'cp1252')
filesystem encoding: mbcs 
Searching for game to manage:
bush.pyo   70 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   72 _supportedGames:  oblivion: e:\oblivion
bush.pyo   72 _supportedGames:  skyrim special edition: D:\Steam\steamapps\common\Skyrim Special Edition
bush.pyo   72 _supportedGames:  fallout4: D:\Steam\steamapps\common\Fallout 4
bush.pyo  130 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  136 _detectGames: Set game mode to oblivion found in parent directory of Mopy:  E:\Oblivion
bush.pyo  171 setGame: No preferred game specified.
bush.pyo  151 __setGame:  Using oblivion game: E:\Oblivion
mods_metadata.pyo   40 <module>: Failed to import the loot_api module: (No module named loot_api)
testing UAC
__init__.pyo 3430 __init__: Fatal error constructing 'Mods' panel.
Traceback (most recent call last):
  File "Wrye Bash Launcher.pyw", line 88, in <module>
  File "bash\bash.pyo", line 446, in main
  File "bash\basher\__init__.pyo", line 4005, in Init
  File "bash\basher\__init__.pyo", line 3646, in __init__
  File "bash\basher\__init__.pyo", line 3425, in __init__
  File "bash\basher\__init__.pyo", line 1737, in __init__
  File "bash\basher\__init__.pyo", line 253, in __init__
  File "bash\basher\__init__.pyo", line 213, in __init__
  File "bash\balt.pyo", line 1686, in __init__
  File "bash\balt.pyo", line 1799, in PopulateItems
  File "bash\balt.pyo", line 1744, in PopulateItem
  File "bash\basher\__init__.pyo", line 767, in <lambda>
  File "bash\bolt.pyo", line 171, in formatDate
ValueError: (22, 'Invalid argument')

 

Edit : Forgot to mention while I had SSE and FO4 Wrye Bash up and running, I went through changing to all the different tabs .. Which seemed fine ( I just didn't dabble with any right click menus or functions with it being quite old, I dont want my current installations going too Pete Tong at the moment - I'm now going back to my nice comfortable nightly build :) )

Link to comment
Share on other sites

Windows 10 Creators update x64 here too (also not the insiders ring). Daidalos' build works with both Oblivion and Fallout 4, and the installers scan is WAY faster (that's a technical term) than Utumno's WIP - so much so that someone should verify that all of the contents are really being scanned. If so, this is a tremendous leap forward. Congratulations! :)

EDIT: It turns out the "someone should verify" would be me. I randomly deleted and otherwise messed with installed files from deep within the most complex installers I could find and the new scan found every one of them. All I can say is, WOW! ...and THANK YOU! :)

Link to comment
Share on other sites

I'm on the Insider's Ring running the Creator's Update that breaks MO/MO2: version 1703  Build 16251.1002.

Edit: OK it was easier than I thought. I was doing some testing with my Skyrim LE installation so I goosed it by moving about 35GB of stuff back into my Bash Installers directory to put it through it's paces.

First my BashBugDump.log:

 

Wrye Bash starting
Using Wrye Bash Version 307 (Standalone)
OS info: Windows-10-10.0.16251
Python version: 2.7.13
wxPython version: 2.8.12.1 (msw-unicode)
Using scandir 1.5
input encoding: None; output encoding: None; locale: ('en_US', 'cp1252')
filesystem encoding: mbcs
Searching for game to manage:
bush.pyo   70 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   72 _supportedGames:  oblivion: D:\Games\Steam\steamapps\common\Oblivion
bush.pyo   72 _supportedGames:  skyrim: D:\Games\Steam\steamapps\common\Skyrim
bush.pyo   72 _supportedGames:  skyrim special edition: D:\Games\Steam\steamapps\common\Skyrim Special Edition
bush.pyo   72 _supportedGames:  fallout4: D:\Games\Steam\steamapps\common\Fallout 4
bush.pyo  130 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  136 _detectGames: Set game mode to skyrim found in parent directory of Mopy:  D:\Games\Steam\SteamApps\common\Skyrim
bush.pyo  171 setGame: No preferred game specified.
bush.pyo  151 __setGame:  Using skyrim game: D:\Games\Steam\SteamApps\common\Skyrim
mods_metadata.pyo   40 <module>: Failed to import the loot_api module: (No module named loot_api)
testing UAC

 

Stats:

Bash Installers:
259 mods
37GB
18,026 files

Skyrim Data folder:
14GB
1754 files

From mods tab to BAIN tab - 1:10 Seconds

That's a first-time scan of new files that weren't in the manifest, so that's really nice. :)

 

Edit 3: Meh my apologies.  Apparently the end of the spoiler tag didn't take so the end of my post with the stats are inside the spoiler.  I can't fix it, but my results were really nice. :)

Link to comment
Share on other sites

Thanks @Daidalos - beta has  a steady slow pace - I need to merge stuff and there are small details to be adressed - if meanwhile you verify the scandir commit it would be very welcome as it does indeed add to performance - Daidalos could you try rebasing over my wip branch and compile this ? @alt3rn1ty's oblivion error is fixed there (actually on dev also but let's go all the way)

Link to comment
Share on other sites

The results so far seem quite encouraging.

@Utumno Ok, I'll rebase utumno-scandir over your wip branch and try to compile it. Should I push it if it works or should I create my own branch of utumno-scandir for that?

Aside from that I have a question regarding packaging wrye bash with the loot api. The wiki guide to making releases has no mention of the loot api, but the build failed when I did not have loot_api.dll in my Mopy\ folder, so I downloaded the latest loot api release from the github page, pasted loot_api.dll and loot_api.lib into \Mopy\ and the build completed. However it would appear that the loot api is not recognized in the standalone version. Is there something else I need to do?

Link to comment
Share on other sites

34 minutes ago, Daidalos said:

I did not however manage to pack loot api with it.

Edit: scratch what I said earlier I must be suffering from the early stages of heat stroke.

I don't need the LOOT API for what I'm doing at the moment with Wizard, Bodyslide, BCF, and MEI testing.

I'm off to do my regular series of tests. I'll report back shortly. :beer:

Link to comment
Share on other sites

OK I did a few rounds of testing. My main impression while using it is how quick it seems. So for my results:

  • BAIN Conversion File handling and progress meter is functioning correctly. CRC of converted mod package matches origin BAIN.
  • Wizards appear to be parsing correctly.  No bug checks thrown or image issues found from either compressed or uncompressed packages.
  • Manage External Installation is functioning correctly.  

Bodyslide data is not being detected however.  This may be due to the commits being slightly out of sorts at the moment, but I wanted to make you aware of it.

Bodyslide and Outfit Studio In 201708021813 - subpackages 00 to 02 that contain bodyslide data are indexed:

Spoiler

WB-20170802-Utumno.PNG.1acea7b903420cb6fae9640d10cf093b.PNG

In this build:

Spoiler

Daidalos_307_merge.PNG.30b44db554332cef3625975ec91c2541.PNG

Everything else I've tested so far has been great. :) I'd like to test scandir with my SSE build since that is going to give it a real work-out. I've got about 100GB of mods and 70GB of data in my SSE data directory. I'm going to hold off for now because without the BodySlide support being rolled in it would orphan about 6GB worth of BodySlide data.

Edit: my bad, I was wrong. I've got about 130GB of mods.  I admit I'm an addict.

Spoiler

59a343f10e065_Mertzsmonsterbashinstallersfolder.PNG.450b1f7c9eba1a191f3022176861d717.PNG

 

Link to comment
Share on other sites

The consensus seems to be that installing additional files for tools will be allowed and if that's what was decided, fine but I have concerns.  My issue is that Wrye doesn't install the EXE files so until you manually copy those to the data folder, you can't even use the files you installed.  Once you copy the EXE files if you run the Clean Data routine all those files will be cleaned out.

I don't understand the screen shot. It shows txt and xml files.  Those files are currently ignored.  I'm also concerned because in the past Wrye installed just files that come from Bethesda, like in the BSA files.  Not every single file type because it belongs to something that is already non standard, and was never allowed to install in the past.  If we start adding all manner of files like txt and xml files as valid files then Wrye will start installing all sorts of additional and unneeded files that aren't related to BodySlide at all.

In the interest of trying to be unbiased, if txt and xml files are required for Bodyslide then add something to the INI file. For example, when NMM and Wrye Bash share the same folder for installers and downloads NMM will create many folders that aren't needed. So sSkippedBashInstallersDirs was added for cache, categories, downloads, ModProfiles, and ReadMe. Maybe add sAllowedFileExceptions=.xml|.txt.  The only problem is, I don't want to start seeing files installed into the data folder whether I want them or not. Will I start seeing Info.xml and ModuleConfig.xml in my data directory?  I don't see how you can include xml and txt files for only the Bodyslide folder. Would you need to add additional arguments to the INI change such as adding sAllowedFileExceptions0=[Bodyslide]|.xml|.txt, sAllowedFileExceptions1=[Caliente]|.whatever|.whatever and the like?  Where the numerical value is added to an array (so users can have more then one and customize it to their needs), the name in brackets is the folder that you can install .xml and .txt files into.  Which should prevent them from installing into other locations.

Link to comment
Share on other sites

@Sharlikran That is all defined here http://wrye-bash.github.io/docs/Wrye Bash Advanced Readme.html#bain-skipped

When a new folder is added to the allowed list, which is game specific, anything except the skipped extensions can be installed in it by Wrye Bash.

We already need .txt to be allowed for unusual needs, the prime example being tree / billboard mods which need to have there data .txt files installed for generating LOD and keeping any particular tree at the correct height in relation to the ground it stands on locally. In that case we have to select Override Skips (where ReadMe's have been chosen to be skipped, this rule needs to be overridden to allow the .txt files) to ensure they install.

I think the exe files in the case we are talking about have to be installed outside of data. And there are no unusual .dll files going to be allowed either (like OBSE / SKSE / F4SE / Pluggy / SkyProc / ScriptDragon ), there are just .nif .osd .osp .xml files, which are all allowed and will not be cleaned by Clean Data

And no similar files would install from say a FOMod folder, because the FOMod folder itself is not on the allowed list so anything in it, is ignored.

 

I personally dont like that CBBE has a Core files sub-package that includes a Tools folder, it would have been better as an optional sub-package that we could choose if we do not have bodyslide installed (its redundant files I dont need), but thats the authors prerogative so I just manually prune the zip of files I dont want installed while I am adding Beermotors wizard to the package.

 

.exe files manually installed would be deleted by Wrye Bash Clean Data command .. but then that is for the user to beware as it always has been using that command (hence the pop-up warning Clean Data gives before going ahead) : Bodyslide is a bit unique/special in its installation requirements I think.

All that has been allowed for bodyslide recently in Wrye Bash is a couple of folders, enabling its data files to install (for mods like CBBE), anything else exe wise has still to be manually installed. The Bodyslide package itself still has to be manually installed.

000.png

( IIRC it was Myk002 who made the good suggestion that we have such a warning for the odd occasion someone may have installed something manually that they may not realise will be swept out, prior to his suggestion Wrye Bash used to just clean data without a warning. People read it or not, their choice, but the swept files are easily recovered afterwards anyway even if they do still mistakenly clean out files they did not realise would be swept, because they also get backed up - In the case of exe's suddenly not working it will be obvious to the user something went wrong recently :) ).

Link to comment
Share on other sites

18 hours ago, Daidalos said:

The results so far seem quite encouraging.

@Utumno Ok, I'll rebase utumno-scandir over your wip branch and try to compile it. Should I push it if it works or should I create my own branch of utumno-scandir for that?

Aside from that I have a question regarding packaging wrye bash with the loot api. The wiki guide to making releases has no mention of the loot api, but the build failed when I did not have loot_api.dll in my Mopy\ folder, so I downloaded the latest loot api release from the github page, pasted loot_api.dll and loot_api.lib into \Mopy\ and the build completed. However it would appear that the loot api is not recognized in the standalone version. Is there something else I need to do?

Wiki needs editing - see issue 329 for details. You need the dll ad the pyd. @Wrinklyninja

Everybody don't use much energy in testing scandir - it will work, we just replace a method for a faster one. The issue is if it plays well with packaging the standalone.

Will see what broke in bodyslide - @Daidalos just force push utumno-scandir rebased on utumno-wip so I can see what's in there

Re: bodyslide (issue 379) - any luck  @Beermotor in pushing the wiki ? There we should have all needed info/details.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...