[UPDATE] BSAs and You

[VaultDuke]

yeah, ok. but suppose I have a BSA and want to check whether that is fine or not? any way to retrieve that data from back from the BSA?

[alt3rn1ty]

No tools that I know of apart from using a Hex editor and figuring out a byte code which is unique to each games BSA format

 

Edit : Did some tidying up of the first post, couple of typo's, spacing and font sizes for clarity, added a link and spoiler reference Fireundubh's findings to the end of the post as to which original bethesda created BSAs did not have any compression when they were made ( which I am pretty sure means some ( if not all ) files in those needed to be in an uncompressed BSA ).

[Utumno]

Bonjour!

 

Does anyone remember here why we go into the pains of resetting BSAs timestamps to 1/1/2006 for oblivion (but not for skyrim) ? Was some kind of voodoo to cure some kind of bug ?

[VaultDuke]

That timestamp would be used for load order in oblivion and earlier

[alt3rn1ty]

Hi Utumno - Wow its been a while since I thought about Oblivion Load Ordering - History on this one is in old OBMM documentation ..

 

A Blast from the past is here : http://timeslip.users.sourceforge.net/obmmm/archiveinvalidation.htm

 

 

Reset BSA timestamps 
Oblivion only loads files from the data folder instead of the BSA if the file was modified more recently than the BSA. After installing any oblivion patches, the BSA timestamp is increased, so many mods will not work correctly. Use this option to reset the timestamps, so all data files are once again newer than the BSAs

( For "patches", read official and DLC )

 

The full page there copied here just in case :

 

archiveinvalidation 

obmm's archiveinvalidation utility contains all the functionality related to oblivions archiveinvalidation.txt file. 

For most cases, the default settings should be sufficent for ensuring correct mod operation. Just check the 'autoupdate on exit' checkbox to turn the functionality on. 

Reset BSA timestamps 
Oblivion only loads files from the data folder instead of the BSA if the file was modified more recently than the BSA. After installing any oblivion patches, the BSA timestamp is increased, so many mods will not work correctly. Use this option to reset the timestamps, so all data files are once again newer than the BSAs 

Mode 
This controls the type of archiveinvalidation file generated by obmm. 

BSA invalidation: 
This is the old official way of handling archiveinvalidation. It works well in most cases, but if a mod replaces something with a common file name then all other files with the same name may turn purple, regardless of the directory they are in. This is an oblivion bug, so using this method is not recommended. Use it only if you have some phobia of allowing obmm to edit your BSA files. 

Universal: 
Creates an archiveinvalidation file containing the line 'Textures/ \s'. This has been proven not to do anything, but some people swear by it anyway, so I've left it in for them. 

BSA alteration: 
This is < was > the default way of handling archiveinvalidation, originally invented by me. In this case, instead of creating an archiveinvalidation file, the BSA archives are modified to prevent oblivion from finding the original file. There are two issues with this method: 
1) Any oblivion patches will detect the changes and refuse to patch the BSA archives. This can be avoided by clicking the 'Remove BSA edits' button before attempting to install any patches. 
2) Some files suffer hash collisions, and cannot be edited by this method. obmm can generate a mini archiveinvalidation file containing only these entries. 
To avoid BSA corruption, you should use remove the BSA edits before doing anything that may want to alter the BSA's. This includes uninstalling/reinstalling oblivion or installing patches. 
In the event of BSA corruption, you can use the 'BSA uncorrupter' utility to restore the BSA to a usable state.

USE BSA Redirection instead, a lot more reliable and safer if you value your sanity .. Some older advice which needs editing is still out there on the internet, do not follow it. OBMM can become lost in its granular management of installed files, and what should / should not be invalidated with alteration ( complicated more by users not strictly adhering to the advice above about using Remove BSA Edits for every mod uninstalled ).

BSA redirection: 
This is the very new way of handling archiveinvalidation, originally invented by Quarn. It's a fire once and forget method that uses an empty bsa as a target for oblivions bugged invalidation code, allowing the real BSA's to work as normal. This is likely to < has > become the new default method once some issues regarding facial textures are ironed out. 

File types to include 
This controls which types of files to include in the archiveinvalidation file. Textures should be the only file type required. Note that including meshes in the archiveinvalidation file can cause oblivion to crash. 

Ignore normal maps 
If this is checked, normal maps will not be added to the archiveinvalidation file. In Edit BSA mode, normal maps in the BSA will not be edited. 

Only include files which already exist in BSA archives 
If this is not checked, all files in the data directory will be added to the archiveinvalidation file regardless of if they're actually overriding something. If it is checked, files will only be added to the archiveinvalidation file if a file with the same path and name already exists in a BSA archive. This option only has an effect in standard mode. 

Only include files with matching extensions 
If this is checked, obmm will check that files have the appropriate extensions for their folder before including them in the archiveinvalidation file. For example, anything in the meshes folder must have the extension '.nif'. This only has an effect in standard mode 

Edit BSA entries regardless of type 
This option only has an effect in edit BSA mode. If this is checked, all entries in the BSA files will be modified regardless of the file type. If this isn't checked, the 'File types to include' setting takes effect on the BSA archives themselves, as well as on the mini archiveinvalidation file. 

Generate archiveinvalidation entries on hash collision 
This option only has an effect in edit BSA mode. If this is not checked, the mini archiveinvalidation file will not be generated. 

Display warning on hash collision 
This option only has an effect in edit BSA mode. If this is checked, a warning dialog will be displayed each time obmm encounters a hash collision. 

Pack face textures into redirection target BSA 
This only has an effect in BSA redirection mode. If this is checked, instead of creating an empty BSA, the contents of the textures\faces folder will be included. 

Autoupdate on exit 
If this is checked, the archiveinvalidation file will automatically be updated every time obmm is shut down. 

Update now 
Click this to update the archiveinvalidation file there and then. It will display a small dialog containing the results. 

 

Ordinarily Wrye Bash just increased plugin timestamps for Load Ordering, so there also needed to be a special context command for the above problem in Wrye Bash aswell as OBMM. I think it should still be in the Oblivion instance of Wrye Bashes context menus somewhere, I am pretty sure it used to be anyway. Possibly made automatic at some stage because there was every reason to do that, and no reason not to.

 

 

Skyrim obviously adopted the newer plugin / loadorder .txt file Load Ordering ( after the crowd shouted at Bethesda loud enough )

 

 

And the date of 1/1/2006 is specifically the furthest back anyone would reasonably expect there to be any plugins out there with a timestamp on for Oblivion, and when Wrye Bash re-orders plugins it gives them all the same timestamp + 1 minute more than the previous plugin in the load order.

 

 

I think at one point it was also tied in with the old archiveinvalidation method of "BSA Alteration" which is no longer used ( preferring these days the much more reliable and trouble free "BSA Redirection" ), and or sorting out issues where another mod manager ( OBMM at the time, more recently OBMMex or NMM ) had set them differently to Wrye Bash

 

 

Another good reference / read on archiveinvalidation is TomLongs old TesCOSI site ( which has now been moved to TheAssimilationLab to preserve and update it ) http://wiki.theassimilationlab.com/tescosi/BSAs_and_Archive_Invalidation

 

And another blast from the past good reference, is Dev_AKM Devnull site, ArchiveInvalidation :

http://devnull.sweetdanger.com/archiveinvalidation.html

If you scroll down to "Technical Details", you will find an explanation on the way Oblivion invalidates entries in its BSA hash tables. BSA Alteration which used to be OBMMs modus operandi, edited the hash tables. So aswell as occasionally losing the plot on what files it had installed, OBMM could really stuff up this procedure too causing some very complicated and sometimes irrecoverable problems needing a full re-installation of the game.

Hence my editing in the spoiler above reference do not take any old advice on using BSA Alteration, its a nightmare waiting to happen. Just use BSA Redirection for Oblivion, as Wrye Bash does by default these days.

 

ArchiveInvalidation

 

 

Everything You Never Wanted to Know And Were Afraid to Ask 

 

 

ArchiveInvalidation Destroyed
ArchiveInvalidation Explained

 

ArchiveInvalidation Destroyed

 

by dev_akm
Updated 1-29-2008

All of the original threads on the Elder Scrolls Forums have now expired and been purged from the system, but this summary of the great work done by the ArchiveInvalidation Explained Research Team lives on! The recent discovery of an entirely new method is now described as well.

Overview

Oblivion normally gets all of its media assets (artwork, spoken dialog, music, etc.) from within a few large .bsa files rather than in individual folders (such as Meshes, Textures, Sound, etc.). The original game and official plugins use these .bsa files (it stands for Bethesda Softworks Archive). These .bsa files are organized internally just like your "Data" folder is.

Although fan-made mods are starting to use .bsa files more often, most mods still place individual meshes, textures, etc. into folders within your "Data" directory. This isn't a problem for new items added by a mod, but it can be a problem for some "replacer" mods that alter original game items. For example, some of the most popular types of mods are "armor texture replacers" and "landscape texture replacer" mods that improve the look of the game by replacing the original "stock" textures.

Normally, this isn't a problem since Oblivion is designed to automatically load any meshes, textures, etc. that it finds in your Data folder, as long as the timestamp (Modification Date) on each of these files is more recent than the timestamp on the original .bsa files. Since the .bsa files are older, items in them get replaced by any newer items with the same name that exist in your Data folders.

In some cases, however, this doesn't work correctly. It now seems fairly certain that this is caused by a bug in Oblivion's ArchiveInvalidation system itself. After more than a year and several official game patches, it seems unlikely that Bethesda will ever correct this bug. This means you will have to use one of several workarounds if you want to install any "replacer" mods.

Further details on this subject, as well as extensive testing results for ArchiveInvalidation problems in general, can be found in the article ArchiveInvalidation Explained (the ESF threads on the topic have expired).

Each of the known workaround solutions has advantages and disadvantages. The first two methods in the list below -- BSA-alteration and BSA-redirection -- are the only recommended solutions. The last two methods -- BSA-invalidation and BSA-extraction -- have such serious drawbacks that they are not recommended solutions. Here's the list of methods along with some description of what's involved in each:

•  BSA-alteration: This was the first true solution to the problem. It is widely used, very reliable, fairly simple, and most Oblivion mod users are by now familiar with using it. The disadvantages are that you must use one of several external tools (not a big deal since you probably need one of these tools anyway), you must update the BSA changes after installing or removing any replacer mods (although most of the tools will do this in an automated way), and you must remember to remove BSA changes before applying any of the official patches from Bethesda.
• BSA-redirection:  This is a newer solution that has proven to be very effective. This method has advantages that make it preferable over BSA-alteration. The biggest advantage is that it does not require any external tools (although it does require you to install some files and you may want to use a tool to help you configure it) and once it's set up, you can pretty much forget about it (no intervention is required after adding or removing any replacer mods). 
• BSA-invalidation: This was the first solution discovered, and as a result was the most widely used method for a long time. However, it has severe limitations and problems and is rarely used anymore because of this. Unfortunately, many early mods still recommend this solution, so lots of users get fooled into thinking it's a good idea. In fact, it's a terrible idea under most circumstances and the only way to make it work reliably is to also use BSA-extraction (see below). You should avoid this method if at all possible.
• BSA-extraction: This solution works well for mod creators (who often need to have the source files extracted from the BSAs anyway). However, it has major disadvantages for mod players and because of this it is not discussed in any great detail here.

Next we'll examine these methods in a bit more detail.

BSA-redirection

This method works by tricking the Oblivion game engine into using its bugged ArchiveInvalidation method on the wrong BSA file. By doing this, it allows all files in the main Textures BSA to be reliably superseded by any more recent files found in your Textures folder.

The easiest way to use this method is to use the BSA-redirection option in Oblivion Mod Manager if you already have it (see BSA-alteration below for more details). Here are the steps using OBMM:

1.    Start OBMM.
2.    Click Utilities.
3.    Select Archive invalidation. You'll get a popup window.
4.    Click BSA redirection (it's the default choice).
5.    Click Update Now.
6.    Close the Archive invalidation popup (click the red X in the upper-left corner).
7.    Quit OBMM or click Launch Oblivion.

Alternately, you can install a mod specifically designed to do BSA-redirection , such as ArchiveInvalidation Invalidated!.

You can also set up BSA-redirection fairly easily yourself if you don't mind a bit of tweaking. In the Oblivion.ini file, there is a configuration entry called the SArchiveList. It lists all of the standard BSA files. It normally looks like this:

SArchiveList=Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa, Oblivion - Sounds.bsa,
Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

By changing the list to include another BSA containing at least one texture, you can trick the game engine into processing the bugged ArchiveInvalidation logic for that new dummy file rather than the real Textures BSA (called Oblivion - Textures - Compressed.bsa). So, the modified SArchiveList should look something like this:

SArchiveList=DUMMYFILE.bsa, Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa,
Oblivion - Sounds.bsa, Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

To do this, you'll need to either use a BSA file that came with some other mod to be your DUMMYFILE.bsa (any BSA file containing at least one texture should work), create one yourself using a tool like BSA Commander (you only need to put one texture in the BSA).

BSA-alteration

BSA-alteration requires Wrye Bash (as of version 0.60) or one of Timeslip's excellent utility programs: Oblivion Mod Manager (OBMM version 0.7.10 or later) or BSA Patcher. What Wrye Bash, OBMM and BSA Patcher do is to sidestep the ArchiveInvalidation problem by making Oblivion think it never had a copy of the textures you are replacing.

In other words, these utilities edit your Textures BSA so that Oblivion cannot find the original version of texture files you have replaced, thus forcing the game to load the replacements instead of the originals. Without getting too technical, this works much like taking a book and scrambling some entries from the table of contents so those chapters can't be found anymore, even though the chapters are really still there -- thus allowing the process to be easily reversed later by restoring the table of contents back to its original state.

This method of invalidation was initially tested by the ArchiveInvalidation Explained Research Team and has now been adopted as the de facto standard solution by the Oblivion mods community. It has proven extremely stable, but you should still make a backup of your BSA files if you have the hard drive space (or a DVD-burner) just in case something goes wrong (like you accidentally try to run both OBMM and BSA Patcher on your BSA files, which will result in a corrupt file).

OBMM and BSA Patcher will keep track of which files have been renamed in your BSAs and provide a "Remove BSA edits" function to rollback any changes they have made to your BSA. 

Wrye Bash

Download Wrye Bash.

Wrye Bash provides a Replacers Tab that lets you easily add/remove texture replacement mods. It keeps track of which files have been renamed in your Textures BSA and provides a Restore button to rollback any changes it has made to your BSA (if you don't like the results for some reason). You can use the Replacers tab in several different ways:

• Manual update: Click the Update button after adding/removing replacers. This also works on any replacers you already have installed manually or via another program (like OBMM).
• Automatic update: Check the Automatic checkbox. The Textures BSA will be automatically updated whenever you add or remove a Replacer. Only works for texture mods installed/removed using the Replacers tab.

You should probably make a backup of your BSA files if you have the space (or a DVD-burner) just in case, but so far nobody has reported any corruption issues.

Wrye Bash requires Python 2.5 and wxPython 2.8.0.1 ANSI to work.

Assuming you have Wrye Bash 0.61 or later installed already, and have already installed a one or more texture replacement mods, the steps you need to take are as follows:

1. Start Wrye Bash.
2. Click Replacers tab.
3. Click the Update button under Archive invalidation.
4. Quit Wrye Bash

Note that Wrye Bash checks to make sure you don't already have any active BSA alterations from OBMM or BSA Patcher before allowing you to use this feature.

Oblivion Mod Manager

Download Oblivion Mod Manager

Assuming you have OBMM 0.7.10 or later already, and have already installed/activated some mods, the detailed steps for doing this in OBMM are as follows:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Directly Edit BSAs;
5. Click the checkbox for autoupdate on exit and/or click Update Now;
6. Click the red X in the upper-left corner of the Archive invalidation popup to close it;
7. Click Launch Oblivion
8. Play!

If you didn't check the box for autoupdate on exit, then repeat this process after each time you finish installing/activating some new mods. You only need to do this once after you finish installing all the texture replacers you want to play with. You never have to worry about it again until you install more texture replacers.

The latest version of OBMM (0.7.11 or later) provides a lot of new options for the Directly Edit BSAs function. The default options may be okay, but you need to at least make sure to check the box for Generate archiveinvalidation entries on hash collision, since that setting is critical.

Here are the ideal OBMM BSA Settings (for 0.7.11 or later):

Using those settings will take care of everything for you.

As already noted, it's completely up to you whether you want to use autoupdate on exit or Update Now, or both. I find that when running with a lot of texture replacements installed, the autoupdate on exit option tends to cause an annoying delay when quitting OBMM or using the Launch Oblivion button. Since I've gotten into the habit of using the Launch Oblivion button instead of the normal launcher, and I don't install new mods nearly as often as I start Oblivion, I decided to avoid using autoupdate on exit and instead always use Update Now after I install a new mod.

If you later uninstall/deactivate some mods, you'll need to use the Remove BSA editsoption. Follow these steps:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Remove BSA edits;
5. Close OBMM;
6. Delete texture files that you want removed;
7. Start OBMM;
8. Click Utilities;
9. Select Archive invalidation;
10. Click Directly Edit BSAs;
11. You know the rest by now

If you're using .omods, you should skip Steps 5-7 and just Deactivate the .omod instead.

BSA Patcher

Download BSA Patcher

Unlike OBMM, BSA Patcher is a command-line application, although you can launch it by double-clicking the executable. BSA Patcher doesn't include as many configuration options as OBMM, but it accomplishes the same tasks and it only requires .NET 1.1, which means it will work for anyone who is unwilling or unable to run .NET 2.0. It will also work with mono, an open-source version of .NET 1.1.

The latest version of BSA Patcher (as of this writing) uses the BSA alteration code from OBMM 0.7.10 and, like OBMM, it will automatically generate an ArchiveInvalidation.txt file for you containing only the textures that could not be safely renamed in your BSA files. It includes support for several command-line options, but you can safely ignore them.

To use it, place BSAPatch.exe in your \Oblivion\Data\ folder and double-click it once to rename files in your BSA. Run it again to restore them to their original names.

NOTE: If you use either OBMM or BSA Patcher, you should not manually alter your ArchiveInvalidation.txt file. Let one of these programs do it for you. No other solutions are recommended. Never use OBMM and BSA Patcher at the same time.

Older Versions of BSA Patcher
If you have an older version of BSA Patcher, make sure you have at least the version based on code from OBMM 0.7.9. You should probably still make a backup of your BSA files before running it, just in case. Run it once to rename files in your BSA and run it again to restore them to their original names.

If you're not sure what all this is about, don't have any mods installed yet, or just need more information on how to use mods in Oblivion, please read the Oblivion Mods FAQ for more general instructions. 

For more information on ArchiveInvalidation, see ArchiveInvalidation Explained. The results of our testing are summarized neatly there.

UPDATES

The 0.7.10 and later version of OBMM correct an issue from previous versions where several files with very similar names in the same directory (like castleintcolumndetail02.dds, castleintcolumndetail03.dds, and castleintcolumndetail04.dds) might get their textures mixed up if a replacer only replaces one of these files. OBMM now generates an ArchiveInvalidation file for only the textures that could not be renamed.

This feature is now also included in the standalone BSA Patcher, so you don't have to create your own ArchiveInvalidation.txt file for those problem textures by hand.

Go get the update from the links above! NOTE: If you previously used 0.7.8 to patch the BSAs, you must restore them with 0.7.8 before installing 0.7.10 (or later). If you forget to do this and then replace your messed up BSAs with backups, you have to delete the BSAEdits file in your Oblivion\obmm folder before it will work correctly again.

OBMM/BSAPatcher Usage Example

The following example assumes you are using OBMM 0.7.10 along with Parallax Cities 1.2 (all three parts) and LBM Weathered Direction Signs 1.3.

After clicking Directly Edit BSAs and then Update Now, OBMM reports changing 821 entries in 3 BSA files and creates an ArchiveInvalidation.txt file containing only three texture entries:

textures/architecture/castleinterior/castleintcolumndetail03.dds
textures/architecture/skingrad/skingradstone06.dds
textures/architecture/bravil/woodplanks02.dds

 

These three entries are the only textures that could not be safely renamed in the BSAs, so they are invalidated here instead.

Other Solutions

If you don't use BSA-alteration or BSA-redirection for some reason, then the only way to convince Oblivion to load these problem textures from the individual folders rather than from the .bsa files is to specifically force it to do this by creating a special file called (you guessed it) "ArchiveInvalidation.txt".

The ArchiveInvalidation.txt lists the relative path (from the "Data" folder) of texture files you want to replace the default artwork shipped with the game.

As a result of the extensive testing reported in the ArchiveInvalidation Explained threads, we now have very clear information about how ArchiveInvalidation works and how it fails to work. Suffice to say that it can get extremely complicated.

My current best practice, based on the above-mentioned research, is that ArchiveInvalidation.txt should only list just the DDS textures from the original game that are being replaced by individual files in your Data folders. In other words, this means it should only contain entries for texture replacement mods. No meshes, sounds, or voices should be listed.

Be aware, however, that many earlier approaches recommended listing ALL your moded meshes/textures/etc. in ArchiveInvalidation.txt, regardless of whether they replace anything from the original game.

If you do use any texture replacement mods, there is a high probability you will need an ArchiveInvalidation.txt file listing these replacements (if you don't use the OBMM's "Directly Edit BSAs" feature). Without it, chances are that many of Bethesda's default textures will continue to be loaded. Also, if you have bad entries in your ArchiveInvalidation.txt file, or even in some cases if your ArchiveInvalidation.txt file is perfectly correct, you may see items in the game appearing purple or even invisible.

Given all these warnings, if you still want to try using an ArchiveInvalidation.txt file, there are several fan-created tools that will generate these file listings for you automatically.

ElChE's Oblivion Automatic Content Validator 1.0.2 (also called Automatic Content Validator) will create an ArchiveInvalidation.txt file for you automatically, and it also does some other very useful things like making sure all of your meshes have valid texture paths. Be sure to get 1.0.2 or later because it adds the option to only generate listings for replacer files. You may need to manually remove any meshes from the list it generates.

Oblivion Mod Manager, which we mentioned earlier, also has an excellent set of options for automatically creating the ArchiveInvalidation.txt. This process executes automatically every time you quit from OBMM or use OBMM to launch Oblivion. Make sure to get version 0.7.10 or later.

If you prefer to create ArchiveInvalidation.txt by hand, this will work fine, but you can probably guess that it will quickly become a huge problem to maintain all the file listings if you try new replacers very often.

If you do create it by hand, here's what you need to do.

Create a text file named "ArchiveInvalidation.txt" in your main Oblivion folder. This is not the "Data" folder, but is one level up from that, in the same folder with OblivionLauncher.exe and Oblivion.exe. The reason for placing it here is primarily to avoid accidentally overwriting it when you install a new mod.

(You can put the ArchiveInvalidation.txt into your "Data" folder if you want. It will work there as well, but be wary of accidentally overwriting it during mod installation.)

For the sake of creating an example, let's say you have two replacer mods, one that replaces the texture for Daedric cuirasses and one that replaces the texture for glass shields. Your ArchiveInvalidation.txt would look something like this:

textures/armor/Daedric/cuirass.dds
textures/armor/glass/shield.dds

Note that textures whose names end with "_g.dds" and "_n.dds" don't need to be listed. Other DDS files that don't need to be listed include those in the Menu directory and those in "icon" directories.

If your replacer mod came packaged with an ArchiveInvalidation.txt file, you can copy and paste the contents of the lines of that file into yours. If you do this, be careful not to accidentally paste over any lines that you've previously added for other replacer mods.

Many people claim to use a "universal" or "global" ArchiveInvalidation.txt. However, extensive testing by numerous people has determined that this approach does not work at all. It is the same as running with no ArchiveInvalidation.txt file. If someone offers advice saying you should use a file like this:

meshes/ \s
textures/ \s
sounds/ \s

Don't believe it. It's a great urban legend, but it doesn't work.

Modder's Resource

I have released a modder's resource containing sample ArchiveInvalidation instructions that mod creators can cut and paste into their readme files. It's a bit out of date now since it doesn't yet include instructions for using Wrye Bash or BSA-redirection, but you can find it here. 

Credits

This information would not exist without the help of a lot of people who contributed feedback, usage guidelines, suggestions, test results, and moral support in this thread, including:

* DoomedMarauder
* Malifrax
* vkaryl
* Zanderat
* DragoonWraith
* Skyfire76
* phoenixamon
* Fyrstig
* Fyndolyn
* MaGISTraTe
* Ismelda Lasombra
* ThePenquin
* Quarn
* and many more. Thank you all!
 

 

ArchiveInvalidation Explained

 

by dev_akm

Preamble
This is adapted from my ESF thread: ArchiveInvalidation Explained (2). Unfortunately, like all the previous ESF threads on this subject, that thread has now expired and been purged from the forum. The summary lives on here.

Last Updated: 10-13-2006 11:40 AM CDT

 

OVERVIEW

There is a terrible beast wreaking mass destruction and chaos in the mod community. It is called ArchiveInvalidation. The purpose of our mission here was to slay this horrid creature and prevent it from respawning.

This goal has now been accomplished, but historians (and a few others, perhaps) may still wonder: From whence did this beast arise? What spurred it forth from its dank lair and into our peaceful community? The answers lie within this document and its related discussion threads.

The problems with ArchiveInvalidation described here most often appear right now for people using more than one armor texture replacer, but it could crop up anywhere Oblivion uses common filenames like "boots.dds" for more than one set of things. If you have not experienced any problems yet, it is probably because your configuration does not meet these criteria.

The extensive testing carried out by multiple users and reported in these threads makes it fairly clear that Bethesda probably never intended for anyone to have to use ArchiveInvalidation in the first place. The vast majority of textures seem to load correctly from the Data folders even without using an ArchiveInvalidation file, which explains why so many users get duped into believing that a simple three-line global/universal/generic ArchiveInvalidation has solved all their problems when in reality it has been proven to have no effect whatsoever (it's the same as having no AI file at all).

In fact, we are proposing that the only reason it's necessary to use ArchiveInvalidation at all is due to a bug in the game that causes some texture replacements to fail to load, while others load just fine. Based largely on Malifrax's contributions, we've isolated these problems to certain common directory names and perhaps even certain NIF files (meshes) that appear to have a different structure than ones which load textures correctly.

Note that you don't need to read all of this if you are using OBMM 0.7.8 or later and you select the option to "Directly Edit BSAs". Timeslip's workaround avoids the need for ArchiveInvalidation entirely in our tests.

If, on the other hand, you do intend to use ArchiveInvalidation, or just want to understand what led to such drastic measures as Timeslip's solution, then you do need to read this.

Hopefully, Bethesda will issue a patch for the textures bug at some point and all of this will become an odd historical footnote in the life of Oblivion. Until then, read on ...

ARCHIVEINVALIDATION TEST RESULTS

1) ArchiveInvalidation.txt (AI.txt) is only needed for replacement textures. Also, it is most often required for replacement textures that are specified from within a mesh (NIF) rather than specified in the CS (icons and landscape textures, for example, will usually load without being referenced in AI.txt).

2) Universal/global/generic ArchiveInvalidation methods do not work. Period. End of discussion.

3) Mesh replacers load regardless of whether they are specified in your ArchiveInvalidation.txt (AI.txt), so you don't need to list any .NIF files. If you do list them, try to only list the specific meshes that are giving you problems, and be aware of the ramifications, as described in #4 and #5 below.

4) A single entry of a common filename like "boots.dds" in AI.txt will invalidate all paths containing a "boots.dds" file that exist in both the BSA and your Data folder. It will do this regardless of whether those paths in your Data folder actually contain a "boots.dds" file and regardless of whether you have an AI.txt entry for that path. *

5) Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *

If all this sounds really confusing, you're not alone. It's not easy to understand, even for computer experts, in part because it's such a crazy way of doing things that it seems in many ways like a bug in the invalidation code Bethesda implemented. Perhaps it is a bug, perhaps it is by design, we have no way of knowing.

* Credit for this discovery goes to Wonder Dog. 

HOW DO I MAKE ONE?

Instructions can be found in the Oblivion Mods FAQ.

Once you get it started, you can list either just individual filenames, such as "boots.dds", one per line, or list entire directory paths, one per line, like this:

textures/armor/Iron/ \s
textures/armor/glass/ \s

In the end, I think discussions about which approach is better - listing files with no path versus listing only paths - probably won't get us anywhere, since both methods are highly imprecise and cause similar errors. Pick one and stick with it or use a mixture of approaches. I don't have enough data yet to make a recommendation one way or the other.

WHAT NOW?

This is a terrible situation any way you look at it. You can extract your textures BSA into your Data folder. That will solve most of the problems, but really it's a terrible solution. If you do this, be careful not to overwrite all the mod files you've already installed. You've been warned.

Probably the only halfway reasonable approach to even begin dealing with this problem is to have a tool like ACV, OBMM, or Wrye Bash check to make sure all replaced common filenames exist in all replaced paths and warn you when this isn't the case.

I think the best we can hope for is a combination of these approaches.

1) Modders creating texture replacers for things with common names in the BSA need to package all the related files with their mod, as many are already doing; if they don't, then their mods would be subject to warnings or blocks from utilities like OBMM and ACV.

This only applies if you are making a texture replacer that affects files with common filenames like "boots.dds", "cuirass.dds", etc. In this case, you must package all of the original textures that you did not change in the directory where the textures reside, such as "textures\armor\iron", along with the ones you did change. Otherwise, the ones you did not change in that directory path will be broken on the user's system.

We've seen this over and over again with mod-makers releasing a 1.0 version of a replacer that only includes their changed files, then (after getting deluged with hate-mail) having to follow up with a 1.1 version that includes all the items in that set regardless of whether they changed the files.

Another solution would be for modders to avoid creating texture-only replacers whenever possible. You can achieve the same result in many cases by altering the texture references in an item's mesh (using NifSkope) so it points to a new directory structure with texture files that don't use the same common filenames. Since meshes are loaded automatically without any need for entries in AI.txt, this avoids the problem entirely.

2) Utilities like OBMM might warn users before they install a mod that it may cause problems, assuming the user takes advantage of the omod format; then the user can decide whether they want to ignore the warning or stop the installation process.

3) Utilities like ACV might warn users after they have installed a mod that they are likely to have problems with certain files (like it already does for meshes); the user can then decide whether they want to ignore the warning or go back and uninstall the mod.

4) Utilities like Wrye Bash might provide some combination of #2 and #3, perhaps with more advanced users in mind.

TECHNICAL DETAILS

The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

- DirectoryStringTable
- FilenameStringTable

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Here is my first attempt at a simplified explanation of how this works. Unfortunately, I just couldn't figure out a way to make the explanation any simpler than this. (I also couldn't get any colors to work, so we'll have to settle for underscores and strikethroughs instead.) If you have an idea on how to make this explanation any simpler or clearer, please post it or send me a PM.

The FilenameStringTable and DirectoryStringTable are initialized with all the textures in the BSA file, so all the paths and names of existing default textures get indexed in the hashes.

A small sample of these two indexes might look like this:

- FilenameStringTable= shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the underscored items reside in the BSA.

If I install one texture replacer for, say, all the Glass Armor, with AI.txt entries, then the hash entries for those textures get marked as being replaced by files in the Data folders. This means these items will now be invalidated, i.e. marked as being in Data the folders rather than the BSA, so now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the strikethrough items have been invalidated, but the underscored items still reside in the BSA.

If you're following any of this so far, you can probably already start to see where problems will arise, since all the common filenames shared between the two armor sets have now been invalidated from the FilenameStringTable simply because we wanted to replace them for the Glass armor set.

Now, if I want to also replace the texture for just the Iron cuirass, I'm going to run into problems because the hash table entries for all the common filenames in the Iron armor folder, including cuirass.dds, have already been invalidated (i.e., marked as existing in the Data folder rather than the BSA) and now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where everything in both sets of armor has been invalidated.

Because the path "textures/armor/iron/" is now marked as existing in the Data folder, and the hash already knows that path is linked to a set of files that includes shield.dds, cuirass.dds, greaves.dds, gauntlet.dds, and helm.dds, then the hash will now expect to find all those same names in the Data folders even though I only wanted to replace the Iron cuirass.

In fact, the problem is even worse than this because Oblivion immediately recognizes the new directory path textures\armor\iron\ as soon as you add it to your Data folder. It doesn't even need to be listed in your AI.txt because the Oblivion automatically scans your Data folders and recognizes that directory path as being one from the BSA that has common filenames which have already been invalidated.

TOOLS

Ultimately, the only really good solution is to use BSA-redirection or BSA-alteration, but if you really want to see for yourself then these tools will help you generate an ArchiveInvalidation.txt file.

ElChE's Oblivion Automatic Content Validator 1.1.0
ACV
- Use the Smart Check feature (in the latest version it automatically ignores meshes; in the previous version, have it open your AI when done, and then delete meshes by hand).

Oblivion Mod Manager (OBMM)
- Haven't tested the latest version, but Timeslip says he has added new AI.txt options in 0.7.6: "It lets you choose exactly which types of files to include, and there's options for ignoring files not in BSA's, normal maps, and files with incorrect extensions. Following MadBat's suggestion, it also deletes empty subdirectories from the data folder. Default settings are to only include textures, and only ones which already appear in a BSA archive. Hopefully, that should keep most people happy." Heck yeah! (In previous versions, use the DDS textures option.)

Wrye Bash also now includes a feature to generate AI.txt, but I haven't tested it yet.

FURTHER READING

Oblivion Mods FAQ

Original Thread: ArchiveInvalidation method tests (now expired and purged from the forums).

GLOSSARY

AI.txt is shorthand for ArchiveInvalidation.txt.

CREDITS

Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:
- DoomedMarauder
- Auctionmule
- Wonder Dog
- Scanti
- Malifrax
- Jarol
- Buerban
- Midnight Voyager
- Zanderat
- Martigen
- MadBat
- rdjeke
- DragoonWraith
- Oblivionmasta
-Quarn
- and many, many others ... Thank you all for this herioc effort!

 

 

 

 

Home | TOTO Main | The Oblivion Quest List | Copyright 2006, 2007, 2008, 2009 by dev_akm. 

You can tell how old the site information is just by reading the entry on Wrye Bash, and the old "Replacers Tab"

 

 

Edit : Reference current Wrye Bash dev, I have a small glitch with Wrye Bash 307.201610020028 - Installed Standalone

On initial Load of Wrye Bash, the plugin info boxes on the right are stretched across 3 quarters of the Wrye Bash screen, until a second later then they go over to the right hand side again where they should be and the Mods tab is displayed correctly.

 

( I should have rejoined Bethesda forums by now, but recently they said they are still moving to new software ( again ), so until they get their crap in one sock I am still only on bethesda.net .. which I dont really use )

[Utumno]

Thanks @alt3rn1ty - well it's fixed in latest, which I still need to pack - will edit this post when ready

 

EDIT: ready

 

307.201610222334

 

See http://forums.bethsoft.com/topic/1606578-wrye-bash-thread-111/#entry25243324 for what's in there - in particular  you need msvc redistibutable x86 from https://www.microsoft.com/en-us/download/confirmation.aspx?id=48145

[alt3rn1ty]

Yep that works great ( fixed the expanded info boxes ) .. I already have MSVC 2015 as a requirement to a GOG game setup

 

 

But another error on clicking the INI Tab =

 

Traceback (most recent call last):

  File "bash\basher\__init__.pyo", line 3482, in OnShowPage

  File "bash\basher\__init__.pyo", line 1621, in ShowPanel

  File "bash\basher\__init__.pyo", line 1625, in RefreshPanel

  File "bash\basher\__init__.pyo", line 1631, in _SetBaseIni

AttributeError: 'OblivionIni' object has no attribute 'path'

 

I obviously dont have Oblivion installed anymore so its looking for an Oblivion ini unnecessarily in my case

 

Will get a full debug and edit this post

 

Edit : Bugdump - And note the second run of Wrye Bash after getting the above error does not launch Wrye Bash, it tries to load and dies

 

Wrye Bash starting

Using Wrye Bash Version 307.201610222334 (Standalone)

OS info: Windows-8-6.2.9200

Python version: 2.7.10

wxPython version: 2.8.12.1 (msw-unicode)

input encoding: None; output encoding: utf-8; locale: ('en_GB', 'cp1252')

filesystem encoding: mbcs 

Searching for game to manage:

bush.pyo   71 _supportedGames: Detected the following supported games via Windows Registry:

bush.pyo   73 _supportedGames:  skyrim: D:\Steam\steamapps\common\Skyrim

bush.pyo   73 _supportedGames:  fallout4: D:\Steam\steamapps\common\Fallout 4

bush.pyo  131 _detectGames: Detecting games via the -o argument, bash.ini and relative path:

bush.pyo  137 _detectGames: Set game mode to skyrim found in parent directory of Mopy:  D:\Steam\steamapps\Common\Skyrim

bush.pyo  172 setGame: No preferred game specified.

bush.pyo  152 __setGame:  Using skyrim game: D:\Steam\steamapps\Common\Skyrim

mods_metadata.pyo   42 <module>: Failed to import the loot_api module.

testing UAC

__init__.pyo 7761 initDirs: Using libbsa API version: 2.0.0

Traceback (most recent call last):

  File "Wrye Bash Launcher.pyw", line 88, in <module>

  File "bash\bash.pyo", line 438, in main

  File "bash\basher\__init__.pyo", line 3985, in Init

  File "bash\balt.pyo", line 1561, in _conversation_wrapper

  File "bash\basher\__init__.pyo", line 3748, in RefreshData

  File "bash\basher\__init__.pyo", line 1621, in ShowPanel

  File "bash\basher\__init__.pyo", line 1625, in RefreshPanel

  File "bash\basher\__init__.pyo", line 1631, in _SetBaseIni

AttributeError: 'OblivionIni' object has no attribute 'path'

 

 

I used the bleeding edge installer to overwrite install the older bleeding edge standalone ( presuming the installer deletes old files automagically and there is no need to wipe out the old setup first )

 

 

Edit 2 : Will watch for further updates on BGS forum topic, and go back to previous Wrye Bash 307.201610020028 - Installer standalone for now.

 

Alsooo - I am on Win 10 x64 bit Anniversary version ( Update for Windows 10 Version 1607 for x64-based Systems (KB3199209) )

The bugdump there reports OS info: Windows-8-6.2.9200

[Utumno]

The name of the variable is irrelevant - nothing to do with oblivion - the traceback is relevant though - I was expecting this kind of stuff - will post an update asap. As for the windows-8 thing that should be a python thing - will investigate - thanks

 

EDIT: 307.201610230208

[alt3rn1ty]

Ok thanks Utumno, I have opened a throw away account back on BGS forum .. And using this post to confirm this post is me ( saving alt3rn1ty for the new forum software whenever that happens, hopefully this life time ) see new bug dumps there

[VaultDuke]

Wait wait, why does the new forum force you to not use your account name on the old one? And why do you make it sound like there might be a new new forum?

[alt3rn1ty]

Wait wait, why does the new forum force you to not use your account name on the old one? And why do you make it sound like there might be a new new forum?

 

Bit of a long story - When they said they would ditch the old forums ( Oblivion modding / Skyrim modding etc etc ), in preference for the new Jive assed rubbish software they adopted, then I thought to myself "given Bethesdas track record of handling security not so well .. I dont wish to leave anything behind in an account that they may not have purged so well in the future that may be scooped up by future attackers on an archived site"

 

.. So I had my old BGS forum account deleted in preparation ( I also gave it a password that was a random generated 64 character mess which cant be associated with anything I have or will have in future )

 

Now as you are probably aware, the whole change over did not go so well, and they halted transferring some forums over. Fallout went over to Bethesda.net for example but TES forums did not until they decided on a better way ahead.

 

Now reference this topic - https://community.bethesda.net/thread/67465

 

Its going to be an update to Bethesda.net, new software to replace Jive, and then re-commence transfer of old forums to the new - Hence .. again.

 

There is talk that accounts now may be preserved, not sure on this so just in case I already have a Bethesda.net alt3rn1ty account, and made a new old forum BGS account in a new name in case making the same name would stuff things up with whatever direction the new new site goes in.

 

I may aswell have just kept my original alt3rn1ty on the BGS forums and not joined Bethesda.net like everyone else, but it just seemed like a disaster incoming and I did not wish to leave any personal details behind on what would become old forums archived that we would not have had any access to thereafter ..

 

.. Thats about the whole of the story

 

// Much off topic recently.

[VaultDuke]

Thanks for bringing me up to speed. Didn't know they were working on a relaunch of the bethesda.net. I was already beginning to wonder, why the elder scrolls section wasn't migrating for the SE launch (and here we are nearly back on topic), seeing how uploads of mods for consoles would probably have to go through their site, which includes bsas (phew, made it) for all but the Playstation. ​

[alt3rn1ty]

Just changed a link for Ethatrons old documentation in the OP reference BSAs versus Loose files "A Little Calculation" from the NIFOpt Documentation

 

Because his site has been down for a long time now, you can still access them via the wayback machine

 

NIFOpt Documentation

 

BSAOpt Documentation

 

DDSOpt Documentation

 

Note for anyone curious : These tools are only relevant for Old Skyrim and other games supported by them, currently at time / date of writing this post, they do not support Skyrim SE new LZ4 compression BSAs.

 

 

To extract Skyrim SE BSAs, we have :

 

Bethesda Archive Extractor by Jon

 

BSA Manager by ousnius

 

Repacking BSAs in the new format is currently only supported by the official Archive.exe for Skyrim SE ( Needs Creation Kit )

[godescalcus]

This thread has information that is crucial for making a wrye bash installation. Since there is a lack of modding guides for Skyrim that use Wrye Bash, mod users like myself have to adapt from guides made for other mod managers. In my particular case, the modding guide tells you to delete empty/dummy esp's (assuming the respective bsa will be unpacked as per that mod manager's default or configurable behaviour). Using wrye bash, I was unwittingly placing bsa files in my data folder that would not be used by the game, because the dummy esp had been deleted. I've had several of those cases which didn't break my game and I wouldn't have noticed, had I not read on the first post in this thread:

On 06/10/2015 at 8:05 AM, alt3rn1ty said:

Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

So one must take care to extract the bsa if the dummy plugin is to be eliminated.

Which leads me to a question: why do modders pack their assets in bsa format when that means a dummy esp must be created, taking up one slot? Why not just archive the loose files and post the mod in that format? I understand that many (if not most) mod users won't be hitting the esp limit, but the loose files, in this particular case, seems to be more universal.

[Arthmoor]

The better solution is to keep the dummy ESPs and put them in the load order where they belong. IMO, any guide which tells you to drop the dummy ESP files OR unpack the BSAs is a guide that is 5 years out of date and should not be followed since it's based on bad information Bethesda themselves corrected with an official patch.

Most of those were written with Mod Organizer in mind, which encourages a number of bad practices in modding due to the way it's designed.

[alt3rn1ty]

In addition to what Arthmoor just said, there is also the performance of the game being affected by slower to load large files when they are all unpacked as loose files - Especially on the newer games like Fallout 4 and Skyrim SE, which have far more efficient decompression routines making it faster to load a small file and decompress it, than to just load a decompressed but much larger and fragmented loose file.

Throughput : Loading the files from a medium into ram for use by the game, gives far better results when the file is much smaller and decompresses at lightning speeds, the whole operation being far faster than just loading the equivalent larger loose file.

Mentioned in the first post in section 4. Addendum: BSA Compression

"Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed."

In the quote from Ethatron

And well, the ability to Load Order textures is also a benefit. If you had two BSAs which contain conflicting textures, only one of those textures in conflict can win and that will be the one which loads last .. You can also mix n match having a few loose files to get really granular on the details, because they will always win and override any conflicting textures in BSAs .. So you could have the majority of winners in the winning BSA, and the few you really want from the earlier loading mod just extract those as loose. This way helps preserve file loading efficiency of the overall setup.

 

To better visualize the comparison of loose versus BSA loading of individual files

Loose large file on SSD (+'s equal time taken to load all associated fragmented blocks of file data)

++++++++++++++++++++++++++++++++++++++++++++++++

Compressed file in BSA (+'s equal time taken to load whole contiguous smaller file, /'s equals time taken to decompress the whole thing into ram)

++++++++++++++++++++++++++++///////

The engine gets the whole of the compressed file out of the BSA and into ram far quicker than the already decompressed loose file off of the SSD.

See?. Times the above operation by thousands and thousands of file reads

[lmstearn]

And much less weight on the file system: Larger Filesystem = Larger risk of something going wrong with it.

[godescalcus]

But back to the dummy esp's, does WB always ghost them? (and I promptly admit that I don't know exactly what ghosting does, only that it's a feature of wrye bash). Or will they have to take up space in the load order? Can you just merge them with something else in merge plugins to make them call the bsa without taking up the slot?

I understand the benefits of bsa's but to lose maybe 10, maybe 20 slots to dummy esp's seems like a lot.

By the way, I just hit the point where wrye bash tells me I have over 325 esp's and bsa's and tells me to enable auto-ghosting (which I did). But what is this 325 limit - does it mean I also have to worry about bsa's taking up "slots" themselves, only of a different kind? In that case, isn't there a tradeoff benefit to extracting them, when you're installing a large number of mods? My current project will have near 500 mods, I'm over 400 now.

I won't debate extracting bsa's as a rule, I accept from those who know better that it's a bad practice. But would there be cases where you just have to do it, to continue modding the game?

[alt3rn1ty]

1 hour ago, godescalcus said:

But back to the dummy esp's, does WB always ghost them? (and I promptly admit that I don't know exactly what ghosting does, only that it's a feature of wrye bash). Or will they have to take up space in the load order? Can you just merge them with something else in merge plugins to make them call the bsa without taking up the slot?

I understand the benefits of bsa's but to lose maybe 10, maybe 20 slots to dummy esp's seems like a lot.

By the way, I just hit the point where wrye bash tells me I have over 325 esp's and bsa's and tells me to enable auto-ghosting (which I did). But what is this 325 limit - does it mean I also have to worry about bsa's taking up "slots" themselves, only of a different kind? In that case, isn't there a tradeoff benefit to extracting them, when you're installing a large number of mods? My current project will have near 500 mods, I'm over 400 now.

I won't debate extracting bsa's as a rule, I accept from those who know better that it's a bad practice. But would there be cases where you just have to do it, to continue modding the game?

Ghosting is for inactive (disabled / unchecked) plugins you either do not wish to be in play .. Or are fully merged into the bashed patch

Read the Context Menu Description

Its currently not as useful for Skyrim as it is for Oblivion. And the reason for that is to fully merge many plugins, CBash needs to be updated for Skyrim, which will not happen until someone with C++ knowledge comes back to being on the Wrye Bash team (it used to be Waruddars thing, but even with his historically long timeouts I dont think he is coming back).

Or .. The normal python patchers are fully updated for ALL the different records in skyrim - The work on those patchers has not really begun yet.

So currently its mostly only useful for skyrim if you have plugins installed which are not active (pointless having them installed in the first place really if they are not active)

 

Now if you ghost a plugin, what happens is it gets renamed. So ..

uselessmodinstalledbutnotcurrentlyactive.esp

becomes ..

uselessmodinstalledbutnotcurrentlyactive.esp.ghost

The game engine will not even look at it because of the unusual file extension it does not recognise it as a valid game file, and so the old thrashing bug is solved

 

For your case its not going to be useful, you are thinking "aha, a dummy plugin is only used to load a BSA, so if I ghost it then it doesnt count <lightbulb>" - Your lightbulb is malfunctioning, a ghosted dummy esp will be ignored by the game engine, and so will its BSA, and the content of the BSA will also be ignored.

The dummy.esp needs to be active to load the associated BSA.

 

There is a simple solution to your problems - Cut down on the amount of mods you are using. 

The game engine has hard limits for technical reasons which no one can get around without completely re-writing the game engine.

Yep aswell as plugin limit in the load order there is also an overall limit of plugins plus BSAs

 

If you wish to extract some of the mods BSAs and do away with a few dummy plugins, thats entirely up to you. Why not create your own BSA and dummy plugin for a bunch of dummy plugins with all the textures / meshes in the same BSA ?, then you have one dummy plugin loading one BSA full of all the resources which were spread among a few of them before. Just make sure they are all dummy plugins and do not have any record modifications to the game (load them up in xEdit) - And if there are file conflicts well thats also your choice of which you prefer to end up in the BSA, there can after all only be one of any two (or more) conflicting files will win.

Personally though even these days with Skyrim Special Edition being 64 bit, I still keep my mod usage low, 8 at the moment :), game runs like a dream compared to older elder scrolls series games.

 

On another note - Are there actually 20 mods out there using dummy plugins ?, and are they all texture mods, and do you need all of them if the majority are just doing the same bunch of textures with different objectives / style / artistic theme - Point being if you have 20 BSAs all full of the same bunch of textures, then you dont need 19 of them because more than likely the majority are conflicting files and not being used if they load earlier than all the others.

 

Another tip, really modding so much you will never be quite happy with the whole setup, and will end up in a cycle of overhauling everything periodically, so your load order will mostly be changing regularly and causing problems with your saves forcing game fresh starts and .. Never actually enjoying the game.

Or are you in so deep modding its a case of "You can actually play this game too! O_o ?"  

[godescalcus]

I'm just trying to follow a guide!

And yes I know what you mean with the "never quite happy" thing. At the moment I'm having fun installing mods, testing, replacing, etc... Really having fun doing this, rather than playing. I'd like to finish this guide with wrye bash (I've done it with another mm), then start to pick out stuff and work those that I really want to keep. You mentioned some interesting ideas. The guide's allows for a lot of texture replacements (even if it does sometimes tell you to only pick out certain mentioned files, that's mostly to avoid replacing those installed EARLIER in the load order... So I do have a lot of replacers replacing replacers. Later on I might do a little digging on those asset mods and see if some could be "merged together" (I mean, arranged in a way that only the assets that you want in the game are archived). In the case of bsa's with dummy esp's, I might learn how to repack them in a new bsa, as you suggested.

Further into the guide there is a lot of plugin merging that does away with many esp's. Maybe I won't have to do without that many bsa's after all is said and done.

There's certainly lots of room for mistakes causing accidental conflicts and bugs, but I bet you have done this in the past and learned your wisdom through it

[godescalcus]

On 24/04/2017 at 7:29 PM, alt3rn1ty said:

BSA.

Maybe other people using the guide you are using also use MO's more advanced (but ill advised if you are not an expert user) features of extracting BSAs and completely undermining the Load Order system that is best left in place, with MOs priorities.

  Reveal hidden contents

I recall a long argument on the S.T.E.P forum where the resident experts on the subject were eventually pulled up short by Tannin for getting it all wrong for a few years (and influencing thousands of misguided people, who form the basis of the majority of MO users today, and their guides .. The damned thing caused a plague of misadventure imho).

Eventually the advise on S.T.E.P was amended, MO had its more advanced and potentially damaging features hidden and turned off by default. So maybe the guide you are using needs more advanced understanding ? (not only of the tools being used, but also your machines capabilities and what you are feeding it to handle which it is maybe not quite capable of - NPCs laying around like lemons sounds like either a bad ini edit to memory allocations, or you are overloading the game with heavy scripts your CPU cant keep up with juggling .. Or another possibility on that could be an old Combat / Anim mod which has been removed at some point which your game save was dependant upon, and you continued playing onwards, though I dont think you would make such a mistake so not sure on suggesting that one - Any significant Load Order change can also result in such problems, surprisingly, especially if you play on afterwards with a save which was dependant upon the previous load order, Crushboss (Animal Tweaks and Combat Evolved) gets a few reports of such issues periodically which are due to those mistakes being made, typically the old Oblivion NPCs going into crucifix pose but still moving towards you like a Monty Python animation can be a result of Load order changes involving Combat mods, and that problem can still happen in Skyrim)

I'm switching this conversation here so as to not be off-topic in the wrye bash thread.

Is there a way to pack huge mods like Noble Skyrim and SMIM into bsa's?

[alt3rn1ty]

If you mean making BSAs for Old Skyrim LE, then BSAOpt or the official Archive.exe can be used.

For Skyrim SE only the official Archive.exe (gets installed with the CK) currently

I dont know what Noble Skyrim is so unsure what would be involved.

 

Edit : Arthmoor has a guide on using it 

 

[godescalcus]

Noble skyrim is just textures, SMIM is textures and meshes. SMIM comes with a plugin file, noble doesn't. I imagine that mods with huge amounts of assets that pop up pretty much anywhere in the game would be good candidates for performance improvement if I archived them? Arthmoor's guide is for CK+Archive.exe, I wonder if BSAOpt would be similar.

[lmstearn]

BSA Manager can also open Oblivion BSAs, but it lacks BSAOpt's packing capabilty. Has there been any progress on the stability of Archive.exe for SSE in the interim?

[alt3rn1ty]

Well according to the latest pack and ship of the Unofficial SSE Patch, the CK borked it and skipped a fix script from the files required entirely, it was the fix to Dawnbreaker duplicates which was recently solved in the pre-release beta.

So at least in that case it was the CK at fault .. and not the archiver itself ?

I dont use the official Archiver (just because I have no need of having the whole buggy CK installed, and not due to any kind of preference)

-----------------------------

 

I forgot to mention here a new tool by Zilav - BSArch https://www.nexusmods.com/newvegas/mods/64745?

Quote

Packer and unpacker for Bethesda Game Studios archive files

Supported games (archive versions):

• Morrowind
• Oblivion
• Fallout 3 and Fallout New Vegas
• Skyrim LE
• Skyrim Special Edition
• Fallout 4 General
• Fallout 4 DDS textures