Jump to content
alt3rn1ty

[UPDATE] BSAs and You

Recommended Posts

Due to Bethesda forums going to a different forum ( Bethesda.net ), reproducing an important topic here on AFK Mods just in case all old topics fall off the face of the internet as Bethesda move on, and the old locked forums are no longer maintained.
 
Original research and credit to Lojack and Wrinklyninja, I am just copying the info to preserve it on AFK Mods
 
---------------------------------------------------------------------------

 

BSAs and You

 

 
Lojack : "We're back, with an update on the information provided in [iMPORTANT] BSAs and You!

First off, make sure you've read [iMPORTANT] Load Order and You, by wrinklyninja.

 

Load Order and You

Introduction

Load ordering is the method used to determine how conflicts between mod plugins (.esp, .esm files) should be decided. If two plugins alter the same game data, then the changes made by the plugin loading later will override those made by the plugin loading earlier. This "rule of one" results in a list of plugins, with those earlier in the list having any conflicting changes overriden by those later in the list. This list is the load order of the plugins.

A game will only load the plugins that are active. Up to 255 plugins, including the game's .esm file, can be active at any one time. Active plugins are listed in the game's "plugins.txt" file, which is stored in the user's local application data folder. Nevertheless, it is useful when working with load orders to consider the load order of all plugins, even if only some of them will actually be loaded. This is both because it is easier to display a single list of plugins than a list and an unordered set, and because modders have engineered methods that allow the changes made by inactive plugins to be loaded by another plugin (eg. Wrye Bash's Bashed Patch). When any such methods are being used, the load order of inactive plugins decides which plugins override others, similar to as if they were active.

In Oblivion, Nehrim, Fallout 3, Fallout: New Vegas and early versions (pre-1.4.26) of Skyrim, load order is decided by the relative timestamps of plugins in the game's Data directory. An installed plugin's load order is therefore an intrinsic property of that plugin.

In Skyrim v1.4.26, a new textfile-based load order system was introduced, in which load order is decide by the order in which plugins are listed in "plugins.txt". This brought with it a fundamental change, in that load order is no longer an intrinsic property of a plugin. This has the result that inactive plugins do not have any load order.


The Solution

The solution agreed on by Lojack (Wrye Bash), Kaburke (Nexus Mod Manager), WrinklyNinja (BOSS) and Dark0ne (owner of the Nexus sites) was that total load order would be stored in a "loadorder.txt" file, itself stored in the same location as "plugins.txt". "plugins.txt" would be kept in synchronisation with "loadorder.txt" so that the order of plugins that the game loaded was the same for both files, but the latter would allow the load ordering of inactive plugins.

Modding utilities would then perform their changes on "loadorder.txt", updating "plugins.txt" to reflect any changes to active plugin load order as required. This provides a common store for the total load order in lieu of the plugin timestamps used by the other games. There are of course a few more technical details to it, but that's the basics.


What This Means For... Mod Makers & Users

The bad news: Any utilities that you use to manage load order may stop having any effect. This is because nothing currently released knows about the change in load order system. In time, utilities may be updated to handle the text-based system as their programmers become aware of it. Until then, Skyrim's launcher is the only way to change load order. Also be aware of the current limitations of the solution, given above.

One very important thing is that if you have a utility that can handle the text-based system, do not use Skyrim's launcher to set load order. This is because it doesn't know about the total load order, so if you change the load order through it, only the active load order changes and synchronisation between the two is lost. The only way for a utility to re-sync the two load orders is to undo the changes you made in the launcher.

The good news: Nothing changes when it comes to making or using mods, it's just business as usual. Just bear in mind the point about the utilities you use to manage load order needing updating, and have patience while the programmers do their thing.

If you want to manually set your load order, the instructions below may be of use:

  • Open your plugins.txt in a text editor. plugins.txt is found in your local application data folder. On Windows Vista/7, this is C:\Users\[username]\AppData\Local\Skyrim\plugins.txt.
  • In plugins.txt, list the plugins you want active in the load order that you want them in. Each plugin goes on a separate line. Master files should go before non-master files. Merged or imported plugins are not active plugins, so should not be listed in plugins.txt.
  • If you also have a loadorder.txt in the same folder as plugins.txt, then update it so that the plugins in plugins.txt are in the same order in both files.
  • Save the edited file(s) and quit.
  • Don't try to then edit your load order through the Skyrim launcher. It has been reported to reset your load order if you set it up manually.

What This Means For... Mod Utility Programmers

The bad news: Your utilities will no longer function when it comes to load order (both getting and setting) until you update them to support the current system.

The good news: I've already done a lot of the work for you. The BOSS API supports both the timestamp-based and text-based load order systems through the same interface, so you don't need to worry about the differences, and includes all the functions required for querying and changing load order and activation status. The BOSS API will be included in the v2.0 release of BOSS, but if you PM me I'll send you a copy if v2.0 hasn't been released yet.

Note that the BOSS API is licensed under the GNU GPL v3.0 license, so your utility's license will have to comply with that to use the API.

If your utility's license is such that you cannot use the API, it's still worth looking at the BOSS API's readme as that contains the information on the textfile-based load order standard. If BOSS v2.0 hasn't been released yet, PM me and I'll send you a copy of the readme.



Final Notes

Please remember to have patience if you're waiting on someone to do something in relation to this issue, eg. users waiting on programmers, or programmers waiting on me to fix an API bug. Everyone who can do anything about this situation is pretty busy, and these things take time to get done right.

Also please don't hate on Bethesda for changing the system. It's better than the old one, and the issues that it brought up are because we're too smart for our own good, coming up with ways to bend the rules (eg. Bashed Patches), so it's wrong to think that Bethesda need to cater for the situations in which they arise.

It's probably a good idea if people post below any utilities that use load order in some way that aren't listed above, so that I can add them to the compatibility list.

Finally, spread the word. If someone's having trouble setting the load order, let them know that it might be to do with this. If you know of a modding utility not listed above that changes or uses load order in any way, let the author(s) know about this. Please be polite while doing so though, I don't want to end up reading verbal abuse resulting from this instruction. When letting utility authors know, don't demand they update their utility, just post a note saying that it doesn't work any more and link to this thread.

Related links:
Thread #1 on the BethSoft forums
Thread #2 on the BethSoft forums
This thread on the Nexus forums
News item for the issue on Skyrim Nexus 

 

With the latest Beta patch for Skyrim officially getting sent out to all clients, there have been some changes to BSA Load Ordering that you might want to know about. Basically, the issues brought up in the previous thread have been fixed. So this post is only here for educational purposes, in case you're curious as to the order of resources loading.

To summarize, the order resources are loaded is:
 

Registered BSAs -> Plugin BSAs -> Loose Files

 

With Registered BSAs being loaded in the order listed in Skyrim.ini, and Plugin BSAs loading in the same order as their plugins, which are loaded in the order they are listed in plugins.txt.

Disclaimer: The game doesn't actually work exactly like this. This is how it logically works. As in, if you had to describe the decision process the game goes through, this is it. Realistically, working this way would be a waste of resources and processing time (why load up every file, when you already know a different one is going to "win"?). It's just much easier to understand if you think of it like this. A good description of how it would actually work is here.

Post from JDFan :

LoJack --- Is this correct or does it actually SEARCH for the textures in the opposite order UNTIL it finds the texture it is looking for ?? ( seems it would make more sense for the game to go through the list backwards until it finds the texture it is looking for and then load the one it finds first instead of searching entirely through every file each time !! )

Also IIRC it also searches RAM first and if the texture already exists in RAM it does not even search the files ( this is why it is important to exit the game when testing so that changes will show up properly otherwise RAM may not update and you'll get different results than when actually playing -- at least this was the case with OB so if you had 2 mods that each used a different version of a texture with the same name sometimes you'd get the vcorrect texture and other times you'd get a different one if the second model had recently loaded and was stil stored in RAM.

While the end result is the same in most cases whether it actually loads every file first and then overwrites again as it finds newer ones it seems that would cause alot of problems and slowdowns vs. traversing the lists backwards until it finds the first instance of the texture (which would be faster and consume less resources ! 

 

 

Contents:

  • Resource Load Order
    • Registered BSAs
    • Plugin BSAs
    • Loose Files
  • What This Means For Mod Users
  • Other Questions
    • Replacing Scripts
    • Directory Thrashing
    • What are all these terms you use?
    • What about Nitpick?
  • Addendum: BSA Compression

1. Resource Load Order
 

Skyrim loads its resources from the Data folder. Makes sense. These resources can also be packed up into a nice neat little archive, called a BSA. These are Bethsoft's custom archive files, and internally they mirror the layout of the Data folder. They're nice, because they keep the Data folder uncluttered, and it makes it really easy to remove every file from one mod.

Ok, but what happens when the same resource exists in more than one place? For example textures/dungeons/imperial/battlemap01.dds, which is the texture for the map you see all the time in castles, with the little flags and stuff sticking out of them. This file exists in Skyrim - Textures.bsa. That's the original file. If you download a texture replacer, you'll get it as a loose file probably, and if you download HD Skyrimmap from Steam Workshop, you'll get that file in a BSA. So which file actually gets used? Well, which ever one is loaded last. The game loads resources in three stages, with the last files loaded being what you see in game:

Needs Investigation: With pre-1.4.26 Skyrim, only Registered BSAs and Loose Files were loaded when the game first loads. That means anything in a Plugin BSA won't take effect yet - but you'll only notice that if you're trying to replace a Main Menu asset, like the logo. Once you start a game, either by loading a saved game or starting a new one, then the Plugin BSAs are loaded. So in the example of Brumbek's Main Menu Logo replacer, if it's loaded via a Plugin BSA (like it is when installed via Steam Workshop), you won't see the spinning logo until you load a game, then exit back to the Main Menu.

So the investigation needed: Is this still the case? I'll get back to you on that once I get the time to test it.

Registered BSAs
First the game reads your Skyrim.ini file. There's a section that tells the game which BSAs to load. It looks like this for most people:
[Archive]
sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa

 

The game will load the BSAs in order as listed, BSAs in sResourceArchiveList first, and sResourceArchiveList2 second. So if you manually register a texture replacer BSA (like the HD Texture packs), but put it at the beginning of the list - they won't load! Because the Skyrim - Textures.bsa will override them of course. If you put it after, or at the end of the list, or in the second entry, then they will load.

Easy enough, but there is one drawback: There's a 255 character limit on each of these entries. This is because the game internally uses a 256 byte character buffer to receive the value when reading it. One byte is reserved for the NULL terminator (a byte with a value of 0, indicating the end of the data), so that leaves 255 characters for file names. If you want to get around this limitation, you'll need to use Nitpick.

Plugin BSAs
Next the game loads any BSAs associated with a plugin. When the game loads an active ESP, it also looks for a BSA of the same name (For Oblivion, there were naming conventions that allowed a single ESP to load multiple BSAs in this manner - this is not the case for Skyrim). So dummy.esp will make the game load dummy.bsa. This is true even if the ESP does absolutely nothing (for example, the official HD Texture packs do this, as well as Oblivion's Shivering Isles expansion). Cool! We found an easy way to make our replacement textures only load when we want! Just deactivate the plugin and they don't load, activate it and they do!

Ok, but what happens if two Plugin-loaded BSAs have the same file in it? Well, it's easy now: whichever one loads last wins (as usual). But now there's a reliable way to control which order they're loaded in: they're loaded in the same order as their associated plugins. This means if dummy.esp loads before another.esp, then dummy.bsa will also load before another.bsa, meaning of course that another.bsawill win, because it loaded last.

Sounds good, but I seem to remember there being other issues with using BSAs...

Yes, nothing major really, unless you're trying to mod your game with a lot of plugins.

 

Due to the way the game indexes plugins and stores the data in your save game, you can only have 256 active mod files. That's because objects are referenced by what's called a FormID, which is a 32-bit (4-byte) unique number, usually written as an 8-digit hexadecimal number. The first two hex digits are used to identify which mod the data came from. Two hex digits can represent a maximum of 256 unique numbers (0-255).

 

Now, Skyrim.esm and Update.esm will always fill up two of those. Another one that's not obvious is your saved game itself - the one in use is always assigned the ID of hexadecimal FF, or 255. That leaves room for 253 other plugins.

 

And here we get to our point: dummy plugins that don't do anything at all, but exist solely to make the game load a BSA? Yep, those take up one of our plugin slots. This might not seem like a big deal, but once you start trying to play with 300+ mods, you'll be faced with the hard decision of deactivating some mods. Luckily, dummy plugins you can just unpack the BSA and use the resources as Loose Files, then you don't need the plugin anymore. Other plugins can be merged via Wrye Bash into the Bashed Patch, further cutting down your mod count. And hopefully an equivalent of TES4Gecko will come out eventually, which will be able to merge mods into a single plugin. This last one however has an impact on your saved games, so it's for the more advanced user only.

 

Needs Investigation: Another small strangeness associated with Plugin BSAs, at least with pre-1.4.26 - they aren't loaded until you load either a saved game, or start a new game. That means if you're still at the Main Menu for the first time, any replacers loaded this way haven't taken effect yet! Woah! Not a big deal, unless you're trying to replace the Main Menu Logo... Oh wait. That mod works fine when installed as Loose Files, or if the BSA is registered in your Skyrim.ini, but it only works as a Plugin BSA after loading a saved game. This behavior still need to be verified now that 1.4.27 Skyrim is out.

Loose Files
This one's simple. If the file exists in the Data folder, it will always win. It will always be used, even if the same file exists in a Registered BSA or Plugin BSA. This is how most texture replacers work, and this is what mod managers like Wrye Bash or NMM are made for. Managing the Loose Files so you don't have to track them yourself. It's easy to keep track of which file came from which installer when you only have 3 mods. But when you get up to 150 mods, each with their own resources, a lot of them overlapping, a mod manager just takes the headache out of it.

2. What This Means For Mod Users
Not much really. There is one minor disadvantage to all this: Steam Workshop uses BSAs by default, even for texture replacers that don't need a plugin. That means you end up with an ESP that is only there to load a BSA, taking up one of your 253 mod slots doing it. If you start getting close to the 253 mod limit, you can cut back by either:

  • (Recommended) Unpacking those BSAs to Loose Files. A useful tool for unpacking the BSA is DDSopt. Once the BSA is unpacked, just pack up those Loose Files along with any documentation into an archive, and install with your favorite mod manager. Now you can deactivate the plugin, saving you a mod slot.
  • (Not Recommended) Register the BSAs. This method you'll use your Skyrim.ini to load the BSAs instead of using a plugin. First you'll need to deactivate or delete the plugin. Next, you'll need to add the name of the BSA to your Skyrim.ini, in the following section:
    [Archive]
    sResourceArchiveList=Skyrim - Misc.bsa, Skyrim - Shaders.bsa, Skyrim - Textures.bsa, Skyrim - Interface.bsa, Skyrim - Animations.bsa, Skyrim - Meshes.bsa, Skyrim - Sounds.bsa
    sResourceArchiveList2=Skyrim - Voices.bsa, Skyrim - VoicesExtra.bsa  :  The reason I don't recommend this method is that it's harder to change the order. This method you need to edit your Skyrim.inieach time you want to change the order, whereas with Method #1, you just either change the Installer Order of the package in BAIN, or in NMM. Also, you'll run up on the 255 character limit pretty quick, which means that you'll need to make use of Nitpick, which in turn requires SKSE. There's nothing wrong with requiring SKSE, but you'll have to wait for a SKSE update after each game update.

Again, each of these tricks only works if the plugin in question is an empty, dummy plugin, solely there for the purpose of loading the BSA. You can check this using TES5Edit or another similar utility, or right click on the mod in Wrye Bash and select 'Mark Mergeable...' to find out if the plugin is empty. Wrye Bash will tell you if the plugin is empty, but it won't say anything at all relating to its "emptyness" if it's not empty.

3. Other Questions
Replacing Scripts
Pre-1.4.26 for Skyrim, you couln't replace a Vanilla Script file with one in a Plugin BSA. This is no longer true, it has been fixed, and script replacers should work as both Loose Files and Plugin BSAs now.

Directory Thrashing
What is it? link. Basically, simply having too many ESP, ESM, and BSA files (even inactive) in the data directory would cause Oblivion to go crazy, bringing your game to a crawl as disk I/O went through the roof. It's why Wrye Bash has the Auto-Ghost feature.

Does it affect Skyrim? Sort of, but not in the old sense of the term. There's a different problem with Skyrim, such that if you have over 508 mod files in your data directory, the engine just plain refuses to load any of them, active or not. To fix this, use Nitpick. Wrye Bash's ghosting will also work here since, Wrye Bash will not allow more than 255 active plugins at once, and ghost the rest, meaning you will at most have 255 files in the data directory with .esm or .esp extensions.

There's also the added problem of plugin INI files (which pretty much no one uses - no fix for this), and part of the old thrashing problem, where the game tries to read inactive plugins. For the last issue, again Wrye Bash's Ghosting feature still works good.

What are all these terms you use?
I used a bunch of terminology here, most of it old, but there's a couple new terms I made up for this explanation.

  • Plugin - any .esp or .esm file.
  • ESM - Elder Scrolls Master. Usually saved as a .esm file, which has the same format as a .esp file, only one bit (as in, one bit of a byte) is different.
  • ESP - Elder Scrolls Plugin. Most mods are ESPs, but some are ESMs. Their format is the same, but the game handles them slightly differently in some specific cases.
  • ESS - Elder Scrolls Save. Your saved games. This is loaded after all the ESPs and ESMs are loaded, so that its changes (like what items are in which containers) can override what the plugins say.
  • BSA - Bethesda Softworks Archive. Sort of like a .zip file, but specific to Bethsoft's games, and optimized (sort of) for their specific usage.
  • Load Order - The order that your mods (ESPs and ESMs) are loaded. When conflicts occur, the last loaded conflict "wins".
  • Install Order - The order that files are installed into your Data folder. When multiple packages install the same file, the last installed package "wins".
  • BSA Load Order - The order that the game engine loads BSAs. When conflicts in data files occur, the last loaded BSA "wins".
  • Plugin BSA - A BSA that the engine loads because an ESP/ESM with the same name is active.
  • Registered BSA - A BSA that the engine loads because it's listed in Skyrim.ini.
  • Loose File(s) - A resource file in the Data folder that's "loose", as in not packaged into a BSA.
  • FormID - A unique ID assigned to each "record" of a plugin. Each object in the game is defined by a record, and each record has a FormID. A FormID is 32-bits, with the first 8 bits defining which ESM or ESP the record belongs to.
  • SKSE - Skyrim Script Extender. A nice utility that everyone should be using. It aims to extend the functions available to scripts written for Skyrim. It's also useful as DLL loader, if one wants to inject some code into Skyrim's game engine.
  • SD - Script Dragon. Another DLL loader, but with a different goal in mind than SKSE. Both can work along side each other.
  • Vanilla - A term referring to Skyrim as distribute by Bethesda and Steam. This means no modifications have been done. Alludes to the "plainness" of the vanilla ice-cream flavor.

What about Nitpick?
I'm still maintaining it. shadeMe may also release updates as well, he has access to the Skyrim Nexus page, so either he or I can update it. The Nexus page also has links to the source code, but here's shadeMe's source and my source, links in case the Nexus is down at the moment.

4.Addendum: BSA Compression
This question comes up often enough to deserve an answer here: What's better, compressed BSAs or uncompressed BSAs?

To quote Ethatron ( Via the Wayback machine, original site is down ) NIFOpt Documentation :
 

Ethatron said

So lets see a little calculation for two operations of reading data from disk, uncompressed and then compressed:

  • Harddisk-throughput: 20MB/s
  • Memory-throughput: 2000MB/s
  • zlib-throughput: 80MB/s
  • uncompressed size: 100MB
  • compressed size: 60MB
This yields:
  • Reading 100MB at 20MB/s from disk takes 5s.
  • Reading 60MB at 20MB/s from disk takes 3s, decompressing 60MB at 80MB/s in-memory takes 0,75s, so a total of 3,75s.
In case the data is very small it's probable that seeking time will start dominating the times:
  • Reading 1MB at 20MB/s from disk takes 0,05s.
  • Reading 0,6MB at 20MB/s from disk takes 0.03s, decompressing 0,6MB at 80MB/s in-memory takes 0,0075s, so a total of 0,0375s.
Seeking may be as much as 0,002s. Nevertheless it doesn't change the fact that reading compressed data is faster as long as the decompressor is faster than the disk. It's a rather theoretic notion, but when using compressed solid archives even the seek-times go down as the disc-head has to reposition a smaller distance while bulk-reading from the same archive.

Also, in reality with a modern system the decompression speed of zlib reach realistic 250-500MB/s (as you can see here), which can be far beyond any SSD-speed.

 

So for the most part, compression is good for you. There are a few cases where this might be detrimental though: At very high compression rates, it seems some files are affected - notably sound files seems to take on a "fuzzy" sound to them in many cases. Textures and Meshes should be fine going to max compression, but be sure to test things if you're going to up the compression on your BSAs. 

 

 

Edit by Alt3rn1ty typo's plus .. : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

I used my powers of awesomeness to determine the archive flags that were used to pack the official BSAs. This information could be important!

Skyrim - Animations.bsa
Flags: Compressed

Skyrim - Interface.bsa
Uncompressed, Flags: None

Skyrim - Meshes.bsa
Flags: Compressed, Retain Strings During Startup
Resource Types: Meshes

Skyrim - Misc.bsa
Uncompressed, Flags: Retain File Names

Skyrim - Shaders.bsa
Uncompressed, Flags: None

Skyrim - Sounds.bsa
Uncompressed, Flags: Retain File Names
Resource Types: Sounds, Voices

Skyrim - Textures.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Skyrim - Voices.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Skyrim - VoicesExtra.bsa
Uncompressed, Flags: None
Resource Types: Sounds, Voices

Dawnguard.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HearthFires.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

Dragonborn.bsa
Uncompressed, Flags: Retain File Names, Retain Strings During Startup
Resource Types: Meshes, Sounds, Textures, Voices

HighResTexturePack01.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack02.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

HighResTexturePack03.bsa
Flags: Compressed, Embed File Names
Resource Types: Textures

Note:
By default, all BSAs have these flags: Include Directory Names and Include File Names. You can't change those flags, so I didn't add them to the list.

 

Credit to Fireundubh on STEP forum for finding out the spoilered information

Share this post


Link to post
Share on other sites

Users of older Mod Organiser ( MO ) versions take note of that quote from Ethatron at the end of the first post
 
There is a recurring opinion some users of MO have, in that they believe extracting all BSAs is a more optimal setup ..

Take two circumstances :

1. Original file compressed ( much smaller ) in a BSA on Hard Drive, or ..
2. Original file decompressed so that it is much larger on the same device

Decompressing a small file into a larger file in memory ( from where the file will then be used ), is executed much faster than reading a larger file into memory from your HD ( even if the HD is a SSD, the throughput is still much faster )
The reason is processors these days are much faster at decompressing a file than a hard drive is at reading a file, so the quicker loaded small file which has to be decompressed ends up being faster in execution than just reading a much larger file from the same device.

Reference the quote
 in "4.Addendum: BSA Compression", end of post above - Ethatron who created BSAOpt and its compression / decompression code did plenty of testing in this department to draw that conclusion, backed up with the math employed to calculate the results found.

The idea that having the BSAs all extracted for more optimum reading is flawed.

The game reads its files much quicker from compressed BSAs.
 

Additionally having all files loose, and therefore more fragmented across the device, makes seek times for all the individual blocks to recreate the file for reading much slower.

It also does not matter that MO uses a virtual device and does not load them from Skyrim data, it still has to load them from a different part of the hard drive where they are stored by MO.

 

The author of MO recognised how problematic extracting BSAs could be technically and so changed the default of allowing BSAs to be managed and extracted, to not allowing it, and separated the ability in a way that made users seek it out to enable it.

 

In other words its an advanced feature that you turn on at your own risk, and cannot blame mod authors for the complicated problems YOU create by turning the feature on.
 

See this topicRamifications of BSA Extraction in Mod Organiser
 

If you read nothing else, read the first post in its entirety, and Post #13 by the author of MO, in which Tannin not only explains why he stopped the ability being a default feature in more recent version of MO ....

 

... but also concurs that BSAs are more optimal in use than loose files. Read the part where Tannin mentions the Pros of using BSAs ..

 

Regarding BSA extraction in MO: I've decided to move this functionality into a plugin that will by default be disabled. This means an advanced user who wants BSA extraction can enabled it and use it as he always has whereas someone who doesn't know about BSA extraction will not be confronted with the choice at all.

Regarding BSA order vs Loose order in MO:

  • MO totally "messes up" this concept, just throw out everything you've read about BSA vs. loose order, it doesn't apply to MO.  In MO, (if you check all bsas in the archives tab) all BSAs are registered BUT if a BSA has higher installation order than a loose file that loose file doesn't appear in the VFS. This means assets are effectively loaded in installation order no matter which format they are in!
  • This also means that no matter if you extract BSAs or not, the bsa order will never be automatically matched to your plugin order!
  • Therefore you have to have to have to follow MOs suggested installation order in the warnings window (Potential mod order problems) to ensure a stable game.

When I suggested I might remove BSA extraction I was also thinking of removing the behaviour in 1, because then we could drop mod order suggestions as well -> loose one feature, get rid of one issue, drop tons of complicated and confusing functionality.
 
More BSA Pros:

 

  • Better performancea lot of files also slows down game startup, especially when using MO )
  • Less disk usagebecause BSAs can be compressed )
  • Faster installationyou save yourself the extraction step )
  • Removes user-controlyes, this is a good thing, at least for mod authors and myself ;) )

 

 

 

I used to have a very weak machine which was only capable of running Skyrim if I employed a lot of tricks to make the game more optimal. I spent a couple of years making a mod which reduced as much of the games textures ( 32 thousand of them ) to more optimal sizes without losing too much detail, and have a lot of practical experience judging the effect of Loose files versus BSAs

 

After a great deal of testing I came to the same conclusion as Ethatron / Tannin / UPP Team and anyone else who has seriously taken time to study this, that the mod is best distributed as highly compressed BSAs for the Skyrim game, even if only for performance gains in my case.

 

Bethesda use BSAs for very good reasons : Which has become more evident recently as we discover the contents of Fallout 4 BSAs, Textures are notably now split ( which is why the Textures BSA is now a special case for extracting and compressing ), all the MipMaps are in a separate file, and the Large top layer image of the texture only loads as required by the game when the texture needs to be more detailled at close range - Otherwise only the MipMaps file loads into memory for speedy cached use at the more often distant ranges you will see the texture in game. Having all of Fallout 4's textures as loose files with a large top layer image with attached mipmap layers in the same file, is definitely less optimal than the vanilla FO4 game setup.

 

 

Update : The above will no longer be a problem in future versions of MO - Version 2 will no longer have the feature to manage extracting BSAs .. See a more recent quote from Tannin :

 

 

 The feature has been removed because it required a lot of complex game-specific and error-prone code inside the hooking lib and caused ton of confusion, bug reports and hatred from other modders.

It made the whole PMOP business necessary which way too many people didn't understand.

The bsa management was supposed to make things more intuitive by giving assets the "expected" order, but in practice it made everything more complicated.

With MO2 my primary focus is Fallout 4 where I don't want to repeat the same mistake and non-gamebryo games where it's unnecessary.

 

 

If you need this functionality for step then you should stick with MO1.

 

Share this post


Link to post
Share on other sites

Is there a similar source for the fallout games? It seems BSAs are handled even different than skyrim pre 1.4 (the UUF3P team had a hard time getting overrides to work from a bsa reliably and had to scrap the idea)

Share this post


Link to post
Share on other sites

No problem its mostly just a copy paste of info that is very good to know, going to edit the second post a bit later.
The subject of using MOs ability to extract BSAs came up again on USKP comments, so I thought it would be good to get that all down somewhere more reliable as a knowledge resource too.
 
Ethatrons quote ( which has been lost otherwise on Ethatrons web site, its been down for over a year now I think and no sign of Ethatron anywhere ), corroborates everyone elses findings with regards BSAs in the "Ramifications of Extracting BSAs" topic, aswell as my own extensive testing of that aspect creating my mod.
 
Reference Fallout, the closest thing ( if there is not a dedicated topic for Fallout - I dont know ) I would think would be the old Oblivion topics on devnull http://devnull.sweetdanger.com/archiveinvalidation.html

 

Edit : For Oblivion users note that BSA Redirection is now long established as the best archiveinvalidation method for that game, Wrye Bash uses that method.

 

 
ArchiveInvalidation
 

 

Everything You Never Wanted to Know And Were Afraid to Ask 

 

 

ArchiveInvalidation Destroyed
ArchiveInvalidation Explained

 

ArchiveInvalidation Destroyed

 

by dev_akm
Updated 1-29-2008

All of the original threads on the Elder Scrolls Forums have now expired and been purged from the system, but this summary of the great work done by the ArchiveInvalidation Explained Research Team lives on! The recent discovery of an entirely new method is now described as well.

Overview

Oblivion normally gets all of its media assets (artwork, spoken dialog, music, etc.) from within a few large .bsa files rather than in individual folders (such as Meshes, Textures, Sound, etc.). The original game and official plugins use these .bsa files (it stands for Bethesda Softworks Archive). These .bsa files are organized internally just like your "Data" folder is.

Although fan-made mods are starting to use .bsa files more often, most mods still place individual meshes, textures, etc. into folders within your "Data" directory. This isn't a problem for new items added by a mod, but it can be a problem for some "replacer" mods that alter original game items. For example, some of the most popular types of mods are "armor texture replacers" and "landscape texture replacer" mods that improve the look of the game by replacing the original "stock" textures.

Normally, this isn't a problem since Oblivion is designed to automatically load any meshes, textures, etc. that it finds in your Data folder, as long as the timestamp (Modification Date) on each of these files is more recent than the timestamp on the original .bsa files. Since the .bsa files are older, items in them get replaced by any newer items with the same name that exist in your Data folders.

In some cases, however, this doesn't work correctly. It now seems fairly certain that this is caused by a bug in Oblivion's ArchiveInvalidation system itself. After more than a year and several official game patches, it seems unlikely that Bethesda will ever correct this bug. This means you will have to use one of several workarounds if you want to install any "replacer" mods.

Further details on this subject, as well as extensive testing results for ArchiveInvalidation problems in general, can be found in the article ArchiveInvalidation Explained (the ESF threads on the topic have expired).

Each of the known workaround solutions has advantages and disadvantages. The first two methods in the list below -- BSA-alteration and BSA-redirection -- are the only recommended solutions. The last two methods -- BSA-invalidation and BSA-extraction -- have such serious drawbacks that they are not recommended solutions. Here's the list of methods along with some description of what's involved in each:

  •  BSA-alteration: This was the first true solution to the problem. It is widely used, very reliable, fairly simple, and most Oblivion mod users are by now familiar with using it. The disadvantages are that you must use one of several external tools (not a big deal since you probably need one of these tools anyway), you must update the BSA changes after installing or removing any replacer mods (although most of the tools will do this in an automated way), and you must remember to remove BSA changes before applying any of the official patches from Bethesda.
  • BSA-redirection:  This is a newer solution that has proven to be very effective. This method has advantages that make it preferable over BSA-alteration. The biggest advantage is that it does not require any external tools (although it does require you to install some files and you may want to use a tool to help you configure it) and once it's set up, you can pretty much forget about it (no intervention is required after adding or removing any replacer mods). 
  • BSA-invalidation: This was the first solution discovered, and as a result was the most widely used method for a long time. However, it has severe limitations and problems and is rarely used anymore because of this. Unfortunately, many early mods still recommend this solution, so lots of users get fooled into thinking it's a good idea. In fact, it's a terrible idea under most circumstances and the only way to make it work reliably is to also use BSA-extraction (see below). You should avoid this method if at all possible.
  • BSA-extraction: This solution works well for mod creators (who often need to have the source files extracted from the BSAs anyway). However, it has major disadvantages for mod players and because of this it is not discussed in any great detail here.

Next we'll examine these methods in a bit more detail.

BSA-redirection

This method works by tricking the Oblivion game engine into using its bugged ArchiveInvalidation method on the wrong BSA file. By doing this, it allows all files in the main Textures BSA to be reliably superseded by any more recent files found in your Textures folder.

The easiest way to use this method is to use the BSA-redirection option in Oblivion Mod Manager if you already have it (see BSA-alteration below for more details). Here are the steps using OBMM:

  1.    Start OBMM.
  2.    Click Utilities.
  3.    Select Archive invalidation. You'll get a popup window.
  4.    Click BSA redirection (it's the default choice).
  5.    Click Update Now.
  6.    Close the Archive invalidation popup (click the red X in the upper-left corner).
  7.    Quit OBMM or click Launch Oblivion.

Alternately, you can install a mod specifically designed to do BSA-redirection , such as ArchiveInvalidation Invalidated!.

You can also set up BSA-redirection fairly easily yourself if you don't mind a bit of tweaking. In the Oblivion.ini file, there is a configuration entry called the SArchiveList. It lists all of the standard BSA files. It normally looks like this:

SArchiveList=Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa, Oblivion - Sounds.bsa,
Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

By changing the list to include another BSA containing at least one texture, you can trick the game engine into processing the bugged ArchiveInvalidation logic for that new dummy file rather than the real Textures BSA (calledOblivion - Textures - Compressed.bsa). So, the modified SArchiveList should look something like this:

SArchiveList=DUMMYFILE.bsa, Oblivion - Meshes.bsa, Oblivion - Textures - Compressed.bsa,
Oblivion - Sounds.bsa, Oblivion - Voices1.bsa, Oblivion - Voices2.bsa, Oblivion - Misc.bsa

To do this, you'll need to either use a BSA file that came with some other mod to be your DUMMYFILE.bsa (any BSA file containing at least one texture should work), create one yourself using a tool like BSA Commander (you only need to put one texture in the BSA).

BSA-alteration

BSA-alteration requires Wrye Bash (as of version 0.60) or one of Timeslip's excellent utility programs: Oblivion Mod Manager (OBMM version 0.7.10 or later) or BSA Patcher. What Wrye Bash, OBMM and BSA Patcher do is to sidestep the ArchiveInvalidation problem by making Oblivion think it never had a copy of the textures you are replacing.

In other words, these utilities edit your Textures BSA so that Oblivion cannot find the original version of texture files you have replaced, thus forcing the game to load the replacements instead of the originals. Without getting too technical, this works much like taking a book and scrambling some entries from the table of contents so those chapters can't be found anymore, even though the chapters are really still there -- thus allowing the process to be easily reversed later by restoring the table of contents back to its original state.

This method of invalidation was initially tested by the ArchiveInvalidation Explained Research Team and has now been adopted as the de facto standard solution by the Oblivion mods community. It has proven extremely stable, but you should still make a backup of your BSA files if you have the hard drive space (or a DVD-burner) just in case something goes wrong (like you accidentally try to run both OBMM and BSA Patcher on your BSA files, which will result in a corrupt file).

OBMM and BSA Patcher will keep track of which files have been renamed in your BSAs and provide a "Remove BSA edits" function to rollback any changes they have made to your BSA. 

Wrye Bash

Download Wrye Bash.

Wrye Bash provides a Replacers Tab that lets you easily add/remove texture replacement mods. It keeps track of which files have been renamed in your Textures BSA and provides a Restore button to rollback any changes it has made to your BSA (if you don't like the results for some reason). You can use the Replacers tab in several different ways:

  • Manual update: Click the Update button after adding/removing replacers. This also works on any replacers you already have installed manually or via another program (like OBMM).
  • Automatic update: Check the Automatic checkbox. The Textures BSA will be automatically updated whenever you add or remove a Replacer. Only works for texture mods installed/removed using the Replacers tab.

You should probably make a backup of your BSA files if you have the space (or a DVD-burner) just in case, but so far nobody has reported any corruption issues.

Wrye Bash requires Python 2.5 and wxPython 2.8.0.1 ANSI to work.

Assuming you have Wrye Bash 0.61 or later installed already, and have already installed a one or more texture replacement mods, the steps you need to take are as follows:

  1. Start Wrye Bash.
  2. Click Replacers tab.
  3. Click the Update button under Archive invalidation.
  4. Quit Wrye Bash

Note that Wrye Bash checks to make sure you don't already have any active BSA alterations from OBMM or BSA Patcher before allowing you to use this feature.

Oblivion Mod Manager

Download Oblivion Mod Manager

Assuming you have OBMM 0.7.10 or later already, and have already installed/activated some mods, the detailed steps for doing this in OBMM are as follows:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Directly Edit BSAs;
5. Click the checkbox for autoupdate on exit and/or click Update Now;
6. Click the red X in the upper-left corner of the Archive invalidation popup to close it;
7. Click Launch Oblivion
8. Play!

If you didn't check the box for autoupdate on exit, then repeat this process after each time you finish installing/activating some new mods. You only need to do this once after you finish installing all the texture replacers you want to play with. You never have to worry about it again until you install more texture replacers.

The latest version of OBMM (0.7.11 or later) provides a lot of new options for the Directly Edit BSAs function. The default options may be okay, but you need to at least make sure to check the box for Generate archiveinvalidation entries on hash collision, since that setting is critical.

Here are the ideal OBMM BSA Settings (for 0.7.11 or later):

obmm_BSA_settings.jpgUsing those settings will take care of everything for you.

As already noted, it's completely up to you whether you want to use autoupdate on exit orUpdate Now, or both. I find that when running with a lot of texture replacements installed, the autoupdate on exit option tends to cause an annoying delay when quitting OBMM or using the Launch Oblivion button. Since I've gotten into the habit of using theLaunch Oblivion button instead of the normal launcher, and I don't install new mods nearly as often as I start Oblivion, I decided to avoid using autoupdate on exit and instead always use Update Now after I install a new mod.

If you later uninstall/deactivate some mods, you'll need to use the Remove BSA editsoption. Follow these steps:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Remove BSA edits;
5. Close OBMM;
6. Delete texture files that you want removed;
7. Start OBMM;
8. Click Utilities;
9. Select Archive invalidation;
10. Click Directly Edit BSAs;
11. You know the rest by now

If you're using .omods, you should skip Steps 5-7 and just Deactivate the .omod instead.

BSA Patcher

Download BSA Patcher

Unlike OBMM, BSA Patcher is a command-line application, although you can launch it by double-clicking the executable. BSA Patcher doesn't include as many configuration options as OBMM, but it accomplishes the same tasks and it only requires .NET 1.1, which means it will work for anyone who is unwilling or unable to run .NET 2.0. It will also work with mono, an open-source version of .NET 1.1.

The latest version of BSA Patcher (as of this writing) uses the BSA alteration code from OBMM 0.7.10 and, like OBMM, it will automatically generate an ArchiveInvalidation.txt file for you containing only the textures that could not be safely renamed in your BSA files. It includes support for several command-line options, but you can safely ignore them.

To use it, place BSAPatch.exe in your \Oblivion\Data\ folder and double-click it once to rename files in your BSA. Run it again to restore them to their original names.

NOTE: If you use either OBMM or BSA Patcher, you should not manually alter your ArchiveInvalidation.txt file. Let one of these programs do it for you. No other solutions are recommended. Never use OBMM and BSA Patcher at the same time.

Older Versions of BSA Patcher
If you have an older version of BSA Patcher, make sure you have at least the version based on code from OBMM 0.7.9. You should probably still make a backup of your BSA files before running it, just in case. Run it once to rename files in your BSA and run it again to restore them to their original names.

If you're not sure what all this is about, don't have any mods installed yet, or just need more information on how to use mods in Oblivion, please read the Oblivion Mods FAQ for more general instructions. 

For more information on ArchiveInvalidation, see ArchiveInvalidation Explained. The results of our testing are summarized neatly there.

UPDATES

The 0.7.10 and later version of OBMM correct an issue from previous versions where several files with very similar names in the same directory (like castleintcolumndetail02.dds, castleintcolumndetail03.dds, and castleintcolumndetail04.dds) might get their textures mixed up if a replacer only replaces one of these files. OBMM now generates an ArchiveInvalidation file for only the textures that could not be renamed.

This feature is now also included in the standalone BSA Patcher, so you don't have to create your own ArchiveInvalidation.txt file for those problem textures by hand.

Go get the update from the links above! NOTE: If you previously used 0.7.8 to patch the BSAs, you must restore them with 0.7.8 before installing 0.7.10 (or later). If you forget to do this and then replace your messed up BSAs with backups, you have to delete the BSAEdits file in your Oblivion\obmm folder before it will work correctly again.

OBMM/BSAPatcher Usage Example

The following example assumes you are using OBMM 0.7.10 along with Parallax Cities 1.2 (all three parts) andLBM Weathered Direction Signs 1.3.

After clicking Directly Edit BSAs and then Update Now, OBMM reports changing 821 entries in 3 BSA files and creates an ArchiveInvalidation.txt file containing only three texture entries:

textures/architecture/castleinterior/castleintcolumndetail03.dds
textures/architecture/skingrad/skingradstone06.dds
textures/architecture/bravil/woodplanks02.dds

 

These three entries are the only textures that could not be safely renamed in the BSAs, so they are invalidated here instead.

Other Solutions

If you don't use BSA-alteration or BSA-redirection for some reason, then the only way to convince Oblivion to load these problem textures from the individual folders rather than from the .bsa files is to specifically force it to do this by creating a special file called (you guessed it) "ArchiveInvalidation.txt".

The ArchiveInvalidation.txt lists the relative path (from the "Data" folder) of texture files you want to replace the default artwork shipped with the game.

As a result of the extensive testing reported in the ArchiveInvalidation Explained threads, we now have very clear information about how ArchiveInvalidation works and how it fails to work. Suffice to say that it can get extremely complicated.

My current best practice, based on the above-mentioned research, is that ArchiveInvalidation.txt should only list just the DDS textures from the original game that are being replaced by individual files in your Data folders. In other words, this means it should only contain entries for texture replacement mods. No meshes, sounds, or voices should be listed.

Be aware, however, that many earlier approaches recommended listing ALL your moded meshes/textures/etc. in ArchiveInvalidation.txt, regardless of whether they replace anything from the original game.

If you do use any texture replacement mods, there is a high probability you will need an ArchiveInvalidation.txt file listing these replacements (if you don't use the OBMM's "Directly Edit BSAs" feature). Without it, chances are that many of Bethesda's default textures will continue to be loaded. Also, if you have bad entries in your ArchiveInvalidation.txt file, or even in some cases if your ArchiveInvalidation.txt file is perfectly correct, you may see items in the game appearing purple or even invisible.

Given all these warnings, if you still want to try using an ArchiveInvalidation.txt file, there are several fan-created tools that will generate these file listings for you automatically.

ElChE's Oblivion Automatic Content Validator 1.0.2 (also called Automatic Content Validator) will create an ArchiveInvalidation.txt file for you automatically, and it also does some other very useful things like making sure all of your meshes have valid texture paths. Be sure to get 1.0.2 or later because it adds the option to only generate listings for replacer files. You may need to manually remove any meshes from the list it generates.

Oblivion Mod Manager, which we mentioned earlier, also has an excellent set of options for automatically creating the ArchiveInvalidation.txt. This process executes automatically every time you quit from OBMM or use OBMM to launch Oblivion. Make sure to get version 0.7.10 or later.

If you prefer to create ArchiveInvalidation.txt by hand, this will work fine, but you can probably guess that it will quickly become a huge problem to maintain all the file listings if you try new replacers very often.

If you do create it by hand, here's what you need to do.

Create a text file named "ArchiveInvalidation.txt" in your main Oblivion folder. This is not the "Data" folder, but is one level up from that, in the same folder with OblivionLauncher.exe and Oblivion.exe. The reason for placing it here is primarily to avoid accidentally overwriting it when you install a new mod.

(You can put the ArchiveInvalidation.txt into your "Data" folder if you want. It will work there as well, but be wary of accidentally overwriting it during mod installation.)

For the sake of creating an example, let's say you have two replacer mods, one that replaces the texture for Daedric cuirasses and one that replaces the texture for glass shields. Your ArchiveInvalidation.txt would look something like this:

textures/armor/Daedric/cuirass.dds
textures/armor/glass/shield.dds

Note that textures whose names end with "_g.dds" and "_n.dds" don't need to be listed. Other DDS files that don't need to be listed include those in the Menu directory and those in "icon" directories.

If your replacer mod came packaged with an ArchiveInvalidation.txt file, you can copy and paste the contents of the lines of that file into yours. If you do this, be careful not to accidentally paste over any lines that you've previously added for other replacer mods.

Many people claim to use a "universal" or "global" ArchiveInvalidation.txt. However, extensive testing by numerous people has determined that this approach does not work at all. It is the same as running with no ArchiveInvalidation.txt file. If someone offers advice saying you should use a file like this:

meshes/ \s
textures/ \s
sounds/ \s

Don't believe it. It's a great urban legend, but it doesn't work.

Modder's Resource

I have released a modder's resource containing sample ArchiveInvalidation instructions that mod creators can cut and paste into their readme files. It's a bit out of date now since it doesn't yet include instructions for using Wrye Bash or BSA-redirection, but you can find it here

Credits

This information would not exist without the help of a lot of people who contributed feedback, usage guidelines, suggestions, test results, and moral support in this thread, including:

* DoomedMarauder
* Malifrax
* vkaryl
* Zanderat
* DragoonWraith
* Skyfire76
* phoenixamon
* Fyrstig
* Fyndolyn
* MaGISTraTe
* Ismelda Lasombra
* ThePenquin
* Quarn
* and many more. Thank you all!
 

 

ArchiveInvalidation Explained

 

by dev_akm

Preamble
This is adapted from my ESF thread: ArchiveInvalidation Explained (2). Unfortunately, like all the previous ESF threads on this subject, that thread has now expired and been purged from the forum. The summary lives on here.

Last Updated: 10-13-2006 11:40 AM CDT

 

OVERVIEW

There is a terrible beast wreaking mass destruction and chaos in the mod community. It is called ArchiveInvalidation. The purpose of our mission here was to slay this horrid creature and prevent it from respawning.

This goal has now been accomplished, but historians (and a few others, perhaps) may still wonder: From whence did this beast arise? What spurred it forth from its dank lair and into our peaceful community? The answers lie within this document and its related discussion threads.

The problems with ArchiveInvalidation described here most often appear right now for people using more than one armor texture replacer, but it could crop up anywhere Oblivion uses common filenames like "boots.dds" for more than one set of things. If you have not experienced any problems yet, it is probably because your configuration does not meet these criteria.

The extensive testing carried out by multiple users and reported in these threads makes it fairly clear that Bethesda probably never intended for anyone to have to use ArchiveInvalidation in the first place. The vast majority of textures seem to load correctly from the Data folders even without using an ArchiveInvalidation file, which explains why so many users get duped into believing that a simple three-line global/universal/generic ArchiveInvalidation has solved all their problems when in reality it has been proven to have no effect whatsoever (it's the same as having no AI file at all).

In fact, we are proposing that the only reason it's necessary to use ArchiveInvalidation at all is due to a bug in the game that causes some texture replacements to fail to load, while others load just fine. Based largely on Malifrax's contributions, we've isolated these problems to certain common directory names and perhaps even certain NIF files (meshes) that appear to have a different structure than ones which load textures correctly.

Note that you don't need to read all of this if you are using OBMM 0.7.8 or later and you select the option to "Directly Edit BSAs". Timeslip's workaround avoids the need for ArchiveInvalidation entirely in our tests.

If, on the other hand, you do intend to use ArchiveInvalidation, or just want to understand what led to such drastic measures as Timeslip's solution, then you do need to read this.

Hopefully, Bethesda will issue a patch for the textures bug at some point and all of this will become an odd historical footnote in the life of Oblivion. Until then, read on ...

ARCHIVEINVALIDATION TEST RESULTS

1) ArchiveInvalidation.txt (AI.txt) is only needed for replacement textures. Also, it is most often required for replacement textures that are specified from within a mesh (NIF) rather than specified in the CS (icons and landscape textures, for example, will usually load without being referenced in AI.txt).

2) Universal/global/generic ArchiveInvalidation methods do not work. Period. End of discussion.

3) Mesh replacers load regardless of whether they are specified in your ArchiveInvalidation.txt (AI.txt), so you don't need to list any .NIF files. If you do list them, try to only list the specific meshes that are giving you problems, and be aware of the ramifications, as described in #4 and #5 below.

4) A single entry of a common filename like "boots.dds" in AI.txt will invalidate all paths containing a "boots.dds" file that exist in both the BSA and your Data folder. It will do this regardless of whether those paths in your Data folder actually contain a "boots.dds" file and regardless of whether you have an AI.txt entry for that path. *

5) Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *

If all this sounds really confusing, you're not alone. It's not easy to understand, even for computer experts, in part because it's such a crazy way of doing things that it seems in many ways like a bug in the invalidation code Bethesda implemented. Perhaps it is a bug, perhaps it is by design, we have no way of knowing.

Credit for this discovery goes to Wonder Dog.

HOW DO I MAKE ONE?

Instructions can be found in the Oblivion Mods FAQ.

Once you get it started, you can list either just individual filenames, such as "boots.dds", one per line, or list entire directory paths, one per line, like this:

textures/armor/Iron/ \s
textures/armor/glass/ \s

In the end, I think discussions about which approach is better - listing files with no path versus listing only paths - probably won't get us anywhere, since both methods are highly imprecise and cause similar errors. Pick one and stick with it or use a mixture of approaches. I don't have enough data yet to make a recommendation one way or the other.

WHAT NOW?

This is a terrible situation any way you look at it. You can extract your textures BSA into your Data folder. That will solve most of the problems, but really it's a terrible solution. If you do this, be careful not to overwrite all the mod files you've already installed. You've been warned.

Probably the only halfway reasonable approach to even begin dealing with this problem is to have a tool like ACV, OBMM, or Wrye Bash check to make sure all replaced common filenames exist in all replaced paths and warn you when this isn't the case.

I think the best we can hope for is a combination of these approaches.

1) Modders creating texture replacers for things with common names in the BSA need to package all the related files with their mod, as many are already doing; if they don't, then their mods would be subject to warnings or blocks from utilities like OBMM and ACV.

This only applies if you are making a texture replacer that affects files with common filenames like "boots.dds", "cuirass.dds", etc. In this case, you must package all of the original textures that you did not change in the directory where the textures reside, such as "textures\armor\iron", along with the ones you did change. Otherwise, the ones you did not change in that directory path will be broken on the user's system.

We've seen this over and over again with mod-makers releasing a 1.0 version of a replacer that only includes their changed files, then (after getting deluged with hate-mail) having to follow up with a 1.1 version that includes all the items in that set regardless of whether they changed the files.

Another solution would be for modders to avoid creating texture-only replacers whenever possible. You can achieve the same result in many cases by altering the texture references in an item's mesh (using NifSkope) so it points to a new directory structure with texture files that don't use the same common filenames. Since meshes are loaded automatically without any need for entries in AI.txt, this avoids the problem entirely.

2) Utilities like OBMM might warn users before they install a mod that it may cause problems, assuming the user takes advantage of the omod format; then the user can decide whether they want to ignore the warning or stop the installation process.

3) Utilities like ACV might warn users after they have installed a mod that they are likely to have problems with certain files (like it already does for meshes); the user can then decide whether they want to ignore the warning or go back and uninstall the mod.

4) Utilities like Wrye Bash might provide some combination of #2 and #3, perhaps with more advanced users in mind.

TECHNICAL DETAILS

The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

DirectoryStringTable
FilenameStringTable

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Here is my first attempt at a simplified explanation of how this works. Unfortunately, I just couldn't figure out a way to make the explanation any simpler than this. (I also couldn't get any colors to work, so we'll have to settle for underscores and strikethroughs instead.) If you have an idea on how to make this explanation any simpler or clearer, please post it or send me a PM.

The FilenameStringTable and DirectoryStringTable are initialized with all the textures in the BSA file, so all the paths and names of existing default textures get indexed in the hashes.

A small sample of these two indexes might look like this:

FilenameStringTableshield.ddscuirass.ddsgreaves.ddsgreaves.ddshelm.dds
DirectoryStringTable textures/armor/glass/textures/armor/iron/

where the underscored items reside in the BSA.

If I install one texture replacer for, say, all the Glass Armor, with AI.txt entries, then the hash entries for those textures get marked as being replaced by files in the Data folders. This means these items will now be invalidated, i.e. marked as being in Data the folders rather than the BSA, so now the hash index will look something like this:

FilenameStringTable shield.ddscuirass.ddsgreaves.ddsgreaves.ddshelm.dds
DirectoryStringTable textures/armor/glass/textures/armor/iron/

where the strikethrough items have been invalidated, but the underscored items still reside in the BSA.

If you're following any of this so far, you can probably already start to see where problems will arise, since all thecommon filenames shared between the two armor sets have now been invalidated from theFilenameStringTable simply because we wanted to replace them for the Glass armor set.

Now, if I want to also replace the texture for just the Iron cuirass, I'm going to run into problems because the hash table entries for all the common filenames in the Iron armor folder, including cuirass.dds, have already been invalidated (i.e., marked as existing in the Data folder rather than the BSA) and now the hash index will look something like this:

FilenameStringTable shield.ddscuirass.ddsgreaves.ddsgreaves.ddshelm.dds
DirectoryStringTable textures/armor/glass/textures/armor/iron/

where everything in both sets of armor has been invalidated.

Because the path "textures/armor/iron/" is now marked as existing in the Data folder, and the hash already knows that path is linked to a set of files that includes shield.dds, cuirass.dds, greaves.dds, gauntlet.dds, and helm.dds, then the hash will now expect to find all those same names in the Data folders even though I only wanted to replace the Iron cuirass.

In fact, the problem is even worse than this because Oblivion immediately recognizes the new directory pathtextures\armor\iron\ as soon as you add it to your Data folder. It doesn't even need to be listed in your AI.txt because the Oblivion automatically scans your Data folders and recognizes that directory path as being one from the BSA that has common filenames which have already been invalidated.

TOOLS

Ultimately, the only really good solution is to use BSA-redirection or BSA-alteration, but if you really want to see for yourself then these tools will help you generate an ArchiveInvalidation.txt file.

ElChE's Oblivion Automatic Content Validator 1.1.0
ACV
- Use the Smart Check feature (in the latest version it automatically ignores meshes; in the previous version, have it open your AI when done, and then delete meshes by hand).

Oblivion Mod Manager (OBMM)
- Haven't tested the latest version, but Timeslip says he has added new AI.txt options in 0.7.6: "It lets you choose exactly which types of files to include, and there's options for ignoring files not in BSA's, normal maps, and files with incorrect extensions. Following MadBat's suggestion, it also deletes empty subdirectories from the data folder. Default settings are to only include textures, and only ones which already appear in a BSA archive. Hopefully, that should keep most people happy." Heck yeah! (In previous versions, use the DDS textures option.)

Wrye Bash also now includes a feature to generate AI.txt, but I haven't tested it yet.

FURTHER READING

Oblivion Mods FAQ

Original Thread: ArchiveInvalidation method tests (now expired and purged from the forums).

GLOSSARY

AI.txt is shorthand for ArchiveInvalidation.txt.

CREDITS

Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:
- DoomedMarauder
- Auctionmule
- Wonder Dog
- Scanti
- Malifrax
- Jarol
- Buerban
- Midnight Voyager
- Zanderat
- Martigen
- MadBat
- rdjeke
- DragoonWraith
- Oblivionmasta
-Quarn
- and many, many others ... Thank you all for this herioc effort!

Share this post


Link to post
Share on other sites

thanks for the link. do you by chance know, which of the listed workarounds are used by the current versions of archive invalidation invalidated and the archive invalidation disabling in NMM? I'm not sure they really still use BSA-redirection.

also, the problem with the UUF3P came from packing all content in a BSA (for faster load times, several compression levels were tested, including none). suddenly a lot of meshes, sounds and scripts would no longer want to work. AII and the patch work properly when distributed as lose files. your link specifically talks about getting lose files to work, not about what to do if bsa content does not want to work. (or did I miss something?)

could it have something to do with whether a plugin is an esp, esm or false flagged esp?

here's a quote from the UUF3P thread:

 

I've played with BSAs a LOT in FO3, and here's what I've found:

1 - None of that nonsense like "sounds can't be used in the same BSA as anything else" is true. Anything and everything can be in the same BSA.

2 - You cannot override a file (like a texture or mesh or sound) with the same name in a BSA. The rules are that loose files are first (with archive invalidation, of course), then come BSAs in the order they are loaded. So nothing in the main BSA can be overridden as it MUST be first. Makes armor replacers a *censored* to make properly in a BSA.

3 - To "override" a file with a BSA, you need to override the object with an ESP/ESM and point it to the file using a different name. For example, to make an armor replacer, you have to override the armor entry and give it a different model file name. So armor replacers CAN be done using a BSA, but they're a *censored* as mentioned in rule 2 since every single entry must be overridden with all new file names. I usually leave the actual file names the same, but put them in a subdirectory so the only change to the overridden entry is to add the subdirectory to the model file name. So when I say it needs a unique name, I mean the entire path + file name, so you can change any part of that - either the path or the file itself.

4 - Voice sound files in a BSA require BOTH the sound flag and the voice flag be set when making the BSA, even if there are no sound effects. Setting only the voice flag fails to cause the engine to use the voice files in the BSA. That might be what caused that notion that sounds and voices had to be in a separate BSA. Sound effects only need the sound flag. A mix of sound effects and voices requires both (duh).

5 - You cannot use compression when making the BSA with Skyrim's CK 'archive' application. Other than not using compression, the Skyrim CK BSA app is the best and easiest way to make a BSA for FO3.

6 - There's a size limit to the BSA, which is the main reason the main game and DLCs use different BSAs for different things. They HAD to make more than one BSA for size reasons, so it was easiest to clump together different things in the same BSA. I'm not sure what the limit is, but have never had a problem with less than about 1.7 GBytes. If you have a monster sized BSA, you may need to split it into multiple BSAs, otherwise a single BSA works just fine for all files regardless of type (excluding ones without a unique name as mentioned above).

Using the above rules, I've put damn near EVERYTHING into BSAs for the mods I use, making turning on and off mods just a simple click in the mod manager. Using loose files for the patch is easy and avoids needing a big ESP/ESM for all the overrides, but for other mods, making a BSA really helps the user. Follow the rules and you can clean up all those loose files so you only every need to worry about the patch and maybe a general armor replacer.

posted @ 20:17, 16 Aug 2015

It seems that BSA's with lower case names will also load after ones with uppercase. This is actually logical in terms of the numerical values of ascii letters. You can, infact, overwrite other files in other BSA's, but they must be made to load after what you are overwriting. Something else to remember is that a BSA may not be parsed correct if it doesn't have an extension. So if your file is named "mymod.esm" then your BSA should follow the vanilla style and have a name like "mymod - assets.bsa".

posted @ 22:24, 16 Aug 2015

apparently it's just horribly broken. thank god we got rid of that archive stuff in Skyrim. and the broken navmesh...

Share this post


Link to post
Share on other sites

BSA files in previous games don't have strict order like in Skyrim. If the same resource is contained in several archives, game won't use it from the last BSA on 100% of occasions.

The other difference is that BSA files are loaded by partial name matching, for example

DLCShiveringIsles.esp

DLCShiveringIsles - Faces.bsa
DLCShiveringIsles - Meshes.bsa
DLCShiveringIsles - Sounds.bsa
DLCShiveringIsles - Textures.bsa
DLCShiveringIsles - Voices.bsa

 

In Skyrim names must match exactly for a plugin and archive.

Share this post


Link to post
Share on other sites

@Alt

Nice write up. But what happened to the [/size] tags?

5 - You cannot use compression when making the BSA with Skyrim's CK 'archive' application. Other than not using compression, the Skyrim CK BSA app is the best and easiest way to make a BSA for FO3.

It's the other way round apparently. Chesko appears to have a script for it on the page. But it would be nice for more info on compression ratios for the file types e.g. voices etc. :)

Share this post


Link to post
Share on other sites

Thank you for the explanation zilav. I'm always baffled when i learn just how much trouble the modding community had to go through with the older titles.

Imstearn, i think there are bsa archivers specifically for fo3 available, that handle them the way it's needed. No need to rely on skyrim bsa tools. And from the looks of it, loose files seem to be preferable anyway.

Share this post


Link to post
Share on other sites

@Alt

Nice write up. But what happened to the [/size] tags?

 

:) Corrected - I have a habit of changing edit mode and sometimes forum software gets confussed as to what should be where

Share this post


Link to post
Share on other sites

Reference Fallout again - I never really got into FONV, much preferred FO3, but even with that game I did not mod it much.

 

I do recall the best Archiveinvalidation method for Oblivion was without a doubt - BSA Redirection.

It was the method adopted by Wrye Bash for that game and worked flawlessly

Share this post


Link to post
Share on other sites

Imstearn, i think there are bsa archivers specifically for fo3 available, that handle them the way it's needed. No need to rely on skyrim bsa tools. And from the looks of it, loose files seem to be preferable anyway.

 

http://www.nexusmods.com/skyrim/mods/247/?

 

gallery_1652_58_6899.png

 

 

Be sure to always check the Game setting, it defaults to Automatic ( Which = Oblivion format BSAs will be produced )

 

BSAOpt is superb - But as mentioned already, beware using too high compression on some files, specifically sound ( xwm format ) files.

 

I think BSAOpts Compression level 3 equates roughly to the official Skyrim Archivers compression ratio, so any of the ten compression levels in BSAOpt higher than level 3 are greater compression than the official archiver can achieve ( and obviously worse for problem files ).

I used BSAOpt Level 10 compression on my texture BSAs for Vanilla Reduced Textures.

 

 

IIRC the problem with sound files is in the following article https://support.microsoft.com/en-us/kb/89879

 

 

How ADPCM Works

ADPCM, commonly termed as a form of compression, is a more efficient way of storing waveforms than 16-bit or 8-bit PCM. It only uses 4 bits per sample, taking up a quarter of the disk space of 16-bit PCM. However, the sound quality is inferior. Because the Windows Sound System hardware only understands 8/16-bit PCM, the computer must compress and decompress the ADPCM into/from PCM, which requires CPU time. 22 kHz mono ADPCM can be decompressed real-time (that is, while playing) on a 386SX/16 megahertz CPU. Higher sampling rates (44 kHz) or stereo files will take too long for a 386SX/16 to decompress, which causes skipping in the audio. 11 kHz mono ADPCM can be compressed real-time on a 386SX/16 computer. To do ADPCM, the computer must have the Audio Compression Manager (ACM) installed.

 

The problem comes due to the files very good compression ( xwm are a form of ADPCM, xWMA, WMA ( Windows Media Audio ) XAudio2 ) - If you read the above quote its the decompression of such files which can be problematic and cause skipping of the audio data. If the files are further compressed inside a BSA ( depending on how heavy the compression ratio is ) the decompression overhead becomes greater and causes the problem

 

I read that link somewhere in a Microsoft forum ( which I cant locate ), and originally the link was to an article which was titled something like How XWMA files work .. And had the same wording as the above quote. I think the xWMA line of compression methods are all just rebadged advancements of the same technology ..

 

.. See also Wikipaedia definition for XAudio2 https://en.wikipedia.org/wiki/XAudio2, which lists the same related audio compression formats in the Features paragraph.

Features[edit]

XAudio2 provides mixing and signal processing for high-level audio APIs such as XACT and as well provides the programmer with high-level functions related to spatial processing.[3]

XAudio2 abstracts audio generation by separating sound data from "voice", allowing each voice to be filtered by programmable digital signal processing and effects processing functions. Voices can be "submixed" together into a single stream.

XAudio2 can natively decompress XMA on the Xbox 360, xWMA on Windows, and various ADPCM codecs on both platforms. XAudio2 2.8, the latest version supports Windows Store App development and can be used in C++/DirectX apps.[5]

Share this post


Link to post
Share on other sites

So BSAOpt > all older game specific packers (edit: It mentiones compatibility improvements for fallout in the latest risky beta; no updates since 2013...)? And how dangerous is using the automatic setting? I think my mods using the highest compression (meshes only) with iirc automatic mode did not exhibit any issues in skyrim

Share this post


Link to post
Share on other sites

If you used automatic to create a BSA, it will be in Oblivion format

 

I did it once and the Skyrim game crashed loading the BSA

 

Remade it in Skyrim format and no problem.

 

There are differences between game formats, so selecting the correct one is important. I think when Ethatron was still active he was going to look into the Auto setting a bit more, but the last thing I saw him say reference the subject was Automatic defaults to producing Oblivion format BSAs

 

No updates since 2013 I dont think is important at all. The program is still marvellous at what it does. The same technology is also incorporated in DDSOpt, it uses the same code for handling BSAs.

Share this post


Link to post
Share on other sites

And people always recommend not to use its bsa handling.

I'll double check which setting i use in bsa opt. Suffice is to say my mod does not crash the game on load. Anyway, it worked flawlessly for me too in skyrim. But i also never touched the beta and stuck with 1.x. I always get a bad feeling when programs are in beta for too long and are abandoned. Especially since the beta has a big 'risky' warning attached and mentions fiddling with the fallout code.

I mean if it says 'automatic' but it doesn't actually to anything automatically i would not call it a software working flawlessly... (especially if the risk of trusting that setting is game crashes...)

Share this post


Link to post
Share on other sites

Thank you for the explanation zilav. I'm always baffled when i learn just how much trouble the modding community had to go through with the older titles.

Imstearn, i think there are bsa archivers specifically for fo3 available, that handle them the way it's needed. No need to rely on skyrim bsa tools. And from the looks of it, loose files seem to be preferable anyway.

It was even worse in Oblivion. If you packed a duplicate asset into more than one BSA that was loading in the game, you risked generating CTDs from that. So with Oblivion you had to be extra special careful to be sure your BSA had unique assets in it that nobody else would be using in theirs.

 

It's basically why Wrye Bash rose to dominance with the BAIN installer. It was the answer to the loose file issues that were a plague upon everything back then. Frankly, it still is for those who insist on dealing with loose files, but why would you now that Skyrim fixed all that? :P

Share this post


Link to post
Share on other sites

So BSAOpt > all older game specific packers (edit: It mentiones compatibility improvements for fallout in the latest risky beta; no updates since 2013...)? And how dangerous is using the automatic setting? I think my mods using the highest compression (meshes only) with iirc automatic mode did not exhibit any issues in skyrim

 

Forgot to mention I am using Beta 2.0.0 - Never tried the "risky" version

 

Here' a screen I made for the comments on nexus to show folk what to select at the top and bottom of BSAOpt when creating a BSA

gallery_1652_58_228926.jpg

 
If creating BSAs - ( As in the screenshot above ) Browse to the folder 'containing' your {textures} folder (ie dont go inside the textures folder) and click "Use Folder"
At the bottom browse to the folder where you want to create the BSA, give it a name.bsa, and choose "Save"
Make sure you select Skyrim in the game menu
Then click "pack"
 
If extracting BSAs - Choose source BSA at the top (browse to it)
Choose destination at the bottom (browse to a folder and click "Use folder" <- It will be extracted here)
Then click "Unpack"
 
 

Edit : For reference as to what not to compress at High Compression levels - See this topic first post ( Any original BSA flagged as UnCompressed by Bethesda, is a pretty good bet contains 'some' files ( not necessarily all of them ) which should not be compressed )

I used my powers of awesomeness to determine the archive flags that were used to pack the official BSAs. This information could be important!

Skyrim - Animations.bsa

Flags: Compressed

Skyrim - Interface.bsa

Uncompressed, Flags: None

Skyrim - Meshes.bsa

Flags: Compressed, Retain Strings During Startup

Resource Types: Meshes

Skyrim - Misc.bsa

Uncompressed, Flags: Retain File Names

Skyrim - Shaders.bsa

Uncompressed, Flags: None

Skyrim - Sounds.bsa

Uncompressed, Flags: Retain File Names

Resource Types: Sounds, Voices

Skyrim - Textures.bsa

Flags: Compressed, Embed File Names

Resource Types: Textures

Skyrim - Voices.bsa

Uncompressed, Flags: None

Resource Types: Sounds, Voices

Skyrim - VoicesExtra.bsa

Uncompressed, Flags: None

Resource Types: Sounds, Voices

Dawnguard.bsa

Uncompressed, Flags: Retain File Names, Retain Strings During Startup

Resource Types: Meshes, Sounds, Textures, Voices

HearthFires.bsa

Uncompressed, Flags: Retain File Names, Retain Strings During Startup

Resource Types: Meshes, Sounds, Textures, Voices

Dragonborn.bsa

Uncompressed, Flags: Retain File Names, Retain Strings During Startup

Resource Types: Meshes, Sounds, Textures, Voices

HighResTexturePack01.bsa

Flags: Compressed, Embed File Names

Resource Types: Textures

HighResTexturePack02.bsa

Flags: Compressed, Embed File Names

Resource Types: Textures

HighResTexturePack03.bsa

Flags: Compressed, Embed File Names

Resource Types: Textures

Note:

By default, all BSAs have these flags: Include Directory Names and Include File Names. You can't change those flags, so I didn't add them to the list.

 

Credit to Fireundubh on STEP forum for finding out the spoilered information

Share this post


Link to post
Share on other sites

the download description for the beta mentions the word "risky". guess I was lucky with only ever packing meshes and textures. it seems the game is less picky with those.

is there a way to check a BSA, to learn which game it was compressed for, whether it contains the right flags and if the compression level is safe?

Share this post


Link to post
Share on other sites

Here' a screen I made for the comments on nexus to show folk what to select at the top and bottom of BSAOpt when creating a BSA

I prefer command line

"bsaopt x32.exe" -game sk -deployment "e:\8" "e:\8\new.bsa"

Share this post


Link to post
Share on other sites

the download description for the beta mentions the word "risky". guess I was lucky with only ever packing meshes and textures. it seems the game is less picky with those.

is there a way to check a BSA, to learn which game it was compressed for, whether it contains the right flags and if the compression level is safe?

 

See my previous post ( edited it a few times and did not notice a page two with more posts had started  :redface: )

 

I have never used the Risky one, just Beta 2.0.0. ( To clarify there are two beta 2.0.0, the risky one has zip support )

http://www.nexusmods.com/skyrim/mods/247/?

 

And see the link / spoiler reference fireundubh

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Support us on Patreon!

×