[UPDATE] BSAs and You

[lmstearn]

Ah yes, that's the one- a complete solution for all games.  And- er:

Spoiler

Quote

Edit : And a bonus question because someone is going to ask this anyway - Will there be a GUI in future for command line phobes ?, if not then at least that project idea is open to anyone with the know how to take it on. Personally I am just as comfortable with command line.

From another of the first_metacarpal fellowship from TypoLand- a gui wrapper might be possible- but only if Zilav okays it.  

 

 

[godescalcus]

On 4/23/2018 at 12:02 PM, alt3rn1ty said:

Well according to the latest pack and ship of the Unofficial SSE Patch, the CK borked it and skipped a fix script from the files required entirely, it was the fix to Dawnbreaker duplicates which was recently solved in the pre-release beta.

So at least in that case it was the CK at fault .. and not the archiver itself ?

I dont use the official Archiver (just because I have no need of having the whole buggy CK installed, and not due to any kind of preference)

-----------------------------

I forgot to mention here a new tool by Zilav - BSArch https://www.nexusmods.com/newvegas/mods/64745?

I've just tried my first go with BSArch by archiving a ported version of Wyrmstooth 1.17B - it seems ok so far from reloading old saves in WT, but further testing is pending on my getting back there.

I built a non-compressed .bsa because I've been reading around that sound files get funky if you compress them. However, the original classic version of v.1.17B published by Jonx0r has a much smaller bsa, less than half of mine, so I assume he compressed it with sounds and all. Can anyone confirm that sounds (music, voice?) shouldn't be compressed? Is it only for SSE and not classic Skyrim - since SSE uses a different compression protocol?

[Arthmoor]

There's recorded voice lines in Wyrmstooth. Compressing that BSA would be a bad idea unless they fixed that issue with SSE.

[alt3rn1ty]

The compression method is part of the issue. The voices in LE and SE already use their own very high compression, compress them again and it becomes a CPU load and timing issue just extracting them twice with different decompression techniques. I know other files have different types of compression in addition to BSAs compressing them aswell (such as textures), but it only becomes an issue for sound files.

Emma's Vilja for both Skyrim LE and SE use non-compressed BSAs, her mod uses something like 3500 voice files and compressed BSAs cause issues in both games.

See also this post and my next post in the same topic.

Basicly : WAV files, which are greatly compressed into XWM files (in Skyrim these are a new form of compression based on microsoft XWMA, which is also a derivative of Windows Media Audio (WMA)), and then even further compressed into a highly compressed BSA = Sometimes they do not get loaded by the game engine or sound like corrupt garbage (like an old dial up modem on a dirty line, or sometimes parts of a voice will go a bit Dalek).

And if you go looking for Microsoft and XWMA encoded files, I think you will eventually get led to MS blabbing about ADPCM (which I believe was a rebranded XWMA format; The reason for that belief is the ADPCM page I just linked reads just about word for word the same as the old XWMA page did (which was removed from Microsoft sites), with a few changes). You will note there that the higher the compression setting for such files, the result is lossy and degraded, and also the interpretation of loop points can be lost the further you up the compression on the file. This sounds all too familiar for those of us who have witnessed the symptoms of the problem first hand, I am guessing but maybe adding BSA compression also has similar effects on the file because you are taking it beyond its original compression level.

It also depends on the power of your CPU (or the CPU of people using your BSA which may be less capable) to on-the-fly decompress the BSA for the file, then to decompress the highly compressed XWM, then to play the WAV (which is not a compressed format in itself).

[zilav]

Take a look at vanilla archives which contain (even partially) any sort of audio files

Skyrim LE

• "Skyrim - Sounds.bsa" - uncompressed
• "Skyrim - Voices.bsa" - uncompressed
• "Skyrim - VoicesExtra.bsa" - uncompressed
• "Dawnguard.bsa" - uncompressed
• "HearthFires.bsa" - uncompressed
• "Dragonborn.bsa" - uncompressed

Skyrim Special Edition

• "Skyrim - Sounds.bsa" - uncompressed
• "Skyrim - Voices_en0.bsa" - uncompressed

 

Every single one is uncompressed. If Bethesda don't compress archives with audio, neither should modders. The reason is not important, generally when modding you try to do the same as Bethesda, or you are asking for troubles even if it seems to work at first.

[Arthmoor]

It's also not just a matter of degrading the quality. The ones you're lucky enough to have load will indeed sound like shit, but the real danger is that compressed BSAs with voice files inside often crash the game instead of doing something useful.

[godescalcus]

Thanks guys, great info as usual. Uncompressed Wyrmstooth worked flawlessly.

I'm trying to pack DynDOLOD's output assets (SSE), has anyone done that yet?

Previously to replacing DynDOLOD's loose files, I checked every file conflict with other loose files and temporarily removed them from the original mods. It was surprisingly easy as only a handful of mods supplied distant object meshes. Given that DynDOLOD.esp loads DynDOLOD.bsa last, I assume there won't be conflicts with archived assets that aren't won by DynDOLOD.

Still, when I loaded an ongoing game with the newly packed DynDOLOD, it crashed on load.

I'm using BSArch to create an archive. Since there are no sounds, I used the syntax "bsarch pack . DynDOLOD.bsa -sse -c".

The reason for the crash could have been that the bsa was over 4Gb, as I have a lot of worldspaces like Falskaar, Wyrmstooth, The Grey Cowl, Moonpath, etc...

Can you guys confirm that the size of the bsa could be the cause of the crash? What would be the limit for SSE? I have this problem on other mods as well (example: Quaxe's Questorium) 

I could try to split the archive into meshes+skse and textures. What other archives will load from the same plugin in SSE besides textures - can I use DynDOLOD - Meshes.bsa for example? Most of my DDL assets are meshes, actually. If I could load part of them on a DynDOLOD - Meshes.bsa and the rest in DynDOLOD.bsa I could keep archives below 2Gb.

Excluding sound files, would the game have an easier time loading assets from compressed archives or uncompressed? I'm assuming from all the bottleneck talk linked at the beginning of this article that compressed would be the way to go. If not, is there still an advantage in using archives over loose files (seeking times, etc...?)

[zilav]

Technically in SSE archives they changed data offset field from Int32 to Int64, so it can address archives of any size. But internally in the engine they could still treat that field as 32 bits, even signed, limiting overall archive size by 2GB. And since none of vanilla archives is larger than that, I assume that 2GB is indeed the limit.

"bsarch pack . DynDOLOD.bsa -sse -c"

There is no such parameter as -c for bsarch, maybe you meant -z for compressed archive?

[godescalcus]

-z is what I used.

[godescalcus]

I know this forum is under TES5 Skyrim, but I wanted to ask about bsa's and Oblivion. Is it good to archive large texture mods like QTP3 Redimized? Does Oblivion handle .bsa files very differently than Skyrim? Is that where the archive invalidation thing matters? If you can point me to the basic stuff on that subject, I'll read it. I've been having good results from preserving and repacking archives in Skyrim, should the same apply to Oblivion? But then, there's a limit to the number of archives Oblivion can load, isn't there?

[Arthmoor]

BSA handling is broken in all games prior to Skyrim. If any files overlap files in another BSA (which QTP3 would do) you'll either get randomly loaded resources or a straight CTD. With the number of assets involved it's not really worth doing. A BSA for an Oblivion mod only makes sense if you have a large number of assets guaranteed to be unique to the mod.

[godescalcus]

Is SSE capable of loading SKSE folders from bsa? 

I finally got around to packing dyndolod's output in bsa. Had to use DynDOLOD.bsa, DynDOLOD - Textures.bsa and DynDOLOD_Dummy.bsa (with its extra dummy plugin) to keep every archive below 2Gb. Ran the game, didn't crash, which means at least the meshes are loaded. There are no purple "texture missing" distant objects either. However, I got constant "DynDOLOD can not find master data in DynDOLOD_None.json" errors popping in-game.

My feeling is that either the whole skse folder, because it isn't a base-game standard folder, or its contents (namely json containers) because they aren't standard game files, are not loaded from the bsa. To test that, I ran the game with only the skse folder as loose files, and bingo! No errors popping.

Could someone confirm or deny that it is the case? Then, only standard game folders and file types are loaded from bsa archives?

[zilav]

Deny. SKSE and it's plugins are not a base part of the game which uses archives for resources lookup (unless they changed something recently).

[Arthmoor]

The game is only designed to load things in the folders Bethesda created. The SKSE folder is not one of those, so anything in it would be ignored inside a BSA file unless the SKSE team added functionality to do that in SKSE itself. Which I don't think they will because it's not really necessary.

[Dark_wizzie]

I'm probably totally off everywhere but... But if a 970 Pro or 900P can read x10 speed of HDD of 2015, and Lzma compression is x4-7 slower than Lz4 but compresses maybe 30-50% less, wouldn't the math flip the other way and make loose faster for SE? I heard in FO4 there is a bonus for using BA2, so not counting that... Compression algorithms aren't going to improve at the same rate as SSDs so with time it would tip the scales slowly over the years. Wouldn't it be possible to test which is faster for loading times? A vanilla game + a giant mod of many, large textures, test loading time from main menu to a save with bsa vs loose version?

 

Also, Zilav suggested just doing uncompressed BSAs for Oldrim. What is the point of that vs loose if it's not even compressed?

 

edit:

I tried to compress a texture folder into BA2 with Archiver2 and BSA with Oldrim's Archive tool and got almost identical results. Am I doing it right, and I would be getting so much faster decompression here even though the file size is remarkably similar?

[lmstearn]

The good idea of BSAs is not to burden the OS file system, when the number of other programs with vast amounts of files on our disks is also rapidly expanding. The BSA files are speedily extracted into OS memory for fast retrieval, else the game engine must interrogate large file system indexes, which also takes processing resource as there is an allocation for a long path name with each file. Moreover, if the game engine used loose files, it would have to be in reserved or dedicated folder other than Data, so not to cause issue with mods, and this can be an issue with fragmentation on HDDs when official game updates are deployed. Official game updates may remove or rename individual files as well, which requires management of version based code- can be a headache when there are many succeeding versions. A BSA file is just one file, after all.

With SSDs, all the above issues are lessened, certainly, and these days CPU crunching and memory transfers are also very fast, so it's not such a big issue as in bygone days.

Edit: With BSAs for mods, it's the same thing- they're more portable, and much easier for file management when upgrading or even downgrading. Should be little difference when using either a fast compress or no compression method at all. To test compression, one might consider using the Skyrim Performance Monitor on loading a mod with a BSA and then the same mod with loose files. Start off in an interior cell and time from exit into an exterior cell, and time it. As the difference will be milliseconds, worthwhile results may only come from something like Process Monitor, however.

[alt3rn1ty]

I havent done any tests recently, and only have one SSD I could use for such which no doubt is not going to be able to compare with the most recent.

But I reckon besides SSDs getting faster as memory technology evolves, so does CPU speed and compression software to take advantage of it. So I think without testing the comparative throughput these days is probably about the same as it used to be when Ethatron tested. Depends on your CPU capability.

I dont know if Fragmentation of files across the surface of a drive counts for anything with SSDs.

For the future I have no doubt that the new trick of splitting up the biggest resources, textures, into big layer being separate from the mipmaps layers for faster / optimal loading of them from within BSAs, will become adopted by all future games from bethesda using the gamebryo engine.

 

Logically, if the games company thought it more optimal to extract the files loose when installing the game for you, and typically everyones machines (more often for gamers always upgrading to better rigs for the next gen games) will have vast capacity Hard drives, then they would extract them for the better performance of the game.

But they dont, so doesn't that tell us something at least ?. To me that says in the game companies opinion, that BSAs are still giving the most optimal way of delivering resources as the game needs them.

[Arthmoor]

I doubt very much anyone is going to notice a speed difference of any sort between BSA vs loose files. That's not why BSAs are favored. They're favored because the game engine is designed to be optimized for their use internally. Which is where you're going to see other performance gains based on how Gamebryo is handling the file I/O, file indexing, etc. BSA files are hashed during startup so they're going to have an advantage there. Loose files are not, which means the game has to ask Windows each time. Yes, it's milliseconds, but enough milliseconds piled on via system calls can eventually lead to fps drain since that's also allocated in milliseconds.

It probably does not matter much if the BSAs are compressed or not. I doubt anyone will notice the CPU hit on that. I usually just go with whatever the SSE CK sets up for that. Compressed or not, it has never failed to work properly. Plus SSE may have some of the texture streaming optimizations that went into FO4 if you're using BC7 instead of DXT1. Those optimizations will not be used on loose files.

Bethesda is also far from the only company that handles delivery of game assets this way. I don't think I've ever seen a single one in modern times where files are stored loosely in the game's data folders. So obviously they're all trying to take advantage of the contiguous file blocks in those archives for some reason.

[Dark_wizzie]

I think the consensus has been to use BSAs, it's just not as clear whether compressing is better or not. It's probably more precise to record via Shadowplay and go frame by frame to see which loaded faster, although that's a time consuming process. It probably also matters which cores are going to be doing the work of extracting files. If it's single-threaded then it'll be slower than when OP tested because single thread perf has increased more slowly than HDD -> SSD + SSD improvements in the past several years.

 

Is there a way to tell if SSE has the texture streaming optimizations which FO4 has? I would've thought, that if we knew FO4 had the function that everybody would know if SSE had it shortly after launch.

[Arthmoor]

Recording the session in itself is going to slow you down some, and it may be enough to obscure the difference between compressed and non-compressed. I honestly wouldn't worry about it. The CK compresses the texture BSA, but doesn't compress the other one. For SSE anyway. I'm not sure what the CK does for FO4 because I use the external archiver for that since the FO4 CK can load a pre-existing archive for upload.

[alt3rn1ty]

FO4 is the one that uses the new split textures. I was hoping SSE Also used it but apparently not. The idea behind it being ..

.. for 95% of the time the game only needs to load smaller mipmaps of any texture, because they are mostly all at a distance where the large base texture layer is not needed .. So why load it. FO4 loads up the separate mipmaps file for each texture and only loads the separate large base layer on the rare occasion its required. Times that by thousands of textures being streamed to memory makes the game loading a lot more optimal than not using this new technology.

I think the official archiver is the only tool that can do the trick of storing the large base layer as a separate texture to its mipmaps when packing them into a BSA. So if the special textures BSA is not being made by the official archiver for FO4, the textures are not being packed optimally for that game.

SSE being an older fork of the same code as FO4 did not get this part of the advances with textures, but did get BC7 format textures. So SSE textures can be packed with any BSA tool I think, and benefit just from the compression of BC7 with barely detectable artifacts (remember the female face normal map of Skyrim LE when it was released in 2011, because of blocky noses people started using face and body textures in uncompressed formats because they looked so awful compressed).

Dont quote me but I am pretty sure thats the way it is on the texture front.

 

Zilav and Sheson told me about 565 textures being very good for Skyrim SE (especially LOD normals, I select 565 while beta testing xLODGen), but only if you are not on Windows 7 and have upgraded to a newer OS that supports DX11 .. Then they are not an issue and look really good.

[zilav]

7 hours ago, alt3rn1ty said:

I think the official archiver is the only tool that can do the trick of storing the large base layer as a separate texture to its mipmaps when packing them into a BSA.

Nope, it is not the only one

Which mipmaps are stored separately (chunks as Beth call them) is determined by Single Mip Chunk Area setting in Archive2, by default it is 512x512px so mipmaps that are larger than that are separate chunks, the rest are stored in a single chunk. There is also Mip Chunk Count setting to limit the total number of chunks per texture.

[alt3rn1ty]

On 12/5/2016 at 8:52 PM, alt3rn1ty said:

~ To extract Skyrim SE BSAs, we have :

Bethesda Archive Extractor by Jon

BSA Manager by ousnius

 

Repacking BSAs in the new format is currently only supported by the official Archive.exe for Skyrim SE ( Needs Creation Kit )

 

@Zilav thanks for the gentle reminder - Correction to the above quoted post, last line :

Repacking BSAs in the new format is currently supported by the official Archive.exe for Skyrim SE ( Needs Creation Kit ), and ..

 

BSArch by Zilav

And as Zilav says, it also supports packing the special texture BSAs

[Arthmoor]

Do we even know if SSE is handling BC7 textures in some special way in the archives? Fallout 4 for sure is, and its archiver should be handling all of that already.

[zilav]

26 minutes ago, Arthmoor said:

Do we even know if SSE is handling BC7 textures in some special way in the archives? Fallout 4 for sure is, and its archiver should be handling all of that already.

No, it works the same as Skyrim LE (and Fallout 3, and New Vegas). The only difference is faster lz4 compression instead of zlib in compressed archives.