B.A.E. - Bethesda Archive Extractor

[ImsumDave]

Yes, Intel Texture Works PS plugin going public does change things. Of course those who use GIMP will still have to deal with the mess that is BC5/3Dc/ATI2. Nvidia Texture Tools 2.08 is still their best bet. It compresses and decompresses BC5/3Dc/ATI2 textures correctly, at least as they relate to Fallout 4 (RG switched when compressed and switched back when decompressed) (RG channel data is stored and decompressed in the order red then green).

 
I've searched for information on the differences between BC5 and 3Dc/ATI2 and not found anything significant. Usually when they are mention together it is to say that BC5 is the DirectX10 version of 3Dc/ATI2. Basically, that they are same, but that does not seem to be completely correct.

 

I think what happened was that the R then G channel storage order that is optional with ATI2 became the standard with BC5. Tools designed with ATI2 in mind expect the RG channels to be stored in the order G then R and usually ignore the additional A2XY fourCC. Tools designed with BC5 in mind expect the RG channels to be stored in the order R then G and decompress/import the RG channel data accordingly. AMDs Compressonator tool seems to be only one that deals with textures with the additional A2XY fourCC correctly.
 
To add to what I posted previously, Nvidia Texture Tools 2.08 uses the ATI2A2XY fourCCs for BC5 compressed textures. Intel's PS plugin uses BC5U (BC5_UNORM). Both switch the RG channels during compression and switch back on import or decompression. Both store and decompress the RG channel data in the proper order (red then green) for BC5.
 
With all that said, the additional A2XY fourCC is ignored by most tools/apps/plugins except for The Compressonator, so leaving things as is and giving the option to use BC5U, as you said, will probably be fine. However, I think adding the additional A2XY fourCC would be best.
 
If anyone's interested here are two tables to illustrate the differences between BC5/3Dc/ATI2 and BC7 support in various dds texture tools.

Edited December 8, 2015 by ImsumDave

[Kesta]

I'm a bit ashamed to put this request right after ImsumDave's post but...

 

Could you make the OpenFileDialog default to FO4's data folder instead of the excutable's one ? I know it's just two tiny lines of code.

[Jon]

@Kesta

Two tiny lines of code where?  Are you assuming that the path is always `C:\Program Files (x86)\Steam\steamapps\common\Fallout 4`?  Because obviously that's not true.  Doing it correctly is way more than two lines.  In fact with NifSkope I've had people say it still doesn't see X or Y game despite there being dozens of lines for detecting the game installs, because for whatever reason the line in their registry is not filled correctly.

 

 

@ImsumDave

 

With all that said, the additional A2XY fourCC is ignored by most tools/apps/plugins except for The Compressonator, so leaving things as is and giving the option to use BC5U, as you said, will probably be fine. However, I think adding the additional A2XY fourCC would be best.

 

I'm confused how exactly this works.  The DDS header does not allow for a FourCC greater than 4 chars long.  Out of all your posts I haven't seen exactly how they're implementing such a FourCC because there is simply no room for it in the DDS header.  The next 4 bytes after the FourCC are the RGB bit count, if the texture is uncompressed.  So technically having a "FourCC" of ATI2A2XY is actually a FourCC of ATI2 + abusing the next 4 bytes of the header by doing something that it's not supposed to.

 

So to me it's a very hacky solution and it seems very proprietary, as only one company's tools bother with it.  I'd really rather not support their decision to abuse the DDS header.  

[ImsumDave]

I'm confused how exactly this works.  The DDS header does not allow for a FourCC greater than 4 chars long.  Out of all your posts I haven't seen exactly how they're implementing such a FourCC because there is simply no room for it in the DDS header.  The next 4 bytes after the FourCC are the RGB bit count, if the texture is uncompressed.  So technically having a "FourCC" of ATI2A2XY is actually a FourCC of ATI2 + abusing the next 4 bytes of the header by doing something that it's not supposed to.

 

So to me it's a very hacky solution and it seems very proprietary, as only one company's tools bother with it.  I'd really rather not support their decision to abuse the DDS header.  

 

Yes, the A2XY code is located in the 4 bytes directly after the usual 4 bytes used for the fourCCs. 

 

Here is a screen of how the dds header with the ATI2 and A2XY fourCCs appears in my hex editor:

 

If the 4 bytes occupied by the A2XY additional fourCC is reserved for other data then I agree that it's ok to leave it out. Although Compressonator is the only tool I found that differentiates between ATI2 only and ATI2A2XY when decompressing/importing textures, tools produced by both AMD/ATI (AMD Compress CLI released mid 2015) and Nvidia (Nvidia Texture Tools 2.08) do implement the additional A2XY fourCC when compressing BC5 textures. ATI2/BC5 being a compressed dds format, the A2XY fourCC is probably ignored in most tools because, as you said, those 4 bytes are usually used for data related to uncompressed textures. I would guess that's also why the designer(s) of the ATI2 compression type used those 4 bytes. They knew it would otherwise be empty.

 

On a side note, alarmingly, I think I may have been thinking about the whole channel swap thing wrong. In reality, BC5 compression stores the red and green channels in a different order (red then green), where as ATI2/3Dc does the opposite by default (optional alternate XY swizzle). My confusion came from the fact that the nvidia PS plugin and the GIMP DDS plugin use the ATI2/3Dc specs. They expect RG channel data to be stored in the order green then red for any texture with the ATI2 fourCC. The Fallout 4 BC5 textures have RG channel data stored red then green, technically not swapped/switched/swizzled. Looks like I will have to amend all the posts I've made on the subject up to this point, lest they continue to spread misinformation.

This comment on a code issue for Nvidia Texture Tools is what tipped me off:

If you look at the DX10 block compression format spec:

http://msdn2.microsoft.com/en-us/library/bb694531.aspx

you will see that BC5 is supposed to have two blocks, the first one encodes the R and

the second one encodes the G channels. That's what I implement in the texture tools.

However, ATI's 3Dc format uses the opposite convention, G and then R.

EDIT: A2XY appears among other fourCCs in this FFmpeg dds image decoder code under a code comment that says "ATI/NVidia variants sometimes add swizzling in bpp." This may not be the only instance where those 4 bytes are used for an additional fourCC. Might not just be a hacky solution. Of course that doesn't change the fact that most tools ignore it.

Edited December 8, 2015 by ImsumDave

[Kesta]

@Jon, no, that would be one. I'm assuming the path is always :
 

const string registryGamePath = @"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\bethesda softworks";
public static string game = "Fallout4";
string Path = Registry.GetValue(registryGamePath + "\\" + game, "installed path", null).ToString();

//other stuffs

openFileDialog.InitialDirectory = Path;

Or similar, depending on what language you used for B.A.E., but core functions are all in .NET

Made it 4 lines to allow to switch between Beth's games ^^

Not sure why other people have their directory messed up for other games with NifSkope, but it'd most likely be because they aren't using x64 system, which is now impossible if they want to use FO4 afaik. But don't bother if that's making things worse, this is really not a priority, just a request from a lazy user 

[zilav]

I was working with Assets Manager in FO4Edit and stumbled on a file that failed to decompress from Fallout4 - Meshes.ba2

meshes\architecture\buildings\decokit\decomainbdam1x1cora01dam03.nif

 

Error was coming from zlib library used by xEdit, I tried to replace it with builtin Delphi's one and still got the same error. Then decided to test with BAE and it writes 0 length file, so obviously the same zlib error. The file table in BA2 however shows that this file has 114160 packed size and 197010 unpacked.

Now I wonder is it some format used in BA2 not supported by zlib or it is a true data corruption. There could be other files too, not tested yet.

[Jon]

This seems like a platform issue, somehow.  decomainbdam1x1cora01dam03 opens perfectly fine for me.  Straight from the BA2 inside NifSkope:  https://i.imgur.com/K9IfGgZ.jpg

 

The code in NifSkope is copy-pasted from B.A.E.,  but I went ahead and looked at my extracted folder to verify it extracted OK:  https://i.imgur.com/HUdBYnW.jpg

[zilav]

Strange, BAE failed to extract for me. Need to test more.

[zilav]

Update: I tried the same xEdit code today and it works now, BAE also extracted that file just fine. Looks like rebooting helped here, but don't have any idea what was that and why it affected only that single file. The mystery of Windows 7 x64 I guess

[Jon]

I was going to suggest some kind of memory corruption.  The program probably extracts it to the same memory address each time and something happens there?  I dunno.  I'm not really great with file I/O and low level memory management.

 

Anyway, I had the exact same thing happen to a different file once.  Debugged it for hours and then it just went away for a while, then came back again.  Didn't happen again after restarting either. 

[Mator]

Hi Jon,

I've been looking into various BSA/BA2 libraries lately to see if I can find a solution for one of my projects.  What I'm looking for is a C++/CLI wrapper to interface with BA2 or BSA files.  BA2Lib does what I need, but it doesn't support BSAs.  I started looking at the source code for BAE to see if I could make a CLI wrapper around it or port the code necessary to BA2Lib to make it work with BSAs, but I've never coded with Qt before and it's a bit confusing to me.

 

How hard would it be to make a CLI wrapper around BAE?  Is it something I could do in a reasonable period of time?

[Jon]

I threw BAE together in a few hours from the NifSkope BSA code which was already a pretty crappy starting point.  I wouldn't rely on it as any kind of dependency, it's simply not written well enough.   BA2Lib appears to be just ripped straight from ba2extract which is by ianpatt,  and that's why it lacks BSA support.

 

I assume you're talking about Smash and if so this is relevant:  https://github.com/wrye-bash/wrye-bash/issues/296

 

To me it looks easiest to upgrade libbsa for ba2, as long as you don't need to open texture-optimized BA2s with your app.  There were pretty much minimal changes to the general archive format.

[Mator]

I threw BAE together in a few hours from the NifSkope BSA code which was already a pretty crappy starting point.  I wouldn't rely on it as any kind of dependency, it's simply not written well enough.   BA2Lib appears to be just ripped straight from ba2extract which is by ianpatt,  and that's why it lacks BSA support.

 

I assume you're talking about Smash and if so this is relevant:  https://github.com/wrye-bash/wrye-bash/issues/296

 

To me it looks easiest to upgrade libbsa for ba2, as long as you don't need to open texture-optimized BA2s with your app.  There were pretty much minimal changes to the general archive format.

 

Not actually talking about Smash.  I only need to get a the nameTable from a BA2 or BSA archive.  That and being able to access the code from a C# application are my only requirements (which can be done fairly easily using CLI).  It's even better if it is C++, as there's a possibility of using emscripten to port this to Javascript at some point in the near future so it can be used in a web application.

 

EDIT: Looking into libbsa.  Finally got around to actually reading your link.  ^_^

[Kesta]

Something I wasn't sure was an actual bug or just me messing with my files way too much, but I just found it's actually 100% reproductible :

When extracting an archive, if you click on Extract and then cancel in the folder selection, it will extract the archive in the root of the HDD (Like E:).

[Jon]

Yeah, I know about that issue.   That fix alone wasn't really worth an entire release though so it's never gotten fixed.  

[Kesta]

Fair enough  

[alt3rn1ty]

For anyone who has not noticed .. Jon has been busy

 

http://www.nexusmods.com/skyrimspecialedition/mods/974/?

 

0.09 is up, and you may wish to have a look at the sticky post in case you are using any files which may not have been extracted correctly

[Jon]

in case you are using any files which may not have been extracted correctly

 

BTW it only affects anyone who extracted with 0.05 or 0.06.  Ironically, prior private builds I shared with several people did not have the issue.  The bug came right before release when I swapped some code around.

 

Suppose I'll actually update the info here.

[alt3rn1ty]

<Wipes brow> Thats me safe

[Graywolf]

Is there anywhere else it can be downloaded from? The Nexus login page is not being very cooperative lately...

[Nico coiN]

Try the direct-download link from nexus.