Jump to content

B.A.E. - Bethesda Archive Extractor


Jon

Recommended Posts

B.A.E.  v0.09

 

SSE Nexus

FO4 Nexus

 

Extracts BSAs and BA2s.

 

 

Known Issues
- Checking/Unchecking files takes increasingly long with the number of files inside an archive.

Version History
0.01 - Initial Commit
0.02 - Significantly faster open times on all BA2s, especially Fallout4 - MeshesExtra.ba2
which took several minutes (now takes ~1s).

0.03 - Width/Height were actually listed as Height/Width in the BA2 so non-square textures were
    incorrectly extracted.
     - Gave window a minimum width/height as Windows 10 users complained of the window being too small.
     - Can open multiple archives via the file dialog, or by dragging to the executable.
0.04 - Cube maps were being extracted incorrectly.
0.05 - Skyrim SE BSA support
0.06 - Added drag and drop to main window
     - Added application icon
- Added About window 
0.07 - Corrected the size sent to LZ4 for decompression, which affected only a very small number of files.
 Please see the sticky on the Nexus page for information.
- Added a file filter which lets you drop a text file with a list of full or partial filenames (using wildcards).
 This was added mostly to simplify re-extracting the files affected by the bug above. 
0.08 - 3-4x faster extraction
- Added a filter to the UI. Searches either filename or entire path, supports wildcards (*)
0.09 - Drag and Drop file extraction.  You can drag one or multiple files from the archive to any location 
  on your computer, including to programs that can open the filetypes you are dragging.
  Ctrl- or Shift- click rows to drag multiple files. The checkboxes do not matter and are used for normal Extraction.
  NOTE: Dragging folders currently not supported. 
 

Special Thanks
ianpatt and behippo for helping with my understanding of the new formats.

  • Like 3
Link to comment
Share on other sites

In other words: The creation engine is actually high-tech now? O.o

 

well, I wouldn't call it "high-tech" but I would say "higher-tech than previous versions" :P

Link to comment
Share on other sites

Congrats on the release!

 

I'm really curious about these texture files and this DX10+ format. I wonder if NVIDIA DDS tools will ever be updated to support it (the plugin hasn't been updated since 2013).

Link to comment
Share on other sites

I havent used it yet - But reference textures :

 

Is it not just going to be a case of repack a new BSA2 containing replacer textures with the same paths as the original, and give the archive a dummy plugin to get the game to load it ( same as the 3 Skyrim High Res DLCs ) ( I did read the part about mips 256 or less being separate from the higher res mips .. sounds like a PITA, even if it is more optimal for the game loading resources )

Ah scratch that, I just had a read of your post in the Big Fallout topic and the problems that will come with "piecing together" textures that do not have headers along with them being entirely new formats we do not have plugins or tools to support editing them yet .. Baarf, that sucks.

 

I'm still downloading the game files at home, and on shift at work, so @ Bethesda - Really bad timing, could have delayed it a week :P.

 

Thank you Jon for taking on such an important project.

Link to comment
Share on other sites

Looks like we have to add subfolders manually in ini for loose files

www.nexusmods.com/fallout4/mods/91

 

Coolness!

 

Though I do think it's kind of silly to require users to do that.  Non-default behavior on this just means mod users have to do more to set up mods for their game.  Awell.

Link to comment
Share on other sites

Thanks for this!  It'll be very helpful in updating BAIN to know where the "allowed" directories to install files are.

 

I'm wondering if this tool is/will be capable of listing every contained file, along with size and (hopefully) the CRC32 of said file.  If not, do you happen to know if the F4SE's teams tool does?  Aaaand again if not, mind if I come back later and pick your brain on the file format to write my own?

 

Thanks again!

Link to comment
Share on other sites

I was showing the compressed size but removed that temporarily.  I need to show the true size.  I don't really have plans on showing the hashes... why do you need it?   And no, the F4SE ba2extractor merely extracts the files.

Link to comment
Share on other sites

0.02 Released

 

Significantly faster open times on all BA2s, especially Fallout4 - MeshesExtra.ba2 which took several minutes (now takes ~1s).

Also, textures which use the BC5 format (_n, _s files) are now extracted more correctly, but less programs are able to deal with these files. I will be adding an option to choose how to extract these in the next release.

 

----

----

 

What's funny about the MeshesExtra issue is that it was basically only that file.  The rest all opened in a couple seconds or less.  Now they are all instantaneous and MeshesExtra takes a sec.

 

I eventually hope to just have a BC5/BC7 decompressor so that I can write out uncompressed DDS.  This is optimal since no one can author BC7 files at the moment (no frontend/plugin exists)  and the BC5 support is almost nonexistent.

Link to comment
Share on other sites

I was being silly and was connecting my model and proxy model (for sorting/filtering) before filling the source model.  This was causing two models to be updated when inserting rows, and also the proxy may have been attempting to sort/filter with every single update.   Additionally I was connecting the proxy model to the view (the visible tree) before filling the model, and both of these were causing an exponential increase (with file count) in time to open the BSA, because of all the updates being sent to the view/proxy.  The pitfalls of using Qt to do all my work for me.. :P

Link to comment
Share on other sites

Looks like we have to add subfolders manually in ini for loose files

www.nexusmods.com/fallout4/mods/91

 

Might be a shot in the dark but... What if it is somehow related to proper Beteshda-made mod manager ?

Link to comment
Share on other sites

The choice of name is somewhat unfortunate :P Well, specifically the acronym

 

It was very much intentional.   Also,  B.A.2.E. would have been a misnomer because it also extracts BSAs.  And the full name is too long for the exe file IMO, so acronym it was, and I decided to actually play up the connection.   The tagline on the description page reads  "Before Anyone Else... there was Bethesda Archive Extractor" which I said in chat as a complete joke, but I enjoyed the double entendre.  I completely detest the word "bae" and the thirty-five thousand backronyms it's received, so you can see this as a judgement on these people if you'd prefer. :)   Or an attempt to reappropriate the word for something less vapid. 

 

Edit:  If I had gone with B.A.2.E.  it would obviously have been pronounced Batooey. :D

Link to comment
Share on other sites

I found unused (well, I suppose) Masser textures in FO4 archives. I compared them with Skyrim ones. They look absolutely identical once opened in Photoshop, but their md5 are different. I only wanted to point out this if it may help to understand/decode the new textures format.

Link to comment
Share on other sites

I only wanted to point out this if it may help to understand/decode the new textures format.

We already know all the formats used.  They're standard DirectX 10+,  the problem with this is that DDS tools are hideously out of date and so most can't open BC5U without being saved as ATI2 (and Nvidia can't even do that correctly) and basically nothing out there supports BC7.  Though I have yet to see a BC7 file but I haven't actually dug very deep.   The engine definitely supports this format though.

 

The issue right now is that I am attempting to decompress the BC5U (and BC7 if applicable) formats to R8G8B8 or R8G8.  RGB is more natural for normal map authoring even though for 3Dc/ATI2/BC5U the Blue channel is completely unused and only get constructed by the plugin during import.    The _s don't have any real use in being R8G8B8 because it's really just a gloss and a specular texture stashed into one, so I could save a bit by writing them to R8G8 though they'd have to end with _s to differentiate between _n BC5U.

Link to comment
Share on other sites

Fallout 4 Tools (BA2 Archive repacker) Youtube

 

HI all,

I have made a tool which can repack ba2 files from Fallout 4, it is still work in progress %5Bcolon%5D%29.gif

Details:
This games has 2 different formats of BA2 files.
1 - GNRL = general format archive
2 - DX10 = texture format archive (found all formats :) i hope)

My tool can, so far extract all formats.
Also my tool can repack files with format GNRL only, also with compression.

Still gonna work on repacker for DX10 archives.. %5Bcolon%5D%29.gif

USE IT ON YOUR OWN RISK!

 

Someone has used?

Link to comment
Share on other sites

Bug to report. The "hand-made" pipe guns use a texture called pipegunpalette_d (textures\weapons\handmade\piperifle\) to determine a diffuse/albedo overlay color. As of v0.02, this is not being extracted correctly. Currently, the extracted texture is 16x64 (widthxheight) resolution, with every 16x16 chunk repeating the same four 16x4 gradient strips. After messing with the texture I determined that only the bottom 16 pixels are being used. I reworked the texture, changing it so that the four gradient strips are aligned horizontally, one after the other. Final texture res is 64x4 and gives a very similar look to the guns in-game as when no extracted texture is used.

 

gif to illustrate

fPtZvFC.gif

 

lanley02 over on the nexus pointed out that non-square textures are being extracted with errors; could be related.

Link to comment
Share on other sites

Bug to report. The "hand-made" pipe guns use a texture called pipegunpalette_d (textures\weapons\handmade\piperifle\) to determine a diffuse/albedo overlay color. As of v0.02, this is not being extracted correctly. 

 

OK this happens with ba2extract.exe from the F4SE team too.  I'm going to have to talk to them about it. 

 

---

 

Fixed it and will release an update soon. 

Edited by Jon
Link to comment
Share on other sites

0.03 Released

 

- Width/Height were actually listed as Height/Width in the BA2 so non-square textures were incorrectly extracted.
- Gave window a minimum width/height as Windows 10 users complained of the window being too small.
- Can open multiple archives via the file dialog, or by dragging to the executable.
Link to comment
Share on other sites

  • 2 weeks later...

Hey Jon, I've noticed something interesting when messing with the ATI2 textures (normal maps and spec/gloss) BAE extracted for me and textures compressed with AMD Compress. Firstly, here are some results from my testing of various options available for exporting/creating dds files:

 

AMD Compress:
BC5 - compressed texture red and green channels do not match source texture (switched)
ATI2N_XY - compressed texture red and green channels do not match source texture (switched)
ATI2N - compressed texture red and green channels match source texture
Photoshop Nvidia dds plugin:
3Dc - compressed texture red and green channels match source texture
GIMP dds plugin:
BC5 / ATI2 (3Dc) - compressed texture red and green channels match source texture

 

Note that F4 textures that use 3Dc/ATI2/BC5 do have the red and green channels switched. Inspecting the normal maps you will see that the Y component is stored in the red channel and the X in the green channel. Typically, it should be X in red and Y in green. Also, as pointed out by Millenia in his thread over on polycount, the spec/gloss maps go spec in red channel and gloss in green channel when using DXT1/DXT5, however this is reversed when using 3Dc/ATI2/BC5.

 

I checked out the files in a hex editor. Here is what I found in the headers of textures compressed using various methods and extracted textures (_n and _s):

 

BAE:

- ATI2

AMD Compress:
BC5 - ATI2A2XY
ATI2N_XY - ATI2A2XY (file is exactly the same as that compressed using BC5)
ATI2N - ATI2
Photoshop Nvidia dds plugin:
3Dc - ATI2
GIMP dds plugin:
BC5 / ATI2 (3Dc) - ATI2

 

Given that using AMD Compress and BC5/ATI2N_XY automatically switches the R and G channels at the time of compression, and that F4's engine expects 3Dc/ATI2/BC5 textures to have reversed channels I think it's safe to assume that the extracted header should be ATI2A2XY. The original uncompressed textures Bethesda has likely do not have switched channels. That is a result of the compression they used.

 

Files with the ATI2 and ATI2A2XY are treated the same by F4's engine. I'm guessing it looks at only the first fourCC (ATI2). Any program currently available that can open these files treats them the same, however that may not be true in the future. It may be better to switch the channels back when importing/opening for editing/viewing then they are switched again when compressing as BC5 or ATI2N_XY. The added A2XY fourCC probably differentiates between textures with switched R and G channels, and those without (ATI2 fourCC only).

 

Most of the above information is wrong to some extent due to my use of the nvidia PS DDS plugin to view the channel swap of textures, so ignore it.

 

Here is the gist of it, stated more correctly. ATI2/3Dc and BC5 store red and green channel information in different orders. ATI2/3Dc stores RG channel data in the order green then red, unless otherwise stated (ATI2N alternate XY swizzle). BC5 stores RG channel data in the order red then green. An additional fourCC (A2XY) is sometimes used for BC5 compressed textures to indicate the opposite XY/RG swizzle, that is to indicate that the channel data is stored red then green instead of green then red. Most tools don't take this into account when decompressing ATI2/3Dc/BC5 textures. A particular tool will either handle all ATI2/3Dc/BC5 textures as if the RG channels are stored green then red or as if they are stored red then green, despite the storage order difference between 3Dc (green then red) and BC5 (red then green). The one exception to this I've found is The Compressonator, which does detect and account for the additional A2XY fourCC properly.

 

See this google doc for a breakdown of how various tools handle ATI2/3Dc/BC5

Link to comment
Share on other sites

  • 2 weeks later...

0.04 Released

 

- Fixes the extraction of cubemaps.
 
----
 
@ImsumDave  I will get around to a response in a bit, probably via our PMs.  The Intel plugin was released so this changes things.  I will probably just give the option of writing directly to a BC5U FourCC now.
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...