The Big Fallout 4 Thread

[Elgar]

It's probably a dumb question, and sorry if you already answered to it : Arthmoor and the UP Project team, do you plan to also create an Unofficial Fallout 4 Patch and to monitor it ?

[Nico coiN]

@Elgar : The game will have plenty of bugs, of course. I may personally contribute in the same way as Skyrim : meshes, textures, placed references and various fixes. Various reports I can't manage, too (mainly all the scripts related stuff). I guess Arthmoor may do so. But leading and monitoring such a huge project is an harassing experience for sure, and I don't know if Arthmoor wants to endorse the same role, responsibility and sometimes 'bashing'. It is humanly hard and exhausting (I sometimes felt the same when providing support to the french community). So, only him can give you a clear answer...

 

It also depends on the players/modders will and interest to contribute. The Elder Scrolls series have a huge amount of fans, but what about the 'recent' Fallout series (I mean since Fallout 3) ? New Vegas had some kind of unofficial patch team quite lately...

[VaultDuke]

Afaik, the current heads of the UUF3P are planning to lead a project for F4.

[Arthmoor]

It's probably a dumb question, and sorry if you already answered to it : Arthmoor and the UP Project team, do you plan to also create an Unofficial Fallout 4 Patch and to monitor it ?

We haven't really discussed it much. I'm not sure how many of the existing team members will even be getting the game, much less be willing to deal with what that entails.

 

I'll just say that I can't rule it out, but it's something that absolutely will not happen before official tools are released.

[behippo]

I just thought I would pop in and confirm that the F4SE team is all geared up and ready to dive into work when the game unlocks. Obviously we are not sure how long it will take to have a tool that folks can use, but we certainly expect to have some form of release a good time before the official tools are out next year.

[Hana]

@Elgar .... no, I've never been into the FO franchise, so will not be on any Unofficial FO4 patch team.

[Elgar]

Thank you all for your replies.

[Eternity]

The .bsa format has been apparently upgraded to ".ba2" format. People in reddit are getting paranoid, but I guess this change is to extend the file size limit, like it was expected. Although, they say too there are 15 or 16 archive files, so maybe they didn't do it after all

 

EDIT: there is actually an image of it: http://i.imgur.com/rjSd1MB.png

They have indeed removed the 2GB file limit, but I wonder if they just updated to only 4GB or if they just felt like using a lot of archives.

[Jon]

Yeah, that's my image, and I'm the main person in that thread.   We had a private discussion here about it but someone posted about it on Reddit so I went ahead and discussed it.

 

If you're the person who responded to me an hour ago about the 2GB limit...  There is nothing about the file headers that disallow it currently.   I'll just repost what I said here:

 

The header is:

struct HEADER {
    char type[4];
    int version;
    int FolderRecordOffset;
    int ArchiveFlags;
    int FolderCount;
    int FileCount;
    int FolderNameLength;
    int FileNameLength;
    int FileFlags;
} header; 

Which means the limit is 2^32 files and 2^32 folders, not 2^32 bytes. Even the folder/file path lengths, if your typical path is 40 bytes long, that's still over 100M files.

[Eternity]

Yeah, that's my image, and I'm the main person in that thread.   We had a private discussion here about it but someone posted about it on Reddit so I went ahead and discussed it.

 

If you're the person who responded to me an hour ago about the 2GB limit...  There is nothing about the file headers that disallow it currently.   I'll just repost what I said here:

 

The header is:

struct HEADER {
    char type[4];
    int version;
    int FolderRecordOffset;
    int ArchiveFlags;
    int FolderCount;
    int FileCount;
    int FolderNameLength;
    int FileNameLength;
    int FileFlags;
} header; 

Which means the limit is 2^32 files and 2^32 folders, not 2^32 bytes. Even the folder/file path lengths, if your typical path is 40 bytes long, that's still over 100M files.

 

But what about the file record? According to http://www.uesp.net/wiki/Tes5Mod:Archive_File_Format#File_Record, that offset is a 32 bits variable. If the BSA goes beyond 4GB (or 2GB if they are using signed integers), you wouldn't be able to place a file after offset 0xFFFFFFFF (or 0x7FFFFFFF), because you cannot specify its offset.

[Jon]

But what about the file record? According to http://www.uesp.net/wiki/Tes5Mod:Archive_File_Format#File_Record, that offset is a 32 bits variable. If the BSA goes beyond 4GB (or 2GB if they are using signed integers), you wouldn't be able to place a file after offset 0xFFFFFFFF (or 0x7FFFFFFF), because you cannot specify its offset.

 

The offset is a ulong, which is an unsigned 64-bit integer.   I don't see where you're getting a 32-bit variable from.   Furthermore, I just created a 6GB BSA with Beth's BSA tool with no issues. 

 

 

 

Note it reports the size incorrectly but this is a UI issue in the BSA tool.

 

--- 

 

Edit:  Updated.   "ulong" is very misleading.  To me, "ulong" means "long uint" which apparently when they said "ulong" they really just mean "uint"

 

Edit2:  OK, there are multiple conflicting sources about the length of "ulong".

 

Edit2:  Interestingly, neither BSAOpt or the code in NifSkope defines the offsets as 64-bit.  So I'm guessing I can't read this texture archive I just created with those programs.

[Eternity]

I wonder if that BSA works at all.
The ulong as listed there is 32 bits (as actually in 32 bits compiler, longs are 32 bits).

 

It is the same field that is specified here as "unsigned int" here https://raw.githubusercontent.com/Ethatron/bsaopt/master/io/bsa.C, so that's 32 bits for sure.

 

And this code, essentially confirms my suspects:

sze = (*ft)->iinfo.sizeFlags & (~OB_BSAFILE_FLAG_ALLFLAGS);
pos = (*ft)->iinfo.offset;

/* fill file structure */
flh.hash = (*ft)->iinfo.hash;
flo.offset = (unsigned int)(fname - &fnames[0]);
fle.size = (*ft)->iinfo.sizeFlags;
fle.offset = fbsa->tell(fbsa->fle);

/* goto source-location */
if (ibsa->seek(ibsa->fle, pos, SEEK_SET))
  return shutdown("Seeking BSA failed!");

They do an absolute file position seeking using "iinfo.offset", which is filled by previous code from the MWBSAFileInfo structure.

 

[Jon]

I wonder if that BSA works at all.

The ulong as listed there is 32 bits (as actually in 32 bits compiler, longs are 32 bits).

 

It is the same field that is specified here as "unsigned int" here https://raw.githubusercontent.com/Ethatron/bsaopt/master/io/bsa.C, so that's 32 bits for sure.

 

And this code, essentially confirms my suspects:

 

Like I said in my edits,  BSAOpt and the code NifSkope uses from it could easily be wrong.  It's third-party code.  But "ulong", to me means a 64-bit integer, so if they meant 32-bit in the UESP they should simply write uint.  I looked it up at several sources and got conflicting answers depending on the language.  uint is hard to misinterpret.

 

 

---

 

Edit:  So anyway, none of this arguing even matters because UINT = 4GB limit, not 2GB limit.   There is no 2GB limit as far as I can tell.   I do believe the  6GB BSA I wrote had reached its limit but I haven't looked through the BSA in a hex editor yet to see what it's writing.

[zilav]

xEdit uses unsigned 32 bits integer for offset. I assume they probably just upped that field to uint64 in that ba2 format.

[Eternity]

All these structure that start with "MWBSA" or "OBBSA" in bsaopt source are read directly from the file, they cannot differ from the size Bethesda uses for them, because then BSAOpt wouldn't work at all. Of course, they can differ in sign, but that's a different matter.

[Jon]

Again I'll ask how a UINT results in a 2GB limit.  UINT means that the offset value can go all the way to 4GB.  I do agree that the 6GB test I tried probably ends up with incorrect offsets.   But I don't see how there is possibly a 2GB limit based on the file records. 

 

I just used the BSA tool to write a 3.56GB BSA.  I then used BSAOpt to extract it.  Absolutely no issues. 

 

 

Comparing the files I archived with the ones I extracted. 

 

So, given that none of the new BA2 files exceed 4GB, I think it's safe to assume that they did not need to change the offset to 64-bit.   Need being the keyword.  Maybe they did anyway, for future reasons.

[Eternity]

If they use signed variables in the engine and the functions they use to seek in a file also use  a signed 32 bits integer, then, there is a 2 GB file limit, at least in Skyrim/Oblivion/Morrowind ("fseek" functions can take negative numbers to seek files backwards).

Using signed 32 bits for file offsets is a very outdated practice, but the code in Skyrim may have been that old. Like I said, BSAOpt structures may differ in sign from Bethesda code. Apparently, BSAOpt seems to have code to prevent files > 2 GB from being created (probably because it wouldn't work in the engine), but they don't seem to have any code to prevent extracting files > 2 GB, since they are using unsigned int anyway. Though I haven't checked if a file between 2GB and 4GB actually works in Skyrim, I'm just assuming that code that BSAOpt places there on purpose to avoid creating them was done for a good reason.

 

Of course, at the moment there is no real evidence that suggests that Fallout 4 increased the limit to use 64 bits variables. They could have just solved that sign issue with 32 bits variables without actually modifying the size of structures. That would grant a jump from 2GB to 4GB. But if they went to the trouble of renaming the file format, I guess they changed something, and it would be a wasted opportunity if they didn't update to 64 bits offsets and removed any file size limit forever.

[Jon]

Yes, so it's an engine problem, NOT a format problem.  And like you said they wouldn't need to update the format, only the engine.

 

Honestly it's looking more likely to me that they just renamed BSAs to BA2, to prevent anything from reading them the "old" way, like you said using fseek() which uses a signed int.

[Arthmoor]

Morrowind, Oblivion, and Fallout 3 were all released at a time when Windows XP ruled the roost. I'm pretty sure the 2GB size limit was an OS restriction at the time. I can't find where that was discussed, but that's how I remember it.

 

Windows 7 has no such restriction on file size and I've seen game archives in things like the Witcher series that are upwards of 9GB, but then, Beth isn't using the RedEngine and their code up to and including Skyrim may still be treating that "unsigned int" as a "signed int". This is what happens when one sticks with Frankenbryo for so long.

 

Fallout 4 necessarily had to do away with 32 bit restrictions since it's a 64 bit program so this limitation shouldn't matter and they probably did the file the way they did out of habit more than anything. There should be no technical reason for them to keep the archives the size they're using for FO4.

[Jon]

This is veering a bit off-topic, and it's a really complicated topic on its own,  but most people have misconceptions about what it means to be "64-bit".  Programs which are 32-bit are not kept from using 64-bit integers, and compiling a program that was previously 32-bit in 64-bit instead doesn't really change anything.   32-bit processors also have no issue dealing with 64-bit values, they just take up two spaces in memory and can only "fit" through the CPU 32 bits at a time (gross oversimplification here).
 
In general, file size limits are pretty much a result of the filesystem, not the operating system.  FAT = 2GB,  FAT32 = 4GB,  NTFS = no limit,  even on XP.
 
Like Eternity pointed out, the engine's seek operations use a signed 32-bit integer, and this functionally limits the BSA file size to 2GB or less even though the file format itself can handle up to 4GB.  This was entirely arbitrary on their part and has nothing to do with Oblivion through Skyrim being 32-bit applications.  For example, I had a look at what NifSkope's BSA seek operation does and it can take a 64-bit integer even though NifSkope is a 32-bit program.  Of course I'm only feeding it 32-bit integers from the BSAs, but it's fully capable of seeking using 64-bit values. 
 
The BSA format has actually contained 64-bit integer records since Oblivion.  Each folder name and filename receives a 64-bit hash which is used for looking that folder/file up.  So, they could have also given the same treatment to the file/folder offsets as well allowing for >4GB files,  but at the time they obviously didn't think they'd ever be going that large. 
 

...their code up to and including Skyrim may still be treating that "unsigned int" as a "signed int". This is what happens when one sticks with Frankenbryo for so long.

 

Not really.  I'm not even joking that they probably just had to change the type in their function declaration.  Say they have a BSAReader::seek( int32 offset ).  They'd have to change the 'int32' to an 'int64' in like two locations (declaration/definition), plus wherever they call seek() make sure they aren't truncating the value they pass in to 32-bits, and they would have automatically had support for >2GB BSA files.  Also, AFAIK, the BSA stuff has nothing to do with Gamebryo and it was entirely their own,  so that's like the one thing we can't really blame on Gamebryo.   

[Arthmoor]

Well, even if it's 2 function headers, they may not have thought it wise to change that in a 32-bit program. Yes yes, I know, they had nothing to fear, but they're not catering to the fearless among us. They had to make sure shit worked without question and changing something like that DOES introduce some element of doubt.

 

That doubt evaporated upon moving to a true 64-bit platform so they could change that if they wanted and not worry about the possible consequences.

 

I've done enough hackish tweaking in C/C++ and 32 vs 64 bit values to know it's not always a cut and dry matter with these things, even on platforms where it shouldn't make a difference. Compilers are not all consistent about it.

 

Just be glad it's no longer an issue and we don't have it hanging over us from now on

[Jon]

In other news.... http://wccftech.com/fallout-4-reports-severe-performance-issues-surface/

 

I'm a-scared.  For one major reason.  Since they are probably "unifying" the experience across PC and consoles considering we need to have the same files so that the mods work cross-platform,  does this mean that if Bethesda is going to start doing "performance optimizations", i.e. making things look worse to save some FPS that we're going to receive these "fixes" too? 

 

Or do you think hopefully that the platform differences are going to be handled by a load-last ESP file?   So if they really have to make adjustments to scripts, meshes, or object references for performance reasons it will stay in its own patch plugin.   I say "load-last" because now with the addition of modding,  having like Fallout4.esm and then Update.esm directly after isn't necessarily going to cut it if there are going to be tons of mods overriding the Update.esm.   Of course maybe modding on consoles will be "at your own peril" just like on PC, and there will basically be no guarantee that stuff they fix in updates will stay fixed. 

 

Edit:  And also,  having Update.ESM as master... how on earth is that going to work if the consoles and PC all get different Update.ESM to address platform specific issues??   Obviously it's not able to work.

 

Also... https://twitter.com/bethesda/status/663443440332591104

 

LOL.  They're pushing back the time in order to combat VPN users.   A lot of people are going to try to VPN into NZ in order to unlock it early. 

[Arthmoor]

Well. I would assume that Update.esm would be platform universal stuff, assuming it exists at all and they don't go back to merging it into Fallout4.esm.

 

A last load ESP enforced by the engine to load game fixes would be disastrous. Sometimes people WANT to override a vanilla thing, and if that vanilla thing is bugged and later fixed, well, there goes your mod that might have fixed the issue as a side effect of its purpose. Hell, this is something that could even render a U4FP mod useless.

[Elgar]

So now all Bethesda needs to do is get their new forum software launched like they said would be done before the game launches.

 

The new forum is still not available, am I right ?

 

What I don't understand is this statement by a moderator :

The FO4 forums will follow the same structure as past games.

In case you haven't noticed, most of them have been live for a few hours now.

 

http://forums.bethsoft.com/topic/1543852-what-new-fallout-4-forums/?p=24465152

 

Am I missing something ? I can't find anything else than http://forums.bethsoft.com/forum/324-fallout-4/

[Arthmoor]

That's because they don't exist. Though, Pluto is a moderator. Maybe he can see something we don't. Might be a good idea to message him and ask.

 

Last I heard they were supposed to fire up the Bethesda.net community to allow us to go in and reserve user names, but clearly that's not happening now and there's still no sign of the new forum coming up.