Jump to content

[BNet - SSE] Memory issue when uploading large mods


Arthmoor

Recommended Posts

If you are currently supporting PC and XB1 uploads on Bethesda.net, there is a large memory usage issue that you'll need to be aware of if you're maintaining a large mod with many asset files being used.

For awhile now I've been running into intermittent issues with trying to do updated for the Unofficial Skyrim Special Edition Patch that often resulted in the CK hanging for no apparent reason. This usually cleared itself after additional attempts and was written off as a server-side issue since no errors on the CK side were generated. Unfortunately with the 4.2.1 update this was no longer the case. For several days it was impossible to upload anything, then somehow the PC version got through but I was unable to do so for the XB1 copy. Much frustration ensued and I was at a point where I was ready to give up.

Uploading to Bethesda.net involves several steps:

1. Load your mod as usual.
2. Log in to the bethesda.net server.
3. Select the platform you're uploading to.
3b. If you're uploading an ESM flagged file, you'll be prompted to select it from a dropdown list.
4. After a brief wait, a screen should pop up asking which files to include in the uploaded archives.
5. Selected files get packaged and then you are presented with a screen asking if you're uploading a new mod, or to select an existing one to be updated.
6. Files actually get uploaded and you'll be prompted to visit the webpage for it for further editing etc.

My problem stemmed from step 4. After selecting the USSEP's master file, the file archive prompt never showed up and the CK consumed every drop of available system RAM and the entire virtual memory swap file as well. This normally results in Windows 10 killing the process but more often than not if I ignored this it would cause a BSOD instead. Since the CK provides absolutely NO feedback on what it's doing or when it encounters error conditions during any of this process, you're left to guess as to why it failed.

Enter Old Man's Beard. One of the gurus on Tarshana's Discord server which is focused mainly on support for the XBox One community (Tarsh Gaming if you're interested). He generously lent his time to systematically picking at a variety of possible causes until he finally realized that it was possible to upload after chopping portions of the USSEP out that were referencing assets.

Apparently when uploading a mod, step 4 assembles your list of files and allocates 8MB of RAM for each asset file it believes you'll need. USSEP as of version 4.2.1 currently has 5854 files to process as assets. At 8MB each, this results in 46GB of memory needed to allocate to the task. Needless to say this far exceeds my available physical RAM of 16GB so Windows went to the swap file to get at the remaining 30GB which was beyond the size limit Windows 10 was allowing in most cases, even though it had been set to be system managed.

After Old Man's Beard played around with his swap file and increased it to larger limits, the CK eventually did present the archive prompt for step 4. Once it did so, all of the allocated memory was released back to Windows and the upload could proceed as normal.

So I set my virtual memory file to a minimum of 32GB, maximum size of 64GB and went to try again. After selecting USSEP in step 3b, I waited. And waited. And waited some more. After about 10 minutes watching memory usage climb to the limits, the archive prompt came up. Sure enough, it had used a total somewhere around the expected 45GB before completing. It's not a nice workaround, but it got the job done.

Bethesda has been notified of this situation. There's no way to know if or when they'll address it.

TL;DR: The asset list for archiving your upload never shows up and the CK hangs. Increase your Windows virtual memory size to work around this.

Old Man's Beard has posted his own thread with his findings on bethesda.net: https://bethesda.net/community/topic/374381/uploading-large-mods-with-sse-ck-1-5-73-yet-another-cause-of-hanging-and-crashing

Link to comment
Share on other sites

5 hours ago, Arthmoor said:

Bethesda has been notified of this situation. There's no way to know if or when they'll address it.

Really helpful.

8 megabytes just to handle the upload of a file? No indication of this new bizarro memory management in the changelogs?

Link to comment
Share on other sites

  • 1 month later...

Mods are considered unsupported so they've never had a reason to use the CK to upload to the website to begin with. Their internal process for updating content is vastly different from ours because it uses Version Control and is tied in with their internal Perforce server.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...