Jump to content

Recommended Posts

Posted (edited)

Hello everyone, this is my first post on this site so I hope posting this question here is fine.

So I’ve been looking into a very specific bug in the game which seems to have been a thing since the very release of the game up until now. It occurs without mods as well as with various versions of the Unofficial Patch. I found plenty of interesting examples and it looks like it happens on PC as well as on all consoles. The bug is even mentioned on some wikis, but there has been nobody who actually explained why it happens or how it can be triggered.

So far, I’ve found examples for four variations of it:

  1. You travel to a settlement (and stay there for a while) and a random object appears next to a crop. This only seems to happen if the crop is assigned.
  2. Advancing a quest accidentally spawns an item intended to be placed in a specific container at a random crop in a random farm instead.
  3. You’re on a farm already and move or plant a crop and a random object spawns right when the crop is placed down.
  4. Shooting a spotlight pops out a random object in addition to the broken light object. I only found one single example for this one and they got it on the exit door of the Corvega Plant interior where you do the first Minutemen Quest.

Spawned objects include:

yw19iqu7lm791.png

Star Cores (most common), Space Suits, specific quest holotapes (from "Long Time Coming" and "The Lost Patrol"), dead bodies of enemies that were defeated earlier on the save, scenery items like walls, Provisioner Robots stripped of all their robot workbench mods, the Beryllium Agitator, legendary items from the CustomItemQuest, DLC03_CustomItemQuest and DLC04_CustomItemQuest quests, tanks and the Yangtze Warhead.

Here are some examples of the bug happening from various players:

It appears like these things are properly moved and not just spawned in. One of the examples mentions Admiral's Friend being removed from the vendor after it appeared in their farm. The most common places the random things appear at seem to be Sanctuary, Abernathy Farm and Oberland Station. Unfortunately, my knowledge of the mechanics of the game aren't deep enough to actually look into this properly on my own. Believe me, I tried.

So far, my best theory is that the game somehow attaches the wrong reference to the FurnitureBase. This results in the object being pointed to spawning next to the crop if an assignment is done or the crop is moved. I was able to confirm this by making a mod that sets this value to the Eddie Winter Holotape and indeed, this will make the holotape spawn exactly like the provided examples above. I don't know for sure how this mix-up happens, but I assume it has something to do with extreme lag on list init or installing DLC or mods on existing saves which change the indexes of the checked lists which results in the wrong index being used, switching FormList [15] to HoloTape [15] for example.

If this is indeed the culprit, I would suggest updating CreateFurnitureMarkers() in WorkshopObjectScript.psc to add a check if FurnitureBase is Furniture and if it is a FormList instead, to check if it's randomly selected index is furniture. You could also check afterwards if the furniture is flagged as a marker though it looks like this flag is not set for a lot of markers, so we'd have to actually set them for all the crop markers.

Any help in looking into this would be very appreciated. To my knowledge, this bug hasn’t been mentioned on this forum yet. If it has been and I’m just bad at searching stuff, just let me know.

Edited by Geta92
Posted

There's a rare engine bug related to the papyrus function PlaceAtMe(). Among its parameters, there are two bools to handle the cleanup of the generated object(s), abForcePersist and abDeleteWhenAble. The vast majority of the vanilla scripts leave these at their default values when PlaceAtMe() is called, i,e, abForcePersist = false (which means that the objects are not supposed to remain in the game world) and abDeleteWhenAble = true (which means that the objects are supposed to be cleaned up as soon as possible). [NB: This makes sense because too high a number of obsolete objects remaining in the game world does not only cause save game bloat, but also other issues down the line. Thus, to avoid problems, one has to make perfectly sure that any object created with abDeleteWhenAble = false and/or abForcePersist = true gets eventually deleted. This obviously requires extra script wrork, but there are scenarios where this is not feasible and,leaving these bools at their default values may be the best choice.]

Now, the engine has a preference for eagerly deleting references that are marked for a fast cleanup, and somtimes deletes them prematurely. This is well known e.g. for magic effects, but it can also affect other objects, and this may even happen when those objects are still held in script variables. This behavior has been confirmed for the script BOS100FightMonitor [NB: this script controls the ghoul fight on the quest "Fire Support"] and is documented here:https://afktrack.afkmods.com/index.php?a=issues&i=28303. The actual bug is not the references being cleaned up prematurely, but the script variables not simply being turned into 'none' (as one would expect should happen when a reference gets deleted). Instead, they turned into bogus references: first, they were replaced by references that were deleted earlier, and later on, they could hold references of random objects previously spawned elsewhere in the game world.

I need to emphasize here that this is a rare bug. Many people, including myself, never had a problem with Fire Support, yet there have always been reports about the quest being broken. Others had it occur on only one of several playthroughs. A closer look at EyeDeck's findings (on the ticket I linked above) reveals another remarkable detail: the AllGhouls array may contain more than a dozen of references and they should all be susceptible to getting cleand up prematurely, but only two of them were actually affected. I wonder what the point is in trying to clean up those references when only a few of them are picked. I suspect therefore that the cleanup itself is the bug, i.e. that it should never have happened, but with the limited information we have, available, that's pure speculation for now. EyeDeck's findings immediately raised the concern of similar bugs occurring on other scripts that generate objects via PlaceAtMe().and keep them in script variables for some time. However, bugs that don't hit you repeatedly, and don't leave consistent error patterns when they eventually do hit you, are notoriously difficult to track down. We knew at least that we should keep an eye on random objects appearing in this context (they would not necessarily have to appear in a physical way though), and this is why your post immediately rang a bell since it suggests a connection between randomly appearing objects and workshops (which are known to spawn tons of objects via PlaceAtMe() and to keep some of them in script variables for extended periods of time).

Fixing this issue (or rather working around it, since only Bethesda can fix this properly) is relatively simple, but we need to know which obects are affected by the garbage collection (since only those can eventually end up being replaced with random objects). I do not believe that it is the damage helpers, because they are linked to their crops and links always give extra protection to both of the involved references. This extra protection was the cause of two vanilla bugs:

  1. The "invisible crop" issue (ticket #20945, #21894): When a crop is removed from a workshop (usually by storing it in the workbench), its damage helper has to be removed.as well. The vanilla WorkshopObjectScript did thiis by calling Delete() on its reference, without clearing the link between the helper and its crop. Due to the link remaining in place, deletion never worked and the damage helper stayed in the game indefinitely. It could be activated by the player and, when doing so, was identified as a crop in the UI. Any attempts to scrap it (in workshop mode) caused an instant CTD:
  2. A reset issue with all wilderness crops (ticket #21894): Every wilderness crop haa a WorkshopObjectScript, and like the workshop crops, it creates its own damage helper on first load, links it to itself and stores its reference in a variable of WorkshopObjectScript. When the parent cell resets, all properties and variables on the script are reset to their default values. The damage helper reference is now 'none' again, so WorkshopObectscript creates a new one. Unfortunately though, the old helper(s) are never deleted: they do survive a cell reset because of the link that's still in place. This bug could create a tremenduously high number of obsolete references in a short period of time.

Thus, if neither a Delete() call nor a cell reset can remove a damage helper, it is extremely unlikely that it can be inadvertently deleted by the engine's garbage collection, at least not while the crop link s still in place. Otherwise, any workshop object - including settlers - could get replaced with a random object because they are all created via PlaceAtMe(), with both abForcePersist and abDeleteWhenAble at their default values. In fact, it is only their workbench link that protects them from getting instantly deleted when the area unloads. As you pointed out yourself, the checks performed by UFO4P_ValidateDamageHelperRef() do not allow for any unrelated object to pass unnoticed as a damage helper. If you expect the damage helpers to be the culprits, your claim that our fix may have beeen invalidated by a mod conflict is a convincing explanation. Another explanation is that the damage helpers simply don't have anything to do with this issue at all.

I should probably add here that the damage helper validation procedures were added to clean up a minor mess made by another engine bug that hit us when we added missing workbench links to a number of pre-placed crops at Abernathy Farm, Finch Farm and Oberland Station. While investigating that issue, we uncovered a number of other bugs related to damage helpers and fixed those in the process too. At the time this happened, we didn't even know of the crop issue we are discussing here, and most of us, myself included, never had the crop issue occur in our games to the present day. I also can safely exclude any connection between the crop issue and that other engine bug that caused the issues with the damage helpers. If you want to learn more about it anyway, have a look at ticket #21039. This should still have a link to a forum thread where all of our findings are documented

I believe that the furniture markers are the culprits. The furniture markers are idle markers that get created by WorkshopObjectScript at run time. They make it that settlers perform gardening animations at the crops that have been assigned as their work objects. Like damage helpers, furniture markers get aligned with their crops, and they get moved when their crops are moved. Though, unlike the damage helpers, they don't get linked to their crops, and without that extra protection, they are much more likely to become targets of the garbage collection. The papyrus log errors reported in ticket #25922 also point in that direction; they come from invalid references populating the furniture markers array.

The bug reports you linked to in your post suggest the following course of events: WorkshopObjectScript creates furniture mrkers via PlaceAtMe() and stores their references in an array for extended periods of time. Those references are not otherwise protected, so they apparently become susceptible to a premature cleanup, Once this happens, the references orginally held in the array are replaced by bogus references; eventually, the references of any object previously spawned somewhere in the game world can end up here, including quest objects. When WorkshopObjectScript updates the positions of the supposed furniture markers, those random objects are dislodged from their original positions and moved to the crops.

Though, there's one bug report that doesn't match the others: the spotlight issue. We always have to keep in mind that the removal of mods from a running game is still a common practice among users (sadly!) and a frequent cause for weird issues in the game that nobody can reproduce or explain. Unless reports by other persons can be found that confirm the issue, we cannot be sure that it is even a valid bug, so we should disregard it for now.

Posted (edited)

Thank you very much for your reply, from what you mentioned, you saw my prior statements as well. That's good, so you know how my thoughts about this changed, too.

I was working on solving this bug on Discord with a friend to make sense of this and indeed, we eventually dropped the thought of damage helpers being the issue and went for markers instead. I made a small mod where I put item references on the furnitureBase reference of crops just to test if this ends up working and the resulting behavior matches the claims exactly. So plausibility of these claims is definitely confirmed at least.

cropScript2.thumb.webp.47ec0200624d62cc063c1f50bb2ae643.webp

CropScript.thumb.webp.0ea3d8fe4b1e780177a67329bf82bf9b.webp

But this actually connects the issue of crops and spotlights, too. I figured out exactly the only way the spotlight issue can be replicated as well with incorrect markers. For the reported result to happen, the spotlight base objects xMarkerForm needs to hold an incorrect reference to a Star Core. That way, a Star Core indeed just pops out once you shoot it down.
StarCore.webp.53b5372e186d4855535f4c87ca0b9f7a.webp
StarCore2.thumb.webp.3b8f6d8c7ef24828330daeb7344a68cf.webp

I was looking for more similar cases of objects spawning in randomly and I found another very interesting example. This person claims that moving a doghouse spawned a legendary weapon (Reba II). The only thing the doghouse has in common again is that is uses markers as well. It just has one so I can only rationalize that that one got cleaned as well and the reference ended up pointing to the leveled item CustomItem_DoNotPlaceDirectly_DN083_Rifle_Reba2.

DoghouseMarker.thumb.png.ea0b14d9cf416f524b379b7cf2efee04.png

As for the example with holotapes spawning in completely unrelated places, I took a look at the Long Time Coming quest. All claims for this quest only ever mentioned holotape #4 and incidentally, that is the only tape of the quest that is not spawned in the overworld just placed on something, it it places in a safe (the Natick Police Evidence Locker). So again, if the reference to that safe got cleaned, it could very well end up linked to a crop marker in Abernathy. It's very strange just how often stuff ends up specifically linked to Abernathy crop markers though. It's like the games favorite fallback to stale references.

Edited by Geta92
Posted (edited)

So I just found this post on Reddit which kind of changes my perspective as to what exactly is pulled as a "random" reference. This other one seems to also confirm it.

image.thumb.png.77d4d6987721ee313d9f913870361fbe.png  image.png.e1fc19fe42524ea5feea5bd5b2375d3f.png

This is very strange. So far I've only seen statics, misc items, corpses, weapons and armor but this guy actually got Barney to spawn. This makes me strongly believe that the references pulled "randomly" are actually kind of specific.
It looks like they're not random at all but exclusively references used to store Quest Aliases.

image.thumb.png.2ec6a500df4e63b8d775da79161b51d3.png

This would explain the frequent pulls from items that are part of CustomItemList, DLC03_CustomItemList and DLC04_CustomItemList. The Star Cores are also defined as an alias in DLC04GZMainQuest.

Edited by Geta92
Posted

This is probably a stretch, but you having mentioned quest aliases reminded me of this report for Skyrim:
 

 

It's yet another scenario involving data being hooked in a manner that should be impossible. It's also not the only time this happened in Skyrim. There was another user years ago who also ran into something similar with a different reference that had gotten itself attached to a random note but was somehow starting the Hearthfire quest for Falkreath.

It's entirely possible then that this is a manifestation of an old engine bug that's simply become worse with Fallout 4.

Posted (edited)

The script you mentioned is actually not only bound to Aetherium Wars (XX004D5B). There is an alias called "QT_D2IntroBook" which points to a reference used in Raldbthar (XX00B61E) which has the same script that uses the Aetherium Wars as a base to copy-paste the looks and text.

image.png.00b604d0e8ffc495c236bef02f71a645.png  image.png.79a642614ea038dd65b58a570c7a38c1.png

If this reference were to accidentally switch the base to another book, you'd get any book in the game visually with the same text it usually has, but with the extra script attached to "setStage DLC1LD 10".
There may be more ref-books. They're hard to find because the Creation Kit doesn't give them as results if you search for the original book name or checked linked objects in the context menu.

Edit: The same applies for the other example of the Red Eagle quest. The quest has a non-persistent alias pointing to a reference to the original book that could also be picked up by something else after being deleted.

image.thumb.png.c617c28d43aa6d3923f228d65c878b94.png

Edited by Geta92
Posted (edited)

Okay, I just found another Reddit post that is interesting. Because it is a bit different. No crops in sight, just a Star Core on the ground with nothing in sight. This happened in the Castle so I thought "what could possibly be on the ground in the Castle that could be garbage collected?". Then it became obvious. The goddamn Mirelurk Hatchlings. Once you claim the Castle, the game despawns the dead Mirelurks. The Hatchlings are not pre-placed since they spawn with a script and will use random non-persistent ref IDs as a result.

image.png.10b756f6eff412f0e645e95edcef8941.png  image.thumb.png.d94221b164c7613bdb91545273fd3c93.png  image.thumb.png.e1042bf91491921faa19ad54953f1863.png

The spot in question should be exactly here. Nothing is located there, but considering that the Hatchlings move, one might as well just have died there. The spot is exactly in between some of the eggs and the back entrance of the Castle so if you were to attack from this side, the Hatchlings likely pathfind over this specific spot. This example seems very similar to the one you linked me. About the intro quest with Paladin Danse with the Feral Ghoul spawns (Fire Support).

Edit: And I found another one. Mama Murphy's chair got warped into a Tato Marker. And again, her chair has an Alias set in her quest. This alias is initially unset and only set when you build the chair.

image.png.8c347d93e975e6b263c01529e153491b.png  image.thumb.png.127d1176db92d43d7e8907404c61f5b7.png

Edited by Geta92
Posted (edited)

I just hit the jackpot. After looking up more examples of the glitch happening on Reddit (which I added to the top comment) and commenting on the posts, I was finally able to get in contact with a person who got the glitch on PC with no mods installed, no DLC and on the current patch. The glitch happened in The Slog with a Carrot. They went there and found Le Fusil Terribles on the ground. They were able to share the save for when this happened with me. I put the save in the attachments of this comment. I also found The Gainer at a Mutfruit in Greentop Nursery.

image.thumb.png.53d4de5ef3b859e5e36eeeffc72f90f2.png  image.thumb.png.266cdfea07e90492222862fe93bb9b11.png

image.thumb.png.15c247fb5e9d7758db28662c9bad5b58.png  image.thumb.png.e050f9699f9db21995ed4fb79c0a1024.png

From playing around with their save for a bit, I can immediately take note of the following things:

  • The shotgun was not spawned in directy (from 001662DE) but from the quest alias ID of the CustomItemQuest. The one in Libertalia is gone.
  • The carrot that spawned the weapon (000F5DF2 [PP]) is a pre-made one that is always in The Slog from the start and it has not been moved from its default position.
  • The carrots marker is fully functional after all. You can assign settlers and they do their animation.
  • The carrots damage helper reference is returning a null pointer exception. The helper itself is still intact though, moving the crop and shooting the old position still destroys it.
  • They are using power armor with the Targeting HUD mod. I would usually not mention this, but some other people who reported the glitch also used this mod at the time so maybe it's making the script problems worse.
  • They have an active follower in power armor who is also using the Targeting HUD mod.

This whole things all seems to be related to this remark Sclerocephalus did on the invisible crop issue:

On 7/23/2016 at 3:15 PM, Sclerocephalus said:

And this is the vanilla code for deleting a marker (in the HandleDeletion() function):

	if myDamageHelperRef
		myDamageHelperRef.Delete()
	endif

I see two issues here:

(1) The damage helper is not unlinked from 'self' (i.e. the crop) before it is deleted. Links make references persistent, so I wonder whether deletion actually works. Maybe it does, but testing this is a waste of time when you can simply prevent this from becoming an issue by unlinking the marker from the crop before deleting it.

(2) Even worse: myDamagHelperRef is never cleared. This will not automatically become 'none' just because the reference it points at is deleted. Instead, it might be ending up at pointing into limbo. On the other hand, it could also represent an obstacle to the deletion of the marker, since its reference remains stored in a script variable.

The carrot likely used a different damage helper reference originally. But for some yet unknown reason, the game tried deleting it. This did not succeeded because it is still linked to the carrot, so the helper was kept intact. Only the reference was removed from the crop, though it was not set to "none" but instead pulled a different address out of its butt. This may have happened multiple times. It is now pointing at D6A but it might have cycled though multiple addresses in quick succession. D5D points to the shotgun that eventually spawned in front of it. This address is only 13 spaces in front of the one it is set to now so I think the crop previously pointed to D5D and then changed the broken ref to D6A. It may have pointed to an address in between as well, but multiple of these addresses are unused so nothing would have happened. 13 is a prime number, so it's unlikely the game passed over multiple addresses in a consistent way though.

Digging through the memory also revealed more damage helpers in this memory space:

  • D5B: CustomItemQuest: Grognak's Axe
  • D5C: CustomItemQuest: Final Judgment
  • D5D: CustomItemQuest: Le Fusil Terribles (Previously referenced by Carrot 000F5DF2 in The Slog)
  • D5F: CustomItemQuest: Champion Left Arm
  • D60: CustomItemQuest: Wastelander's Right Leg)
  • D62: CustomItemQuest: The Gainer (Currently referenced by Mutfruit 00159FF3 in Greentop Nursery)
  • D63: CustomItemQuest: Reba II
  • D64: CustomItemQuest: Big Jim
  • D6A: Empty Reference (Currently referenced by Carrot 000F5DF2 in The Slog)
  • D6E: Ash Pile of a dead Synth under the rocket engine where Paladin Danse gets roasted
  • D71: Damage Helper Ref (Mutfruit 00159FF3 in Greentop Nursery)
  • D78: Damage Helper Ref (Tato 000E04F1 in Covernant)
  • D7F: Damage Helper Ref (Tato 000A8CCC at Diamond City)
  • D8B: Damage Helper Ref (Mutfruit 001BA691 at Greentop Nursery)

Saves.zip

 

Edited by Geta92
Posted (edited)

I'm just gonna comment on the current progress on this subject because there were some breakthroughs.

image.thumb.png.cf97a0afcf7ddc6692198568f40cdafb.png  image.thumb.png.fcb094d3b77daf76e27d817ff1056515.png

Turns out this bug is very likely related to crops having had multiple damage helper references. It may happen with other types of marker references as well (since we got the spotlight and doghouse example), but so far we only confirmed damage helpers. After using a mod to replace the model of damage helpers with a bright red oxygen tank, I was able to detect that every single crop that was known to link to quest items turned out to use more than 1 damage helper reference in the past. One even had 3 (the one that spawned the Libertalia Shotgun). Next to it there exist the Shotgun (D5D) and a regular Damage Helper (3C46) but it's ref var currents points to D6A (which doesn't hold an object so nothing appeared). Another Mutfruit and a Carrot had a functional but unlinked damage helper but linked a different object with no 3D in it's reference. This seems to at least confirm that myDamageHelperRef indeed cycled through multiple values during the crops existence and the game did in fact not garbage collect the actual helper. It's still there, just not referenced.

By knowing this, I was able to detect lots of affected crops in the save I was provided. But 95% of the affected crops simply have additional functional damage helpers. One is linked, all others not, but they are still functional using the correct model. Although it is definitely noticeable that when more than one damage helper is created, one additional ones all used address more different from the surrounding crops, implying there were indeed created a a different time. These new references can somehow point to locations in memory where quests dumped their quest alias references. If this happens to be one that is free for grabs, the extra helper is properly created but when not, the actual "inhabitant" of that address gets moved to the crop instead. If the address links an empty reference or something that cannot be moved, you wouldn't actually notice the bug at all.

I could confirm that the range that contains the Star Cores in Nuka Word ist pretty much right in order of the items this save spawned in if it would have installed the DLC in the first place (I just checked the addresses in front and after the pulled ones).

Edited by Geta92
  • 10 months later...
Posted

I haven't really looked into this bug any more than I already did but somebody got it again and they posted about it here.

I find it interesting they mention re-installing the game and the DLC. I assume this may be related to the bug happening?

If the game had to remake all the quest aliases for the DLC, they would occupy very different areas than usual.

 

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...