Jump to content
Sign in to follow this  
Sharlikran

Wrye Bash Wiki Discussion Thread

Recommended Posts

On 1/29/2020 at 11:22 AM, Leonardo said:

Why bother using the Wayback Machine when the old BSF are still accessible.

http://forums.bethsoft.com/forum/13-the-elder-scrolls/

Because as I already mentioned the Bethesda forums are being taken down.

image.thumb.png.1000b592aa592d2fdcce28a771a43572.png

Now as to what I was asking bout, again I said don't tell me that it's just there. I do not see it or anything at all. I get a calendar and that's it. There is nothing to click on that takes me to any sort of content. I am asking for assistance to preserve the forums if possible so if you know how to capture all this with the wayback machine then do it. If it's like you say, all automatic then we are going to loose the forums entirely. Go to the forums on the wayback and look for this page specifically. Look for post BAIN Mod Installation Projects on page 30. Then also for Nehrim and Mods II on page 115.

Those posts I have archived so I don't need them they are just an example.

Share this post


Link to post
Share on other sites

We've got a command sequence that we think can download the entire forums but the sheer amount of data is proving problematic: it will grab 500,000 to 700,000 so it would take a couple hundred continuous hours to download. We're working on tweaking the command and trying to figure out a better way of doing it, including doing it in batches to allow us to do it on multiple systems, along with asking people for help who may have the resources on a remote system as the quantity is a bit much for a home system.

Share this post


Link to post
Share on other sites

I want to provide an update on things.

Leo and I are trying to get in touch with Fliggerty over at Great House Fliggerty as he runs the Mod History sites to see if he could provide some assistance. Additionally, I've contacted the Internet Archive but I haven't heard anything back from them. I also emailed an old contact and got some suggestions on how to handle things.

In the coming days, we're planning to formally ask Bethesda if they could provide the public data. Failing that, we'll need to backup the data ourselves. Regarding that, we may need to run a funding campaign and pay for a server and possibly storage space so that we can download everything and convert it. Additionally, we would need someone to manage the server aspect (I can't do it). The storage would only be temporary as the conversion process will allow the data to be much smaller than half a million HTML files.

We're still working on the optimal settings for wget (we have something that should work but we're just trying to optimize the process right now). Regarding the conversion process, it looks like we may need to use a combination of pandoc and custom code to adjust the file names. That's certainly doable but would take some time.

Share this post


Link to post
Share on other sites

Were you planning on doing this with static HTML copies of the pages or were you hoping to do it on an actual forum instance with real data if Bethesda allows that?

Share this post


Link to post
Share on other sites

I haven't given much thought to the final format yet as the focus has been primarily on getting the data (I figure that's the most important part). Though, I've been thinking of something like Markdown for the final version but nothing has been decided yet. If we can't get the data from Bethesda, the transient format would be static HTML files which could then either be hosted or converted into another format. If Bethesda provides a database dump, then that would certainly change things somewhat. It would be nice to keep the forums open to help keep discussions linear but I'm not sure if they would allow the community to take it over, even if that means converting the data into SQL (if they would let us). It's something we could ask them, though.

Share this post


Link to post
Share on other sites

I suspect they'd have serious legal issues with the community taking over a copy of the forum that's not hosted on their servers. Especially privacy concerns, GDPR, you name it.

There's also the 800 pound gorilla in the room: The old forum is now an unlicensed instance of IPB 3.x and would need to undergo a major core upgrade on top of all that.

Getting static HTML dumps may be the only viable option.

Share this post


Link to post
Share on other sites

I've gone ahead and requested the data. We've got about 48 hours until the old site is gone. We've requested an extension but we may not get it.

 

If we do, we'll need to try and run the command as soon as can but we'll need the server resources to do so.

Edited by deaths_soul

Share this post


Link to post
Share on other sites

@deaths_soul are you going to change all the names associated with posts that you copy, to something random? ( like gamesas do http://www.gamesas.com/vegas-pro-forum.html )

I'm not entirely sure I (for one) am happy with all my old content being copied to another site ( again, but at least in gamesas case they changed associated names ).

What I posted at the old beth forums, though public, was only intended for that site given the T&Cs and data protection they had in place. I have not agreed to anything on your destination site wherever that is going to be.

Will the destination site also be proof against google searching like the old site was?

Do you have the finances to chase up and squash anyone else doing the same and copying your site?

I am not against the idea of preserving modding history, and personally dont think I have anything particularly needs protecting .. Its just the idea that you are copying a lot of content wholesale that not everyone may be happy with, there may have been casual conversations about personal / family / profession / medical details for example which was not google searchable on the old beth site so people were more comfortable letting a few details be known within that community only (I am pretty sure I can remember a few such conversations). I'm sure you know that the internet has ways of figuring out through writing style or time of posts and content can be connected with another ID the same person made on other sites and link them for data gathering purposes. You also did not respond to Arthmoors mention of privacy concerns which surprised me, but instead you are going ahead with it. Thoughts?.

Share this post


Link to post
Share on other sites

I'll finish the reply in the morning but basically I'm not going to be doing anything with the data without explicit permission. I'm most concerned about saving it, even if that means it remains private.

 

I didn't feel that it needed a reply as my initial plan was just to archive the data, making it public only with permission. I hadn't even thought about hosting it as part of a forum until he mentioned it, I was simply planning (with permission as my first post said) to convert the data and toss it in a compressed archive.

 

I'm trying to stay out of too much drama. Drama is part of why I left in May of 2018. I'm only doing this as it sounded like people wanted help backing up the data. If that's not longer the case, I can certainly keep my hands off.

Edited by deaths_soul

Share this post


Link to post
Share on other sites

May be best to just leave it alone at this point. Even if legally speaking the only party you actually need permission from is Zenimax since everyone who signed up already agreed that the data posted there was theirs to control.

If there are indeed only 48 hours left before Bethesda terminates the old forum entirely then I guess that's that and it should just be let go. That's not enough time to do anything of value with any of it, and it would be nigh useless in any other presentation than a new forum instance - which btw is going to be impossible to acquire legally since IPB 3.x went EOL 3 years ago and is no longer being distributed.

I know Sharlikran wanted to get certain things off of the site before it died, so maybe the effort should be focused on salvaging individual posts or something instead, and then reposting those threads elsewhere?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

Support us on Patreon!

Patreon
×
×
  • Create New...