-
Ever wanted an RSS feed of all your favorite gaming news sites? Go check out our new Gaming Headlines feed! Read more about it here.
So How DO Companies Lose Game Source Code?
- Thread starter Ragnorok64
- Start date
I went down the rabbit hole of this before. Not even into Dr. Who at all, but the story behind this is pretty fascinating. Apparently the first 10 years or so of the TV show are lost forever, they were live or some such. Audio recordings for some of the episodes exist because audio was easier to keep, but there never was any video to actually archive in the first place. Today, these early episodes exist as slideshow recreations, only because there was a few independent people who had these special machines that could take pictures of their television. It was a technology they developed for various applications, but they used a lot of Dr. Who episodes as they were broadcast to sell to the BBC. The BBC bought the television scans in bulk when one of the inventors of the machines died, and held onto them for like 20 years until fans discovered them. They've slowly pieced them back together with the recorded audio to make crude simulations of the early episodes. Some fans even go as far as to animate these simulations, using flash techniques.
The fundamental rule of storing data (in any format) is that at some point the number of copies available to you will reduce by one. If at that point you only had one copy, you now have zero copies. Data is only safe for as long as you're actively managing multiple copies of it.
This is still really poorly understood, even in some IT circles. It was even more poorly understood in the past.
With games I think there's another factor, which is that it wasn't unusual for different versions of games on different systems to be written independently of each other. There's YouTube videos that compare the various ports of some older games across different systems and a lot of the time they're quite different, and were often done by different teams. Developers generally weren't going to think about the issues people might have re-creating their games ten or twenty or thirty years in the future, but even if any of them did think of that, they might have just assumed that if the game needed to be made on a new machine in future, there'd just be a new version written for that new machine.
There'll be lots more (mostly untold) stories like this as well. Almost all companies in the industry with a significant history will have been through some combination of key employee changes, building moves, acquisitions, financial issues, natural disasters (including small-scale ones like fires, flooding, power outages, etc.) as well as any of the other dozens of things that can screw up a backup regime even if it's being done properly.
This is still really poorly understood, even in some IT circles. It was even more poorly understood in the past.
With games I think there's another factor, which is that it wasn't unusual for different versions of games on different systems to be written independently of each other. There's YouTube videos that compare the various ports of some older games across different systems and a lot of the time they're quite different, and were often done by different teams. Developers generally weren't going to think about the issues people might have re-creating their games ten or twenty or thirty years in the future, but even if any of them did think of that, they might have just assumed that if the game needed to be made on a new machine in future, there'd just be a new version written for that new machine.
You've gotten a few examples already, but just to add to them, the assets for Killzone on PS2 were stored on a tape that was kept in a shoebox, in an employee's cellar, without any kind of proper indexing. Guerrilla were lucky to be able to produce a PS3 remaster, and that's a high-profile game that was only made 15 years ago.
There'll be lots more (mostly untold) stories like this as well. Almost all companies in the industry with a significant history will have been through some combination of key employee changes, building moves, acquisitions, financial issues, natural disasters (including small-scale ones like fires, flooding, power outages, etc.) as well as any of the other dozens of things that can screw up a backup regime even if it's being done properly.
Yep, and a lot of these problems were way worse back in the '80s and '90s. Hard drives were failure-prone, floppies could be magnetically corrupted and yet were often kept in electromagnetically noisy environments, and a lot of the other options were finicky or expensive. Even if companies were following relatively good practice at the time it's very easy to imagine a high percentage of them running into issues and losing backups, or winding up with backups stuck on something silly like ZIP disks.
Raid storage was a thing back then too y'all are making it sound like high capacity data storage was rare or even tough which is definitely not the case.
I think this has a lot less to do with technical side and more to do with complete mismanagement and bad decision making.
I think this has a lot less to do with technical side and more to do with complete mismanagement and bad decision making.
I mean, I consider myself a huge Doctor Who fan but let's not pretend many of the lost episodes weren't rubbish, lol.
Absolutely, at the end of the day it is a question of having good practices in place, but we shouldn't underestimate how advanced in technology have made such practices much easier to enforce. Backing up assets off-site was harder due to limited networking speeds, projects that were big in scope like Final Fantasy VII needed really large backup drives solutions that were really expensive at the time and the software to facilitate such versioning was not really a thing (both git and svn was not a thing in the 90's).
For some projects versioning must have been absolute nightmares with the limited tools available at the time.
I was thinking, "Holy shit!" and then I read the next few lines in that Wiki article and saw they ended up scrapping everything and starting over anyway lmao
Nope.
I was lead backend dev on an archival data storage project for long time horizon storage of research data for some major institutions. RAID is great for high availability of live data but it sucks complete shit for long-term archival purposes. Conventional RAID will happily and speedily propagate file corruption across the mirror and at best with a lot of fucking work at higher levels you'll detect that hours or days later during periodic fixity checks.
Archival is a very different thing than live-data mirroring, and you want very different tools for it. Tape is still more or less the gold standard here. RAID controllers are flakey as shit, and soft RAID suffers from the fact that your array gets fucked anytime a bug in the kernel messes up the mirroring.
Last edited:
It makes it much more difficult to ever do anything with the game.
Let's take Beamdog: there is no new version of Icewind Dale 2 because the source and assets were lost (other than reverse engineering a new client, there's no real way to work with it now, and no hope for an update or remaster). They also wanted to improve character sprites for their BG releases but couldn't because all the original assets were lost.
Backups fail. The storage goes bad, or gets misplaced, and that data is gone forever.
That's not semantics — RAID alone does nothing to solve the problems described in this thread. It only helps as an underlying part of the solution and the other parts were a bigger problem in the past
I'm not a StarCraft fan, but I'm aware of the fanscene. I really feel for them, when that disc with the source code got returned instead of released publicly. That would have released led to some dope ass shit.
It's so hard to stay on top of keeping everything backed up if it's your fulltime job in 2019, but back then the cost and difficulty was a lot worse. Like early films, no one thought it mattered either and no one thought it would be worth anything in the future.
The biggest thing is that when you're busy making games, and rushing to get them out, no one is around to take the time to back shit up. You're just focused on the next thing. Final Fantasy 8, 9, and 10 were one year apart each. Like holy shi. O_O
The biggest thing is that when you're busy making games, and rushing to get them out, no one is around to take the time to back shit up. You're just focused on the next thing. Final Fantasy 8, 9, and 10 were one year apart each. Like holy shi. O_O
This was an under appreciated comment. Everything ages. Everything eventually fails. Thus, the process of keeping backups alive forever is a process of constantly checking in on your vault, testing for errors, and going from there. In perpetuity. Forever.
I'd like to read the account of a Hollywood movie vault keeper. I imagine they will have made a fresh copy of every movie in a studio's vault over and over again At a regular pace and has seen at least a few tragedies.
Tape back up was the only remotely affordable method of long term data storage back then and it had its own host of problems. Still the main reason is simple; games were a commodity and not an art.
Same thing happened to TV/Radio/Pop Art/Comics/Movies/etc early on.
Same thing happened to TV/Radio/Pop Art/Comics/Movies/etc early on.
So, there are a bunch of reasons for how source code and other assets can get lost or effectively lost over time. These include simply not archiving in the first place for time or budget reasons, archives going bad or having hardware failure, physically losing archive tapes/storage, archives tapes/storage getting damaged by water/fire/crushing, the archiving process itself being flawed or restoration not being possible, accidental deletions, intentional deletions to save storage space or otherwise because the content was not perceived as valuable, losses from viruses or external attacks, tools/build processes becoming outdated over time or lost making the code/assets useless, and having lost knowledge or documentation on where or what is stored. Things like company growth and reorganizations, staff turnover, and relatively rapid evolution of software and hardware technology exacerbate these problems.
Our game company has been going for over 20 years now, and pretty much experienced almost every one of the issues I list above at some point.
OP
OP
Can you share anything about "external attack" that sounds wild.
At my work we once lost a ton of data when someone ran a bad database command on a production server. We went to restore from backup only to find that the nightly backup had not run successfully for weeks. And this was at a major company, where there are people whose entire jobs (in theory anyway) are to maintain the database servers and ensure backup integrity. Imagine how bad it is when no one is even trying.
Means you are unable to recompile your game for lets say another platform. say you made a PS2 game in the day and want to re release it on another systems years later. with the original source code you can make some easy changes to make it ready for another platform, when you loose the source code you would need to re engineer your game, which takes significantly more time and borderline impossible in current gen games.
I'm speaking to the multiple references in this thread of storage capacity being an issue.
Even if you manage to reliably reproduce the structure of the original source code (a pretty substantial challenge in itself for heavily optimized languages like C or C++), a lot of the most useful information is often in the comments and symbols, the former of which is basically always lost unless the code is run directly from source (making it sort of a non-issue to begin with), and the latter is commonly removed, probably especially in older software, for size and obfuscation reasons.
In our case, we got hit by a virus on our network when there were just 3 of us in the first couple of years starting out. We didn't have a central server, and I think we lost a few days to a couple of weeks worth of work each.
There are plenty of potential external threats game developers face in the vein of denial of service attacks, disgruntled ex employees, viruses/virus ransomware, competitor espionage, GamerGate style network attacks/hacks, etc which could disrupt operations in general and potentially impact backups and archives. For us, we've only had a couple of minor incidents in these areas, and nothing thats had meaningful impact or hit our archives.
I know, I've watched/listened to many of them but it's clearly a far cry from experiencing the episodes with the video. Some of the classics that survived would seem dumb if we only had the audio recordings, but the video sometimes elevates the material.
(but then again, I'm a fan of Classic Who, and Troughton, the 2nd Doctor, is my favorite, so it hurts everyday to think that most of his episodes are lost forever ;_; It's such a joy to see him in action, the scripts and audio don't do him justice)
It wasn't like that.
There's a ton of NDA to those things, but for example. Think about a system that is responsible for your money and other for you card.
You do have the knowledge that someone spend 5 bucks buying candy, because it's storage in the user transaction history system, but there's communication between systems, and each of is responsible to save their on information, but the information inside the debit card node like, verification code, authentication codes, profile checks, server time, server response, machine ID, credit card number is totally lost, because it's not transactions history duty to save.
On a metaphorical level, it's having a huge puzzle but missing one piece. You can see the image, but you still lost something.
Storage was a premium back in the 80s/ most of the 90s and companies never would have thought that gamers wanted to re play old games again.
i make videos for youtube and storing the source files of those videos takes massive amounts of space. it's only this year i've been able to keep the source video since i have massive new HDD for that and the videos are really short.
now take this and multiply it by 100 or 1000 and that's a way to see how much troublesome it'd be to backup the source code and assets of a game, especially before the 2000s when memory and storage was way more limited.
it's more of a miracle certain companies actually have the souce code of their games. Nintendo comes to mind since they've shown they do have the source of several of their games from those days (example: OoT 3D on 3DS was built using the N64 version's source code to add in the new assets and features the 3DS version got. that's why glitches from the original game work in the 3DS version since the base game is still the same but, as pointed out by others, you wouldn't be able to port the code like this without the source code)
now take this and multiply it by 100 or 1000 and that's a way to see how much troublesome it'd be to backup the source code and assets of a game, especially before the 2000s when memory and storage was way more limited.
it's more of a miracle certain companies actually have the souce code of their games. Nintendo comes to mind since they've shown they do have the source of several of their games from those days (example: OoT 3D on 3DS was built using the N64 version's source code to add in the new assets and features the 3DS version got. that's why glitches from the original game work in the 3DS version since the base game is still the same but, as pointed out by others, you wouldn't be able to port the code like this without the source code)
Dalek Master Plan, Power of the Daleks and Evil of the Daleks are better than most of the modern era of the show. Heck these are episodes that have the Daleks as more than just stupid comic relief. Look I get that their are a fair few clunky episodes from the 60s but I've always enjoyed the quirks of the early show. And I'd easily watch this over the worst episodes of the modern era...
Babylon 5 (the TV show) famously planned forward to make it future proof by saving the film reels, but they only rendered the effects at broadcast quality because it was a slow process (this being in 1993-1997). They did indeed save all the original files since it would be faster to re-render in the future when higher resolutions were needed.
Then the original files were lost.
Then the original files were lost.