Data Backup Digest

Buzz Lightyear and Woody are two of animations most famous characters. You’d be hard-pressed to find someone who didn’t know the pair. After a hugely successful first outing with Toy Story, the animation giants at Pixar decided to follow up with a sequel. But did you know that they almost lost two months of work thanks, in part, to faulty backups?

One day on the production of Toy Story 2, Oren Jacob (former Chief Technical Officer of Pixar) was looking at a folder directory on the computer with some of his co-workers. Said directory contained art assets for the character of Woody. When they refreshed the folder, however, they noticed that some of the files had vanished. This kept occurring, until eventually the directory was no longer valid. They jumped up into the parent folder and noticed disappearing files there too.

Unbeknownst to them at the time, but realised later, a command had been run on the system which cleans out unwanted files. However, the command had been run at root level and it was making its way through the entire structure of the project. As an aside, Pixar never investigated who ran the command and as to whether it was accidental or not, which speaks for their company culture.

Jacob ran the staff in charge of the servers and told them to pull the plug – a big no-no when the servers are connected to so many machines. Nevertheless, he had to stop the files deleting themselves somehow. When they rebooted the machines and analysed the data the team found that 90% of the film had gone.

At this point in time, Pixar already knew about data loss. They suffered some lost data while working on A Bug’s Life, but the team were able to restore from backup. The company’s backups were stored on tape, but the trouble was that these backups weren’t tested properly. It’s best practice to continually ensure that your backups are actually functioning, as Pixar soon found out.

When checking the tapes, Pixar discovered that they’d already reached maximum capacity (4GB – the film ended up being 10GB in the end). As such, while the data was being written to the tape, it was removing the older files as it did so. Even the error log, which would have flagged the problems, was zero bytes.

They used the backups and continued work for a week, thinking everything was fine, but they soon found that certain elements in the film were broken. Only then did they realise what had happened with the tapes. Cue panic.

This is where the supervising technical director, Galyn Susman, steps in. Having recently given birth, she’d been doing some work from home. As such, she had a full copy of the film stored on her machine. While the backup on her machine might have been a few weeks old, it was far better than anything Pixar had at the time.

Jacob and Susman carefully brought her machine into the studio. They plugged it in and found the backup and verified 70000 files. However, 30000 of these had to be validated by human eye. Pixar staff worked around the clock to comb over the films and make sure everything was well.

As we know, the story has a happy ending. Toy Story 2 was released. Indeed, it was released on time. But Pixar learnt a huge lesson about backups that day. It’s all well and good to have a system that backs your files up, but that system is useless if the validity of the backups isn’t being checked. You can’t restore from disaster if your data isn’t there.