Data Backup Digest

You might not realize it, but there are some companies that are currently storing enormous amounts of data. We're talking about datasets so large that they can't be measured in megabytes, gigabytes or even terabytes. In many cases, these vast data stores consume several petabytes worth of raw data. And, in most of these cases, that amount is growing each and every day.

The National Oceanic and Atmospheric Association, or NOAA, is one of these companies and, as luck would have it, they want to share this information with other officials within the private weather industry. Unfortunately, sharing such tremendous data troves is proving to be more difficult than meets the eye.

Understanding the Problem

It's not simply a matter of uploading this data to a public server. Not only would this make it accessible to everyone – including potentially nefarious actors – but there aren't many public servers available to store such amount of space.

Jonathan O'Neil, director of the Big Data Program with NOAA, provided some insight into how big these data troves are in a recent interview: "NOAA sits on 200 to 300 petabytes of data on any given day. NOAA collects data literally from the surface of the sun to the bottom of the ocean. We have datasets that are potentially underutilized. There may be an unknown data set to solve an unknown problem."

He continued his statement by saying: "Data has become the new valuable resource. The government considers data to now be a strategic asset of the federal government. There’s been recent legislation — the Evidence-Based Policymaking Act — which requires NOAA to make its data open and requires us to make it more available to the public, and that’s really what the Big Data Program is trying to accomplish."

Exploring a Potential Solution

As you can see, the act of sharing such a large amount of data is far more nuanced than it first appears. However, IT officials with NOAA do have a potential solution: the cloud. But hosting this data on a cloud server has some inherent challenges, too – including the ever-present obstacle of finding a service provider to host such a large amount of data.

Zach Goldstein, CIO with NOAA, is confident that cloud-based storage is the only solution, as he revealed in a recent interview: "Imagine how much data would be available and how much faster it could be used and how much better it could be used if, instead of getting it off of a website or an FTP site, you could get it from the cloud, which has scalable speed and is blinding compared to what we are resource to do in our archive."

Unfortunately, the biggest roadblock to storing this data on the cloud is found with the service providers themselves. To date, none of the major players like Amazon, Google or Microsoft have stepped forward with an offer to host these tremendous data troves.
Whether the NOAA ultimately chooses one of the industry leaders or an alternative option, it seems, at least for now, that they're sights are set on the cloud.