NASA officials predict that the agency’s latest state-of-the-art satellites – launching soon – will generate an unprecedented amount of data that will be difficult to manage and optimize with NASA’s current software. Therefore, NASA researchers have turned to cloud computing to make the most of that treasure trove of information.
Currently, NASA satellites send data back to ground stations where engineers turn the raw information into information that people can utilize and understand. That dataset is then sent to an archive that keeps that information on servers. Typically, when a researcher wants to use a dataset, they log on to a website, download the data they want, and then work with it on their machine.
Processing the raw data increases the file size; this isn’t a huge problem for older missions that send back relatively smaller amounts of information. However, NASA officials expect that the quantity of data will grow, making this process unfeasible.
“Five or six years ago, there was a realization that future Earth missions [would generate] a huge volume of data and the systems we were using would become inadequate,” Suresh Vannan, manager at the Physical Oceanography Distributed Active Archive Center for NASA’s Jet Propulsion Laboratory (JPL), said in a statement.
The Surface Water and Ocean Topography (SWOT) mission – slated for a 2022 launch – is expected to produce 20 terabytes (TB) of science data a day. While the NASA-Indian Space Research Organization Synthetic Aperture Radar (NISAR) mission – slated for a 2023 launch – will generate roughly 80 TB daily. Together the data collected from these missions is enough digital storage for approximately 250 feature-length movies.
With missions like SWOT and NISAR, NASA’s current data management infrastructure is not feasible. For example, suppose a researcher wanted to download a day’s worth of information from the SWOT mission onto their computer. In that case, they’d need 20 laptops, each capable of storing a terabyte of data. Or, if they wanted to download four days’ worth of data from the NISAR mission, it would take about a year to perform on an average home internet connection.
“Working with data stored in the cloud means scientists won’t have to buy huge hard drives to download the data or wait months as numerous large files download to their system,” Lee-Lueng Fu, JPL project scientist for SWOT, said in the statement. “Processing and storing high volumes of data in the cloud will enable a cost-effective, efficient approach to the study of big-data problems.”
The first satellite to this cloud system is the Sentinel-6 Michael Freilich satellite, part of the U.S.-European Sentinel-6/Jason-CS mission. Working with Sentinel-1 data in the cloud, engineers produced a colorized map showing Earth’s surface change from more vegetated areas to deserts.
“It took a week of constant computing in the cloud, using the equivalent of thousands of machines,” Paul Rosen, JPL project scientist for NISAR, said in the press release. “If you tried to do this outside the cloud, you’d have had to buy all those thousands of machines.”
However, NASA officials clarified that utilizing cloud computing does not replace how agency researchers work with these datasets; it simply makes working with datasets more efficient.
Additionally, NASA officials realized that the current space available to store and archive these datasets is minimal. For future missions expected to generate a large quantity of data, it’s simply not enough.
“We just don’t have the additional physical server space at JPL with enough capacity and flexibility to support both NISAR and SWOT,” Hook Hua, a JPL science data systems architect for both missions, said.
However, by utilizing cloud technology, NASA expects infrastructure limitations not to be as much of a concern since it won’t have to pay to store mind-boggling amounts of data or maintain the physical space for all those hard drives.
“This is a new era for Earth observation missions, and the huge amount of data they will generate requires a new era for data handling,” Kevin Murphy, chief science data officer for NASA’s Science Mission Directorate, said.
