The State Department’s Office of U.S. Foreign Assistance Resources has redesigned its main Web portal for making data on U.S. government foreign aid available to lawmakers, international aid organizations, and the general public. But a major effort is underway behind the scenes to improve the quality of the data collected and the site’s analytics capabilities.
What began in 2010 as a basic site offering budget planning data has since evolved to a major data-driven portal that plays a critical role in the U.S. participation in the International Aid Transparency Initiative (IATI). Today, ForeignAssistance.gov is the main tool for improving transparency into the $38 billion the U.S. spends on foreign aid and is driving data standardization across 22 Federal agencies that are involved in foreign assistance.
“The purpose of the site is to give U.S. taxpayers and the general public a sense of how much money the U.S. government is spending on foreign assistance, in which countries, and in which sectors, like health and agriculture,” said Dennis Vega, managing director for Planning, Performance, and Systems in the Office of U.S. Foreign Assistance Resources. “It’s not only humanitarian assistance but development assistance and security assistance as well.”
Although the site started with budget data, it now includes detailed transaction data down to the award number and implementing partner for each agency. According to Vega, data is available from the top 10 agencies that account for 98 percent of all foreign aid spending. The goal is to move the site out of beta status in the next few months and to integrate data from as many of the remaining 12 agencies as possible this year.
As a signatory to the IATI, the U.S. is required to report foreign aid data on a quarterly basis. And that means collecting massive amounts of data from agencies with different systems and different definitions of the same data fields.
The U.S. Agency for International Development (USAID), for example, provides ForeignAssistance.gov more than 300,000 lines of data each quarter. “It’s a huge amount of data. We could do more complete data sets manually, but we tried to stay away from that because you can’t replicate it every quarter,” Vega said. “It would be a huge burden. So we try to use existing data systems, make improvements to those systems, and supplement them where we can.”
But the effort has not been without its technical challenges.
There are 204 data fields that ForeignAssistance.gov reports on in support of international aid transparency efforts. According to Vega, the agency sends a template to all of the agencies. Most of the agencies report back using Excel spreadsheets, with one agency using XML (eXtensible Markup Language). For those agencies that aren’t able to report in XML, the data must be ingested manually.
“The goal is to be able to get it all automated,” Vega said. “One of the issues that we ran into was that everyone calls the same data element something different. A big example is that…different organizations have different regional breakdowns. The State Department is the one that we use. But the State Department has Western Hemisphere as a region while the U.S. Agency for International Development calls that Latin America and the Caribbean because they don’t do anything with Canada.”
Likewise, USAID refers to Asia as a region while the State Department breaks that out into South, Central, and East Asia, and the Pacific.
“That’s pretty basic, but when you go into the transactional contract language to describe what somebody calls an award or a project or a program, it’s different across the board,” said Vega. “So we spent a lot of time working with interagency groups on what we call the hierarchy of primary data and what data elements go there.”
But because the reporting has to be done quarterly, the process has to be as automated as possible. “So we use the financial systems for the transactional data. And a lot of them weren’t built for this purpose,” Vega said. “A lot of the financial systems, especially in the more domestic-focused agencies, don’t collect data that is helpful for us. It’s not that there’s anything wrong with those systems, it’s just not what their purpose was. For example, from an audit perspective it doesn’t matter to them what country or sector an award is for, but for our purposes it’s a pretty important part of seeing where all of the money is going.”
Moving forward, Vega’s team of developers and data analysts must determine where they can do as much data mapping as possible and try to work with agencies to improve the systems they’re using to collect the data.
A big effort this year is to give agencies the ability to do quality control on the data they are providing. In conjunction with that effort, ForeignAssistance.gov plans to introduce new capabilities that will make the data more customizable on the site, including new data analysis tools that people can use to manipulate the data instead of having to download all of the data and enter it into an app outside of the system.
“None of the reporting agencies are reporting complete data right now, so figuring out where the gaps are is another important goal,” said Vega. They’re also working through the State Department’s e-Diplomacy office to figure out who is using the data around the world and how they are using it.
“One of the challenges we have is the burden on people to provide the data,” Vega said. “Because the systems aren’t there and because we’re still trying to work through how people are using it there’s a lot of human capital that goes into providing the data sets and producing the data. It’s hard to get people to buy in without seeing an immediate benefit. That’s one of the challenges that the aid transparency community [at large] is dealing with.”