How’s this for an emergency drill within the eye of the swirling coronavirus storm? As America social distanced, Federal IT has literally brought us together – as a government, and as a nation. As government IT operations achieve steady-state amid the demands of the pandemic, let’s begin to lift the veil on that story, starting with the Small Business Administration (SBA).
Small Business is Big Business at SBA – CIO Q&A
As volumes spiked at hospitals across America, the SBA performed triage on our nation’s hemorrhaging economy, providing life-blood loans to the biggest segment of America’s economy – small businesses. You may be surprised to learn that small businesses make up 99 percent of our country’s businesses, and account for about half of total U.S. employment.
In an action-packed fortnight, the agency delivered $350 billion of emergency loans (and counting…) to keep small businesses alive. It tripled its staffing to meet the national emergency. The SBA issued 56 disaster declarations. The agency scaled up website capacity to meet a 100-fold traffic spike, while – incredibly – also improving page load-time.
But it’s not just about more, the SBA also wanted to deliver less – money to crooks and cyber vulnerability. That’s why the agency geo-fenced its loan portal to detect loan applications submitted from overseas – and there were many. It identified and took down malicious websites and social media feeds, including two that impersonated the SBA administrator. All that, and, of course, delivering almost 100 percent telework on day one.
In an exclusive interview with MeriTalk, SBA Chief Information Officer Maria Roat and Deputy CIO Guy Cavallo tell the story of how the agency’s technology operations have been put to the test, improved on the fly, and pointed toward better performance no matter what the future might hold.
MeriTalk: Can you provide some metrics to give us a sense of the enormity of the work that SBA has done?
Roat: For starters, the Paycheck Protection Program (PPP) totaled $349 billion of loans; we did that in under two weeks. Our administrator signed 56 disaster declarations in six days, one for each state and U.S. territory. Economic injury disaster loans totaled $5.2 billion in four weeks. So that’s moving loans and loan guarantees very fast.
MeriTalk: What about the surge in web traffic?
Roat: Our web site – SBA.gov – is in the cloud, with scalable infrastructure. In normal times, we average 600 to 800 people on the site at any one time, but when President Trump tweeted out SBA.gov as the source for loans on April 3, we instantly had 93,000 users hit the site, and it scaled to 825 percent larger than it is under normal conditions. So we have seen a record amount of users hitting the site over the last month and a half, but because we use caching, there was actually a decrease in load time because the cache was used so heavily. In normal times you look at data on the site, and it’s about a terabyte a day, but the content delivery network served up 25 terabytes on that day. It was huge.
MeriTalk: How do you protect that site from people that shouldn’t be using it?
Roat: We know that people applying for PPP loans should be coming from the U.S. and its territories, and we put a geo-fence in place for loan portals to protect against spam and DDoS attacks. We geo-fenced that to only allow IP addresses from the U.S. and its territories to use the site, and we were able to cut off traffic that wasn’t supposed to be there.
Before we implemented the geo-fence, we saw a lot of loan requests starting to come from the Middle East, and other parts of the world. Once we saw that as a problem, we fenced it off to eliminate non-U.S. traffic to the loan portals. But the rest of the world can see everything else on the site.
MeriTalk: Those are great success stories from a cloud and security perspective. Are there others?
Cavallo: Before we moved to cloud cybersecurity tools, we would report two, three, or maybe five instances to US-CERT each month from our on-premises tools. When we turned the cloud on, we went to a hundred. Now with the right tools to be able to report threats, they brought down hundreds of malicious websites. That doesn’t only help SBA; it helps every citizen around the world because who knows what malicious directions those websites are sending. We are now one of the leaders in doing that, so the team had the tools to be able to do it again. The foundation we’ve laid over the last three years allowed us to move so quickly.
Roat: Last year, we launched SBA Connect, our authentication portal for external facing users that leverages login.gov. It replaces a legacy system for small businesses that was about 20 years old. Because we laid the groundwork with SBA Connect, when we stood up the LenderGateway for the PPP portal to take in new small and midsize banks that had never done business with SBA before, all we had to do was build on a portal with the web forms in front of it and put in the APIs to the systems on the back end. We already had the authentication at the ready. We brought that portal up in eight days – it was many hours of development and testing, and we were able to deploy quickly.
We knew the Economic Injury Disaster Loans (EIDL) – the $10,000 advances, as well as the loans – were going to be huge. We stood up the intake for that in seven days with a software-as-a-service solution.
We had a legacy system for disaster loans – non-COVID loans. We wouldn’t let them turn it back on after it went down for maintenance one night. We took their database – about a terabyte – and moved it up to the cloud. We accelerated the decommissioning of their on-premises system by six months. That was actually kind of cool.
Cavallo: Standing up these new information systems, having the elasticity of the cloud, was critical. There is just no way we could have handled the volume on premises. Encouraging that office to migrate to the cloud six months early was … I don’t think we have to convince them now. They are happy with it.
Roat: We were getting 10,000 emails a day at peak. We built on our existing capability to stand up a customer service hub in seven days. The emails auto forward to the hub and are automatically assigned to customer service agents as open cases. It allows them to manage, track, assign, and do the analytics for the volume of email that’s coming in, as well as some of the applications people are emailing. We’re on our fourth iteration of it right now, putting in the next wave of case routing capabilities where we look for certain keywords in each email and overall customer sentiment to make sure customers get to the right team faster.
Cavallo: When something big like this happens, we’ve had a tendency in our program offices to cut up a shared mailbox and tell the public to email their applications there. Well, we’ve exceeded the storage limits of Office 365. The initial request we got was to add another mailbox. A mailbox wasn’t the right solution, because it was getting flooded and 150 people were sharing it. So we put in place the workflow and the customer service portal that Maria talked about, which is really what you need to manage this amount of traffic. Emails get lost, and cases that are assigned do not get lost.
Roat: One more on the security front – not only was it geo-fencing but also the work of our security operations team that’s focused on the financial sector. They’re looking at the landscape and paying attention to social media, and what’s coming into the agency with malware and phishing. Some of the results – we’ve worked with DHS and taken down eight websites that were fraudulent and malicious, and we’ve taken down two fraudulent Twitter sites that were impersonating the SBA Administrator. The security operations team lives for this, and they’ve been busy.
MeriTalk: Looking back over the last seven weeks, what are you proudest of, what have you learned, and what would you teach others about the experience?
Roat: I am most proud of the entire IT team, the hours they have put in, and the work they’ve done that’s all about the mission and serving small businesses. The work they’ve done and what they’ve turned out in a short period of time is just incredible. And it’s not just my staff but also our vendor partners, because they jumped in to help us. Everybody understood the importance of what we did and how technology can really help when we needed help in getting loans and money out to small businesses.
What have I learned? I knew this already, but it comes back to the team and the ability to move quickly. You say go and they run, we were able to deliver several products very quickly. We cut through the red tape. You know where the roadblocks are and you just plow right through. You’re not doing anything that’ll get you in an orange jumpsuit, nothing illegal, but boy, can you move fast when you need to.
Cavallo: In addition to the role of SBA in COVID-19 response, what I’m really proud of is the work that we did the last three and a half years to get us to this point.
We were able to have the whole agency telework on day one. If we were struggling to implement telework, we wouldn’t have had the staff time freed up to be able to build the applications to undertake new programs. Basically we had everything in place, and it may look to others that we just flipped a switch. Well that was a switch that took three years to build, and it allows everybody to be productive, and to have access to collaboration.
Internally, we were able to set up a virtual command center using a collaboration tool that we had in place – again, we already had it versus trying to start and buy all this stuff and implement it during the pandemic. It allows any of the senior leaders working externally from the building to have the full experience of being in the command center, view dashboards and share documents.
Getting all the legwork done ahead of time for telework allowed us to then do these very special things that Maria’s talking about to build information systems and get loans out. I don’t think we could have done that without already having this other part laid out.
MeriTalk: You went to near 100% telework on day one. How many employees is that?
Cavallo: We’re probably at more than 99% teleworking. We have some auditors in our Denver office that are on desktops because their work is done in a secure location. Today, about 6,000 employees are teleworking. That includes staff we’ve added over the last 60 days – we’ve almost tripled the staff.
MeriTalk: If you had the benefit of your current experience three months ago, what would you have told yourself at the beginning of the pandemic?
Roat: Maybe something about getting our contracts in place if I had known this was coming, and probably getting ahead on some of the things where we needed support. And getting ahead on staff, because even as we flipped to telework, we had to add staff to support disaster funding disbursement. We had to go to OMB for that request, and it takes time. So I’ve gone five weeks killing my team on the operation side, working on operations, the circuits and infrastructure in the background, but had I known I would have all that set up ahead of time.
Cavallo: If we could go back in time, I would have tried to push harder to get some of our vacancies filled and some of our contracts in place, but a lot of those are not necessarily within our control, and we were pushing hard.
MeriTalk: Similarly, if you were advising the whole Federal IT enterprise, would there be any different wrinkles in that?
Roat: To the extent that term/temp FTE hiring authority could be made available sooner for mission support, that would be helpful. It was great that OPM came out with the guidelines. Also, having contracts in place with head room on their contract ceilings for surge support is beneficial to bring in contractor staff quickly.
Cavallo: We are where we are today because we’ve worked really hard to use all of the tools that we have at our disposal, particularly Office 365. Again, if we had just tried to start this a little while ago, we wouldn’t have the teleconferencing in place, and we wouldn’t have the collaboration tools in place. When you buy something, you use 100% of it, and we owned Office 365 so we said let’s use all of it.
MeriTalk: Telework has obviously been a success for SBA. Where have you seen challenges, if any?
Roat: It’s funny, when everyone started teleworking, the first reaction was the sky is falling, and some of that had to with access to printers. I have held a pretty hard line on that because of security concerns, and so far there have only been four exceptions for people needing printers, and those people have valid reasons. The security concern is you are working from home, and what if somebody sees the stuff you’re working on? I cannot verify the security of the at-home wireless printer. We work with small businesses that provide us with a ton of data and I cannot put that information at risk.
Cavallo: When Maria granted the exceptions for printers, we also had to create a process to allow only for limited use, and then take it away again. There is a clear cyber threat with very cheap printers that have WiFi capabilities built into them. It’s a scary situation and we said we’re not going to do it.
Outside of that, the biggest technology challenge was that everybody assumed all software would work through a remote connection – such as our call centers have – but we found that does not work through our secure connection. So we had to come up with a different alternative to be able to do that.
MeriTalk: What do you think might change in government or in society as a result of all the technology work that’s been done to deal with the pandemic? Will we stop doing something – like printing – or will there be new things that we’re going to carry forward?
Roat: I think the expectation is that your government is going to be digital. Rather than going to an office for services, the public is going to expect that they will be delivered digitally. I think telehealth will accelerate for the medical community. For the Federal workforce, I’m hoping that that people will rethink what it means to work. It’s something you do, not where you are.
Cavallo: I think this has moved the Federal government to where the millennials and Gen Z want us to be. They’re used to doing everything digitally and being on a phone, and they couldn’t understand why we have our paper forms and have to go to offices to file them. It’s shown everybody that we don’t need to print everything. We can digitally sign.
MeriTalk: Can you contrast what it was like to be in the first week of the pandemic with how you are operating today? How has life has evolved over these several weeks?
Roat: We were putting in 15-hour days or more, and I pulled at least one all-nighter. We’ve deployed new capabilities, rolled out some new capabilities and now we’re enhancing some of those and adding on and iterating as we go. We’re still back-to-back with phone calls and meetings and collaboration, but once the laptops started shipping, our portals came online, and we start received applications and processing money, we moved into more of a routine on the disaster activities.
Cavallo: If this becomes a very long-term thing, it has helped reinforce that we don’t need to have our employees here in D.C. If we can get some great cloud engineers or cloud developers that don’t want to move to D.C., we’ve shown our HR office that we can manage people remotely. We know what they’re doing, and they’re working very long hours. I think it’s changed the whole methodology of how we’re going to work and how we’re going to hire in the future.
Roat: Because we were just inundated the first few weeks, we divvied up the workload. Guy handled the requests for IT approval and the funding, while I kept up with the development of the projects and portals. Guy was behind the scenes doing a lot of the coordination with procurement and acquisition, and he gave me updates.
Cavallo: We’ve always worked that way – divide and conquer. That’s unique about our relationship. I had to buy the laptops or the White House was going to call, and Maria took care of the portals and getting the loans out.
MeriTalk: What has been the reaction of the front-line staff to these technology advancements you’ve rolled out on the fly, like the customer service hub?
Roat: Oh, they’re all over it. We made their lives a whole lot easier. They had to hire fewer people to sort through emails because we did it automatically for them.
We rolled out Microsoft Live Events, which can support up to 10,000 attendees. That was really well received because of all of our outreach to small businesses across the country. Sometimes the little things make a big impact.
Cavallo: With hurricanes Harvey, Irma, and Maria, our old methodology that Maria and I inherited was that when SBA surged up employees they would put them physically in a call center or a building and SBA would buy or rent more and more space. We showed them back then that we could give them virtual desktops, and that they didn’t have to all be sitting together. So now, when we went to full telework and with most of the surge staff also being teleworkers, it wasn’t brand new. This time, in fact, they expected us to do it.
MeriTalk: Any shout-outs to other Federal agencies or state can local governments that you’ve collaborated with during this time?
Roat: In the Federal government, a big shout-out to Suzette Kent. She knew the pressure we were under at SBA, and she was a big help if I needed something or had a question. Also to my fellow CIOs. Across the board, they were there to help. Early on, not knowing what way the winds were going to blow on the pandemic, I had two agencies standing by if I needed support in the field. We even got a couple of people from another agency with specific expertise.
MeriTalk: Did you see a shortening of the distance between the mission and IT in this period?
Roat: We deliver to enable the mission. I think we proved to the front office and our new administrator what we are capable of. I’m not sure the distance shortened, because we’ve always sought to deliver against mission – the business requirements.
MeriTalk: Thank you so much. We really appreciate your time and everything you’re doing.
Read more Federal success stories from the COVID-19 pandemic.