Federal IT has taken the ultimate stress test during the COVID-19 pandemic – and has withstood the strain. Fortified by ongoing modernization efforts, agency CIOs and their teams have ensured the delivery of vital government services during unprecedented crisis. As the new normal paves the road to recovery, MeriTalk is chronicling those success stories. Today, we go into the eye of the storm with the Department of Veterans Affairs (VA), the second-largest Federal agency and largest civil agency.
VA Outraces Demand in Virus Crisis – CIO Q&A
Even in the best of times, the VA has a huge mandate with a department budget of $220 billion – to provide healthcare, benefits, and services to support the nation’s nearly 20 million veterans. Its largest unit, Veterans Health Administration (VHA), offers healthcare at 1,440 facilities under an annual budget of more than $80 billion. The agency’s Veterans Benefits Administration (VBA) delivers a host of other benefits and services for vets, and the National Cemetery Administration takes care of burial sites and services across the United States.
In these extremely trying times, VA has met the health crisis with a remarkable expansion of IT-driven patient care capabilities, thanks to modernization of key underlying service architectures. VA is innovating to outpace service demand during the coronavirus pandemic and to prepare for whatever lies ahead. Here’s some of that story by the numbers:
VA expanded its telehealth capacity tenfold, to handle up to 35,000 appointments per day. That includes a boost to 17,000 concurrent sessions thanks to cloud service expansion. Traffic on its va.gov website has spiked to 12 million hits per month, while users of its MyHealtheVet patient portal have risen 30-40 percent.
The agency scaled up telework capability from 10 percent to 35 percent of its 400,000-person workforce – and is prepared to go higher. It ordered 225,000 laptops and relocated half of its end-user operations out of VA medical centers to ensure continuity of operations. And by harnessing data in its business intelligence service line and corporate data warehouse, VA can provide a single source of truth around the coronavirus for the entire department, as well as build out its digital experience information and applications on VA.gov, thanks to Chief Technology Officer Charles Worthington and his team.
Its reward for a job well done? The VA’s score on the American Customer Satisfaction Index is up three points in the last three months, and eight points over the last year. That puts it on par with some of the most respected corporations in America.
In an exclusive interview with MeriTalk, VA CIO Jim Gfrerer covers how the agency prepared for unprecedented demand and continues on the modernization fast track.
MeriTalk: As the CIO of a large agency with a unique mission to support veterans, please tell us a couple of your largest priorities and successes in this COVID-19 pandemic. What are you proudest of and what surprised you the most?
Gfrerer: We’re proud of the rapidity and the agility with which all our employees and vendors came together to attack the problem. That wasn’t really a surprise, but I was very pleased with the commitment of our vendor partners and how much they really leaned into it. Everyone was scrumming on the problem really hard, trying to find ways in a disrupted supply chain to help us grow exponentially.
One of the things I stressed to our vendor partners early on at the most senior level was VA’s “fourth mission” – supporting national, state, and local emergency management, public health, safety, and homeland security efforts. I spoke with a senior leader at Cisco, and said, “I hope it doesn’t happen, but you may find one of your non-veteran Cisco employees or family members ends up getting treated at a VA facility.” I think that added an extra sense of urgency for everyone.
We were concerned initially with the Defense Production Act, since we were not formally part of that effort. But we’ve been able to work with the General Services Administration (GSA) and our vendor partners to find the right fit for what we need.
MeriTalk: Can you provide some metrics to illustrate the success of your work during the pandemic?
Gfrerer: Let me start with one that doesn’t get a lot of attention – information. We are the Office of Information and Technology. Our ability to harness information and data for our customers is really important. One of our biggest successes is our business intelligence service line and our corporate data warehouse, including our national bio-surveillance tool. We’re able to use our corporate data warehouse feeds with VHA (Veterans Health Administration), and then harness the data around a whole ecosystem. That gives us a single source of truth for the department.
A lot of people don’t fully understand how much goes into getting that data right, so when agency leadership is looking at that – or national leadership in the case of the White House Coronavirus Task Force – they have that confidence in it. From a metrics standpoint, our ability to expand that and work with VHA and others to provide that single source of truth was a real standout.
MeriTalk: How about progress on telework?
Gfrerer: Before the pandemic on any given day we had about 35,000 or 40,000 teleworkers – which out of 400,000 agency employees is not much. The vast majority of those are in our VBA (Veterans Benefits Administration). But in early conversations with VHA, we became concerned that we were going to see upwards of half the department working remotely.
When we looked at our virtual private network and Citrix Access Gateway – which are two channels to deliver remote access – our VPN was very scalable, and we were able to work with Citrix quickly to do some things to modernize our four TIC gateways.
The VPN is dependent on the number of government furnished equipment (GFE) laptops. We were prepared to run upwards of 200,000 remote machines, but we’ve plateaued at about 138,000 on any given day in VA. So for telework, we are operating at about triple the pre-pandemic level, and we could probably increase that by half again.
One of the benefits of a pandemic is you get a chance to stress test your system in production. We had the cooperation and patience of the VHA and VBA to stress test the VPN and the Citrix Access Gateway over a week. As a result, we are now looking at a much better infrastructure, more cloud-based and probably going to Windows Virtual Desktop to give us all those advantages and scalability of the cloud.
We were certainly a lot smarter on May 1 than we were on March 1, in terms of what the needs were going to be.
MeriTalk: Are there other metrics you can share?
Gfrerer: The next is around telehealth – we’re basically looking at a ten-fold increase of service capacity since the pandemic began. For daily appointments, we are upwards of 30,000 to 35,000 right now, from a pre-pandemic figure of the low single-digit thousands.
For concurrent users, we have greatly expanded capacity – to between 10,000 and 11,000 concurrent sessions. With our cloud expansion this month, we think we will be upwards of 17,000 concurrent sessions. Our mantra throughout has been to stay ahead of VHA and VA’s demand, and I think the metrics have proven we’ve been able to do it.
MeriTalk: Do you have figures for the number of web hits you are getting, and how websites are performing?
Gfrerer: We have approached 12 million hits per month on the va.gov site. What we’ve been able to do on digital transformation and experience has been pretty phenomenal.
Our principal focus is to serve the business lines for VHA and VBA. But I’d also point to a 30-40 percent usage increase in our MyHealtheVet application (www.myhealth.va.gov/). That’s our patient portal, and one that I use myself as a veteran. We’ve certainly stayed ahead of demand. Sometimes it takes a pandemic to accelerate adoption of those platforms, and we’re certainly seeing it here.
MeriTalk: What are the biggest lessons you’ve learned since the pandemic began?
Gfrerer: We talked about staying ahead of demand; that’s a big lesson.
Certainly VA as a culture was not postured for remote access, but that’s changing. I think every Federal agency and commercial entity is revisiting that right now. I saw that Facebook and Google are telling their folks to be prepared to be remote for the rest of this year. We are certainly having those conversations. If we had not done some of the things that we had done in the cloud to better position ourselves over the past two years, I think it would have been a much more bumpy ride.
I’d say the same for telehealth. We don’t have a monopoly on cloud adoption, but I think VA is one of the leaders. Upwards of 80 or 90 of our applications are now hosted in the cloud.
In looking at our data center optimization initiative, we have joined our on-prem and cloud folks in our infrastructure operations pillar. And we’re talking about how we can accelerate our plan going forward to reduce our on-prem, accelerate our cloud migration, and review what metrics really matter.
The other lesson is the criticality of having a relationship with your commercial vendors, and remembering the old adage, “you can’t surge trust.” We were in a superior position with our vendor partners, knowing what our business was, knowing what our demands were, and knowing some of the personalities. When we picked up the phone, people already know who we were, what our mission was, and what we were going to ask for. They were already leaning into what we needed.
MeriTalk: Have you seen a change in the cyber threat landscape, and how have you dealt with that?
Gfrerer: We know that in the midst of any crisis, threat actors are going to try to take advantage. Within our enterprise security program, we believe that the individual employee is probably both the strongest and weakest link, and so we put a lot of emphasis on education. We added our Cofense PhishMe button just in time. We put a lot of emphasis on making sure people are attuned to the cyber threats out there. The individual employee – it starts and ends with them.
We certainly paid attention with CISA and the Health and Human Services Department when they had their big DDoS attacks early in March. That’s another great lesson – our partnership with CISA and how we collectively maintain security. So knock on wood, we have good security around our medical devices, and we have a good cybersecurity culture in the department. We’re going to continue to put a lot of effort into that.
MeriTalk: How has the CDM program performed for you in this period?
Gfrerer: We’re an active participant. And our partnership with CISA – and everything they do for us at the TIC gateways – that really is the center of gravity.
MeriTalk: How do you measure agency performance at the hospital and business level, in the trenches?
Gfrerer: Our American Customer Satisfaction Index scores have gone up eight points in the past year. They’ve gone up three points just in the past three months.
So what does that portend? We’re not doing anything other than serving our customers really well. I think that’s the one thing that’s been really tremendous coming out of the VISNs (Veterans Integrated Service Networks) – the level of support has only gotten better and stronger, and now our goal is to maintain that, especially in an environment where the demands are going to go up.
MeriTalk: Knowing what you know today, what advice would you have given yourself three months ago?
Gfrerer: The obvious areas are telehealth and telework. Those are the most quantifiable – and you can’t fake those. Either you hit the metrics and deliver or you don’t. I was very pleased with how the team swarmed the problem. We used a model that was similar to Mission Act, where we had a master scrum leader – Paul Brubaker, who’s our Deputy Chief Information Officer for Account Management. And we organized very quickly around those 10 teams. That doesn’t necessarily answer “What would I tell myself?”, but that [Mission Act] experience really guided us – certainly guided me – in terms of how we attack the problem.
MeriTalk: The VA has a lot of employees that can’t telework, because they are front-line people. For the portions of that workforce that can’t stay at home, are there any IT adjustments or mobility accommodations that you could make that would change their lives a little bit?
Gfrerer: That’s a great question. Jack Galvin, who’s our Associate Deputy Assistant Secretary of IT Operations and Services, Dewaine Beard, Director of End User Operations, and I saw it even from our IT folks – the desire to move toward mobile. You can see that with VHA; you’ll certainly see it with OIT as we go to ServiceNow with mobile applications. You can see where mobile is really creating additional efficiencies for IT, and you’re going to see that accelerate. For VHA, we are putting more cycles into contactless personal identity verification (PIV), so devices for the Federal ID card have proximity instead of physical insertion into an endpoint in order to activate it.
MeriTalk: How do you look at intra-government collaboration and cooperation? Are there greater opportunities for that kind of collaboration?
Gfrerer: I don’t think any of us could really foresee the type of continuity of operations that this pandemic created.
We’re going to have to talk about supply chain and the Defense Production Act, and how that’s going to go in the future. We would have been in a real world of hurt if the pandemic had spread a lot farther and faster. There’s a lesson learned there for the workforce: You absolutely have to have periods where you test your continuity of operations plan and your individual readiness. A lot of people in the Federal government don’t take that individual preparedness and readiness to work remotely very seriously. Hopefully we will all take it much more seriously after this.
MeriTalk: How are information and best practices being shared across agency IT teams?
Gfrerer: I think Suzette [Kent] did a great job. She had us brief on how we worked with Cisco at our TIC gateways. And how we’re able to essentially double our capacity by breaking mirroring and then by working with Cisco quickly to put the additional equipment in place, and working with the carriers to double the bandwidth. Other departments and agencies were able to brief their stories too.
The old joke is, “We’re all going to make mistakes. Let’s just make different ones.” A number of department and agency CIOs with whom I spoke were able to get a couple of steps ahead by virtue of what we had experienced. And certainly there are some other lessons learned that I observed from Rajive Mathur at SSA and Eric Olson at Treasury.
MeriTalk: From your perspective, what systems have worked best? And do you have lessons learned that you could share in IT modernization, cloud, cyber, or authentication, for example?
Gfrerer: We’ve been putting several instances of our healthcare information system into various cloud platforms. We can do some things in terms of cloud hosting to even further enhance and improve performance around those. We’re also looking at how we create an instance of that healthcare repository that might focus on our fourth mission, bringing in additional clinicians and patients from the civil sector. We know it’s a priority, especially if you expect there will be subsequent waves of this virus or any other.
Luwanda Jones, Deputy Chief Information Officer for Strategic Sourcing, and her team have done a great job engaging and educating the vendor community on how OIT supports this part of VA’s mission.
We know that coming out of the pandemic, VBA is expecting a pretty significant increase in additional and supplemental claims. When you look at the scalability of all these systems, we’re just in a much better position now to anticipate and accelerate and to be prepared for that demand as it hits us.
But I remind folks: We are literally in a black swan event. It is so rare and so unusual that you really can’t depend entirely on your past experiences. You have to think very critically about this. An example of that is – internally – we made the extraordinary step of almost immediately taking upwards of half of our end-user operations staff and putting them in an off-site setting. That was for force protection, to get them out of the medical centers, keep them healthy. There was also a value proposition around the ability to do their ticketing from home. We also use them to augment our commercial enterprise service desk for the absolute deluge we’ve got around remote access.
MeriTalk: Thinking back to your days in the first week of the crisis, how are your days different now? Are you fully entrenched in this “new normal,” and what does that look like?
Gfrerer: I would say we’re at the beginning of the beginning of our new normal. The urgency of the pandemic, in a first-aid sense, is an apt metaphor, especially in a healthcare emergency. Stop the bleeding, start the breathing, treat for shock – everything’s kind of stabilized now. We worked our way through the remote access challenges. We know what the brighter, better future is, and that’s probably around VPN and Windows Virtual Desktop. We’re going to start to provision and get to that architecture. If we were to get hit with it again, even in July, or in the early fall, we would be able to accelerate very quickly and very efficiently from a cost and infrastructure standpoint.
MeriTalk: Looking at the big picture and what’s next, what do you think will change in our government and our society moving forward as a result of this pandemic experience?
Gfrerer: In every department and agency, the CIO and the CFO are going to have to have a conversation about whether we are appropriately resourced to address these sorts of contingencies again. And do we have the modernized and scalable environment so that if the next contingency is not quite as generous in terms of onset – if it’s a little more urgent and impactful – will we have the opportunity to scale up faster to not only put our workforce in a sustained remote environment, but to also continue to deliver services. I think those are the hard questions that agencies will need to ask themselves. We’re starting to have the conversation. Speaking of which, I’d like to thank my Principal Deputy, Dominic Cussatt, for keeping a focus on normal operations to keep the many digital modernization and transformation programs moving forward and maintaining our proactive posture and timelines.
MeriTalk: Would you like to give any shout outs to team members at VA or others across the government?
Gfrerer: Certainly Suzette Kent for her leadership and engendering collaboration across the Federal space. And our colleagues at GSA and those that had to work through the Defense Production Act challenges. Jack Galvin, and his entire team. Paul Brubaker, who’s our scrum master who led those 10 work streams every day and represented us in the healthcare operations center. And certainly Susan Perez, who was my executive in charge for COVID. She worked with the department as our lead executive every day to keep IT needs front and center to make sure we could support the enterprise.
MeriTalk: The industry conference model where you would go to a tech show and you would see technology is really not happening now. How would you and your team function without conferences, as we have known them, and how do you envision interacting with industry?
Gfrerer: I guess I’m kind of old school. There’s no substitute for being there. I would cite the Silicon Valley trip we did in January as a very good example. The ability to take 20-plus of our executives offsite for a solid week and to meet with the industry teams at our partners in Silicon Valley – you can’t put a price on that in terms of experience and learning and future development. That was really helpful for everyone as we entered the pandemic and had to work with them much more closely.
Read other Federal success stories from the COVID-19 pandemic.