The United States Postal Service (USPS) recently opened up a third round of free COVID-19 rapid tests to send to households across the nation. But before the program was announced, the Office of the CIO had just three weeks to stand up a site that had to handle up to 20 million orders an hour.
USPS Chief Information Officer (CIO) Pritha Mehra explained how the test-ordering site went at the end of last year from ideation to reality at the AWS Summit in Washington D.C. on May 24. Mehra was part of the leadership task force established for the project by Postmaster General Louis DeJoy, and she shared how her team quickly stood up the technological aspect of the operation.
“[The Office of the CIO (OCIO)] developed all the systems … whether it was the ordering system, the fulfillment system, the customer care system, the visibility, the tracking systems, you name it,” Mehra said during a keynote address. “This was a very exciting initiative. We’ve never done anything like this before, but it was very daunting.”
After admitting that her personal nightmare included mock headlines like “CIO Delays COVID Test Website,” Mehra said her team’s biggest goal was figuring out how to keep the website up and running.
“That’s all we talked about on the CIO team was, ‘how do we keep this website up?’” Mehra said. “We knew we had to stand up a website. We didn’t know how many orders an hour, we didn’t know how many orders concurrently.”
When USPS was looking to stand up the site, the tech team was initially told to prepare for up to one million orders per hour. However, she said that in December 2021, when OCIO was looking to build and stand up the site, rapid tests were in high demand and short supply across the U.S.
Realizing that building the site to take on one million orders per hour might not be enough, Mehra said the team looked at five or 10 million orders per hour as a goal before settling on 20 million orders per hour as an eventual standard.
“We first looked at our own postal store, which is our storefront from which you can order stamps and shipping supplies,” she said. “And we geared it up. We just kept throwing servers at it, hundreds of servers. I’m not kidding – 400, 900 servers at this test site – and then we started testing.”
She said the team was able to start testing within the first week and was testing critical components of the site before it went live. Once the on-prem site was able to handle one million orders per hour, the team designed a cloud application and sent it to two separate cloud platforms for redundancy purposes.
“For performance reasons, we limited our real-time synchronous process to only two things,” Mehra said. “Velocity check – make sure that this dude hasn’t ordered before – and validating the address – make sure this is a valid address. And so, that’s what kept it really simple. Everything else was done asynchronously.”
To keep the application secure Mehra said the team relied on a variety of Amazon Web Services native services to protect the site. She said the team did the most work adjusting the firewall application to fit its needs.
“So what did this do for us? At peak, we accepted 10 million orders an hour [and] 54 million in the first two days,” she said. “We have delivered over 380 million test kits and … I believe we’re now at 100 million orders. How’s that for a success story?”
