Shutdown caused surge for E-Verify, but modernization helped

U.S. Citizenship and Immigration Services needs a way to keep digital records for the constant stream of people looking to work in the U.S., and moving E-Verify to the cloud has been a big help. The public program for employers to verify people’s work eligibility receives about 60 million or so cases a year, and Eric Jeanmaire, division chief of Identity, Records and National Security Delivery, said it’s only growing in popularity.

In 2017 USCIS and partner Excella Consulting won an Igniting Innovation overall award from the American Council for Technology and Industry Advisory Council (ACT-IAC) for their work to modernize and improve USCIS verification services. But at that time USCIS had wrapped several pilot projects and was only a year into its modernization goals, Jeanmaire said.

“But at this point, now to go even further into the future and our another roughly year and a half into the program, we have modernized the end-to-end E-Verify case management piece, the real bread and butter of E-Verify,” he said on Federal Monthly Insights — Digital Transformation. “When individuals submit that employment case, it goes all the way through our modernized, more modular application to vet someone’s employment eligibility and comes back and gives the response.”

Modernizing E-Verify is meant to make the program more scalable in light of two major concerns: Routine business growth year-over year, and potential legislation around making E-Verify mandatory.

“So the combination of those two items really made us rethink the architecture of E-Verify,” he said to Federal News Network’s Jason Miller on Federal Drive with Tom Temin. “And in kind of [led] us … to a put E-Verify into the cloud and leverage the elasticity of the cloud for us to scale up in a more horizontal fashion, rather than us having this all-in-one application that’s vertically scaling, meaning we’d have to basically rip it out and replace it with a bigger server every time we need to build or handle a wide traffic load.”

So USCIS started to break apart the application in the strangler approach, so named by Martin Fowler in 2004 as a way to handled refactored code from a large web application. Divide the app into different functional domains and replace them with new, microservices-based domain implementation one at a time, as IMB Developer explained on their blog. All the pieces that account for case processing are now modernized, Jeanmaire said. But USCIS never got a chance to move beyond simulation for testing the new throughput for the modernized system.

“Most of our tests so far, I’ve just been through simulating traffic through a tool called J meter, where we have effectively shown that we can scale numerous fold-up in traffic, but have never actually had the opportunity to take on a very large number of real cases in production in a magnitude larger … than normal,” he said.

Then the partial government shutdown happened in December, which no one quite expected. E-Verify had to be turned off, locking employers out around the country. But companies still needed to hire workers, and Jeanmaire said this created a backlog of cases.

Whereas normally USCIS processes about half a million cases per week, the first week after the shutdown it was processing more than 2 million cases. At its peak, when the agency turned E-Verify back on it saw nearly 1,500 cases per minute, versus the normal rate of 100-150 cases per minute. Luckily, he said, the system was stable enough to handle it.

But moving to the cloud is nerve-wracking even without a surge in traffic like that which USCIS experienced. Jeanmaire said the agency had a “war room” going to monitor the situation.

“Something we’d learned over doing numerous deployment is being prepared to really watch the data as it comes in and being less reactive and more proactive,” he said. “That way we understand what’s happening in the system and can kind of react in real time.”

He also said the last year has seen staff reevaluate what they’re monitoring beyond basic CPU utilization and memory or storage.

“But what’s really the health of the data flow through the application? You know, where’s the current bottleneck? You know, what is the health of our downstream partners?” he said. “I think without that monitoring, you’re right, we would have had a very nerve-wracking day or week, their posts it down. But luckily, the entire time we felt very much in control and had a very good idea to help the system.”

Copyright © 2024 Federal News Network. All rights reserved. This website is not intended for users located within the European Economic Area.