On Friday afternoon, a government contractor called L3Harris was continuing work on upgrading an aviation telecoms network.
As it swapped out copper lines for fibre at a remote location in the US, L3Harris installed a router – a piece of equipment which basically directs traffic across the network.
According to Duffy, that router started flooding others in the air traffic control system, sending 1,000 connection attempts every four to five seconds.
As a result, 17 out of 22 air route traffic control centres in the US experienced “instability” on voice and ground communications, said Bryan Bedford, administrator of the Federal Aviation Administration (FAA).
It took the aviation authority 45 minutes to identify where the problem was because maintenance is logged manually on paper.
This meant the FAA had to “pore through” the physical documents “to figure out who’s working on what”, Bedford admitted.
As a precaution as it tried to track down the cause, Bedford said the decision was made to ground all flights.
Aviation analyst Miles O’Brien described the US system as “a house that’s about to be condemned”.
He told the BBC: “I think most of us could look at this and go ‘this is a system in the United States that we have allowed to decay’. It is worse than a fixer-upper.”
Duffy said the system needed a complete overhaul, estimating a $50bn price tag.
In the meantime, he said authorities would work to determine whether it was “incompetence” or faulty equipment that led to the outage.
He also said there would be “re-evaluation” of allowing upgrades to be done during the day.
In a statement on Saturday afternoon, L3Harris confirmed it was working with the FAA to modernise the airspace communications infrastructure and said it was supporting the agency with its investigation.
“L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously,” the firm said.
Juliette Kayyem, US assistant secretary of Homeland Security under President Barack Obama, questioned why an upgrade was done on one of the week’s busiest days for travel.
“You don’t do it on a Friday when there’s lots of air traffic controllers,” she said
“You try to find ways to minimise its upgrade so you’re not maybe doing it across the board in a single moment but you also have to have systems that are ready to step in if the upgrade in any way begins to threaten the system,” Kayyem added.
A spokesperson for L3Harris said is it “supporting the FAA in its investigation to understand the conditions that led to the 13 November ground stop of our nation’s air travel”.
He added: “L3Harris takes the nationwide ground stop and its impact on travellers, airlines and air traffic operations seriously.”