What to Know
- For the second straight weekday, subway riders' evening commutes were turned into nightmares Monday thanks to an MTA tech problem
- Due to a switch malfunction caused by heat, multiple subway lines were either suspended or faced long delays
- Commuters going between Manhattan and Brooklyn were hit the worst, with some lines completely stopping all service between the boroughs
This story was originally published on July 22, 2019 by THE CITY.
Some MTA board members bugged out Monday over a computer glitch blamed for Friday night’s subway meltdown as agency documents confirmed the fiasco was no isolated incident.
Board member Larry Schwartz even raised the possibility of a cyber attack or other criminal activity.
But as THE CITY first reported Sunday, the breakdown marked the latest in a string of software problems with the computerized Automatic Train Supervision system that routes trains along all but one of the numbered subway lines and the 42nd Street Shuttle.
“This was a software bug and because that bug… was also in our backup servers, that’s why they went down as well,” New York City Transit President Andy Byford said at a committee meeting Monday. “That bug automatically powered down our servers without notice.”
Local
System Cost $230 Million
Starting in 1997 and running through 2008, the MTA spent close to $230 millionto install the ATS system, which brought real-time countdown clocks to subway platforms on most of the numbered lines and gave dispatchers the ability to control train movement remotely.
The system has been flagged for problems at least 13 times since June, including a pair of so-called “major incidents” this month that contributed to more than 180 subway delays.
“We have on occasions had problems with ATS, but nothing of this magnitude,” Byford said.
The latest bug arrived at the start of a weekend heatwave, crippling service on the 1, 2, 3, 4, 5, and 6 lines, as well as the 42nd Street Shuttle, for nearly 90 minutes. Officials said the cause differed from what was behind a July 6 failure of Automatic Train Supervision, which, according to another incident report, delayed 61 trains.
That breakdown, Byford said, was caused by a single piece of code.
“When we did the downloads and we went through the logs with our supplier, one digit, one illegal character, for whatever reason, caused an illogical command to be given,” he said.
Since then, Byford said, a “patch” has been applied.
“Things often go completely wrong because of just one tiny piece of code,” said Sarah Feinberg, an MTA board member who chairs the transit committee. “Which is why companies are supposed to check and recheck and have multiple people reading every single piece of code, every single line of code, testing, retesting and checking for glitches all the time.”
Past Glitches Snagged Service
Friday’s nightmare was caused by a different computer problem in the same system.
Tim Minton, an MTA spokesperson, said software upgrades and maintenance activities related to Automatic Train Supervision are ongoing.
“Engineers, both at the MTA and our software vendors, are vigilant to test prior to deployment, detect and resolve problem patterns that may develop,” Minton said. “Glitches are addressed promptly, with the goal of providing uninterrupted service.”
That was not the case on July 6 and at other times. Documents obtained by THE CITY show that Automatic Train Supervision failure played a part in several other major incidents on the numbered lines dating to October 2018. Among them:
On March 22, 130 trains were delayed when a “disruption in the connection of the ATS-A system” occurred for 51 minutes just after 9 am.
On March 21, 85 trains ran late after “a software event caused a disruption in the connection of certain Automatic Train Supervision-A consoles.” The disruption lasted 74 minutes.
On October 21, 2018, a “system-wide ATS-A failure” delayed 87 trains.
“Appropriate delay announcements were made,” each incident report noted.
‘No Suspicion’ of Sabotage
THE CITY’s report struck a nerve with City Council Speaker Corey Johnson — who called the outage “unacceptable” and demanded an “explanation on how it won’t happen again.”
The report also prompted Schwartz, an MTA board member and a confidante to Governor Andrew Cuomo, to question the “credibility” of THE CITY. Schwartz called for “outside expertise” to be brought in to examine Sunday’s incident, as well as an investigation by the MTA Inspector General, to see whether foul play was involved.
“When you mention the word ‘bug’, you know, I want to make sure that, and with law enforcement as well, one, was this a criminal activity that took place?” he said.
Byford ruled out any system sabotage.
“There is no suspicion that this was either a virus and there’s definitely no suspicion or concern that this was in any way a law-enforcement issue, a terrorist issue, some sort of hacking or cyber attack,” he said. “That’s absolutely not the case.”
He apologized again for the subway meltdown.
“The bottom line is, we gave people a pretty horrendous journey on Friday night,” Byford said. “I feel really badly about that.”
This story was originally published by THE CITY, an independent, nonprofit news organization dedicated to hard-hitting reporting that serves the people of New York.
