Transcript: Mayor Adams Updates New Yorkers on Impact of Crowdstrike Outages to New York City

July 19, 2024

Video available at: https://www.youtube.com/watch?v=8lgbg_vVrtE

Deputy Mayor Fabien Levy, Communications: Good morning, everybody. My name is Fabien Levy, and I serve as deputy mayor for Communications for the City of New York. Early this morning, our office became aware of a global IT outage impacting city governments, companies, and other organizations around the world.

We first became aware of the situation at 12:40 a.m., and at 1 a.m. in the morning, our team started convening together to deliver for New Yorkers. New Yorkers need to know that their city government moved quickly to assess the incident and protect critical resources. Chief Technology Officer Matt Fraser even conducted a series of television and radio interviews throughout the morning to keep New Yorkers informed about the outage, starting at 5:40 a.m.

While we are continuing to monitor the situation, emergency operations and crucial services, including our 911 system, are working properly. To tell you more about the incident, we are joined this morning by Mayor Eric Adams, Chief of Staff Camille Joseph Varlack, Deputy Mayor for Public Safety Phil Banks, Deputy Mayor for Operations Meera Joshi, Deputy Mayor for Strategic Initiatives Ana Almanzar, New York City Chief Technology Officer Matt Fraser, New York City Fire Department Commissioner Laura Kavanagh, New York City Health + Hospitals President and CEO Dr. Mitch Katz, New York City Department of Citywide Administrative Services Commissioner Louis Molina, New York City Department of Environmental Protection Commissioner Rit Aggarwala, First Deputy Commissioner of New York City Emergency Management Christina Farrell, New York City Police Department Chief of Information Technology Ruben Beltran, New York City Department of Education Deputy Chancellor for Family and Community Engagement Melissa Aviles-Ramos, MTA Authority Chair Janno Lieber, and Con Edison Director of Regional and Community Affairs Ruth Fasoldt. Without further delay, I’ll turn it over to Mayor Adams.

Mayor Eric Adams: Thanks so much, DM Levy. I often recall, although this was not, it was not a cyber attack, I recall when I was briefed by the former mayor, he said one of the major concerns is dealing with any form of IT outage, including if it is a cyber attack. We need to be prepared. With that in mind, Chief of Staff Camille Joseph Varlack has been hosting a series of drills. We just recently did one to deal with any type of IT outage slash cyber attack. It was good to see how quickly the team coordinated… CTO Fraser called me around a little after midnight last night and gave me a preliminary briefing that what was taking place and reached out to the other team members and everything kicked into gear. We see the results of that from not only the notifications he made this morning, but also the coming together of the team. The blueprint was already in place. All we had to do was to execute on it.

I just want to thank all these city agencies, as well as our partners in other agencies like the MTA. We have we’re here to update New Yorkers on this global I.T. outage involving CrowdStrike. CrowdStrike is a cybersecurity software prevalent in computers across the globe. Their goal, as the chief technology officer would explain, is to identify if there is a problem and to immediately take action. Early this morning, they sent out a software update that inadvertently took systems offline. They sent out a software update that inadvertently took systems offline.

This was not a cyber attack that was a hit or a cyber hit on our city infrastructure. Our city’s IT and security teams, led by the Office of Technology and Innovation, have been working nonstop to troubleshoot problems and restore security. The CTO, Matt Fraser, has been directly in touch with CrowdStrike to get real-time updates and identify the fastest path to getting all systems up and running as quickly as possible. I just really want to emphasize this was not a cyber attack. This was an update that took systems offline.

This is why the preparation is so important that the chief of staff is in charge of doing to make sure that we can immediately respond if there is an IT disruption of this magnitude or if there is a cyber attack. We have to be prepared, and this is what the drills are for. There are no calls being held or missed, and there is no backlog at the FDNY and FDNY is reviewing to ensure this remains the case. We want to ensure New Yorkers that 911’s call systems have not been impacted. Our infrastructure and emergency operations, they are all in place and we are going to continue to do that.

Life saving complaints that come through 311 or city agencies are being prioritized. Our traffic signals are working and are fully operational as is the Staten Island Ferry. Our water systems are fully operational. Our Summer Rising program will continue as scheduled, including bus service. Although we want to tell parents, you may see disruption on the bus tracking app. The systems are still operating.

We expect to continue to see cascading effects of the outage throughout the day. New York City Office of Technology will be doing everything possible to minimize potential service disruptions and restore service to our system as soon as possible. Our city agencies’ IT department will be prioritizing this work today to get things back up and operating online based on the priority of those particular systems. We want to make sure New Yorkers are safe and they can rely on the city services they need.

This is a developing situation. We are assured by the coordination that’s taken place from all the agencies involved, as well as the entities that are responsible such as CrowdStrike to make sure we can get this underhand without a minimum amount of interruptions in our city.

Deputy Mayor Levy: Thank you, mayor. Next we’ll hear from Chief Technology Officer and Commissioner of the Office of Technology and Innovation Matt Fraser.

Matthew Fraser, Chief Technology Officer, Mayor’s Office of Technology and Innovation: Good morning, everyone. To give everyone a brief detail in terms of what happened and how we responded, at approximately 12:20, between 12:20 and 12:30, we started to see impacts from systems across the city, manifesting in different places. Like usual, when something like this is one of the reasons why we have New York City Cyber Command and we have a citywide operation center, so we can detect these things as quickly as possible and dispel what’s a threat versus what’s not a threat.

So as we started to see the symptoms manifest, we contacted our partners, both at Microsoft and CrowdStrike. Subsequently, we got down to the root of the issue. The thing with technologies like this is in order to be safe and be able to respond to threats that evolve in a continuous basis, you need tools that are capable of being updated in that way. As a result, a tool like CrowdStrike, which gets updated real time, when they push a patch, if that patch goes wrong, this is a perfect example of how bad that can be.

One patch goes out at about 12 o’clock a.m Eastern Time, and it’s pushed between 12 and 1.30 a.m. Eastern time. In that period of time, we had a number of workstations, a number of computers and servers that were impacted. After notifying CrowdStrike and working with them, by 1:30, they had stopped deploying that patch. Since then, we’ve been in a state where we haven’t seen any additional machines impacted. We’re now in a state where we’re going through the recovery.

Unfortunately, the recovery at this moment is a fairly manual process that requires physical touch to most of the machines. We have a team of people that are working across the city in real time, trying to get things back up and running as quickly as possible. As the mayor said, most of our critical needs are in place. Most of our critical services, all of our life safety-related services remain up and running. There’s no impact to 911, no impact to 311, no impact to police, fire, EMS, radio or dispatch operations. Everything is running as expected. In addition to that, our operational technology networks and our SCADA networks, things that supply our water systems, things that operate our traffic lights, none of those systems are also impacted.

Today is a good example of how some of the planning that we’ve done in advance helps protect the city and ensures continuity of our critical services. As we’ve seen across the globe, this is not an issue that’s limited just to New York City. We have everyone from the airlines to financial institutions that are experiencing the same challenges. The good news is that despite the challenge that appeared today, our most critical services remain up and running, and we have a pathway to recovery to bring everything else up.

Deputy Mayor Levy: Thank you, Matt. Thank you for joining so many TV stations and radios this morning to keep New Yorkers informed. Next, we’ll hear from MTA Chair Janno Lieber to hear about what’s going on in the subways, trains, and buses.

Janno Lieber, Chair and CEO, Metropolitan Transportation Authority: The MTA activated our incident command system and our emergency management procedures in the late morning hours, and we established quickly that there is no impact to subway services, there is no impact to the bus system, no impact to commuter rails, and no impact to the paratransit operations.

The impacts are limited so far that are customer-facing are limited to one portion of our countdown clocks on the B Division, which is the letter trains, are not operating. As old New Yorkers know, the trains are coming. They’re coming on regular schedules, and you can count on it. You just may not have the countdown clock available. Otherwise, the tolls are being collected. Bad news for some people, I know.

Every single operating system is functioning. You may not have up-to-date information, just like the countdown clocks are not 100 percent. You may not have up-to-date information on exactly when commuter trains are arriving, but all of that is coming back online pretty promptly. As I said, all of the customer-facing impacts have already been evaluated and mitigated, and all of our systems are running. I want to thank the mayor and the entire New York City team for their cooperation. I’ve also been in touch with the governor’s team and briefed her on everything that’s going on. Bottom line, the airline system may be in disarray, but New York City’s public transit system is going full speed. Thank you, everybody.

Deputy Mayor Levy: Thank you very much, Janno. Next we’ll hear from Dr. Mitch Katz from H + H.

Dr. Mitchell Katz, President and CEO, NYC Health + Hospitals: Good morning, everyone. Dr. Mitch Katz, I’m the CEO and president of New York City Health and Hospitals, and I’m happy to report that not only are all of our hospitals functional, but all of the hospitals across New York City are functional.

Every hospital is required to have a plan and to practice what would happen if computer systems went down, because we deal with life and death and have to make instant decisions about prescribing a medicine, running a test. Every hospital has to be able to run without computers, and some of us are even old enough to remember when everything ran without computers. So, we know how to do it. There is a manual override process for when you have to not use the computer. You go to paper. But, throughout the city, because most of the hospitals, including ours, use an electronic health record called Epic, which did not go down. Overwhelmingly all of the systems are running just as they should.

Patients should keep their appointments. They might experience delays because some of the individual workstations have to be brought up manually, as you’ve heard. Not all of them are brought up. People should have a little bit extra patience with their doctor or nurse, but people should not neglect their care. The hospitals are all running fine and able to do their function. Thank you.

Deputy Mayor Levy: Thank you, Dr. Katz. We’ll take some on-topic questions now.

Question: About the system update and how it caused the problem the city’s experiencing now, is it something in the software update, the patches you were talking about, or was it like, was it… How it was implemented? Could you just give us a little more detail on how things went awry there?

Fraser: Yes, sure. The way these technologies work is that there’s a control file that gets deployed into the agent that runs on every computer and server. When that control file got deployed, part of the control file was corrupted. So when it went to execute and apply the update, the Windows system itself panicked, which is what causes the blue screen that says that they commonly referred to as the blue screen of death. That update that went in, that came as a control file into the agent, is the thing that triggered the symptoms that we’re seeing.

Question: That corrupt file was on the CrowdStrike end of things?

Fraser: That is correct. That is correct. The way these agents work, because we have malicious actors that work 24/7 around the clock, there are new signatures that appear every day. These tools stay connected consistently, so that as new updates come, they update, so that our systems can be as safe as possible. Now, part of that means that those control files get updated sometimes multiple times a day, on a daily basis, weekly basis, so on and so forth. So as those files come in, it’s normal to see that type of update. What’s not normal is the code that was corrupted as part of it.

Question: What types… You mentioned some workstations went down. What types of workstations, meaning, is this, processing vouchers on an administrative level for example? I’m not saying that’s what happened, but what types of workstations went down? And is there some sort of bifurcation to ensure the critical services didn’t go down, or was it just luck?

Fraser: In this case, there’s no luck. It’s good planning and practice, which kept our emergency services and our lifeline services up. The way that New York City controls its infrastructure, our most critical services, 911-related, 311-related, we isolate and keep it in a separate environment, and we control what updates get pushed into that segment of the environment to ensure that in a situation like this, when something happens, it doesn’t take down our most critical systems.

Question: There’s no automatic updating on those critical services?

Fraser: There’s automatic updating, but it’s controlled so that it only happens within certain periods of time. We have a sandbox where we test those updates to ensure that when they get applied, if it impacts, it impacts something that’s in a test environment, not in a production environment. And the other cases, for the machines that have access out to the internet and the basic workstations that many people interact with, those are updated in more real time because the risks to those machines are much greater than the ones that are in the public safety side of the world.

Question: What workstations were those? What type of jobs will New York City impact on them?

Fraser: What I’d say is that there’s going to be a myriad of impacts, which we are currently assessing, and those workstations are in places like TLC, DOB. They’re just regular computers that are out in most of the agencies.

Deputy Mayor Meera Joshi, Operations: Just to clarify that complaints as they come in through other 311 or to the agencies are always prioritized. Those high-prioritized complaints are being addressed. So, even if there was a period of time when the workstation is down or there is a lack of interface because of the outage, we are manually going in and ensuring we have all of those priority complaints, and they’re being addressed across the city, especially with the first responder agencies.

Question: Does CrowdStrike know how that file got corrupted, or the implementation got corrupted, or are they still trying to figure it out?

Fraser: So after we discovered the incident and we started to see the impact, with working with CrowdStrike, they were able to correlate the impacts that not only that we saw, but the global community saw, with an update they pushed, and they found it with timing. They aligned it with timing. Now what they said was that they tested the update in their testing environment, and none of these symptoms manifested, but something happened somewhere in between when they went to test it and when they went to deploy it, something changed or something got corrupted, which is why we see what we see right now.

So CrowdStrike has confirmed that the issue related to their control file that was updated, and they said that is why they stopped deploying it, and they rolled back to the previous version.

Question: They’re still trying to pinpoint what went wrong in a way.

Fraser: Yes, I’m sure they’re looking in their quality control process, but I would defer to CrowdStrike to have them answer what’s going on in their internal process.

Question: Just to be clear, are there any impacts to fire or police services at all?

Fraser: No. Emergency services, meaning if you dial 911 or you call for support, there are no impacts to calls for support. 911 operations are working, dispatch operations are working, and emergency communication via radio are also working.

Question: One more, as New Yorkers go about their day, is there anything that they can expect to see, any specific disruptions?

Fraser: Can you repeat that one?

Questions: As New Yorkers go about their day, are they going to see any specific disruptions?

Fraser: As we mentioned, the essential services and life safety-related services are not impacted. However, there are other services that are in the city that may be impacted. So if you go in to file for a permit or you go in to request some other service, you go to pay a bill online like a water bill or a parking ticket, you might find yourself in a position where those services may be offline temporarily. But as I said, we’re moving through and we have a pathway to recovery, and we’re confident that soon we’ll be back to a normal operating state for the services that may feel slightly impacted.

Deputy Mayor Levy: Water bills are fine.

Fraser: You can still pay your water bill.

Deputy Mayor Philip Banks III, Public Safety: When we have situations like that, we automatically go to what emergency services that impact New Yorkers on an everyday basis. I think Chief Fraser actually explained it properly.

When it comes to the FDNY and the NYPD, there are some back issues what were dealing with now that won’t have an immediate impact on the day to day services, there’s some arrest processing issues that are taking place that were on top of, there’s some camera issues that are affecting DOC, DEP, NYPD.

The average, 99 percent of them won’t be impacted by it, and we’re pretty confident that the impact we will have will be cleared up before it does have an operational impact on, in fact, the everyday public. We’re looking at this. We’re working on it now. While we’re here meeting, the mayor has a whole team, all the agencies are all talking, coordinating together. We’re pretty confident, in fact, that we jumped on top of this.

As the mayor said earlier, I just want to reiterate, and the chief of staff, a series of meetings. So when this came up and we went into mode, this was something that we did, I think, two or three weeks ago. It was almost like it was an exercise that we had planned for. It came out in fact, and that’s why we’re going to see minimal disruptions to New York City. The team is working well.

Question: MTA related, which is why I wanted to get it in. Any sort of update on when the somewhat minor impacts, ETAs and whatnot, will be resolved by MTA? When are New Yorkers expecting…

Lieber: It’s all related to the same technological issues that the chief technology officer outlined in detail. We’re anticipating… CrowdStrike and Microsoft get their resolution, it’ll translate through pretty quickly. In the meantime, New Yorkers know where to go. Regular bus service, regular subway service, regular commuter rail on schedule. We’re keeping to that. And they can count on the service being provided. Thank you.