Baltimore ransomware nightmare could last weeks more, with big consequences

It's been nearly two weeks since the City of Baltimore's networks were shut down in response to a ransomware attack, and there's still no end in sight to the attack's impact. It may be weeks more before the city's services return to something resembling normal—manual workarounds are being put in place to handle some services now, but the city's water billing and other payment systems remain offline, as well as most of the city's email and much of the government's phone systems.

The ransomware attack came in the midst of a major transition at City Hall. Mayor Bernard C. “Jack” Young assumed office officially just days before the attack, after the resignation of former mayor Catherine Pugh, who is facing an ever-expanding corruption investigation. And some of the mayor's critical staff positions remained unfilled—the mayor's deputy chief of staff for operations, Sheryl Goldstein, starts work today.

To top it off, unlike the City of Atlanta—which suffered from a Samsam ransomware attack in March of 2018—Baltimore has no insurance to cover the cost of a cyber attack. So the cost of cleaning up the RobbinHood ransomware, which will far exceed the approximately $70,000 the ransomware operators demanded, will be borne entirely by Baltimore's citizens.

It's not like the city wasn't warned. Baltimore's information security manager warned of the need for such a policy during budget hearings last year. But the final budget did not include funds for that policy, nor did it include funding for expanded security training for city employees, or other strategic investments that were part of the mayor's strategic plan for the city's information technology infrastructure.

This may take a while

In a statement to press on May 17, Mayor Young said:

I am not able to provide you with an exact timeline on when all systems will be restored. Like any large enterprise, we have thousands of systems and applications. Our focus is getting critical services back online, and doing so in a manner that ensures we keep security as one of our top priorities throughout this process. You may see partial services beginning to restore within a matter of weeks, while some of our more intricate systems may take months in the recovery process… we engaged leading industry cybersecurity experts who are on-site 24-7 working with us.

Some of the restoration efforts also require that we rebuild certain systems to make sure that when we restore business functions, we are doing so in a secure manner.

City officials have provided few details about the extent of the attack, as the city is cooperating with an FBI investigation. But it appears that the ransomware was triggered on some systems in the early hours of May 7, when email service was suddenly interrupted. The city's response to the attack has thrown many city services into disorder or shut them down entirely.

The attack was first reported by Baltimore's Department of Public Works, when the department's official Twitter account announced that its email access was cut off, and it reported phones and other systems were affected soon afterward. As it became clear what was happening, the city's Office of Information Technology team shut down nearly all of the city's non-emergency systems to prevent the further spread of the attack. It’s not clear how widespread the ransomware was within the network, but the city's email and IP-based phones were among the systems affected.

City officials have stressed that emergency systems, such as police and fire department networks and the city's 911 system, were not affected. The 911 system suffered from a ransomware attack last year when some firewall settings were disabled during maintenance. But the Baltimore Police Department was dependent on the city's email servers, and surveillance cameras around the city have been affected by the network shutdown. Nearly every other city department had services interrupted as well.

Real estate purchases cannot be closed, though Mayor Young said that a paper-based workaround for handling closings would be put in place by today. Water bills and other city charges (including parking tickets and citations from the city's speed camera and red light camera network) cannot be paid. And many city workers have had to resort to using their own laptops without a connection to city networks, as well as personal e-mail addresses and cell phones, in order to get work done. Other tasks are idled completely or have gone back to paper-based processes the city was in the midst of trying to eliminate.

A thankless job

The mayor's Office of Information Technology has been struggling to regain its footing over the past two years after a string of fired chief information officers—four consecutive CIOs were fired or forced to resign over a period of five years. Frank Johnson, who now holds the titles of both CIO and Chief Digital Officer for the city, was hired in November 2017 after leaving a position as a regional vice president of sales for Intel. Johnson led the development of a digital strategy for the city that aimed to bring Baltimore's IT spending more in line with those of similarly sized cities and transform its IT practices. According to a 2018 strategy document, Baltimore spends about half of what other cities budget for IT, and the Office of Information Technology only controls about one percent of the total budget; most of the IT spending is part of other department's operational budgets.

Until the ransomware attack, the city's email was almost entirely internally hosted, running on Windows Server 2012 in the city's data center. Only the city's Law Department had moved over to a cloud-based mail platform. Now, the city's email gateway has moved to a Microsoft-hosted mail service, but it's not clear whether all email will be migrated to the cloud—or if it's even possible. While Mayor Young said the city had data backups, it's not clear how widely backups were implemented. And Johnson would not say whether there was a disaster-recovery plan in place to deal with a ransomware attack.

Some of Baltimore's systems are hosted elsewhere, including the city's primary website, which is hosted on Amazon Web Services and operated by a contractor. But the city almost lost that website last week, and not because of ransomware: the contract for operating the site had expired, and the city was delinquent in its payments.

Tracking down how and when the malware got into the city's network is a significant task. The city has a huge attack surface, with 113 subdomains—about a quarter of which are internally hosted—and at least 256 public IP addresses (of which only eight are currently online, thanks to the network shutdown).

"We engaged leading industry cybersecurity experts who are on-site 24-7 working with us," Young said. "As part of our containment strategy, we deployed enhanced monitoring tools throughout our network to gain additional visibility. As you can imagine, with approximately 7,000 users, this takes time."