S2:E7 – The One With the Christmas Day Bombing: A Data Infrastructure Nightmare
Season 2

 
 
00:00 /
 
1X

The 2020 Christmas Day bombing in Nashville, Tennessee, caused obvious physical destruction to the city’s bustling downtown area—but what about the trickle-down effect of the physical destruction? Join our Cybersecurity and Data Privacy team for this episode of It Figures as they discuss the ramifications of such an event on infrastructure systems supporting everything from businesses to governments and families to hospital systems and how entities can protect themselves from future risk and outages due to events beyond their control.


Intro:

From Carr, Riggs & Ingram, this is It Figures: The CRI Podcast, an accounting, advisory, and industry focused podcast for business and organization leaders, entrepreneurs, and anyone who is looking to go beyond the status quo.

Jimmy Woodall:

Good morning everyone and welcome to another CRI Podcast. My name is Jimmy Woodall. I am the firm’s consulting service line leader. Joining me today are Tyler Mills and David Mills, and they are partners in our firm’s IT audit practice. They are my go-to cybersecurity experts and we’re very glad to have them. So today we’re going to talk about something that happened recently and actually personally affected to me and several others that I know.

Back on Christmas day, if you are in and around the Nashville area, there was a bomb that went off early Christmas morning on 2nd Avenue between Commerce and Church Street. We won’t get into all the reasons for the bomb or whether happened there, but we’ll talk about the effects of the bomb, right? I certainly don’t want to get into motives, but we’ll talk about the effects of the bomb and the primary effect was, again, it obviously damaged businesses. Damaged like 41 businesses.

It was a suicide bomber, so it killed the person who set the bomb in a recreational vehicle, and it injured like eight others. But 41 businesses were damaged, but one of the primary businesses was the AT&T Switching Station that’s there on 2nd Avenue. A switching station, for those of you that are old enough to remember and have been around this area in the old South Central Bell era back in the ’50s, ’60s and ’70s, it’s kind of the old place where if you see the movies where the little old ladies are switching and connecting your phone calls, right?

Well, over the years obviously technology has developed, and that’s a major AT&T data center now, and it contains a lot of network servers and data systems there for a very large area. So this explosion messed that all up, right? So there was fire damage, there was a water main that went off and so there was water damage in there. It lost power and basically you had AT&T telephone services, you had AT&T cellular services, you had AT&T internet services. Their U-verse television, and then this thing called FirstNet that they’ve got that is their nationwide wireless public safety network. All of that is in that area, particularly for … It’s a major hub for the Southeast, and so if you were in Tennessee, if you were in Kentucky … I’m in Southern Kentucky … North Alabama, North Georgia; you were affected by this. Personally I was here on Christmas day and we’ve got AT&T for everything. So I couldn’t get my … My cellphones wouldn’t work. My internet wouldn’t work. All of my streaming devices and networks; Netflix, Hulu, so forth, none of those would work.

The kids’ video games for PlayStation, all those things that they got for Christmas would not work. We couldn’t do anything, so we were kind of living back in the Dark Ages it seems like several days. But outside of my personal experience there, there were a lot of other folks who were affected a lot more seriously. So you had emergency response 911 calls for Middle Tennessee, Southern Kentucky, North Alabama were not working. So, if someone was in trouble and needed to call 911 they couldn’t because the services went down. The air traffic controllers in the Memphis Airport had issues because of the cellular and mobile networks, and so a lot of traffic had to be diverted to the Nashville Airport. There were long delays in flight movement in the Nashville Airport as well, so you had a lot of connections and issues there. You had department stores … some of them didn’t open at all, but a lot of stores, businesses large and small in all these areas could not use credit cards.

They had to go to cash-only, because their POS systems and their credit card systems couldn’t get online because of the effect of this issue right here. ATM machines, there were reports of those … and again, in this area, Kentucky, Tennessee, Alabama, Georgia, that were not working. Hospitals. I know of several hospitals that were on the front lines. You had nurses who had to enter patient information a hundred percent on paper because of the patient record systems, and it was part of their network and their network was down so they couldn’t use anything there. There were reports as far north as I’m aware of in Louisville, Kentucky, where entire school districts did not have internet. Now again, it was over Christmas so that didn’t matter as much, but still that was kind of the range of what happened there. Government offices, same thing. Again, we mentioned the 911 calls, but government offices had issues closing. At one point the Nashville Police, their phone lines were not working or having issues.

So the real consequences to this, and kind of shows our dependency on one telecommunications site. So those are some of the effects of the things that we had, and so now I want to go over to the experts. I’m kind of finished talking about the story and my personal involvement, so I want to go over to our experts, David and Tyler. Guys, what are some more of the dramatic effects that businesses and governments and local entities like this have in a situation like this? How do they deal with it, and what can they do to help protect themself from something like this going forward?

David Mills:

Well you know, Jimmy, really, we kind of talk about this a lot, and it’s the recovery capability. And a lot of times when these things occur, it’s actually a scenario that they hadn’t truly planned for and tested. So it’s always a good thing to have a good disaster recovery plan and business resumption plan, but things like this when they occur … hurricanes, tornadoes … these huge emergencies really make it difficult to understand what all the idiosyncrasies are to actually getting back up and running.

Doing a little reading on this, it looks like they were using a battery backup, and when the batteries got wet, that basically eliminated the power that they needed to maintain their systems. So again, that’s from the fire hoses and things like that, so they may not have had a sprinkler system in the room where the batteries were, but as a result of emergency services trying to put the fire out, actually water did impact that. And that’s a hard one to really plan for.

Give AT&T some credit; the services were back up fairly soon. I think as of the 27th they had most of the services back up, but the problem are things like emergency services. I know that the 911 board was rather upset that it took longer than they thought it should take to get the actual 911 services back up. And then it’s all the downstream effects. I mean you take, for example, Jimmy, the thing that you experienced. That’s a downstream effect that you just wouldn’t think would occur.

So a lot of times when you’re trying to plan for this stuff, that’s what you’ve got to consider, is some of the worst-case scenarios. So you know, Tyler, a lot of times we talk to clients and things like that about worst-case scenarios. I don’t know that we’ve talked about bombings specifically, but oftentimes there may not be redundant carriers for businesses, and they’re really only used to one that carrier.

Tyler Mills:

Yeah, that’s certainly possible, that there’s only the one carrier in the area. We often preach the importance of cellular networks in a situation like this, but then you have a situation like in Georgia; the T-Mobile circuits actually ran on AT&T backbone, so there was service interruption under T-Mobile on a completely different mobile carrier when this occurred. So it’s pretty far-reaching now.

It’s always imperative when you have something like a hospital, something that the connectivity … it really can end up as a life or death situation. It is really important to figure out that second, completely independent method of connection, even if it is some kind of cellular connection. But that’s part of the due diligence piece, is making sure that the backbones are not the same, and in this case and in a lot of cases it was here, but you would hope that AT&T and a cable provider is available.

Usually that’s your second connection. As you go through the telecom to get either DSL or your MPLS, and then you have just a consumer grade internet connection through the cable company that you can back up to, at least to run at minimum capacity, just to kind of get stuff done. So we evangelize that everywhere we go, and as part of that, also, you want to make sure that you’re testing that as well, and that failover, making sure that that actually works.

So when this happens, because it’ll happen without anybody warning you it’s going to happen and it’ll probably happen at a pretty inopportune time like Christmas, so you want to make sure that stuff is tested for sure.

Jimmy Woodall:

Tyler, that’s funny that you say that. You’re talking about the backups. I was talking to, again, a few of my neighbors here around town when all this was going on, and they were saying well that’s kind of what you get for doing these package deals with AT&T where you’ve got everything running on AT&T. Like they may have AT&T cellphone service, but they had their internet through a cable provider, and again, those like us who we were all loading up and going and sitting in Starbucks parking lot trying to get an internet connection on our phone because we couldn’t get anything. Couldn’t get cell service either, so …

Tyler Mills:

Yeah. It’s the same reason why I don’t put the magnet wallet on the back of my phone. If I lose my phone, I don’t want to have also lost my wallet, and that’s sort of the philosophy I go with when I try to explain this to clients, about why you need to make sure that there’s true independence between those two vendors.

Jimmy Woodall:

That’s a great analogy.

David Mills:

Yeah, it really is kind of, again, a difficult thing to do, but it depends on what level of impact it has. We always go through these things called business impact analysis. When we’re talking about hospitals and we’re talking about 911 services, obviously what we refer to as time to recover becomes very, very minimal. You want those services back up as quickly as possible. Unfortunately, there may not be anything available. We still don’t have a ton of multiple connectivity options for people.

And then there’s cost involved in maintaining a connection that really is never used. So it’s awfully easy to say well, the risk is low that AT&T will completely go down, and therefore we’re not going to spend the money we need to have that secondary capability. And this is a good example of when it might even go down. One other interesting fact here is not only was AT&T trying to deal with the outage and the damage from the smoke and things like that, and the fire, and the water, it was a crime scene.

So they had to also tread lightly, if you will, around everything. They were trying to get back up so that the investigators weren’t impeded.

Tyler Mills:

And it’s a pretty populated area, too.

David Mills:

Oh gosh, yeah, absolutely. Right downtown. I mean it’s a lot of folks there, and it’s not like it was for a little while there. There were a lot of people that live in the downtown area, so this is like their walking areas and things like that. But again, it’s difficult to say well, we’ll have to work through a crime scene in a recovery plan. But things that a lot of times people don’t consider, like let’s say you’ve got a business near a railroad crossing and they have a chemical spill for some reason near your location.

They may shut down a mile radius or more, and so you can’t get in to do any type of work or recovery efforts if it also somehow damages where you … if there’s obviously a train derailment or something and it damages where you are and impacts your business. These are things that we probably don’t consider enough risk, I think. I think we kind of think, “Oh, well that’ll never happen,” but yet over and over again we see it happening to a lot of people, especially hurricanes, tornadoes, thing like that. So-

Jimmy Woodall:

That’s right, David. I mean you bring up something, and Tyler brought up something a second ago about timing, and your point here when you say you don’t know that these things are going to happen. Yes, this was a bomb, and of course we hope that nothing like this happens again, but the city of Nashville downtown has experienced multiple tornadoes that have come through there recently.

In 2010 when I was still living there they had the major floods, like once-in-a-thousand-year flood that flooded all of downtown Nashville. So it’s not that there’s not events that could happen to make these things occur, right? I mean …

David Mills:

Absolutely. And so obviously the location is … when people choose locations to put switching centers, a lot of times they’re in places that the old … like you were saying earlier, the old switching terminals, buildings, are still being used. So it’s real estate that the company owns and they’re putting it to use. So they may not be in a hardened location or something like that that you would see some of the newer bunker-type data centers.

It’s super difficult to say, well, don’t put that back in downtown Nashville. Well, there may be a lot of mitigating circumstances that would require them to put that back. So how do you plan for something like that maybe occurring again? It’s super difficult, and when you’re talking about infrastructure you’re talking about large expenditures for governments and companies.

So the backup process and the recovery process for anybody is always something that you’re never really quite as sure of as you would think. I mean we have situations that occur all the time at clients that they had not expected, or that they hadn’t counted on. Things like when you have the banks that have ATM machines. It’s nothing for a tornado to tear an ATM machine up, and your recovery plan may say well, we’ve got to get a guard there to guard the ATM machine, and you start to call in the guard resources and lo and behold you hadn’t talked to them in a year, and the phone number’s changed.

So one of the things I think we need to learn is you don’t want to wait till this occurs to actually test everything that could be tested in a scenario. Don’t you agree, Tyler?

Tyler Mills:

Yes. I think it’s really important to remember that you need to test early and often. You don’t want to get caught in a situation where something happens that you could not or did not foresee and you didn’t plan for. A really interesting piece of this to me, and we haven’t really gotten into much of it, but a lot of people, myself included as someone who works in the audit profession, the question I had was where was the audit oversight?

Where the audit committee? The government committee that was making sure that if something happened to this data center that stuff could continue to operate? And there actually is that metro oversight in Nashville, and this has never … Reports that are publicly available are fairly heavily redacted, but this is something that I think that it seems like they relied very heavily on AT&T’s ability to recover, and didn’t take any of that responsibility back on themselves.

They said, “Okay, we’re using AT&T. They have these redundancies in place” and et cetera, et cetera. They never really thought that maybe they need that extra bit of recoverability on their end, and I think that caused a lot of heartache in the Nashville metro, anyway. And there’s some talk in congress now about further oversight into that process.

There are a lot of industry lines that we work in where there is no true requirement. There is no regulation around having to be able to recover or be able to … require a business resiliency plan or anything like that. There are a few. In the banking industry, the FDIC does expect you to have put that in place, tested the plan, tested it again. They’re very, very rigid.

But in the healthcare industry, really and truly, the requirement is that a plan exists. Not necessarily the plan’s any good, just that it exists. So this actually, to me, should be a pretty good place to start when you’re talking about maybe implementing some real-world change, some regulations making sure that this doesn’t happen again.

David Mills:

Yeah, no doubt about that. That’s truly where I think the 911 board was so upset. I think one of the board members said he was led to believe that this type of thing just could not happen. And we tend to sort of take everybody at their word for these larger companies that do know what they’re doing, but that they have thoroughly tested, and when they make statements that this type of thing would not occur, we just tend to believe them and move on.

So some of these areas, like I said, there’s no one real regulatory body that talks about information technology and the controls around our communications and cybersecurity and things like that that really flow down to all of these areas. And so a lot of times you’ve got things in healthcare like HIPPA and even performing high trust certifications where they are in fact being required to prove that they tested their recovery processes, but yet there’s no real requirement for them to actually do that on a truly regular basis, even annually.

Some things they’ll do, and then they’ll do it again, and they’ll test and audit and they’ll do again in two to three years. Well, a lot’s changed in the IT world in two to three years. Even annually we see that a lot has changed and things that could impact down the communication lines, that they’re not really being addressed. Not only that, but there’s a lot of changes that occur with these companies, especially the larger companies.

They buy smaller companies, the smaller companies merge together and make a larger company. All of that is kind of a recipe for, well, maybe not checking things out when these mergers occur between these types of companies to make sure that all these controls are actually in place. So there’s a lot of moving parts to this, and whereas the financial auditors have things like requirements for financial statements, for a lot of different mechanisms, they’re not required for all mechanisms.

So it’s kind of really hard to say that they didn’t follow regulations or regulatory processes, because they may have followed those to some degree, but we don’t have really much of a way to know what occurs. And the thing that always concerns me is when these companies are being bought and sold, where is the continuity after that purchase for controls testing? Things like that.

Jimmy Woodall:

Yeah. We’re just going to start to wrap things up here, but I wanted to kind of circle back. We’ve talked about the effects that occur to some of the emergency response areas; to the hospitals, to the businesses that are I guess more integral to people’s lives and public services and so forth, and we talked about the processes where a lot of these folks should be evaluating hey, what safeguards are in place, what processes are in place, what regulations are in place to address the possibility of an issue like this happening again.

One thing that I would tie in and ask about again is some of the industries that we mentioned that were affected like just retail stores, and we’re not just talking about like the large department stores. We’re talking about all the way down to small mom & pop places, and again, it’s a matter of timing; what if this had happened just a few days prior to that? A week prior to when it actually did, before Christmas?

You’re talking about disruptions to these folks’ businesses, these retail businesses that are in their busiest time of year, right? And so it’s affecting their sales. It’s affecting their ability to continue as a business. And so anything you guys want to add, again, as we wrap up here? Again, we’ve talked about the more public stuff, but what about these folks? Our clients out there that are listening.

David Mills:

Yeah, and you know, Jimmy, the thing about is is this could have affected one of those retail businesses that were nowhere near the bombing. So that’s kind of what we’re talking about here, is these things have far-reaching effects. But from a client perspective, I think one of the biggest things to do is to have an understanding of how your communications work for your business.

Ensure that you’ve got some redundancy built into that system, even if it’s something that you have to pay a little more for that you don’t actually use, but have those things built in. And then to test with realistic scenarios. Not just something that’s really easy to test and put on paper, but utilize realistic scenarios when you’re testing and actually make the phone calls that you need to make.

You may not be able to actually shut something down and switch over, but you can certainly bring up your redundant communications path and test the connectivity to that. So Tyler, anything to add?

Tyler Mills:

Yeah, I think one thing that we don’t talk about a lot in things like this, when we do podcasts or webinars or whatever it is, is what Jimmy brought up; that truly small business. Two, three people, retail businesses that are affected by stuff like this, and what can they do when it’s not economically feasible for them to have a backup connection, et cetera, et cetera.

There’s a couple of pieces. In 2021, it’s hard to imagine anybody of that size using anything other than Square or something similar to take payments. It’s so much easier to set up. There’s no hardware that you have to rent from a bank. So if you’re already in that world, then the thing to consider is, okay, well my WiFi works through this company. My cellphone works through the same company. Maybe I need to consider changing what I can change.

If I’m on AT&T’s cellphone network and also AT&T’s landline internet, it might be worth maybe taking a look at moving to a Verizon or a Sprint or somebody independent of AT&T so that you always have that connection and your credit card machine can function, because you can just use the WiFi hotspot on your phone or use the cellular connection on a tablet to run those transactions.

I think that’s a really important piece, and something that I would bet most people don’t consider, but it is really important. No matter what size of a business you are, you really want to make sure that you can continue to do business in case something like this that’s totally out of your control happens, especially around a big sales time like the holidays.

David Mills:

Jimmy, how long were you down actually at your house? I mean I know-

Jimmy Woodall:

We were down here for three days at my house, and then there was some interruption there for a few weeks after that. It was a specific time every day you could clearly tell that they were working on the network. But you mentioned this David; those folks at AT&T were put in really, really bad situation and they did their best. They brought in some mobile towers and diverted some traffic some to some other places, I think, maybe down to Huntsville and different areas.

But yeah, they were put in a bad situation. It was just a bad situation all around. But again here as we wrap this thing up, I think the important message here is, that I know these guys have conveyed, is that if you’re a business, if you’re a small retail business, if you’re in the healthcare industry, if you’re a governmental entity, whatever, it didn’t matter. Across the board folks were affected. Individuals like myself.

So you may want to take a look at what these guys have said. You may want to take a look at the network capability that you have, and what is your ability to continue? To up and run and do business and keep going in case of emergency, public service if you’re a government, in case you’re just trying to run your business if you’re a private industry. What is your capability to do that if the unthinkable happens like it happened here?

We wanted to bring you this podcast today and talk about this because this is … a lot of times we talk about theoretical events. This wasn’t theoretical. This actually happened. So I hope you’ve enjoyed it. Guys, you got anything you want to add before we sign off?

David Mills:

Jimmy I think you summed it up really well. Just consider all your options as far your recovery capability, and I wouldn’t use the option of “This will never happen to me.” I think we’ve seen that not work many times in the past.

Jimmy Woodall:

That’s right, and you don’t want to have spend Christmas with your family playing Monopoly that you got when you were a kid back in like 1985. Like it’s all over-

David Mills:

Yeah, the old analog games ha! Ha!.

Jimmy Woodall:

That’s right.

David Mills:

That can be a little bit difficult.

Jimmy Woodall:

When you’re sitting there have to look at PlayStation games that you can’t play. So with that, we’ll sign off. Thanks everybody for listening, and stay tuned for more CRI Podcasts.

Outro:

If you want more CRI insights or are interested in learning about our firm, please visit our website at cricpa.com. Thanks for listening to this episode of It Figures, the CRI Podcast. You can subscribe to It Figures on iTunes, Spotify, or wherever you prefer to listen to your podcasts. If you liked what you heard today, please leave us a review.