The Hacker Mind Podcast: Hacking Industrial Control Systems
Can criminal hackers shut down a city’s electrical grid? Well, nothing’s impossible. But how might it actually happen? And how might we defend ourselves?
Tom Van Norman, co-founder of the ICS Village, joins The Hacker Mind to share the group’s upcoming plans for RSAC and DEF CON, where they will again present present virtual scenarios and hands on physical models of industrial control systems in order to expose hackers to their inner workings and to provide them with best practices to prevent potential threats to health, life, and safety.
There’s a classic trope in fiction, that a criminal hacker somehow gets information that allows them or their team to take down a power grid of a major city. Sometimes the fiction is too good to be true. The reality is much more complicated. But to say that something like that can never happen … well, that’s not correct either. Don’t believe me? In December, 2016, the lights went out in Kyiv, Ukraine. Here’s the CBS evening news:
CBS: Nearly a quarter of a million people lost power in this small Ukrainian city when it was targeted by a suspected Russian attack last December. vassal him Chuck is the electric control center manager and told us when hackers took over their computers, all his workers could do was film it with their cell phones. It was illogical and chaotic. He told us it seemed like something in a Hollywood movie.
The very concept of criminal hackers taking down the power grid of a world capital does seem like the plot of a Hollywood movie. And this what, as a journalist, excites me, it’s figuring out what really happened, and then it starts to sound really boring. Like any other criminal hack.
CBS: The hackers sent emails with infected attachments to power company employees, stealing their login credentials and then taking control of the grid systems to cut the circuit breakers at nearly 60 substations.
So this power grid attack is sounding more credible. This is ransomware, starting with a phishing attack. So the reality is that there was a team of criminal hackers, and like all intrusions, this attack didn’t just start in December 2016; it began months before it was executed. Here’s Robert M. Lee, CEO of Dragos, on Vice TV:
Lee: So it all started about six months previous December's about six months previous. There was phishing emails sent out so operators at Power Grid were getting emails about variety of different events going on in Ukraine when they opened up the email and a piece of malware called Black Energy three was dropped to the system that enables you to steal off credentials, usernames, passwords, things like that from the network, and then were able to come back in over that six month period. They spent that time researching and understanding the environment. So it wasn't this story that we hear sometimes thrown around about Lightspeed net speed cyberattacks, you know it was human adversaries doing research on the environment, so the attack starts. When they did that. They also had a piece of malware called killdisk position on the systems so that when the systems rebooted, it would kick off deleting all the files and deleting all the systems. Wow. So while the operators are trying to recover, they're also dealing with the fact that all their systems are going down. And then in the midst of all that, they basically blew the bridges to the substation
This was a coordinated attack, over months, with an arguably political goal in the end. But to protect these systems, we need to understand these systems. So there’s a need, a definite need, for information security professionals to have access to industrial control systems -- not virtual, but actual hands on systems -- so they can learn. In a moment I’ll introduce you to someone who is trying to do that--bring ICS equipment to security conferences.
Welcome to The Hacker Mind, an original podcast from ForAllSecure. It’s about challenging our expectations about the people who hack for a living
I’m Robert Vamosi, and in this episode I’m discussing industrial control systems-- not just power plants but hospitals and even an odd cookie factory as well. We’ll also find out more about when someone says it’s an attack on critical infrastructure, what does that mean? And we’ll hear about the ICS Village, not just at DEF CON but RSAC and other conferences.
Perhaps no legitimate discussion of Industrial Control Systems can be had without discussing Stuxnet. Kim Zetter’s book, Countdown to Zero Day, is perhaps the definitive book on the subject. In it she describes not only the political backdrop but also the technical details with great detail. Stuxnet targets supervisory control and data acquisition systems. Instead of just targeting zero days in the Windows operating system- which it did -- it then infected Siemens Step7 systems, causing the fast-spinning centrifuges to tear themselves apart. It was a masterful piece of malware, carefully crafted to achieve a specific goal. And there were a handful of researchers who worked to understand how it functioned. Here’s Liam O'Murchu of Symantec on 60 Minutes:
Wallace/ O'Murchu: is one was a very large operation. It would have taken a lot of money, a lot of time, a lot of expertise to build something like this. And really that leaves very few candidates for who could have been behind it. You're really looking at a government agency from some some country who's politically motivated and who can afford to put the money and the time into building a threat like this and who has the insider information from a uranium enrichment facility that would facilitate building a threat? Intelligence Agency? Probably Probably who has an interest in setting back the Iranian nuclear program? Yes, that narrows it down quite a bit. It does narrow down and you can see straight away the political motivation there. Essentially, yes. What's the Fed is trying to do is it's trying to slow down or stop the enrichment of uranium which can be used for atomic bombs. Were you ever concerned about your safety? This is the first time we've ever analyzed something that was so politically charged. We were looking for details about uranium enrichment facilities when we were doing our research. So it was there was times when we when we were worried. Yeah. One point you've told to colleagues, if I turn up dead and I committed suicide on Monday, I just want to tell you guys, I'm not suicidal. Yes, well, I was joking at the time, but those sort of thoughts were occurring to us that we may be followed or that people may be interested in the information that we have. And they may not want us to disperse the information that we have, or they may be interested in to find out how much we know, how close are we to finding out who's behind this. So those were the sort of things that were on our mind. We were thinking that we may have some information here that people don't want discussed, or that they would be interested in finding out how much we know
Stuxnet was, for me, the first time I really realized that there are other operating systems in production and suddenly programmable logic controller (PLC) became part of my everyday vocabulary, as did Siemens Step7, the name of the PLC operating system that had been attacked by Stuxnet. Of course our guest on this episode, he already knew all that.
My name is Tom Van Norman. I'm the co-founder of ICS village.
Vamosi: The ICS village is one of the original villages at DEF CON. Think of a room in a larger conference where people of like interests gather to hear speakers, see demonstrations and participate in themed Capture the Flag events. A village is like a mini conference within a larger conference and it is not just at DEF CON, ICS village is also at RSAC, Hack the Capital, AvergerCon, BSides, and many more. So, what exactly is meant by Industrial Control Systems?
Van Norman: industrial control systems are the systems that every industry is going to use from your manufacturing to your chemical, your food and beverage, your power plants. They are the systems that bridge, your physical world, your pumps, your valves, your motors, robots, things like that, to a computer world so we're taking the physical, physical inputs and outputs connecting them to some sort of a some sort of controller and that's where all of our programs are, are kept, are created to, to automate things to control things to make things happen. Whether it's keeping the lights on or changing your stoplights and know your intersections to making clean water. They are found in every industry in some shape or form.
Vamosi: At the time of this podcast there is renewed fighting between the Ukraine and Russia. Physical war. But also war over the internet.
Van Norman: So just with everything going on with Russia and Ukraine and everything, not that that's all in the news. So it shouldn't be. The just today that the article came out, I believe it was the DOJ that released about the critical infrastructure hacker from Russia. It's real. It's, you know, we've been saying this stuff for 20 years now, that critical infrastructure has been under attack and everything. Not all we have incidents, there's certainly you know, enough use cases or you know, and enough articles out there, but now it's now it's getting real now we had a coming home. We can't go amongst weeks really without seeing more activity groups that are being released. From you know, different companies. So for the longest time, I think it's been we'll get to it, we'll get to it. Now that is here. We get phone calls, emails, inquiries all the time. What can we do if you haven't done anything the last two years you're not going to be able to do anything today. People don't there's a lot of companies that are how do I start getting along with everybody else and it's unfortunate that it's like that but everybody is talking about it now.
Vamosi: And a lot of these systems do not use the familiar Linux Mac Windows operating systems. They are unto themselves their own things
Van Norman: That is correct. So these are all embedded systems fronting some kind of archtops or real time operating system. Occasionally you'll find a you know Windows Embedded you'll find some sort of Linux but these are all usually sandboxed off from the user. I know for the user, the engineer, the engineer or the technician that that's going to set these up and program them or even the operators and interface with them. They're going to have some kind of graphical interface. They don't really have access to the operating system behind to make any changes to make any updates. The updates are done through firmware, firmware updates that we get from the vendor.
Vamosi: the last time we spoke you were saying that not an industry like power or it was more of this concept of process control that was of interest.
Van Norman: Right right. So the process controls part of it. That's where you know, your water plants come in, that's where your chemicals, your oil and gas, your big manufacturing facilities. That's what that's what they're going to do. They're going to take your field devices and say it's a level in a, in a tank, they're going to put that into your control system. They're going to start and stop that pump, you know, open and close a valve to maintain a level or, you know, speed up a conveyor liner or do something by your process control part is no more manufacturing things.
Vamosi: Sometimes there’s this feeling that the sky is falling and part of that I think may have been precipitated through the Hollywood idea of what that would look like-- you know, the entire East Coast being blacked out all at once. Here’s the thing -- the power grids are not connected. That type of scenario that we know doesn't necessarily have to happen because the systems aren't hooked in that particular way.
Van Norman: That is correct. Is it impossible? I hate to say anything is impossible, especially with the accuracy that we're seeing today. And you know, things that are going on? Probably now, it's highly, highly unlikely. But you know, I think everybody's really cautious to say, Now that'll never happen. You know, we just look back at COVID Who would have thought that the world would shut down the way it did two years ago. So I think people are really hesitant to say No, that'll never happen because in the last two years we've seen that it doesn't ever happen. But is that balance right? Is it balance between the monkey that the myths and all of that to explaining the risks explaining the probability explaining what can what can happen and you know, in an honest way,
Vamosi: I think the nuances are more interesting the the realities of what can fail and could happen. You know, it isn't the blackout scenario. It's individual pieces of it that fall apart and become a nightmare for that company or that utility or whatever.
Van Norman: Absolutely. The, you know, everything is dependent upon everything that we just look at shipping microchips. Who would have thought that, you know, we would have a supply chain issue do we do microchips like we do today. If we follow that back? You know, it's going to go back to just a couple components, a couple things that just aligned up and happened correctly that impacted so many different industries. So, explaining that showing that and thinking outside the box on Hey, what happens if this does happen if that substation over here does go out? Yes, we have a smaller blackout or smaller, you know, area that's impacted. But what happens when you know you lose a microchip or or something else and now that just the cascading effect is just tremendous? Yeah, just just the whole supply chain issue is huge.
Vamosi: So we're talking a little bit about some of the consequences of what's going on in Ukraine and Russia and so forth. There was the ViaSat that was attacked, and the consequence may have been that the German wind turbines were affected, but in reality, it was just the monitoring of the turbines that are still working properly. That subtle nuance is sometimes missed in the news and to me, that's a lot more interesting.
Van Norman: Absolutely. You know, it wouldn't be newsworthy if it was completely factual. Probably not, at or you know, probably not to the masses, but you know, to, to the person looking for their details. It is, you know, we go back to A, we look at Colonial have 18 months ago now, I, so, that was a building that, that that that intrusion had to happen to billing. It didn't happen in the control system, but they're interconnected.
Vamosi: In June 2021, malware caused the colonial pipeline, which provides gas to the southeast corner of the United States, was shut down in an abundance of caution. Here’s the PBS News Hour:
PBS NEWS HOUR: Judy Woodruff: The federal government today confirmed that a Russian criminal group is behind the hacking of a crucial energy pipeline. The Biden administration said it is working with a Colonial Pipeline Company to deal with a cyber hack and its effects. Colonial shut down its pipeline, the largest of its kind in the US, after the company learned it was the victim of this cyber extortion attempt. We and Brian is back now with the latest on that story.
Brian Bangham: Judy, the FBI said a group known as Dark Side is responsible for this cyber attack, which used what is known as ransomware. Ransomware is malicious computer code that blocks and owners access to their computer network until a ransom gets paid. Colonial operates a 5500 mile long pipeline that carries almost half the jet fuel and gasoline that's delivered along the east coast. The company has so far refused to say whether it paid any ransom, but said it hopes to be largely back online by the end of this week. So far, the impact on gas prices has been small. But this attack is just the latest example of ransomware incidents in the US. By one estimate in just the past year, more than 113 federal, state and municipal agencies 500 Plus health facilities and more than 1600 schools, colleges and universities have all been attacked with ransomware
Van Norman: They don't know what happened. If anything happened to the control system, they couldn't track things they couldn't, you know, there's things that were going on there. So they decided to shut that pipeline down. The hack was on that pipeline. As far as I know, everything I've seen. There's no bad actors or adversaries on the pipeline didn't happen. The billing system they decided to shut it down. So just the interaction and the interdependencies is just amazing. Right now, you think about shipping a manufacturing company, making widget pallets. Come off the line, they're, well if their warehouse program or if their shipping program can't make the labels to put that pallet where it needs to go, or what trucking needs ago, that manufacturing line has to shut down because there is no space that gives a stack of pallets that don't have labels on them. What's in there? So that you know, that shipping system is not part of your industrial withdrawal system. It's on a corporate network. But without that printer, that label printer, your entire process is shut down.
Vamosi: But even something like Stuxnet, where the United States may have had a hand in that it was revelatory that you could cross an air gap system you could go into a very proprietary type of machine and disable it. As long as systems are air gaped,w ’re okay? Yet there’s Stuxnet proving that that is not true.
Van Norman: Yes, well, you did. The whole thing about air gapped systems are Garnon. Anyway, an air gap system is just a high latency network connection. At some point in time you have to cross that boundary with a USB or a CI burn DVD or something you got to get information in and out at some point in time. And when that happens, as we seen what Stuxnet you know that that USB Drive contain what
So do the other nations then look at that and try to learn from it? Do these events in other countries spark concern in the United States or in Europe?
Van Norman: Oh, absolutely. I think we're, we're all learning if you're not learning from it. It's a missed opportunity. That huge missed opportunity. How often do you have one country invade another country? And, you know, a superpower? invade a neighbor? Yeah, it's one of those things where watching, listening, learning and everything.
Vamosi: Okay, so let's talk about the ICS village a little bit. How did it come about? It seems to me that this was one of the first villages at DEF CON.
Van Norman: Yes. So I think we are going on our eighth year now. I had to look back at maybe nine years now but we'll go with eight. So we were one of the first ones that came about you know there was a few other ones. The wireless Bill has been around for quite some time. There's other ones such as the car hacking village and stuff but ICS village we were probably one of the longest running and nonprofit organizations out there but when it comes to villages we started it we excuse me, let me start. Let me start over that. So the ICS village started about eight years ago at DEF CON to bring education awareness and exposure to industrial control systems. Technology Security. It started because we go to conferences and where we read articles and you know, magazines, newspapers, whatever. And people are talking about hacking control systems taking POCs and what we quickly realize is they don't, they've never touched a PC. They have no idea what these control systems are, how they work. Their security researchers know that maybe they have firmware or maybe they found a program or something somewhere. It's legitimate work, but it's pretty pretty obvious pretty quickly that people don't know what those controllers are, what a PLC is or what to control. And so we decided to put together ICS villages around the world now. We do international events now and expose people to control systems to the technology to security, what happens or how these systems go together, what why they do what they do, how they work and things of that nature.
Vamosi: who is attracted then to this village you said you wanted to educate? So who are the people that are coming to visit?
Van Norman: We we get people all over from you know, students in academia we get controls to people to work on to control systems, engineers and technicians who want to learn more about security. We get the InfoSec people that were on enterprise systems, we get them that come because they want to learn more about security. We also get the ICS security community that comes to understand more how things work and are different vendors, how they play in with everything and you know, the audience is pretty wide which is fantastic. We have from technicians all the way to upper management, to senior leadership and companies that get involved.
Vamosi: the barrier to entry for someone who's interested in this, for example, you know, I have a laptop that runs Linux and so I can get into like network security. I can do those basic things, but if I want to do some ICS work, what do I need? Do I need to go out to eBay and buy some equipment or how would I even get started along those lines?
Van Norman: So the we get that question all the time. Where do I start? How do I start? You know, you can go to eBay you can buy this stuff the problem going e-bay and buying a controller is I know you need the software. You know a lot of times it's software is not free. You're not gonna find the software normally on eBay. So you have to go back and find where the distributors are. How to Buy that software. A lot of it's spent all this preparing Terry so you have to there's a learning curve with that. There's a huge cost and everything. Going to an ICS village event. We we expose you to all that we have trainers that we bring running young, give it a little USB with a with all the require software on it was kind of crazy because we're security conferences were giving a USB with with you know, AWS as a Ubuntu VM runs on it and people quietly take them to computer running. And you have to question that, but there's they're so eager to learn it and to do it. And a lot of people you know, have burner laptops at conferences anyway, so maybe they just aren't worried about it. And there's more and more colleges, colleges, universities, community colleges, that are that have programs now. There's also so many virtual conferences. One of the things that came out and COVID is everybody's doing hybrid this year. I can't think of any better completely. There's probably a few totally virtual ones out there but hybrid conferences and a lot of the hybrid ones are are for free. Occasionally you'll find ones that are not but there's a there's quite a few that that are for free. But But back to your back to your initial question their eBay shirt does ramp up time note is pretty long. We find people all the time that hey, I bought this thing off eBay. Can you help me set it up? I'd love to help you set that up. But it's going to take us, you know, six hours to set that thing up because we had to get the software we have to just spend on time so coming to one of the ICS village events will expose you to all of that. We're gonna be bringing our trainers around now to different different events this year. interact with it and you know good go go from there.
Vamosi: One of the things about a village is that you can have physical models. So I remember visiting the village and you guys I think had a model a water treatment plant. And you could manipulate it and see the the results in more or less real time.
Van Norman: Absolutely. So we display that you're referencing. We take that locally. We had that before. The world shut down a couple years ago. So we went from, you know, Pennsylvania where it's where we had at the time only to Clayton back, but that goes to a lot of our conferences. We are working on some smaller kits where we don't have to ship that large one but the nice thing about that is it shows people how the control systems go together and how the process actually works. You know I mentioned Process Automation before where we do have Level Transmitters and three phase pumps and variable frequency drives and how these systems all work with one another. We have vendors different vendors in there from Phoenix Contact Allen Bradley to Siemens, a Schneider draggers and clarity and rizoma and the list goes on and on but with the different technologies that we show, how they all come together, how they work, why they're important, why some of them aren't that important. So that's also another important thing to know, debunk some of the myths that are out there. Some of the marketing stuff, certainly not to throw shade on anybody, but maybe you don't really need all of those things that are being sold.
Vamosi: One of the best ways to learn is through experience and when you can’t have a working model, you can have what’s called a capture the flag. It’s a jeopardy style series of questions often themed around an incident that has happened. And as you progress down the board to the more challenging questions, you ware solving the problem. In the case of RSAC 2021, it was a cookie factory, which will return again in 2022.
Van Norman: Yes, so for RSA coming up. i We have a couple of cool things that we're doing. We're doing a tabletop tabletop exercise that's I believe, an hour long Max with 30 people for a conference that we just finished. This week. We did a cookie factory capture the flag. And with that, you know the capture flags all all virtual now. We went from only hosting it at a conference to now when we're at a conference, we run up for the conference but it's open. It's open to anybody but the cookie factory CTF is pretty neat because it has some reverse engineering in it. It has basic cybersecurity in it. It really has. It has modules or challenges for all skill levels all everyone can go there and play it. The nice thing about the CTF the way we designed it is we have hints the more hints you take, the more points get deducted from your challenge. But the more hints you take it gets you closer to that answer. So if you're into to win it, you don't want to take any hits unless you really know they can. If you're in it to learn things. Take this hence, because it's going to give you it's going to lead you right to that answer without giving you the answer. So it's worked out very well and it's great to see people spend hours going through this and then we get statistics on how many people finished questions, how many people get wrong and things of that nature. And when you look at it like hey, this person is in it. To learn things, which is pretty, pretty cool gratification here that people are using it to learn.
Vamosi: basic scenario the cookie factory like how your questions relate to what it is that they're presented with?
Van Norman: https://hacktheplanet.ctfd.io/ is where you go to, to play this conference. So what happens is our cookie factory shut down. We can no longer operate the cookie factory we have to go through and figure out why. Why did that process stop? We up all of the questions or all the players have to go and start answering these questions in a chronological sequence. So we have easy questions all to harder questions. And to give you an example of some of the questions. We have everybody starting out with the initial access category. So we have received CPU server addresses that we give a PCAP out and look at that. network traffic and figure out where the CPU server is. Your breach happened. We want to identify what's going on. Analyze that PCAP then we go to my persistence module. So persons that are the adversary got into your network. Now how did they gain persistence? Well, this is where we're going to start analyzing some firmware. We're gonna look at post post exploitation tools. We're going to go and look at how they have that adversary, keeping that foothold in there. Then from there we go to a we're brute forcing username passwords or credentials. So we give out log files and other artifacts in order to do that. Then we're going to do some lateral movement within the network. Try to identify what that lateral movement is, how it's been done, and why it's been done. Then we're going to go through and you know, we're going to do some discovery on a network. So the CTF really has a wide range of all the way down to exploitation. But the cool thing is we also have trivia. We have some basic things where you know, we throw them in there, if you're getting hung up on say an exploitation or a firmware challenge to keep people more engaged in moving. We'll throw these trivia questions throughout the CTF. It just breaks the pace up and, you know, if you're stuck on something we don't want you discouraged. Here is a trivia question that just has a little bit more fun to that to the game with Cesar. We've talked about their partnership before. They have summer skins that are out in Idaho National Labs. We have challenges for their skins also. So they went and they set up a remote connection into their cellar skids. We have questions in our CTF engine here that they answer but they use their physical hardware on wheels and also have some IoT stuff. So we have a with one of the other partners that we have is a Grimm was a founding member of it. They have howdy neighbor howdy neighbors IoT house that we build the and then we connect to the to the CTF
Howdy Neighbor seems friendly enough, but this is a connected house. It's a smart house. There’s a lot that can go wrong, and it’s up to you find that out.
Van Norman: Yeah, it's a smart house and I you know, we have everywhere anything from networking to crypto to hacking smart devices. We have a toaster that tweets out to players that we look for the tweet and then tweets part of the challenge with howdy neighbor though, howdy neighbor is all either smart house or IoT technologies. So Grimm built that all about five years ago now and I say it's all commercial off the shelf stuff that is built into the CTF, you know, its web hacks, it's reversing its crypto into things that you can do things that you can hack. So how do you neighbor is more hacking, then the ICS part in the ICS part we go from, from securing the hacking to trivia to the full circle. Things here and same thing with a C sub part of this the proceeds of skids call them their same way they have some hacking, they have some defending and bring everything full circle.
Vamosi: I know you said the landscape is pretty broad. Let's talk a little bit about the government side of things. You've had some partnerships with the Department of Energy and also with Sisa.
Van Norman: Absolutely. So we were pretty excited about this method of partnership. The Sisa one they have an escape room. Next keep in mind right there, like they have an escape room that we are or that they are going to bring to Def Con this year. We're also bringing some of them they call them seller skids to RSA. So they have some fantastic programs that they're running that we are integrating with ISIS villages. So the nice thing about that partnership is it's mutual to both parties. We have a platform where they can go and show their stuff. We both align a lot with you know community outreach and, and some stuff of that nature. So we're really excited to work with both organizations.
Vamosi: Then there's the commercial side as you indicated that there is more shall we say mundane uses of ICs.
Van Norman: Yes, so with the commercial side, I you know we get involved with a lot of different vendors. The vendors range from product control system vendors to security vendors such as there's me drag was clarity that the network detection vendors but we we also get involved with vendors such as a garland technologies that that go in to produce span aggregation, network visibility hardware, working with working with the vendors, really brings a different light to things. Everybody has, you know, their point of view on what they're, why they're important or what they're doing is important. valid points. However, we like putting everything into operation and showing people hey, this is zero trust technology. This is how it works. This is one of our sponsors. You know who has remote connection technology or zero trust technology or network visibility technology. We can listen to marketing all the time. Monitors true, a lot of it, you know, is needed. But now let's go and put it in a network and actually use it and show that end user Hey, this isn't a marketing pitch. At the village, we don't we don't market any of our sponsors. Or sorry. We market the sponsors. We don't go and sell any other technology. We wanted to go out and show people how technology works. And if they say hey, this company one of your sponsors meets that. That's great. But we don't sell anything.
Vamosi: Would you say that maturity is is okay. You mentioned that some people may not have started and they need to start now. But is that a large group of people or is that a smaller group? And really, we're somewhere along the spectrum.
Van Norman: Some industries are a lot better than others. You know, they there's some industries that are very mature. There's some companies that are very mature. There's unfortunately a lot of companies that are not really depend on you know, the size of the success of the company, the industry that they're in. The variables go on and on and on, but it's also not surprising when we go into a larger company and find flat networks. You find literally flat networks or outdated systems or email running on HMIs. Yeah, it's quite interesting. When you find that stuff.
Vamosi: Flat networks lack segmentation. That means if someone gains remote access, they have the run of the entire organization. They can go from a vulnerability in the heating and air conditioning system to the financials -- which is what happened with one of the Target breaches. Some of this, I think, is legacy . Utilities are monopolies because who else is going to lay miles and miles of wire or fiber optics or if it’s not a utility, who else but the company itself that will likely have that manufacturing center. But as Tom just said, there’s cases where someone is running email on their Human Machine Interface - their HMIs. And so, with the internet, all of that now becomes accessible. So perhaps there is a sort of a technological shock that like these were designed to be isolated systems. And organizations with control processes just didn't realize that now we're connected and we're remotely doing all this stuff that there's a consequence to that. The risk still isn’t clear enough.
Van Norman: I don't think that the consequences clearly define what your risk is. So we talk about now hey, that's really bad to have your systems configured that way. It is. It is bad however. It's not a risk. That we have to accept. It's a reset the customer already did that the owner has to accept and if that risk isn't clearly explained or identified. I think it just gets missed. It gets lost. Historically with InfoSec it's been Hey, the sky is falling. We you know we had to do this horrible thing if you know it said how can you run that? HMI away that you know windows seven on it still? We're Why didn't you upgrade that firmware or patch that? Well, I didn't patch that controller because there's no network connection to it. It sits on a seat in the control panel and that doesn't one function and I don't need to patch it. I don't need to do anything because my risk is very low. Yes. It has a very high BraunAbility bad bad vaudevillians to it, but there's no connection anywhere. So it's that balance between explaining that risk. And we're identifying that risk to somebody that has to own that risk.
Vamosi: beyond ICS I'm hearing healthcare and IoT you mentioned that at RSA you're going to have a healthcare thing is that starting with ICS and then getting into larger healthcare or just staying focused on the operational part of healthcare.
Van Norman: So it's more around the ICS portion of healthcare. So you go to a hospital and the amount of ICs and in a hospital's just amazing from your building. You're building management systems to your What are your power your now the list goes on and on and on. Those are all industrial control systems. They're the same systems that you know are going to run your chemical plant coach differently however you're gonna find. Building Automation controllers from you know the leading manufacturers at your hospital than you do in your skyscraper or your pharmaceutical plant. So your pharmaceutical plants have clean rooms. So you have, you know, positive negative pressure to keep stuff out of your manufacturing environment. That hardware is the same type of hardware you can find in your hospitals.
Vamosi: And again, that goes back to companies and governments recognizing that risk comes in all different flavors. It doesn't have to be the digital Pearl Harbor event that shakes everybody. It can be the little things like you said about the billing system at Colonial Pipeline that causes a problem.
Van Norman: Yeah, we're talking more and more about risk and interdependencies on systems than we ever have. It's extremely important to talk about and to bring light that Hey, you, you don't have to patch all your systems. Your systems don't need to be patched. Not all of them. Like I said, Do you have that control panel on the wall with no network connection on it? That controller has a really bad vulnerability, but it's not connected to anything. It doesn't sit there and it does its job and that's it. Where your other system might not have any vulnerabilities. No, no vulnerabilities in it. But if that thing goes down, your manufacturing just completely stops. So what's more important: patching that controller wall with no network connectivity or making sure you have a backup of your printer that has to print the shipping labels so your manufacturer like continue? It's all identifying the risks.
Vamosi: Are there enough people doing ICS work today or is there a massive skill shortage for what we actually need?
Van Norman: Yeah. I don't think we'll ever have enough people. Is it a massive shortage? That's also really subjective. I say, you know, my day job. I try to hire people all the time and you know, we get applicants here and there that that's a tough one. I hate to say that we it's a massive shortage. I hear that all the time. We need 100,000 people. We need, you know, 10s of 1000s of people where they're going to work. We snap our fingers and say okay, here's 100,000 more people that know about InfoSec in that we know ICS security. We train them, we develop the years 100,000 People all right. What are they going to do? Where are they going to work? Can we use them? Probably. I don't think anybody's really thinking what happens when those people do appear? Yeah. Is it a shortage? Yes. How big a shortage? I don't know. I don't
Vamosi: let me let me spin that another way. If you're thinking of going into security is ICS security a up and coming field that you would benefit from?
Van Norman: Absolutely. So one of the things ICS really just does is the education and others we love to mentor and collaborate and and bring in people with little to no experience and introduce them to things. So they can get a job in ICS security. The cool thing about it is it's not your typical, it's not your enterprise stuff. Now we have real things that are moving or you know, you can see the stuff you can, you can feel it, you can see it and it's not just your virtualized servers that you'd find on the enterprise side. So they're, they're hot, they're cold, they're dirty, they're, you know, they make noise but they also produce something that we all need to survive. I think that's one of the things that drives or interests everybody that I know and insecurity is the stuff that comes out the other end. Well,
Vamosi: Does ICS require a specialized team to do this. It's not just your your basic network stuff. It's beyond that.
Van Norman: Oh, absolutely. So you didn't need people with specialized skills or talent to do that type of work on control systems. When you're dealing with health and life safety. If somebody you're doing a penetration test on on an enterprise side and your corporate web page goes down, that's a bad day, especially if you're, you know, Amazon and your servers. That's a really bad day. However, if you are a chemical company and you're doing pen test and big into your ICS and they blow something up that's I'd have to say valuers day. So it's a you know, you have to watch health, life, and safety. Part of that. You just can't go take your laptop run. You know, pick your favorite exploitation tool on that network. You know, you can't scan it and do your test there without knowing what you're doing without a lot of coordination with the owner. The operators and owners of that process. TV is back to help life and safety. You can hurt people you know, kill people and hurt the environment.
Vamosi: Are there general best practices that you would issue? I realize that each industry is going to be different but are there blanket best practices?
Van Norman: The blanket best practices but you know, from network hygiene to make sure you have your backups but not only have your backups but test your backups, make sure they work and not not just your software backups, but you have critical spares. It could be that it could be a PLC, it could be that label maker that printer for your shipping labels. Make sure you have a backup spare. It might not be a cyberattack that's going to happen, maybe it's lightning, maybe it's like maybe somebody has a little fork truck. Maybe somebody spilled something on it. Things happen. Make sure you have backups. Make sure both software and hardware, network hygiene. You have practice, you have policies that people are following, they are tested and then you know down to Incident Response what uh What are you gonna do when something does happen? If you know, the received more and more threat mounting engagements or people talking about rounding contracts trying to look for an adversary or, you know, bad things on your network ahead of time. That's becoming a more popular discussion right now. Test your test your secure security controls, make sure they work, do that penetration. Test, do that, you know, Purple team. Test on everything.