- Welcome to "The Cyber5" where security experts and leaders answer five burning questions on one hot topic in actual intelligence enterprise. Topics include adversary research and attribution, digital executive protection, supply chain risk, brand reputation and protection, disinformation, and cyber threat intelligence. I'm your host Landon Winkelvoss, co-founder of Nisos Managed Intelligence company. In this episode I talk with leading security practitioner, Nate Singleton, who's most recently the Director of IT and Governance and Risk and Compliance at Helmerich and Payne. We discussed the conundrums of operational technology within gas and energy sectors, including risks downstream and upstream. We also compare the aggressive and constant need for interconnectivity on the information and operation technology sides of the house, to show that events like the Colonial Pipeline ransomware attack are probably only just the beginning for future attacks against critical infrastructure. We also discuss what more the major oil and gas companies can do to help smaller companies, critical in the oil and gas supply chain. Stay with us. Nate, welcome to the show, sir, would you mind sharing a little bit of your background for our listeners please?

- Sure thing, so after some time in the military and undergraduate and graduate school, I went to work for the US Department of Energy, working in electric power grid security covering about four or five states here in the Midwest. When I left there I went to a oil and gas oilfield services company, they didn't have a cybersecurity program so I was brought in to build their cybersecurity program up from zero to as future as we could get it. Had a great time doing it, got to travel the world a bit, report to the board of directors, talk to the big cyber insurance underwriters, all that sort of thing, so it was a great experience, really enjoyed it. Now I'm doing some independent consulting work for smaller clients just to keep myself busy for a little while.

- I appreciate your range of experience, for sure, and I think that's exciting to get into the evaluation, the conundrums, frankly, of really dealing with operational technology security in energy and oil and gas today. So just starting off based on your experiences, providing an overview, if you don't mind, just of the general challenges within ICS and SCADA security.

- Sure, so taking it one step back, when you start talking about ICS and SCADA in general, there's still a lot of debate and, like with the rest of the IT landscape, there is some friction between security and IT. Now we're gonna add a third leg to that stool and put OT in there. There's friction between these three groups and so building those relationships becomes paramount to helping secure the environments and there's a lot of education that has to occur. Whenever you start talking about OT, especially, these systems are all designed for exceptionally long lifespans. In the IT world you buy a server, you expect it to last three to five years. These guys are building systems they expect to last 20, 30, 40, 50 years, maybe longer than their career, right? Because of that you also have, you build this system, it's 20 years old, it still runs on 20 year old technology including some of the underlying servers that directly support the PLCs and those end points there. So, we're talking really old versions of Windows, old versions of Linux. The upgrade costs are massive. The human capital required to do those upgrades is massive because it requires complete reprogramming, so a lot of financial people time, and then you're gonna probably take some significant downtime on these resources. So you're talking drill rigs, you're talking pump stations and parts of pipelines, you're talking segments of a refinery, what have you. It's also, there's lot of commonality but it's very esoteric when you start getting into the weeds between IT and OT, special protocols you use, that sort of thing. Also, whenever you start looking at it, downstream, that's the exploration and production side, so you have all the drilling companies. You have the big majors, that we all have heard of, plus smaller companies, so you're talking about your Exxons and your Shells and all the oilfield services companies, all their technology is different, potentially substantially so, even inside of their own organizations. If you just look at drilling itself, you have land-based rigs, which all of them can be very different going from full ethernet all the way down to basically wire wrap, plug and play type things. You have drilling platforms on the offshore and then you have drill shifts, and they're all completely different. That's where you start getting the problems. If you're a Cisco engineer, it should be fairly easy to go figure out Cisco at a new shop. But even if you're an engineer for this type of OT equipment, because of the way that they've programmed it, configured it, it's gonna require a whole different learning curve to go into that new shop and understand what they've done there. That's where you start getting a lot of the conflicts and like with most of the end point devices, like LiO or IoT, even printers and stuff, there's intelligence there. There's enough intelligence to do something malicious but it's not gonna run your current generation end point security products. So how do you go about securing them? The difference between OT and IT truly is the devil's in the details. OT gets very esoteric once you get past that, go from layer two to layer one. That interface there, so that's going from those backend HMI, data logger type server systems to those industrial control systems, that whole interface portion and then down into the actual machinery pumps, what have you, that you're controlling. That's where it starts to get very unique and very separate. That's where it really starts to distinguish itself from traditional IP and the devil in those details will kill you every time. And in some cases literally kill you.

- Can you give an example of that?

- Sure, easy case. If I'm using your classic vulnerability scanner or network scanner, where you're plugging your Nmap's, your Nessus, your what have you, right? I'm scanning a network segment. If I scan the standard issue Windows machine it's gonna pop back with whatever details it's gonna fetch up, right? If I hit one of those PLCs, it could cause a derangement in the process, so it could cause it to shut down. It could cause it to crash and crater. It could cause it to spin up a process. If something just goes straight shut down, the world will probably not end, it just locks things down unless it's shut something down that is allowing pressure to bleed off, in which case pressure could build up, you have an explosion. If I start something up that shouldn't be started up and there's a guy standing in the wrong place at the wrong time. In the company that I was recently at we had drilling rigs, if a guy's in the right place and following procedures and everything's working right, he's perfectly safe standing right here at this time. However, if all of a sudden something spins up that should not be spun up, he could get pulled into the works and literally it could rip him in half and it wouldn't even blip on the torque meter. That could happen all because of a scan.

- Because of a scan.

- Because all of a sudden the PLC picks up because it thinks it's time, for whatever reasons, it's time to start this process up, and that is a bit of reality. You do it wrong, you can shut down segments in the power grid and let the chaos ensue. And that's an electric side. You know, waterworks side, you can shut down water to a town. The real problem here, and the reason why you have to have such a metered and careful approach, is because if a computer goes down, it's a frustration but if you've got your proper backups everything spins back up properly. If these things go down they impact the physical world and they impact it in a very direct way. Water gate's open on dams. There's not some dude with a winch or a handle that's spinning the thing around, they push a button or it's part of a whole overall control process. The water gates lift our dam and water spills down the rest of the river, so all of a sudden you have a wall of water going down the river and nobody's expecting, there's guys down there fishing. Oil and gas, pipelines, all of a sudden bunch of pressure going through 'cause they're pushing product, all of a sudden you jam up one end of that and all that product hits there. Let's face it oil and gas, the products we're talking about, a lot of times they're a little bit flammable. All of a sudden there's an explosion because there's a rupture, something causes a spark and now you have a fire, or at least a major oil spill.

- So, basically, a simple scan you're really talking about. And because there's not the redundancy in place, a simple scan for vulnerabilities really has to have the utmost coordination almost across the different sectors of who's operating that to ensure that the fail-safes are really in place. Is that basically what you're saying?

- Basically, yeah. The point of it is, is that there is a lot of redundancy in the network but as you're going through the scan, most people want to hit a scan off, they just blast out across an IP range, right? If you do that there could be consequences and severe consequences because you may hit all of the PLCs in that range and that may take up your primary, secondary and tertiary redundancies real quick.

- In the modern state US economy, certainly within the US technology economy, you're very familiar, just like I am, of the move fast and break things mantra of really pursuing endless connectivity. And a lot of people could argue that that state of mind within the VC private equity and investment communities of pushing for that connectivity has us where we are in the current cybersecurity weakness state that we live in and that we're around, the whole world is now going through. And, of course, that's been particularly exacerbated with a lot of the recent ransomware events. Is there a similar problem within, I mean, you mentioned earlier that these systems are being built for 30 to 40 years, whereas a lot of technologies in the current Silicon Valley landscape are not built for that. They're probably built for a lot less time, like you just said. Is there a similar problem of connectivity that you see play out within ICS and SCADA environments?

- Yeah, absolutely. Just like when you start talking about the commercial, off-the-shelf kit that you buy for your home, or for your office, a lot of that stuff. And then it's getting better in the industry, I think, as a whole but a lot of that is not built with security in mind, right? Arguably you can go down to the design of processors, how secure are they really from a design perspective? When you start looking at that rush to market and building those new functionalities, security, a lot of times, takes the backseat because they want to just sprint and get this new widget out there. We see that in the ICS SCADA environments as well, and your OT environments, but then where you start adding additional levels of complexity is that again you go back to that, some of this kit has been out there for 20 years plus. When you're talking about they're running the big beast of Windows NT on these old outdated servers and whatnot, the cost to upgrade are just egregious in some cases. And then you're mixing and mashing all that stuff together so you have this latest and greatest whatever over ethernet device. You have stuff that's running off serial ports or pre serial port, running through a DAC, a digital analog converter, and that just creates a big mess because most companies are not gonna be completely homogeneous. They're not gonna have the same thing throughout their entire network, as far as the same type of SCADA kit and everything else, or ICS kit, so that's where we see a lot of those issues. And when you start talking about ransomware or hitting, some of the ransomware events we're seeing, whenever you have something that a simple vulnerability scan can take down, then something like a ransomware event, or any malware event, becomes potentially catastrophic. I know of instances where an inadvertent Windows patched because things were misconfigured on a domain site and an inadvertent Windows patch brought down the system. It started acting funny, could've gotten somebody very seriously hurt. I know of an instance where a malware came off of a vendor's laptop, got into a OT network and there's this little process that they were running, required a lot of heat. Well, when everything shut down rapidly, it wasn't properly cooled and this destroyed millions of dollars, millions and millions of dollars worth of equipment as well as took down that whole production line for a long time so that impact over business, as well. I think where the big concern that I have, from when you start coming about this let's move fast, break things kind of approach, is that if I move fast and break things and it's a printer, that means I don't print my document, I have to go to another printer in a corporate office. If I move fast and break things in ICS, in industrial control systems, or SCADA, or whatever that means I get somebody killed potentially and I think that's a big concern, it's a big distinguisher between the two.

- Pulling that thread a little bit, talking the Colonial Pipeline incident. If ransomware actors have the ability to take down a pipeline across the East Coast by just accessing the billing system on the corporate domain, they didn't even get to the OT environment. Is this problem gonna get worse, really, or is it gonna get better?

- It's gonna continue to get worse, that's the short answer. The long answer may require a little bit of a tinfoil hat but it's geopolitical in nature, is my opinion. Well, a lot of these big ransomware actors are operating in countries that protect them as long as you don't hit some resource inside of our country, right? So, therefore they can be pointed, directed. Again, my opinion, a little bit tinfoil hat, conspiracy theory but they can be pointed and directed at potential targets of interest. So, I think this is going to continue to get worse across the entire spectrum of IT. We have a lot of weaknesses and a lot of vulnerabilities and we've still got a lot of work to do to catch up. As everybody knows, we're being outspent by bad actors and by orders of magnitude. That whole issue is only exasperated whenever you get down with the OT world because, again, you're talking to upgrade a simple device on just a few hundred systems, computer operates on a modern OS and you can put in some more modern protections against it, that cost can get into tens of millions, hundreds of millions of dollars very rapidly. So I think this is gonna continue to get worse before it gets better. It's gonna require a lot of investment and a lot of time, and it's gonna require vendors to take a personal stake in this, to harden their own systems. And even once they do that, then you've still got to go back to that initial investment to upgrade those systems to the latest and greatest, which is gonna require a lot of time, money and effort.

- Let's go down that path in terms of talking about the vendors, we all know who the big vendors of ICS and SCADA generally are. And I've talked to a lot of other folks that say if you want to really attack SCADA, you don't attack the device you go attack the administrator on the IT side that has access into the device. I'm just curious from your perspective on that. But overall when we really talked about what are the basics for securing OT security, is it fundamentally different from securing a Windows corporate domain, for example? And we all know that that's firewalls, it takes end point detection response, that takes segmentation, that takes patch management, that takes application security, a long list of certainly things to cover there. Is it similar in that nature? And again, realistically back to the cause, are you really protecting the administrator or are you really protecting the device or is it a combination of both?

- It's a combination of both to a point. If you separate them out and you have all the entire stack that's required to run, we'll go with the quote unquote, modern OT systems. There's a lot of that's gonna be similar, maybe even identical, to what you need to do to properly secure a traditional IT environment. And this even gets into, especially when we start talking about, maybe more so, that segmentation of the network. What is proper segmentation? The zero trust networking concepts? And anybody who works in the OT sector that's familiar with the Purdue model which segments out the different layers from the machinery, where we're talking about pumps or switches or relays or generators or what have you, at the base level up to the next level being PLCs, the next level up being the HMI's data loggers. You start getting to get control systems that are running on your traditional Windows, Linux type machines and you go up to the rest of the levels, up till you get to that corporate environment. Because at the end of the day you've got to get that reporting out of that control system environment to corporate so you can bill your customers, or because everybody wants to see how well we're doing today. So there's a lot of that that gets into, if you build out a zero trust network and you have the security and you have the segregation between network nodes and you've got traditional Windows systems on there or Linux systems and you're able to run traditional antivirus end point protection stash. Or your in the next generation than what you'd traditionally be, those next generation endpoint protection products, the network monitoring products, et cetera, then you're good there. Multi-factor authentication in any of those Windows or Linux machines, absolutely. But at some point you also probably need to have a separate domain because you cannot patch the Windows and Linux machines that are talking to those PLCs, which are then controlling the machinery on the far end. Because if you do, again, any patching, any software pushes, any whatever, is gonna cause you some potential serious issues. You don't want the guys checking their email and surfing the internet on the same machines that they're using to, from an engineering perspective, to modify and update all of the control system portion of the network there, so you've got to do all that separation out. When it comes down to the PLCs, again, you look at a lot of IoT devices, there's limited options that you have for really securing the device itself. So you've got to go into that network layer of security, whether it's a purpose-built network security product along with the more traditional firewalling that's controlling port protocol, connectivity back and forth, because there's nothing you can do on that end point to protect it, and then you have to double up on your controls and your security on those HMI's and data loggers and historians, which are that initial layer of Windows or Linux machines.

- In terms of threat intelligence, where is threat intelligence potentially useful in that type of scenario where it has the appropriate context to be useful? I would probably think that there's only so many actors that have the capability to ultimately navigate to OT, what's the kind of context that you like to see? But from a private sector and the OT perspective, what do you like to see in threat intelligence that you can actually use to take action against? If that makes sense?

- Yeah, sure. So of course you have your standard issue IoC, IP addresses, all that sort of stuff that's constantly changing anyways but you set the direct feed into your firewalls, your IPS, your ideas share. All the different tools, including any of your monitoring systems that can go, oh, look, we see this thing trying to connect to IP address whatever. I think that at its most simplistic level is still very valuable in the ICS environment. Part of that reason is that once you set, ICS OT environments are really built around that availability concept. It really is a set it and forget it kind of build out. So if I can constantly be updating my network protections and the protections, the intelligence of the protections around that OT environment, then I really increase my potential of catching them before they get in. Again, you go with that whole Defense in Depth concept, catch them before they go in. I think one thing where we're lacking, although it is coming together with the ISACs and some of the other industry groups with the public private partnerships, is that near real time intelligence of what they're doing, how are they executing these attacks? If I can get the dissection of the attacks that occur, then working with the OT engineers, if we understand that look, they're using these type of functions to create havoc in an OT environment or to penetrate an OT environment, then we can work to, over time, modify the way the environment is set up enough to mitigate some of those issues. If you always have this function left open on this type of equipment, they can use that to kick off a run inside the system of some kind. Well, if we don't ever use that, then we can lock that down. We can secure that overall system a little bit better. And that goes along with the whole concept of assume breach, right? Assume that the bad guys are already in your network. If we can change it to where even if they're in the network, we add on layers of defense and we reduce the functions that they would normally use to attack these types of devices or impact the overall control systems, then that gives us a quick heads up. That's a lot of bang for a little buck. Otherwise it's gonna come down to providing as much intelligence as we can get to just try and do the block all concept which, as we all know, at the end of the day there's always gonna be another hole that they're gonna come in on. Block all's not gonna work for you.

- And also that there's just so much noise that you just don't have the time in the day to ultimately put rules in place for all those types of blocking techniques. Then that, I guess, gets to the next part of really. And I'm just using comparisons to the financial industry, so there's a lot of financial industry players that take a much more aggressive and proactive approach to help downstream, to help those smaller types of companies. Of course, there's certainly things that everybody can do better. Take MasterCard, right? We had MasterCard on the show about a couple of months ago and they are very aggressively going after, and they want to be known as one of the premier organizations in the industry that tackle digital crime. They want to own that space. Do you see the major oil and gas companies taking that same approach and having a level of security that they are then replicating downstream to smaller clients that ultimately are in the supply chain that could potentially wreak havoc? Or do you see that the major oil and gas companies need to step up and do a little bit more? Where's the maturity there from that perspective?

- So to begin with financial services, the financial services sector is light years ahead of the energy sector. They were some of the first ones attacked. They were some of the first ones to really use the big computing systems. They've done a lot of investment in security technologies and not only just buying them for in-house use but also, quite literally, investing in some of these startup groups. Energy, O and G particular is lagging. We've seen that big upswing in security in the electric sector, unfortunately this was done through compliance. And I say unfortunately, compliance is not a bad thing. Not anti-compliance. But the way it's set up the compliance then becomes all encompassing. It takes up so much time, energy and effort to prep for that next audit, that it does impact your ability to get security done. The oil and gas industry is not on a regulatory compliance-driven track at this time. There is some with the pipeline companies, with the refineries, but a lot of it is attempting to do this internally, depending on your customers, depending on where you're at, in that from upstream, midstream, downstream, where you're at in that whole path, that whole supply chain to get the finished product to the consumer. They're starting to layer on requirements, it's supply chain third-party risk type of requirements. This does present some challenges because a lot of times they don't necessarily understand the systems that are being used by the third party and they're trying to go with a one size fits all scale, which of course presents a lot of challenges. But it's that first step and they're trying to take it and they're trying to drive, you know, this is like your big integrated companies, like your Shell's and your Exxon's and those guys. They're trying to push anybody else's vendor through this process, it's got a lot of maturing to go but they are working down it. But I think, unlike your MasterCard's and your Visa's and some of these other guys who are really sponsoring and trying to drive that improvement to reduce those cybercrimes. To start with they're working on a traditional IT network. When you start getting to that OT stuff, it starts getting esoteric and there's a lot of concerns that we have to address from a safety and a stability standpoint. But I think what a big chunk of the problem is is that the majors, the big, huge companies are still, in a lot of ways, trying to get their house in order. Whereas your big financial companies, financial services companies, they've been working at this for a much longer period of time and their house is in pretty good order. Not perfect, but nothing ever is. I still think we've got a ways to go in the industry before the big majors can do more to help other companies. That's just where they're at from a maturity level. But there's another side to this and that is that inside, especially when we start talking about oil and gas, there's a very strong desire to be independent and do it ourselves, our way. It's a very big cultural thing in there. You know what? We will stand on our own feet and we're not gonna get into anybody else's business but we're gonna work with them, so that's gonna have to be overcome as well. Plus, depending on what's going on in the market conditions today with oil and gas. Oil and gas today is not doing well when you look at the overall market so without the money there, it won't happen anyway. So we have to get the maturity level to a certain point. We have to get over the cultural influences of bootstrapping yourself up and then the market's got to be in a place where they're financially capable of providing that support.

- And then, final sub-question on to that, do you think that the majors would even be successful without the help of the big suppliers? Do you think that they're even gonna be successful in combating these types of threats without their buy-in, as well?

- Holistically, no. I think they can make a big dent. There's a lot of things you can do because if you set up your overall global network properly, those OT environments are very isolated in a very containerized compartment of that overall network. There, of course, is gonna be connectivity and people who say, oh, we have a completely air gap network. I just like to, you know, I'll nod and shake my head, okay, if you say so. I've not ever seen one but I think you can do a lot of isolations, however, there's always still an attack vector in there someplace. To be at a very high level, like the financial services level of security. Microsoft is very interested in security right now, Dell, HP, all the major players that the financial services use are very interested in security and so, they're being more successful. I think OT does have a few buffers between them, hopefully, between them and the outside network but to get to that same level of success, the vendors have step up, have to get very driven about security and start being secure by design from the outset.

- Nate, we appreciate your amazing expertise here. Thank you for joining the show. For the latest subject matter expertise surrounding managed intelligence, please visit us at www.nisos.com. There we feature all the latest content from Nisos experts on solutions ranging from supply chain risk, adversary research and attribution, digital executive protection, merger and acquisition diligence, brand protection and disinformation, as well as cyber threat intelligence. A special thank you to all Nisos teammates who engage with our clients to conduct some of the world's most challenging security problems on the digital plane and conduct high-stakes security investigations. Without the value the team provides day in, day out this podcast would not be possible. Thank you for listening.