Could the US grid be brought down by Chinese hackers? To find out, I talked with Patrick Miller, who helped write the cybersecurity rules for the bulk power system and became the first person with federal authority to enforce them. We get into what's actually been hacked, why those "rogue devices" in Chinese inverters are less sinister than they sound, and why squirrels still do more damage than hackers.
As clean electrification proceeds, more and more of the US economy is going to be dependent on the electricity grid, and the electricity grid is going to be more and more dependent on power electronics, software, and computing.
If there’s one thing we all know about computers these days, it is that they can be hacked — accessed and controlled from the outside by bad actors.
Fears about the vulnerability of the electricity grid to hacking and manipulation have been steadily rising in recent years. Most of the power electronics — the inverters, transformers, and controllers — used to build the electricity grid come from China. If you listen to the doomsayers, they’ll tell you that China is embedding malware on all these devices, with the goal of creating some sort of apocalyptic kill switch that could take out the entire US grid at a stroke.
How much of that cybersecurity threat is real, and how much is hype?
I’ve been meaning to get to this subject for years and I’m excited that I have the perfect guest with me today. Patrick Miller helped write the original cybersecurity rules for the US bulk power grid—the NERC Critical Infrastructure Protection standards—in the 2000s. Then he became the first person in the country with delegated federal authority to enforce them, auditing utilities on the government’s behalf. Since then, he has founded a nonprofit, run a Department of Energy program, and built a consulting firm, all to advise and educate utilities and regulators worldwide on cybersecurity threats.
It’s an unusually broad range of experience and it has made him a prized voice in the field, a level head in an area filled with uncertainty and fear. I’m eager to talk to him about how cybersecurity threats might manifest in our increasingly distributed grid and how grid engineers and regulators should be managing those threats.
Timestamps
00:00 – Introduction
02:47 – What state utility commissioners get wrong about cyber risk
04:50 – Real attacks on the grid so far: Ukraine, Poland, and the US
06:57 – IT versus OT, and why grid devices are hard to protect
10:30 – The NERC CIP standards: scope, requirements, enforcement
17:40 – Distributed resources outside the CIP perimeter
20:54 – Dropping the threshold to 20 MVA, and federal jurisdiction
29:12 – Chinese inverters and the commodity board
36:09 – Volt Typhoon, Salt Typhoon, and China's intent
39:26 – Data centers as a new attack surface
43:19 – The trade-off between security and speed
47:44 – Cyber-informed engineering and analog safeguards
54:05 – AI on offense and defense
1:02:21 – Squirrels, balloons, and physical threats
1:04:24 – CISA cuts, CIRCIA, and harmonizing the rules
Resources
People & Organizations
Patrick Miller (LinkedIn - Personal Site)
Additional Orgs Mentioned:
Company & Industry News
NERC inverter-based resource registration initiative enters final stretch for asset owners
CISA tells critical organizations to prepare for cyber outages
Books & Articles Discussed
NARUC and DOE - Cybersecurity Baselines for Electric Distribution Systems and DER
NERC - Petition to Revise the Rules of Procedure for Inverter-Based Resource Registration
Patrick C. Miller - Written Testimony before the US-China Economic and Security Review Commission
CISA - Cyber Incident Reporting for Critical Infrastructure Act
Related Volts Episodes
Text Transcript
David Roberts: All right then. With no further ado, Patrick Miller, welcome to Volts. Thank you so much for coming.
Patrick Miller: You bet, David. Thank you so much for having me on the show.
David Roberts: Really excited for this. This is, something I’ve been circling around for a long time, as I said, and I’m excited to dig in. So you, among other things on your extensive resume, you run a program for NARUC, the National Association of Regulatory Utility Commissioners, a training that brings state utility commissioners up to speed on cybersecurity. I guess where I’d wanna start is when those commissioners come in, I suspect that they, like me, like I suspect most of my audience, are... don’t know much, and have all kinds of weird preconceptions. So I’m curious, sort of like, what’s the first thing you kinda tell them to get grounded, and what sort of misconceptions are they carrying around with them?
Patrick Miller: A great place to start. Yeah, I do a lot of work with NARUC to help the state commissioners. As you know, where the federal regulations don’t reach, it’s pretty much left to the states, and in a lot of cases, you can end up with like, you know, 50 different directions.
David Roberts: Yeah, we’re gonna get into that later.
Patrick Miller: Okay. So, typically where I start with them is they come from varied backgrounds. They’re almost always a legal background- or in some cases they may have some infrastructure. They may have been a-- I’ve seen CFOs and other, like even general counsel from various utilities, whether it’s, you know, telecom, water, gas, electric. You start with the fact that they probably don’t understand all the technical things, and the typical questions they come with, they usually sit on one side or the other of a pretty, you know, wide divide. They either think that, you know, everything’s fine, nothing is hackable, this is all a bunch of hype- or they think, oh my God, everything is hackable and China is here to eat your children. You know it’s just, you know. So it’s kinda one or the other. There’s a few that are in the middle, but it’s a slim few.
David Roberts: That’s funny. That’s funny. Well, hopefully we can find some, chart some course in between, in between those.
Patrick Miller: That’s my goal.
David Roberts: Yeah. Yeah. So before we get into some of the details, maybe just to level set, I kinda wanna know- Are we talking about something that might happen or something that has happened? Like, have there been notable cybersecurity attacks on grids that have had negative effects, or is this mostly something that people are worried could happen?
Patrick Miller: Well, there’s a bit of nuance in the answer. There have been attacks on grid companies. We do know that. There’s public evidence of that. Most of these have been situations where the attacker is broken into the system and hasn’t done any damage yet. They just pre-position themselves. They’re holding that access. So in theory, they would use it for something, you know, bad or nefarious later on. What we don’t have is, direct examples in the US where, you know, actors have broken in and actually directly caused damage. We do have evidence of this in Ukraine, more than once, so at least a couple of extra- uh, exercises there, and then one in Poland recently. And when I talked to my contacts in Poland, they say they’ve got a long list of things they just haven’t published yet. So there are some threat actors that are definitely breaking into power systems and definitely causing direct damage, direct harm, blackouts, that kind of thing, just not in the US.
David Roberts: Just curious, in Ukraine, is that, are those state actors? Is that Russia doing that or do we know?
Patrick Miller: Yeah. No, it’s definitely Russia. Yeah, and same with Poland.
David Roberts: So this is s- this is something that’s being used as a warf- a proxy for warfare kind of thing.
Patrick Miller: it is. It’s a proxy for warfare. It’s also a proxy for influence. So, you know, Poland is the major channel that Ukraine gets supplied through, so-
David Roberts: Hmm...
Patrick Miller: Poland’s also been under heavy attack from Russia as well, with basically the same system, same type of attacks.
David Roberts: Okay. So for the US, though, we’ve documented access, but we’ve not yet had anyone, like ‘cause a blackout, say, or something like that.
Patrick Miller: Right. Right.
David Roberts: Okay. So I thought I’d start with a basic distinction here. I think a lot of people, when they hear the word cybersecurity, they think about IT. They think about information technology, protecting passwords, protecting data, preventing identity theft, that kind of thing. But a key distinction here is that, is that cybersecurity in the way that you are dealing with it is mostly about OT, operational technology, physical processes, and those are different. That’s a different approach. So talk a little bit about that distinction and how it, how it’s meaningful for cybersecurity.
Patrick Miller: Sure. Sure. I mean, IT still plays a role. I mean, information technology is how any company runs these days. You pretty much can’t work without your email and your corporate systems and your data. You know, all your customers are in there, all of your billing is in there. In the OT world, in the operational technology world, it’s the actual interface to the physical world. So, like the sensors that take things like fan speed or oil viscosity or temperature. So this is like, you know, think of how much voltage is on the wire, for example. That’s sensed by a digital component now. And the OT is what directly touches the physical world. Like, it’s the last line. So you would click on a screen somewhere, like a picture of a breaker, you click on it, and it will open an actual breaker out in the field. And that travels from someone’s IT system, like a, you know, standard workstation. It’s a Windows machine with a program on it that runs a, kind of an operational window into what they’ve got in their field. So they’ve got transformers and breakers and all the things they use to operate their power system. They can operate those from, like, a control center, through this kind of IT/OT partnership versus send someone out in the field to actually go open that breaker, right? So that’s the automation that we get with this OT. But these systems are not like, um... The OT systems are not like, you know, Windows or Unix or a database or a web server. They’re literally purpose-built devices. They do one thing. They, like they open the breaker. They sense the voltage. They don’t have, like, a, an operating system. They’re not... You know, they’re basically just what’s called firmware and a configuration file. You just turn it on, and it reads a configuration, and it does its thing. So they’re very interesting. And, um-
David Roberts: It seems like on, it seems intuitively like that would be a little simpler and maybe a little easier to protect. Is that right or wrong?
Patrick Miller: it’s easier to hack as well, so, because there’s not a lot of... I mean, they weren’t really built... I mean, think about this. When we first built this out, they were never really connected to anything, right? They were just by themselves. So now that we’ve networked everything and we’ve made control centers that can control, you know, everything in the field, so you know, generation assets, transmission assets, distribution assets, with all of that connectivity, you end up with devices that were never really meant to be connected that have some kind of connection to them. So they weren’t designed, to be protected in the same way. So we have to come up with really unique and interesting, you know, ways to isolate them and limit access and that kind of thing because I mean, realistically, you know, to update them, you have to reboot them in almost all cases. If this thing is required for the grid to run, you don’t just reboot it because you get a blackout.
David Roberts: Right.
Patrick Miller: So you end up with, you know, you gotta wait for the right moment. So you have these, this kinda mishmash of various things. So our main protection is just to isolate the heck out of ‘em and architect it so that it’s very difficult to get to them unless you know what you’re doing.
David Roberts: Hmm. Okay, talk a little bit about these, we’re gonna start with the bulk transmission grid. We’ll get to the distribution stuff later, which is really my true interest. But let’s talk a little bit about the NERC CIP standards, the Critical Infrastructure Protection standards. So you were instrumental in developing these and then in enforcing them. They’re about the bulk transmission grid. Just give us a little texture of like what do they involve? Like what do they require of people? And they’re sort of unusual. I mean, the grid world, the electricity world is full of all kinds of standards that are mostly voluntary. These are not voluntary. They are mandatory and enforceable. So talk a little bit about what they consist in, what they ask people to do, and how they are enforced.
Patrick Miller: Sure. Let’s start with who is required to follow the law ‘cause everyone thinks, “Oh, it’s the whole US power grid.” There is no distribution in scope at all, and it’s only some generation and some transmission. So there’s a handful of exclusions for transmission, so it covers a pretty big, you know, area of the transmission system. Generation, if it breaks a certain threshold, which, you know, to give out some numbers, it’s 75 MVA aggregate behind a single point of interconnection, they have to register with NERC, and they’re required to follow the laws. There’s a lot of generators out there that aren’t 75-MVA, so, and a lot of the times they would even generate facilities to be 74.5 so they could get right under the threshold. But that’s changing. We’re taking that threshold down. It’s gonna go down to 20 MVA and connected to actual distribution environment. So that’s-- the threshold’s getting lower because a lot of those smaller generators have actually had an impact. So that’s who’s gotta participate. The rules they have to follow, it basically, they have to declare, which systems are critical, and these are systems that if for some reason they were, there was a problem, whether you needed it or it got hacked, if it would cause an impact to that system within 15 minutes, that’s a critical system. You gotta protect those. And protecting them means you gotta have security policies in place. You gotta background screen your people and train your people. You gotta restrict who has access to those systems, and you gotta remove their access when they leave. You have to put ‘em inside of a, what, it’s called an electronic perimeter. It’s a firewall, really, and most people understand what a firewall is. It’s kind of a way to protect the systems from other dangerous networks. Then you have to have a physical boundary, so you gotta lock the systems up. You can’t just, you know, let anybody have access, and track who has access to ‘em and-- ‘cause honestly, if I can get my hands on it, I can hack it. Then you do the systems themselves, so you secure each individual system with everything from, like, password security to anti-malware to logging and tracking and monitoring and then of course you have to have incident response plans. You have to test your plans. You have to have backup plans in case you need to restore a system and test those. And then you gotta res- you restrict their configuration, so if you wanna make a change to the systems, you gotta track ‘em all, approve ‘em all. And then there’s the information about the systems that if hackers got it would be really bad. You have to protect that information. And then there’s supply chain security, so you don’t wanna buy from, like, North Korea or China or-- So you gotta, like, “Well, do we really wanna buy from these people? We’re taking a big risk here.” The next piece is the, communication between control centers. There was a Chinese hack recently on-- They basically hacked all the different telecoms. So control center to control center, you have to encrypt that so if someone were to somehow get in the middle of that communication, they couldn’t see what’s happening.
David Roberts: Hmm.
Patrick Miller: And then the last piece is, what, there’s some physical security, additional stuff for, like, really big, sort of transmission substations. The last one that just got adopted is. The monitoring the traffic between those critical systems as they talk to each other so we can see what they’re saying to each other, and that’s the last piece. That pretty much covers the, what the requirements are for the body of the standards.
David Roberts: That seems like a pretty... That’s a lot. That seems like a lot.
Patrick Miller: It is a lot.
David Roberts: So these are like, so these are mostly large entities being asked to do this, right? Large power companies, transmission-
Patrick Miller: Even some smaller ones. I mean, if you have transmission, like if you break 100 kV, and you’ve got transmission, there’s a chance you’re in scope.
David Roberts: Hmmm
Patrick Miller: And if you’re a generator above 75, then you are in scope. So it actually covers a fair number, of the utilities. It’s just which systems, fall into what they call, you know, high, medium, and low ‘cause there’s some categorization in CIP. Most of the systems are in the low impact space.
David Roberts: Hmm.
Patrick Miller: So they don’t have to go through all of those things. They gotta go through some of those things, but if they’re medium or high, they gotta do all that.
David Roberts: This is, and this is a little bit of a subjective question, but like how big of a pain in the arse is that? I mean, if they’re building systems that are 74.5 MVA, this probably indicates that they’d rather not take all this on. Is this something that you can like hire a guy-
Patrick Miller: Yeah...
David Roberts: who will do this for you? Or is, I mean, like how elaborative a process is this? How much time does it take for them to do this?
Patrick Miller: It’s pretty elaborate and what they really expect is they really expect kind of a level of diligence and the reality is if you wanna participate in the, you know, grid critical infrastructure ride, you have to be this tall to ride the ride. And it’s, the expectation is high for a reason and you know, when I was, you know, auditing it and, you know, writing violations and enforcing, we took it very seriously in the fact that no, this is the North American power system. Right. You have to keep the lights on.
David Roberts: Right. Right. Right...
Patrick Miller: So yeah, you’ve gotta have a real program with real people doing real things, and you gotta take it seriously.
David Roberts: And presumably these fines are big enough to really, these are not just things that they could absorb and brush off.
Patrick Miller: The number that gets thrown around is $1 million per day per violation. That’s actually gone up with inflation. It’s over 1.8 million per day per violation. That’s never been seen though. So what they typically do is they’ll write the penalty, and if the utility is like, “Oh, okay, what if we put all this stuff in place and we fix these things and we put this money toward improvements, can we take that cost down?” Usually that’s how the settlements work out so that it’s less of a, you know, go spend a bunch of money on the penalty and then go spend a bunch of money on, you know, fixing it. They try to get them to incentivize the fixes instead. So it works out in a pretty good way where it drives the incentive to do better.
David Roberts: And who is this we that is doing the enforcing? Is this the federal government? Is it some department of the federal government? Who’s the enforcement arm?
Patrick Miller: It’s a branch. So, it, the authority comes out of FERC, the Federal Energy Regulatory Commission. They have basically delegated this down to NERC, which is the North American Electric Reliability Corporation. They’re now called the ERO or Electric Reliability Organization. So they do the enforcement stuff, for FERC. And there are six regions of NERC, so whatever region you’re in, your region audits you.
David Roberts: Hmm.
Patrick Miller: And if you’re in a bunch of regions or all regions, there are some com- uh, utilities that are big enough to actually cover all regions, you get audited. They kinda take turns, and one of them will lead, but the rest of them participate. So yeah, you get, you get audited by your region, and then that goes upstream to NERC and to FERC, and then the penalties come back down.
David Roberts: Okay, interesting. Okay, so here’s the, here’s the bit I’m really fascinated by, which is the CIP standards were designed for the bulk power system, but, you know as listeners of this podcast are very aware, all the action these days is out on distribution systems. All the stuff that we cover every day here, rooftop solar, batteries, EV chargers, virtual power plants, sit outside that sort of regulatory perimeter and are, and in states, as you say, beyond FERC jurisdiction. And so, you know, you get something like a virtual power plant, you get the size and scale of a power plant that might have come under the bulk, the bulk standards. But instead of a single point of, you know, regulation or access or physical whatever, a VPP consists of thousands of devices scattered all over thither and yon, which just, like, intuitively to me sounds like a cybersecurity nightmare. Like you have, your attack surface, as they say, is, has become, you know, exponentially larger. So I’m just curious, like, what are grid s- what are the s- what’s the cybersecurity community thinking about this? How do you reach these distribution grids? What sort of standards do they need, and who’s gonna enforce them? Like, how do you standardize an approach when you have 50 different regulatory bodies involved?
Patrick Miller: Yeah. This one has been a really interesting challenge. So you’ve got some examples where those smaller assets, whether they’re, you know, run by VPPs or in some markets like in Texas where it’s kind of a QSE market, and the-- what we now have just lumped them into a big term called inverter-based resources because it kinda captures all of them, under this umbrella term. So there have been inverter-based resource, we’ll say challenges, disturbances, you know, also known as blackouts, in Texas a few times as a result of these. So, they’re one of the good examples. There’s been some other disturbances and things in the California area. It’s caused some strain in some other ones. We’ve got a pretty good-
David Roberts: Now, are we talking about accidents, or are we talking about attacks?
Patrick Miller: No, these are just, like, grid physics problems.
David Roberts: Right. Right. Right
Patrick Miller: Yeah, so not attacks, let’s be clear. So what they did, just because those grid physics problems, manifested, they were like: Well, what if someone, like, caused this to happen with a hack?
David Roberts: Mm-hmm.
Patrick Miller: That would be even worse. So those two things converged, and basically NERC went out and did lots of studying and invited lots of different participants and stakeholders to the mix, and they came back with this inverter-based resource study. And as part of that, they took that threshold that we were talking about, that 75, MVA for generators down to, 20 MVA. So, and they-- It used to be 75 MVA connected to a 100 kV system, which is basically transmission.
David Roberts: Mm-hmm.
Patrick Miller: They took that down, and now it’s 20 MVA connected to a 60 kV system, which is distribution.
David Roberts: So give a, give us a system. What is 20-- Like, what, would that, would that include, like, a residential rooftop solar generator? Like, how big is that?
Patrick Miller: It wouldn’t include s- this is like a small scale, uh... Like if, like if a whole, a small neighborhood, for example, or a, it’s mostly commercial operations. It’s not likely gonna be rooftop. That it’s, it was mostly generated for people that maybe had some battery storage that they were feeding back in, for example, and they were on the market as a result of that. So it was trying to capture those that are big enough to matter, and they looked at all the way down to rooftop, like, “Where do we draw the line?” So they took it up to 20 MVA because they figured this is big enough in aggregate, if someone were able to control it could cause problems. So based on both the grid physics and the security aspects, they’ve drew this line for what’s called inverter-based resources. They now have to register. They have to do some things. They don’t have to do security things yet, but that’s likely gonna be coming in the future. But we’re starting down the path of roping them into those federal regulations so that there’s less of a state-by-state mix, because there, as you’ve s- probably covered, various states have their own rules about how things are done.
David Roberts: Yeah. How does that work? I mean, how does the f- how do the feds even have jurisdiction here? Doesn’t, isn’t that violating sort of the kind of state federal boundary that everybody cares about so much in this, in the grid?
Patrick Miller: Oh yeah. This has been, it’s been hotly contested. I mean, it’s, I mean, nothing short of a bar fight, really. But what the result of this was, and FERC has, pretty much put their foot down and agreed with it, is that in these situations, it’s affecting an interstate transmission system-
David Roberts: Mm...
Patrick Miller: as a result of their ability to influence it with those smaller resources. So the aggregate possibility has, they’ve -- And they’ve actually done a pretty good job of proving it with physics. It’s not just a speculation. They can, they can show their math.
David Roberts: Yeah. This is, that line has become more arbitrary, every day, just around the physics, just around the technology of everything, and I’ve been wondering. I’m sure this is a species of problem that is gonna pop up more and more, that line between what is and isn’t interstate, what is and isn’t, bulk, is fuzzy at best.
Patrick Miller: Yeah. Yeah, and we haven’t even talked about, like, VPPs that could control assets all over the country.
David Roberts: Yeah, right. I mean, a VPP that’s aggregating resources across multiple states. So you say right now it’s about physics. What would, I mean, I guess the sort of the nightmare scenario here is like you gain, you know, a hacker or whoever, you know, and I, and I again speaking here from a very low base of knowledge, but like a hacker gets access to one battery through an inverter and like via that creeps into the other batteries it’s connected to and creeps its way. Do you know what I mean? Like, it gets in a back door through one thing and then accesses the whole system through that thing, which makes me think then that like you have to ensure that every one of those batteries has protections. Like, is that the threat here?
Patrick Miller: Y- y- yes. To be short and sweet about it, yeah. Well, we did some work with, NARUC, as we mentioned earlier. We did what’s called the cybersecurity baselines for distribution and DER aggregators. Because those DER aggregators are a lot like a VPP in a lot of cases. They aggregate at a point much like a VPP does. So when you’ve got virtual power plants and distributed energy resource aggregation points, you have at your fingertips a very large amount of potential to affect- you know, regional problems, just because you could say, well, you know, if you did it across the whole board, yes, that would cause problems. But if you wanna focus in a certain area and cause a problem at the right time to cascade things, you could, you could do that. So we wrote some standards, and it was basically some very lightweight things. You know, like have passwords, don’t connect these things to the internet directly without a firewall. Really simple stuff, kind of basic things like, you know, if you’re gonna have remote access, put some multi-factor.
David Roberts: Mm.
Patrick Miller: Give them a one-time, you know, phone code to actually log in so that you can’t just like log in with default credentials.
David Roberts: Are there, are there any of the-- ‘cause one of the things that pricked my ear about the bulk, about the bulk standards is, that I hadn’t really occurred to me, ‘cause, you know, I was just thinking about kind of the technology angle of it, but the personnel angle is also obviously a thing? Are there any of those personnel-based standards gonna be applied down at the VPP level, the aggregator level?
Patrick Miller: So far it hasn’t gotten to that stage. And states can certainly go do this on their own, right? And that’s kinda what it was designed for. When we wrote these, what we call the baselines, we based them on, a DHS work called the Cyber Performance Goals, and it was just lightly tailored to fit the distribution environment and for DER aggregators. But it was designed so that states could use it as a model to actually go write their own regulation, and it would give them something to work from. A blank page is always the hardest place to start.
David Roberts: Yeah, right.
Patrick Miller: And the ones that wanna go further and do, you know, go their own route and do their own thing, they’re gonna do that anyway. But there’s a bunch of them that don’t really have anything to work with, so that was designed to give them something, and hopefully, honestly, get some degree of uniformity in the approach.
David Roberts: Yeah. So it’s like a floor, a floor then, a federal floor that states can build on if they want to.
Patrick Miller: At this point it’s just a suggestion. It’s a nice-to-have.
David Roberts: Interesting. And this is another just sort of intuition of mine, which is that once you’re down to, like, 20, that 20 MVA level, you know, you’re not talking about individual residences, but you are talking about some pretty relatively small operations. Like if I’m a, you know, I could have a warehouse, you know, with a bunch of solar panels on my roof and a stack of batteries, I could break that threshold.
Patrick Miller: Yes.
David Roberts: So you’re-- I just feel like there’s a limited amount you’re gonna be able to demand from like a mom and pop, like somebody who owns a warehouse. Like these people are not going to be... You’re never gonna get them to be cybersecurity experts. You’re never gonna get them to hire dedicated personnel or employees just to focus on this. Like I’m just wondering like how far can you get? I- d- don’t eventually you need to start building the safety into the devices themselves? Is that not sort of like where this needs to go eventually?
Patrick Miller: Yes, and we’re also trying to do that at the same time because the way the system is designed, as you’ve already well articulated, there’s not an easy way to do this. You can’t just make all the warehouses out there with rooftop solar and, you know, warehouse-level rooftop solar and batteries, which most of them kind of will break that threshold. You can’t make them become like cybersecure and follow these federal guidelines and get audited, for example.
David Roberts: I mean there’s too many of them
Patrick Miller: And there’s not enough people to do the audits, right? So, you know, on the reverse side of that. So what we’re also trying to do is to actually, you know, do this at the supply chain level. So I’ve testified to Congress on this and some of my other counterparts have. Dr. Stewart is another really smart one, Emma Stewart. She’s brilliant on this subject. Probably the, I would say the global expert on this really. We’ve taken a strong push to get the device-level security to a place where you don’t have to be a cybersecurity expert. And this is not an easy thing to do because we buy a lot of our equipment that, you know, not from the US. So imposing rules on China is a little challenging.
David Roberts: Yes. Yes. And I-- and also I would imagine that like for every barrier you put up to hackers or bad actors, you’re making some incremental trouble just for the people who are just trying to use the device. You know what I mean? Like one more little gateway for the, for normal everyday use of the device, which I’m sure like the people using the devices are gonna resist somewhat and the manufacturers are gonna resist somewhat. So this does seem like a really sticky wicket.
Patrick Miller: Yeah. It’s not an easy one, but, I mean, we have to try all fronts, and if we get an inch out of all the fronts, we actually can cover a larger amount of territory. So it just takes a, you know, takes a whole village of effort to try to make the needle move really.
David Roberts: Well, you’ve set me up for my next question, which is, as I think people probably know, most inverters, you know, we’re talking about inverter-based resources. Most inverters come from China right now. There was this big Reuters investigation in 2025 which said that inspectors found all kinds of little undocumented communication devices inside solar inverters and batteries, and I guess, like, t-two questions. A, what do you do about what I imagine are now hundreds of thousands, if not millions, of those Chinese inverters that are already installed somewhere and running? Like, you can’t yank them all out. You c- there’s no practical way to audit them all. What do you do about those? And then what do you do about the new ones? You c- as you say, we have no jurisdiction over China. Like, how do we exercise any control over what China puts in those inverters?
Patrick Miller: Yeah. I’m gonna keep this at a non-technical level ‘cause it gets really technical really fast. So in general, you’re absolutely right. There’s a, there’s millions of these things out there, and they do have a lot of undocumented communications components in them. But I wanna make sure we’re really understanding what that means without any hype whatsoever. When China manufacturers a lot of this equipment, they literally do it on what’s called a commodity board. So you can imagine in your mind like a motherboard, right? It’s a- you know, it’s a, it’s a computer component where the chips and the memory and all those things sit, and they’ll buy literally millions of these at once. Mm-hmm. And they’ll buy them, you know, from a commodity manufacturer that just makes, you know, boards, literally, like generic board. And on that board, it’ll have all the things just in case you need them, whether you use them or not. They don’t buy specific boards with just these components. So when they g- uh, I mean, 99% of the ones I’ve seen that they’ve gone out and found stuff, it’s just been the commodity board. Yes, it came with a radio, it came with a modem, it came with a place for a SIM card. There wasn’t a SIM card inserted, but it has all those components on there in case the buyer of these commodity boards wants to use that functionality.
David Roberts: I see. I see. So not nefarious inclusion of these, right?
Patrick Miller: Not necessarily. So whenever you see those, you kinda have to take that with a grain of salt and think, well if there’s millions of these things out there, did they just use commodity gear and they’re just not, you know this is just part of the board that maybe it’s used, maybe it’s not. Now, they have found some of these things phoning home and, you know, the routing traffic through China, so that’s a different discussion. That’s at the software level, you know, up where the applications that you’re using, to manage these devices and that kind of stuff. So those are, those are different components, and we, there’s some supply chain there that we can impose because it’s software, right? We can inspect code. We can enforce certain rules, on how code is developed and what’s, you know, certain, like, transparency of the code. We need to look at the code before we’re willing to allow it, these kinds of things. And we’re not just gonna let updates from China happen without someone taking a look at it before it goes in, those kind of things. Well, yeah, I mean, you- So there are some gates we can put in place...
David Roberts: Well, yeah, I mean, you say there might be these elements on the boards just because the boards are sort of generic and created for maybe lots of different purposes. Could those, could those components of the board that aren’t being used be activated later? Do you know what I mean? Sort of remotely activated. Like that’s, I guess, is the worry.
Patrick Miller: Yeah. They’re there, and they could be activated. But they’d have to have... I mean, you could activate them, but you’d also have to configure them. Like, it’s hard to explain, but when you, when, just because you turn something on doesn’t mean it’s ready to be used. Like even a phone, you can turn a phone on, but you have to give it a SIM card that’s got an activation code, that’s got, you know, all these things have to happen before it’ll actually work on a cell network. I mean, you could get a phone out of the box and turn it on and it just doesn’t do anything. So those pieces of gear are there. That doesn’t mean even if you did enable it, that it would automatically work. So it does, it ups the ante in terms of just like flipping a switch and getting this like rogue network of these things to go do stuff.
David Roberts: Right. So are we currently trying to impose standards on these imported inverters? Like, is that, is that something that’s happened, is gonna happen, is in the works?
Patrick Miller: Yeah. States like Texas and Florida have, like, banned certain Chinese gear. The feds have banned certain Chinese communications gear. What we’re trying to do, at least myself and some of my friends that are pushing this, we’re trying to get a place where we can actually just get transparency into what they’re selling. It doesn’t make sense to go rip and replace all this gear. They actually make quality stuff. I mean, we used to have... A long time ago, China made, you know, garbage, and we... It would break, and you’d have to go buy new stuff. They actually make some pretty, you know, fairly high-quality gear now.
David Roberts: Yeah, I’m familiar from the EV space that that story is no longer that story no longer applies.
Patrick Miller: Yeah. So but if, by and large, a lot of the stuff we have out there, it’s gonna work. It’s gonna work as designed. It’s got, you know, it’s gonna fit within, like, the engineering specs that it says it’ll do. So with that, is there a way to basically live on a diet of poison fruit? And that’s really kind of the approach we’re gonna have to take is what if we needed to, I don’t know, repurpose this, you know, so we can take out some of the Chinese com- you know, software components or firmware components and put our own on it? That’s an option we’ve got where we could just basically just say thank you for the equipment. We’re gonna repurpose it to our own needs and we’re gonna put our own software and firmware on it and run it that way. Or we can force anything sold wherever it comes from to just have certain degrees of transparency. We can, we can inspect it, and that way we can sample batches and say, okay, you know, X number of them to a high degree of certainty we think are not tainted with some sort of bad code. So there’s different avenues that we’re trying to push. None of them are, you know, like a switch. None of them are easy. None of them just, you know, roll out and make everything, make the problem go away.
David Roberts: Well, hasn’t the Trump administration discussed a, discussed a wholesale ban of these, which I assume is like—
Patrick Miller: absolutely a bad idea...
David Roberts: not a good idea.
Patrick Miller: Yeah, not a good idea, no. W-we, I mean, what we typically do now is when we buy this gear and we don’t have these assurances, we just isolate the heck out of it so that it just can’t talk to anything. Like it doesn’t get to talk to anything except through this very slim channel, and we inspect every single thing that it does, says. You over-monitor it so that just in case something did go wrong, you’d know. So you can’t prevent it from happening, but you can certainly detect it. I mean you can prevent a lot. You just can’t prevent everything. But what you can limit it down to, and then you just detect on that narrow band of what’s actually allowed, and it gets you a little bit more assurance that things are gonna be okay.
David Roberts: I guess I’d just ask flat out, like, do we have reason to think that China wants to or is trying to do anything nefarious with these inverters? Like, just flat out. Like, is there-- do we have evidence, not just sort of vague suspicions, but do we have evidence that they are-- want to do or trying to get away with something?
Patrick Miller: Uh yeah, short answer is yes. There’s a campaign called Volt Typhoon. And the one I mentioned earlier where they broke into the communications network is called Salt Typhoon, so this is the Typhoon series of actors. Volt Typhoon was, um... And it’s still ongoing, and it’s not like it was, you know, like one group and then we, you know, sent them home and they went away. It’s an active campaign that is still ongoing, and they are trying to f- And they’ve-- we’ve actually found them, and there’s documented evidence of this, it’s public record, where we found them embedded in utilities and various different systems, and they ha- Like I said, they haven’t done anything bad. What they do is they don’t break in and come in the front door with, like, a marching band behind them. They don’t announce their presence. They don’t do-- They don’t drop ransomware. They sneak, they come in very quietly, and they immediately hide. And they don’t even drop, like, malware, so they’re not using-- Once they’re inside, they’re not using malware. They’re using the tools that already exist, so we call it living off the land. They’re very good at this, and it’s extremely hard to detect, but we’ve found a few cases where we can see what they’ve done. It shocked everybody because we were quite surprised at how good they were first, but also how deep they’d gotten embedded. But yeah, so there’s documented evidence that they’re actively trying to do this. They have done it. They’re still trying.
David Roberts: But do we know why? Like, what are they up to? What do they envision doing with that access?
Patrick Miller: They haven’t openly communicated this to us, so we-
David Roberts: I would, I would imagine...
Patrick Miller: all we can say is our best guess is ostensibly it is to exert influence over us, whether there is a particular trade situation, whether there is a particular Taiwan situation, you know, those kinds of things. So it is, they’ve got us in a very tight spot, if we ever, you know, decide to take action against them in some way where they needed to leverage that.
David Roberts: But it does seem like if they came out and said, “Hey, guess what? We have access to a bunch of your energy resources, and we’re gonna shut them down if you don’t do XYZ,” that would amount to something like a declaration of warfare. Like, that would cause the US to freak the hell out, I would imagine, and do all kinds of crazy things. That would not be a small step to take. It’s not like a trade, you know, that would not be... That would be more than a trade dispute.
Patrick Miller: Yeah. One would think. So far we’ve found them, like, in our communications n- like literally almost every single communications backbone, all the major telecom providers, and they, like, they admitted it. They didn’t just, like, shy away from it and act like they didn’t do it. No, they openly admitted it, and we didn’t do an act of war then. So I mean, the, so the, you would think the answer to that question is yes, but maybe is probably the response. I’m not really sure.
David Roberts: Wild. Wild. I guess it would look very different if they actually did something, right? I mean, the fact that they haven’t done anything sort of, I guess, lets all this kind of simmer in the background. Let’s talk about data centers. This is something I did a, I did a, an episode recently, you know, NERC, the Reliability Corporation put out some alerts about data centers. Just because the way data centers are built, they’re very sensitive to perturbations, they’re very sensitive to grid conditions, and they are prone to just switching off and dropping off the grid all of a sudden, which if you’re a, you know, gigawatt data center, you know, gigawatt of demand disappearing all of a sudden can destabilize the grid quite badly. And so that was, those worries were mainly about just the normal operation of these things, like the way they operate is somewhat of a threat, somewhat of a, you know, potential threat to the grid. But then it occurs to me as I’m researching this, like, what if you caused that to happen on purpose? Like, what, you know, what if you got access to those systems and made that happen on purpose? You could destabilize grids via data centers. So is the profusion of large data centers, is this yet another sort of attack surface, and is anyone doing anything about it?
Patrick Miller: Yes. That, what’s called a Level 3 NERC alert, for those that wanna look it up, it was for what’s called the computational load entity. And we’ve looked at these things, and this has been an issue in some areas. Texas is another example again. They have a lot of Bitcoin farms, and they would basically automatically shut off, so they would just absolutely draw enormous amounts of power until, you know, Bitcoin prices or whatever coin they were mining, whatever thing they’re doing, whatever crypto situation they’re in, would hit a certain price threshold, and they would just drop, and it would just swing their grid hard. So they’ve put some rules in place there. And then we saw some other situations that, I mean, recently was the one in PJM. A data center dropped 3 gigawatts-
David Roberts: Good Lord...
Patrick Miller: of power at once and swung. They can typically stabilize these things. I mean, most disturbances, we can stabilize them within seconds to m- you know, a few minutes at best. This took 10 minutes to stabilize, which is a very long time in power curves.
David Roberts: Yes, this is unprecedented. This is not something that has happened in history. This is a new th- a new thing in the world.
Patrick Miller: Yeah. So when we look at these things, as I mentioned, the IBRs, we look at it not just from a-- We start with the grid physics components, and we look at this from an all hazards perspective. So cyber threat, backhoe, whatever the threat is, if that system drops or comes back online and starts drawing and we didn’t expect it- Um, those things can cause grid, you know, problems. So we take a look at this from an all-hazards perspective, and cyber is one of those hazards, and we’re looking right now at how do these computational load entities affect the grid, and what do we need to do to figure out how to balance this? ‘Cause typically you have to have some sort of reserve capability or ride-through capability. Other, like, grid physics components, whether it’s, like, static VAR compensators, and they’re larger grid components that can absorb these swings, and, you know, kinda help us ride it out without, like, lopping off a big area just because you’re having a disturbance.
David Roberts: Right. Didn’t Texas just pass some ride-throughs standards re-re-just recently?
Patrick Miller: Yeah, they did. And we’re looking at a bunch of ride-through stuff for the NERC standards as well.
David Roberts: So those would also guard against a deliberate malicious taking those things off the, off the grid, like-
Patrick Miller: Yeah. Intentionally by design. Like I say, when we’re-- I’ve been on the standards writing process with NERC for 20-something years now. We look at it from, we call it all hazards, whether it’s a cyber threat, whether it’s a squirrel, whether it’s a backhoe... you name it. Or earthquake, flood. What happens to the grid physics, and how do we actually stop that from being a problem?
David Roberts: Interesting. Interesting. Here’s sort of like a broader question. One of the other things I was worried about as I sort of approached this whole subject, and I contemplate the sort of, as I said, the sort of incredibly distributed attack surface we are developing on the grid- which is that there’s got to be, just intuitively, there’s got to be some trade-off here between sort of security and speed. Like we, you know, we want to decarbonize the grid quickly. We want to electro-we want to electrify quickly. There’s a great deal of urgency behind doing that as quickly as possible, and I just think intuitively, like the more standards you put in place, the more requirements you put in place, the more personnel are involved, the more audits are involved, et cetera, et cetera, that is inevitably going to slow things down somewhat. How do you think about that trade-off?
Patrick Miller: I also, speaking as a recovering regulator, I’m a big fan of regulating only absolutely the bare minimum that you have to regulate- and then let other forces operate above that threshold. So you will see things like, business agreements. If you’re an unreliable partner, you’re probably not gonna get a lot of business. If you’re trying to grow, a bunch of, you know, build a bunch of assets and grow a bunch of business, your insurance company is not gonna wanna do business with you, and you’re not gonna get insurance for your plans if you can’t show that you’ve got some degree of responsibility in the mix. So there’s other forces that push the, these components around a bit, too.
David Roberts: So just sunlight, just exposing who’s doing what you think is, does some of the work?
Patrick Miller: Well, it’s gonna be hard to hide ‘cause you know, if a disturbance happens, we typically know where it came from. Um- uh, like in the Spain blackout, for example, one of the bigger problems they had, they were able to come to a fairly rock solid conclusion, but there were still little gaps here and there because some entities just said, “No, we’re not gonna give you our data,” and some entities just said, “We don’t have the data. We don’t log it.” But in the US, you’re kinda required to keep a lot of data and, you know, we use a lot of data on our, just our operations, and we even, like, resell a lot of our data to third parties and that kind of thing for being a data broker, an operational data broker. So we have a lot of data. If something happens, we’re gonna be able to reconstruct that, and honestly, if a disturbance or a blackout happens, FERC is gonna show up, and it’s not gonna matter. You’re gonna have to prove it and show what you did, and you’re gonna get, you know, you’re gonna get that black eye or you’re gonna get the stigma, for operating, you know, rogue, or, you know, irresponsibly. So there are some things that will help. Are they perfect solutions? No. Does it prevent it from happening? No. But every time we’ve tried to prevent a problem through regulation before it happens, there’s a lot of rightful pushback that says, “Prove it. Show me that this is a real problem.” So until we have, unfortunately, it’s a, it’s always a rear view mirror approach.
David Roberts: Yeah. And when you say you’re gonna find who’s responsible, are we talking about- The entity running the power plant, the manufacturer who sent the equipment, the software company that wrote the software or some of, some of each? Like who’s typically, who are you gonna find that was at fault if you, if something goes wrong?
Patrick Miller: The answer to that is yes. And for different reasons, right? Like if you caused a problem where my insurance company is trying to prove that it wasn’t us, but it was somebody else, they’re likely gonna send in some forensics teams to go figure out what happened. And if there’s software liability or hardware liability, they’re gonna chase that liability chain as far as they possibly can because, you know, hell hath no fury like an insurance company that’s been scorned. So you know, that’ll happen. FERC will come in. They’ll wanna do typically like a 1B investigation, especially if it causes a blackout, they send in a blackout investigation team, and those are some solid engineers that go do that work. They’re good. It’s like an NTSB investigation. So, you know, from all different angles, there will be sunlight on that event.
David Roberts: Mm. Okay. Well, this again sets me up for my next question really well, which is I think one of the sort of like disaster movie scenarios that people have in their head is that you get access in one little spot, and then, and then there’s some sort of cascade and, you know, next thing you know, the whole East Coast is out or the whole country’s out or whatever. And so, I know a lot of your work and a lot of focus has gone into thinking about just how would you contain an event if it happened. Like how do you... And you call this cyber informed engineering.
Patrick Miller: Engineering. Yeah.
David Roberts: So just that, that’s like what, how do you build your devices and your systems so that things don’t cascade, basically, I think is the simple way to put it. So talk a little bit about what that means. Who’s responsible for that, and what does it look like?
Patrick Miller: Sure. This is putting in some, we’ll call them analog safeguards. So we put in some things that just aren’t digital. They’re just literally an electromechanical thing, and we used to have, like, electromechanical relays and protection systems, and we’re looking at putting those things back in.
David Roberts: Interesting.
Patrick Miller: Now, they don’t, they don’t operate with the same level of, like, precision that we can do with digital stuff, like a PMU or a phasor measurement unit, unbelievable precision in terms of what it can protect. An old electromechanical one, eh, not nearly close to that. But, you know, it sure does come in handy if someone hacked the system, right? So-
David Roberts: If that’s all you’ve got, it’s better than nothing, I would imagine.
Patrick Miller: Right. So we’re looking at those kind of things, and a lot of other just kind of really physics-based behaviors. So, like, one of the best examples, I go back to, like, a water system ‘cause it’s easy to visualize, where electrons just aren’t as easy. So, like, if the concern is, too much chlorine getting dumped into the water, because someone hacked the system and said, you know, “Put all the chlorine in right now,” what we’ve done is let’s make the pipe that delivers the chlorine super tiny. So even if you set the, you know, 5 billion parts per million, it just couldn’t do it. Let’s make the pump a little tiny pump that can only deliver so much at once. Let’s make the battery supply for that pump only operate at a certain amount, right? So the pump just can’t over, you know, can’t over pump for example, and the reservoir that feeds it, let’s make it really super small. So there’s ways to do this, you know. Now correlate that to the grid physics size, side. There’s ways to put in these analog gates and checkpoints where, you know, key areas are key, whether it’s, you know, generation resources or large transmission through, throughput. We can do this on that scale and just... And it’s relatively inexpensive to put these things in, and it gives us a lot, kind of a safety underpinning. So think of, like, a float valve that operates based on pressure versus a digital sensor. So we’ve got-- We’re looking at those approaches. But, you know, you have to remember, we don’t have one grid. We’ve got four.
David Roberts: Yes. Yes, and each distribution grid, in some sense, is its own little thing.
Patrick Miller: Is its own, right? So, I mean, we got the HVDC, A- sorry, AC/DC ties that break up all the grids, so you couldn’t just black out the whole country at once or all of North America at once. That’s just not a reality despite movies. And then the, I mean, when I get the conversation around could they just hack all the things, and I’m like, if you really went out and did, like, an asset inventory of just all the things that are out there on the grid right now, even the really important ones, and you just got those, the number of different devices would absolutely blow your mind. Like, it’s an incomprehensible number. Their ability to hack that diversity of systems is just, it’s impossible. Well, it’s astronomically improbable. I won’t call it impossible. Even with AI, it’s just improbable. So could they do key points and cause regional problems? Yeah, that, that’s a concern, but nationwide or continent-scale blackouts are just not a realistic thing.
David Roberts: And so when you talk about these physical, these analog sort of gate limiters and type things, is that something that’s a manufacturer responsibility? Is that at the device level, or is that at the system level? Is the utility the one doing that? Who’s overseeing that?
Patrick Miller: Yeah, this is typically the utility. Yeah, this is typically the utility that puts this in so that they can basically protect parts of their system that are very difficult to replace. Because reality, one of the bigger concerns we’ve got is, I’ll keep it simple, the, what are called protection systems relays, they’re designed to keep the grid from burning down, basically. Like, no uncertain terms, like literally burning down. So, and this is keep transformers from exploding, all this kind of stuff. If we can, if we can put those re- you know, like a, like a, for example, an electromechanical or dumb relay- on some of these really expensive transformers, another really expensive piece of equipment, even if they did hack it, this piece of equipment’s gonna keep it protected. So those are the kind of things that we’re looking at. Really big generators, for example, obviously you could, there’s ways to desync and resync a really big generator, and you can actually torque the shaft and, there’s even some that have modeled like actually getting the generator to go boom. That’s a problem. Yeah. So you put these kinds of pla- these, devices placed in the right spot there to keep the generator safe as well. So this is to keep our big components that take a long lead time to build, that are really difficult to replace or fix, we’re protecting those critical components with this kind of protection.
David Roberts: Got it. So even if someone got in, caused some event, we have some reasonable confidence that it would remain reasonably localized.
Patrick Miller: Yes. I mean, now, like I say, is there still a chance for problems? Sure. And is every big utility doing this for their critical stuff? Eh, some are some aren’t. So, you know, it- there’s no rule to do this, no requirement to do this yet, but it’s those that I think most of them are doing it out of the financial incentive to if replacing that transformer is gonna cost us, you know, how many hundreds of millions of dollars and a minimum of a two- to three-year lead time- and now we’re up against, you know, Google and Facebook and Amazon supply chain... who are buying the equipment out in front of us, so we can’t even get in the front of the line if we wanted to. So yeah, they’re-- I think most of them are realizing this is just a good business decision, too.
David Roberts: Right. Well, once again, you’ve teed me up for my next question, which is about AI, and of course, when everybody th- when anybody thinks about computing these days, that comes up, which is, which is that, you know, maybe the number and diversity, of devices would be daunting for any sort of human hacker, but of course, now we’ve got these giant super intelligent robots who can just go at it without pause. So in your mind, does AI heighten the threat here? And then conversely, on the other side, are, is anyone putting AI to the task of building better defenses? What is the current role of AI in all this?
Patrick Miller: That’s another podcast in and of itself really.
David Roberts: I bet.
I’ll keep it to the two key things. AI is really good at finding vulnerabilities. So we are concerned about its ability to find vulnerabilities in key equipment that we’re concerned about. So yes, it will definitely accelerate that arms race. At the same time, we’re using AI to patch those vulnerabilities. So the manufacturers can, well, they’ll see there’s a problem, and they’ll write a patch for it, and we can... The challenge is this is, like, grid equipment, so we may have to do things like, “Okay, general public, are you okay if we have, like, a blackout window and we schedule it? Because we gotta patch stuff.” Because most of the time you can’t just, like, take these systems offline.
David Roberts: Yeah, like a operating system update.
Patrick Miller: Yeah you know. Yeah, gotta reboot.
David Roberts: It’s, a it’s enough of a pain in the butt on your, on your personal computer, but if the whole, if the whole grid has to do it.
Patrick Miller: Yeah. If you gotta reboot a section of the grid, that’s a problem. So we’re looking at ways to try to work this into operations. Some of it’s gonna be redundancy or, you know, moving the grid, the power over to a certain line while you know, restart those systems and try to get, build this into our operations and our behavior practice, ‘cause we haven’t done this yet, right? We gotta figure out how to make this work. So we’re in the process of that as an industry, and obviously the manufacturers have definitely, the light bulb has gone off, and they’re reacting, and they’re using AI to write patches for the vulnerabilities that the bad guys are finding. Now, the next piece is, one of the bigger concerns that is not, we don’t have an answer for yet. It’s the ability to model the grid was pretty difficult in the past. It took a lot of computing horsepower. AI is actually really good at this, and you can model large sections of grid, if not the whole grid. You can overlay other infrastructures like transportation and communications and gas and-
David Roberts: Mm.
Patrick Miller: the ability to cause a multi-infrastructure cascade by key component attacks in just the right number and frequency of basically like the sequence of infrastructure is to cause a larger problem. So we’re worried about that. It’s being discussed. It hasn’t really... Like we’re trying to model it ourselves and figure out, oh, you know, if they’re doing it, we should probably be looking at these same kind of key weaknesses and how do you get these cascading interdependencies down, this kind of stuff. So the thing that started is DHS CISA, C-I-S-A, put out what’s called CI Fortify.
David Roberts: This is the, Cybersecurity and Infrastructure Security Agency. I wrote that down so that I wouldn’t have to remember.
Patrick Miller: I love how it’s got security in its name twice. That’s really secure. Yeah. But they put out one called CI Fortify. The goal is at least if there’s a problem to get these, and it’s not a requirement yet, this is still a suggestion, but to have these environments disconnect. Just what does it take for you to island yourself off and we, you know, we, in the business we call it turtle mode. How do you how do you like clamp down the shell until the danger passes and you peek open and you see that things are okay and then you can, you know, pop your arms and legs out and keep going. So it’s trying to get, owners and operators of all the different infrastructures to go do an exercise on what’s a day without connectivity look like, what’s a week look like. Can you, can you lock down in turtle mode and still operate and keep essential services running even if not everything is working but enough is? So we’re at least trying some approaches.
David Roberts: So these are like water systems, transportation systems-
Patrick Miller: Gas, electric...
David Roberts: whatever, all the, all the d- gas systems. Man, that’s a whole, that’s a whole new Pandora’s box right there.
Patrick Miller: Yeah.
David Roberts: Trying to get multiple infrastructures to coordinate their behaviors just seems like it’s a whole new layer.
Patrick Miller: Well, if they can even do it on their own, great. Well, let’s just start with them, figure this out, what it looks like in your own little environment. And then we’ll look at what it does when, in terms of the interdependencies, that cascade out. But, yeah, I mean, think of, like, the dependence of electricity on the gas, right? We have a lot of gas generators, especially now for AI. There’s almost all of them are gonna be gas generators. So, it’s amazing to try to think about this and wrap your head around it. But like I say, that, that’s something we are at least discussing. We’re starting with, you know, at the asset owner level, could you do this? Could you operate? And then we’ll figure out what is-- Well, now, when you’re doing it, what are the interconnectivity, what’s the interdependency issues? And then we’ll take, kinda take it out from there. But, it’s on our minds.
David Roberts: It’s interesting. It’s hard not to, you know... It’s hard to hear about this stuff and not get a little sci-fi dystopian about it, about a future where sort of like our massive super intelligent AIs are just battling their massive super intelligent AIs. And at some point, like, it’s gonna be hard for any human to even really know what’s going on anymore, to wrap their head around any of it. You know? Like, eventually it’s just gonna be the robots the robots battling over our infrastructure.
Patrick Miller: That’s why we’re trying to put in a lot of that cyber informed engineering, underpinning so that, you know, even if the cyber causes problems, physics is still physics, you know?
David Roberts: A couple of final, couple of final questions. One, that I’m just curious about. One is, and maybe this is naive to even ask it, but like, are we trying to sneak-you know, are we trying to sneak stuff into China’s systems too? I mean, I kind of assume we are.
Patrick Miller: I mean, if we’re not, then shame on us. I mean, and I really think those are kinda table stakes for nation states at this point, so.
David Roberts: Geez.
Patrick Miller: And we’re remarkably capable at what we do. I mean, we’re arguably the best in the world at cyber anything, so one would have to believe that that’s a reality. Yeah.
David Roberts: It’s just gonna seem like the, all the, all these forces are pushing toward, you know, kind of reversing the trend of the last century almost towards greater trade and greater independence or interdependence. Like, it seems like this is gonna wanna cause countries to want to domestically manufacture everything so that they can keep an eye on everything. You know what I mean?
Patrick Miller: Yeah. I think there’s, there is a push for that right now. There’s a lot of balkanization of manufacturing, of you know, sovereignty of various things. What we’ve seen in most cases that actually solves the problem, and I’m working on various different channels. I’ve got operations in Europe as well. It’s about, it’s transparency so that there are open frameworks, and everyone operates against the framework, and everyone can see what’s in there, and transparency. Think about what happened with encryption. There used to be, like, you know, export rules on certain types of encryption. Well, now we just make it so that it’s you know, it’s open. Everyone can see it, and it gets all the transparency that’s needed, and it still works. So there are ways to do this where you have transparency into how it’s done, what’s being done, and everyone can agree upon kinda what the standards are. And then once... And there’s, you know, IEEE. There’s IEC. There are bodies that do this already and do it very well. So yeah, I think it’s gonna... We’ll have a push where everyone goes into their little hole for a while and realizes-... “Okay, we can’t do this. We just can’t s- we just can’t survive like this.” So the pendulum will swing, and then it will have to swing back, and all of us that are pushing the pendulum back the other direction, we’re all going, “Look, you know the solution to this is transparency, so let’s just start there now.”
David Roberts: Interesting. Another question is, when you read, or at least when I’ve read about attacks on the grid that actually do damage, it’s squirrels or, like, rednecks with rifles shooting- shooting at transformers. Like, how do you, how do you rate the physical threats to the grid relative to the cybersecurity stuff?
Patrick Miller: Orders of magnitude worse than the cyber. Orders of magnitude, without question. Yeah, absolutely. It’s been, it’s largely squirrels, raccoons, snakes-... uh, scorpions. I’ve seen a whole bunch of different things. Mylar balloons, believe it or not, are actually a bigger threat than cyber. Um- You know, so, you know.
David Roberts: Has anybody weaponized Mylar balloons yet?
Patrick Miller: No. But there was, there was, like, a Twitter account called, like, Mylar Squirrel, and it would track all the, attacks on, well, just things that happened to the grid from, like, squirrels to balloons to raccoons to... Yeah, I think it’s, it stopped, but, it’s far bigger issue. I mean, and it causes a lot more damage, than cyber.
David Roberts: And you think the vulnerability is greater? Like, I mean, that’s what’s-
Patrick Miller: From the physical standpoint? Oh, absolutely, yeah. I mean, you can physically dam- Well, that’s what’s I mentioned earlier, the physical protections for big transmission in the CIP standards, there’s a specific standard for that. Like, you actually have to protect the transformers from gunshots. Like, you have to put up ballistic protection for the transformers. Yeah, because they’re, I mean, that’s, it’s a bigger threat.
David Roberts: Yeah. I wonder, that’s another one that just seems like as the grid distributes, as, you know, everything moves outward, again, you’re just like, you’re not gonna get every warehouse to hire armed guards and you know, put up, put up ballistic fences. Like again, your just, your attack surface is so huge.
Patrick Miller: It is. One of the good things about that is if you distribute the attack surface to a much, you know, more diffuse, set of assets, you actually take the threat away in a lot of cases ‘cause they physically can’t get everywhere.
David Roberts: Right. Right. Any single attack will be, it’ll have smaller consequences, I guess. Okay, so let’s wrap up with this question. Right now the aforementioned CISA, the Cyber Security and Infrastructure Security Agency, is getting cut much like much of the rest of the federal government is getting slashed its budget and staff. I’m just curious your sort of overall assessment. You’ve written an essay actually saying that critical infrastructure cyber regulation is headed in the wrong direction. I’m sort of curious your overall assessment, like are we doing this correctly and what is, if not, sort of broadly speaking, what is the course correction you would like to see?
Patrick Miller: Oof.
David Roberts: Again, a whole other pod if we- if we fell into that.
Patrick Miller: That is a whole other pod. I’ll keep it short. I do lament the cuts to CISA. It is-- American infrastructure needs this desperately. We’ve got a mix of things. The regulators are all different. We need something that is a centralized and a harmonized effort that helps all of them, and most of what they’ve done so far has been fantastic outreach. They’ve done some really good information products that have helped a lot of people. They raised the floor for a lot of those- those asset owners that you mentioned that just don’t-- They can’t whole-- They can’t hire a security team.
David Roberts: Mm-hmm. Right.
Patrick Miller: But they can take this template and apply it- because it’s free from CISA, and CISA did the work for you, and you don’t have to wonder, “Is this gonna work?” You’ve got an authoritary, authoritative source to get good stuff that helps you and raises the floor. So I would love to see more funding in CISA. I would love to see more activity from CISA. I don’t want them to become a regulatory agency, but I love what they do with, you know, their known exploited vulnerabilities list, which this is a different thing. When you hear of a vulnerability that happens like, you know, Microsoft says, “Oh, we have a problem. Go do something,” what CISA does is if no one is exploiting this, okay, probably lower priority. But the minute someone’s actually, like attackers are actually using this, it goes on this special list that CISA maintains. That gives us the priority and ‘cause when you’ve got 100 vulnerabilities to deal with- well, which ones matter? This tells you. So that kind of work is absolutely useful, and they need to be doing more of that. Now, the follow on of kinda where should we go and what would-- if I had one rule that I could change, and I get this question as a former regulator and someone who writes a lot of regulation all around the world at this point. I like the CIRCIA. It is C-I-R-C-I-A. It’s an incident response reporting act that was put out. It’s gotten a lot of negative attention ‘cause people don’t wanna report their incidents, and I get it. No one wants to air their dirty laundry.
David Roberts: Right. Right. Yeah, I guess it’s embarrassing and it-it-
Patrick Miller: It’s embarrassing, right? And in some cases, you gotta put it on your 10-K right now, and you gotta report to the SEC. But people get around that, and their lawyers step in and they, you know, they water it down. My issue with this is we don’t know what is working. We don’t know what the attackers are doing. We don’t have any actuarial data without this kind of reporting. We need something there. And I’m not saying like, you know, make them divulge everything, but we should have a at least some data that we get that’s useful, like a handful of things. Because the corollary is if, you know, we know that if I, you know, eat bacon cheeseburgers every day and I smoke and I never exercise and I never move, you know, Patrick’s likely to die at age 45 of these conditions because we have, you know, hundreds of years of actuarial health data. We don’t have anything like-- We’re basically guessing. So without this what is happening, what’s working, we don’t know where to put our money and our protections. I think this would actually really benefit us. If we could do one thing, we need to see what’s actually happening.
David Roberts: And is that a federal-
Patrick Miller: Because it helps us make regulations. It would be-- It’s a federal law.
David Roberts: Is that a federal rule?
Patrick Miller: Yeah, it’s a, it’s a federal rule for critical infrastructures and there’s, you know, a handful of them, that get roped in first, that kind of thing. But we don’t... Right now it could happen and, you know, if it wasn’t a big enough issue, they could just hide it and which, you know, I get companies may wanna do that, but the rest of the industry and the rest of the populace, we need this data to know where we’re, where, what’s working and what’s not. It helps regulators, it helps insurance, it just helps everybody all around.
David Roberts: And when you say that cyber regulation is headed in the wrong direction, what do you think we’re... What do you mean by that? What do you think we’re doing wrong?
Patrick Miller: I think we are, we need to harmonize a lot of our regulations. Like, each individual infrastructure has their own, you know, authoritative body and they’re all going in different directions. They’ve all got different penalty structures. They’ve all got- If you are a company that owns a gas infrastructure and an electric infrastructure or an electric and a water infrastructure, you got different, an entirely different compliance organizations and different experts. The software companies have to write different versions of the software. I mean, we could standardize a lot of this stuff because realistically, I mean, this is all the same stuff. It’s flow control, whether it’s gas, electric, water, transportation, it’s all flow control. So we’re kinda doing the same things. The technologies are all roughly the same. We could make this a lot easier for a lot more organizations, and then we could even introduce easier guidance. So we wouldn’t have guidance for just, like, the gas operators or the electric operators. We’d have guidance. So we need to get to a place where we’re not doing this in such weird silos. And I get there are differences, don’t get me wrong, but there’s a core component of this outside of those, you know, fairly small number of differences that could be done in a uniform way with a lot less wasted time and effort and money.
David Roberts: Interesting. All right. Well, this has been so fascinating. Thank you so much. Thank you so much, Patrick. I feel like 100%-
Patrick Miller: Absolutely...
David Roberts: 100% better grounded at this point. Thanks for walking us through it.
Patrick Miller: Sure. Anytime.
