April 1, 2021

Zero Trust: Fast Forward from 2010 to 2021


John Kindervag, who is widely considered to be the creator of zero trust model in 2010 (currently works at ON2IT)


Topics covered:

  • Which malware is scarier, state-sponsored or criminal?
  • How do we approach cybercrime mitigation at Google?
  • How do we actually track malware? Don’t we need “attribution” for it?
  • What are the most useful telemetry sources for study in modern malware?
  • Does ransomware have a bright future?
  • Where do you see threat actors making the biggest investments?

Do you have something cool to share? Some questions? Let us know:


Anton: Hi there. Welcome to the "Cloud Security" podcast and thank you for joining us today. Your hosts here are Tim Peacock, the product manager for threat detection here at Google Cloud and me, Anton Chuvakin, the reform [inaudible] analyst and member of Cloud Security team here at Google Cloud. You can find this podcast wherever podcasts are distributed at our website whenever we finally launch it. You can also follow us on social media at Our guest today and please pay attention this is important, is John Kindervag at ON2IT, formerly at Forrester [inaudible] and Palo Alto Networks. Who was the first to define the concept of Zero Trust in his 2010--think about it. 2010. 11 years ago paper. So we have a few questions mostly focused on Zero Trust and of course its past and the future. So let's start from a somewhat painful but necessary question. Let's define Zero Trust. Perhaps contrast how you define it back then and how you think about it today.
John: So zero trust hasn't changed, right? It was still a fight against the old trust model where we had trusted parts of the network and untrusted parts of the network as we would see implemented in say an old Cisco PIX. And so you had to define policy based upon a trust level. So the internal interface of a PIX was trust level 100, the highest level. And the external interface was trust level zero, the lowest trust level. So you could go from a high trust level to a low trust level without any policy. And I thought having that variable is painful and it means that there's no outbound rules and it's highly insecure. And we allow people to have access because of this trust model. And so trust is just a human emotion that we've injected in digital systems for no reason at all. And people confuse all the time human trust and digital trust. I mean, going back to 1984, Ken Thompson, who we all know as the co-creator of Unix in his Turing Award speech that year talked about the problem of trusting trust.
Timothy: Yeah.
John: So trust is something that shouldn't be in digital systems and that was the main thesis of the report. And then it led to how do you build systems like that. But mostly it was about thinking that this concept of trust actually incentivized bad behavior because all data breaches and almost all negative security events that have ever happened, the root cause is the trust model. You think it was a spam email but it exploited the trust model. You think it was ransomware. It exploited the trust model. Snowden and Manning were insider attacks who exploited the trust model. So that was my fight and then it's led to a lot more stuff.
Timothy: I think that that's one of the more wide-ranging attempts to define something on the podcast so far and I want to celebrate that. We've made it eight episodes without bringing up on trusting trust. So you've--you've broken that barrier. I think it's never going to be the same for us. But what is Zero Trust? Could you do that again?
John: Zero trust is elimination of the broken trust model that exists in digital systems today.
Timothy: Okay, I like that. That's concise. And how has that changed since 2010?
John: Well, it hasn't changed as Chris Hoff says. There's just a lot of people who lost the plot, right? So they'll say, "Oh, the goal of Zero Trust is to make a system trusted." No, it's not. It's to eliminate the concept of trust so we look and constantly validate it. Part of it was inspired. I was talking to a CSO many, many years ago and I said, "What's your strategy?" And he says, "Well, it's trust but verify." And I said, "Okay, trust I get. That just means you're not going to do anything but what are you doing for verification?" He said, "Well, we actually don't verify because that'd be rude 'cause they're trusted." And so then I started, "Why do you do trust but verify?" And he said, "Ronald Reagan said so." And so I looked it up and Ronald Reagan never said trust but verify. He said a Russian proverb and since I've got Anton on here, he is going to say it in real Russian 'cause I butcher it every time I say it. So the actual phrase is what, Anton?
Anton: It actually rhymes in Russian. It's. It sounds exactly like trust but verify but in Russian it rhymes so that's the cool part.
- But it means the opposite thing. And so you can see the video of Gorbachev and Reagan and they're both laughing, right? He's making a joke. So trust but verify was always a joke that Ronald Reagan made--made but we of course, yeah, we need to do trust but verify because that great cybersecurity expert, Ronald Reagan said so.
Anton: I now know, by the way, why when I showed John the questions specifically the next question John said, "We're going to argue about this one and now I know why." So the question we prepared was this. Sometimes I hear people say that zero trust means you cannot trust anything which is, in my opinion, possibly mistaken was false because you need to trust the Zero Trust system itself and other things like identity data and point state data. I think from John's definition, I'm off. Maybe there is a way to build a system that will just work and you don't need to trust it. So could you help us here? What needs to be trusted for the Zero Trust system to function if anything?
John: In digital systems, we say trust is a four-letter word, right? So don't use the word trust. Trust is binary. Once you say, "Well, I don't trust Anton very much anymore." He's untrusted, right? But I can say I have confidence in the system. So I'll substitute the word confidence 'cause it's measurable across a spectrum. And then when you talk about the system, right? I can measure the input and output of the system and see whether I get the expected results. This is something that we do, like, in supply chain or something, you know, you're testing if I send an input to a device, am I getting the proper output? Because I'm going to see somewhere inside the packet that something has been changed of what I expected to see. So we can test that and measure it and validate it. So the word I tend to use is validation. How have I validated a system so that I can have an extreme level of confidence that it's doing what it's supposed to do? My saying is trust is a vulnerability and it must be mitigated. And the reason is is because people exploit it. As soon as you're trusted, then nobody does anything. And I was in Switzerland but in the German part of Switzerland, and everybody was saying, "Oh yeah, it's like, you know, or some long German word. And they were trying to explain why people do this trust thing. And finally, they were all talking amongst themselves about how do they translate this very long word into English. And finally, the guy said, "Lazy." People are just lazy so it's easier to do this, right? And so now revalidating the system and I also have a saying that people aren't packets, right? So we anthropomorphize everything. We would say colloquially, Tim and Anton and John are all on the network but none of us are on the network right now. A packet that is asserted to come from a machine that is asserted to being used by us is riding across the network. People aren't packets, right? So maybe in the movies "Tron," "Lawnmower Man" people are on a network but even in "The Matrix," they got to plug in, right? So it's those simple things that cause people to rethink things and then they will lock stuff down. I mean, look at Snowden and Manning. What did they exploit? Did they have any great hacking techniques? Did they create new malware? No. They exploited the trust model. Once you're on that network, SIPRNet you have access to everything on SIPRNet.
So Snowden is setting up all these SSH tunnels. No one is noticing that 'cause nobody's looking at his packet post-authentication. And so we need to validate the asserted identity of the user, right? That's an assertion. It's not a real user. And then we need to constantly look at that traffic and see if it's behaving correctly and apply policy at layer seven about that behavior, that expected behavior. And then, of course, we need to look at that authentication because authentication has entropy attached to it, right? So those are different ways that you can look at it.
Anton: So when I used to say that for ZT system to function, you have to trust identity system, identity validation. I really should not have said it. I should have said for the Zero Trust system to function, you need to have a system, whereas the asserted identity can be reasonably seen to belonging to the right person or some such thing.
John: Yes. There's that philosophical question of ontology, right? So I was talking to somebody from the--let's just say the intelligence community, the IC. And I was joking. I said, "Well, hey, if somebody comes up and puts a gun to my head and makes me get up from my computer and then starts using it, do they suddenly become me because the computer is equal to me. And so now they're John, right?" And he laughed and he says, "Yeah, that happens a lot more times than you think." And I said, "It does not. What are you talking about?" And he says, "John, it's not a real gun. It's a virtual gun." I can say to you, "Hey, either you get me this intellectual property or I'm going to tell this dirty secret about you." Or I can say, "Hey, if you give me this intellectual property, I'm going to pay you a million dollars." It's just traditional spy craft into the digital realm, right? So Robert Hanssen was trusted. Aldrich James was trusted, right? So identity is an assertion. It's fungible and it has entropy. And it's important but it has to be constantly validated and then it has to align with the policy-derived behavior expectations.
Timothy: So steering a little more tactically here in a way from the philosophy of all this, you've mentioned identity systems as a prerequisite, behavioral expectations, policy. When you've seen organizations do a ZT project, what are the things they need to do to start with? How do they make it go right? What are the most manageable and important things for security teams and IT teams to grab onto at the start?
John: Yeah, I tell people you need to know nine things to do Zero Trust.
Timothy: Nine.
John: They're exactly nine things. There are four design principles. First one is align with the business, right? What is the business trying to do? Focus on business outcomes. Second one, design the system from the inside out. What are you trying to protect? That is the most important question. If you do not know the answer to that, it doesn't matter how many controls you have. You just get into what Rick Holland calls expense in depth, right? Instead of defense in depth. It's expense in depth
Timothy: I like that.
John: The third thing we do is control access, right? So you do access on a need-to-know basis, you know, privileged access. All those things we talk about, but that's not enough, right? Snowden and Manning proves that that's not enough. Then we inspect and log all that traffic and we do it at layer seven. So we can apply a policy at layer seven that includes access, control, and inspection. So Zero Trust is ultimately a layer seven policy statement that has to be enforced by layers seven controls because all attackers know how to bypass layer three and layer four controls. They all do, right? And so that's pretty trivial for them. And so if you say, "Well, in my cloud, I'm just going to turn on a stateless ACL in iptables." The Nation-State attackers are going to laugh at you. Oh, you think you're protected. Mmm. Great. Be happy with that. So those are the four design principles, and then there's five steps on how you deploy it. So the first thing is you focus on a protect surface. So what I've taken is the concept of attack surface and completely inverted it. And instead of saying, "What are all the threats out there? What do I need to protect?" It's a much smaller thing. I can shrink it down. Orders of magnitude is something that is very small and easily known. So I usually use a metaphor of how the Secret Service protects the president of the United States. You know, like at the inauguration parade, there are four people in the protect surface, the president, his wife, and whoever else, his children, whoever was there. And that's all the Secret Service cares about. That's all that has to survive Inauguration Day. And that protect surface is decoupled from transport. So sometimes they're going to be on Air Force One. Sometimes they're going to be on the helicopter, Marine One. Sometimes they're going to be in the limousine, The Beast. Sometimes they're going to be walking around. They might be on a rope line and doing handshakes but that's still a protect surface that the Secret Service understands. And so what you put in a protect surface is what I call a DAS element. That's an acronym that stands for data that you need to protect. Anything sensitive. Assets that you need to protect, IT/OT, IOT, IMOT, IIoT. They're all the same. They are all run on TCP/IP, don't they? And then the third thing we protect are applications that typically run sensitive data, ERP, CRM, HR. And the fourth thing are services. That's the "S". So I think of DNS, DHCP, active directory network, time protocol. Things that are incredibly fragile. So we take a single DAS element. We put it into a single protect surface and we build it out one protect surface at a time so we can do this incrementally, iteratively, and non-disruptively. So that's step one. Just know what you need to protect. If you don't know that, none of this stuff is going to work. It doesn't matter what you do. The second thing is you map the transaction flow so you understand how the system works. And I learned that the hard way when things went south a couple of times because we're working on a system that nobody understands, right? What's this server here? Wow, we probably don't need that. Let's take it out. Well, it was integral to the system. So understand how the system works. The third thing you do is you architect the Zero Trust environment no matter where it is. It could be in a public cloud, private cloud, on-premise. It could be on an SD-WAN. It could be on an endpoint or it could be in a SAS environment. Just what do I need to protect? Where is it? And then figure out what's the right architecture. So Zero Trust is a bespoke or tailor-made architecture for each protect surface. Then the fourth thing we do is create policy. I have a method about that called the Kipling method. Who, what, when, where, why, and how. It's my personal homage to Rudyard Kipling 'cause he gave us that idea in a poem in 1902. "I keep six honest serving-men. They taught me all I knew. Their names were What and Why and When and How and Where and Who." So that's where we get that and that resonates across all languages in all cultures so far at least. Everybody has that concept. And so I can define things and I have documents written about that. And then finally we monitor and maintain it. So we take all the telemetry. You know the first time I ever met Anton, he was wearing a button that says, "I love logs," right? I heart logs?
Anton: I love logs. Correct.
John: I mean, he's the only person I've ever met who actually loves logs. He does.
Timothy: I actually feel pretty similarly.
Anton: I wrote the freaking book about them.
John: Yes, he did.
Timothy: I'm with you, Anton.
John: Now's the time. Now is the heyday for logs because now we can correlate them all. And now that we're seeing what's going on in the system, we can refine each of those five steps and make an antifragile system out of it. So if you've ever read the book "Antifragile?"
Timothy: Oh, Nassim Taleb. We're going all over the place with the authors today. This is great.
Anton: Yeah, totally.
John: So the more that system is under load, the more we're going to see how we need to refine it and we go, "Oh, we need a new controller. We need to update policy." And so we've got this feedback loop and so Zero Trust is an antifragile system.
Timothy: Wow.
John: And so once you've done those five steps for one protect surface, you just do it for another one and another one and another one. And I have a thing called the learning curve where I talk about how you do it one protect surface at a time. You start with low sensitivity resources early on so that if you break something, it's safe. So I call those learning protect surfaces. And then you get up and you do some practice protect surfaces, you know, things that have a little bit more sensitivity. Not mission-critical. I'm going to get better at it. We need to practice. So Tony Scott, the former CIO of the US Federal Government and Microsoft and VMware and Disney, he's become a friend of mine and he told me this. He says, "I tell people you get to Zero Trust the same way you get to Carnegie Hall. You practice, practice, practice."
Timothy: Yeah, yeah, yeah, I love it.
John: After that, that's when you do the high-value assets, the keys of the kingdom, the crown jewels, whatever you want to call it. And I used to tell people, start with those crown jewels and then I found they were too fragile and there was nobody in the company or in the organization who knew how it was built. And you would see things go south. So give people some experience before they do this, and then you just follow those five steps. So I've made it simple. I've given you nine things. And I read other people talk about it and I go, "Whoa, I don't understand this, and I created it." So I can't imagine what the average, poor person out there who's trying to figure this out with all of the competing stuff. So I'm calling what I do authentic Zero Trust, right?
Anton: Cool. You have the brand authentic Zero Trust.
Timothy: I like it. Locally sourced, artisanal Zero Trust.
John: Artisanal Zero Trust. Yeah, there you go.
Timothy: There's actually some really cool parallels here between what we heard from Max on episode four and what we're hearing today.
Anton: Yeah, yeah.
Timothy: In particular, I'm hearing similarities in the agile and the repetitive approach to this. That you, like, learn how to do it on something low risk, and then build up the flywheel of skill here. I also really like the parallelism between understand your data and your assets at risk and protect surface and do one thing at a time. This--this is a well-considered model.
Anton: Yes, yes.
John: Well, thank you. Thank you. Well, it's even longer than 2010 because remember I spent two years researching it, talking to lots of people, validating it, test marketing it with speeches and webinars before I ever wrote the paper, right? So what you learn in the analyst business is how to do primary research, how to ask the right questions to the right people, go to the original source which a lot of people don't do. I mean, I went to a ton of people. People who were experts. Poke holes in this, right? And those people all helped. And so, you know, that's what you learn to do when you're in a place like Gartner or Forrester or whatever because they give you the freedom to think big thoughts. And they teach you to a methodology on how you can do that in a way that doesn't make you look like a nutter. And you can create things. You couldn't create this kind of stuff in a university, right? Imagine if you had to do peer review and all this stuff and everybody say--they'd say the same thing that they said to me early on, "That's not the way we've always done it." That was the...
Anton: Yes.
John: Big objection. That's not the way we've always done it, right? And imagine Einstein trying to come out with something today, right? No-no, that can't be right 'cause that's not the way we've always done it and that's kinda the--
Anton: And he would face outrage in social media. I'm sure.
John: Right, right, right. What do you mean that space and time are flexible, right?
Timothy: I will say that this is--this is a great learning experience for me today because I'm finally seeing the value that these analyst firms can bring other than really annoying-to-deal-with acronyms. You know my life as a PM, I feel so beholden to the analyst firms. But it's really nice to see the real value that they bring to the community here.
Anton: So apparently, I failed to express this but then John shows up and he's like, "Dude, this is how you do it." And Tim was like, "Yeah, totally. Very clear."
Timothy: Exactly.
Anton: Okay. Thank you, John.
Timothy: Exactly. Anton and I have been doing this for months now. I never got it but now I do.
John: Oh, well, good, good.
Anton: Well, I'm not offended at all.
John: Anton is a much more brilliant person than I am so I have to make things simple that I can understand, right?
Anton: So one other question we sometimes ask some people is this. What is the first step in the Zero Trust implementation? I think your previous question kind of answered it but the funny part is from other people who encountered some difference of opinion. So for example, somebody said asset management. Somebody else said identity. Somebody else said what you said, namely, the crown jewels. Well, it's kind of connected to assets. Do you have a pithy, short answer about what's the correct four-step in a Zero Trust implementation that has the cha--highest chance of success?
John: Determine what you need to protect. I mean, it's that simple. I was just on this big call and they had all these people who had various opinions on Zero Trust. And they were all mostly from vendors and I was from a vendor from the Palo Alto Networks. But they were all giving their opinions about what they need to do and I'm just sitting back listening, you know, 'cause it's kind of fun. And finally, I just interrupted and I said, "Excuse me, what are you guys trying to protect?" And it just ground to a halt. No one had ever considered that because everybody was trying to position, "Here you need my endpoint protection 'cause it's better at Zero Trust than anybody else's." And I was just like, "Who knows? Maybe that's true but that's not the question here. The question until we know what you need to protect, we will always fail, right?" And that's why I have that DAS thing. And that's why I have the learning curve because don't start with the high-value assets because you'll probably fail and you'll blame the failure on the Zero Trust model instead of the fact that you don't understand how your system works, right? So give yourself an opportunity to fail. I have a friend who's a CIO in Florida and he wrote an article for Security Roundtable for me because he actually has a degree in classical organ, right?
Timothy: Where is this going?
John: Well, think about he had to practice to be a classical organist. He didn't just decide, "Oh, here's the keys. And now I'm going to go play a concert," right? He had to practice. And we train people in our business but we don't give them the ability to practice and learn what they're doing and that's the issue, right? And so give some people the opportunity to practice. That's where it's going. You know you need to practice something to get good at it.
Timothy: Okay, I actually really like that. There's a kind of, like, almost defending the notion of Zero Trust against the failure to begin with so you can actually succeed as you get through the tough part. So speaking of defending against failure, what's the most spectacular failure you've seen in a ZT project? And also, what's the most spectacular success? What are the ends of the spectrum here?
John: The biggest failure I've ever seen was in a retail environment. Somebody as we were going through this process said, "What's that old server there? Let's pull that out." And this is for the POS system, right? So we're getting back to PCI 'cause I'm a recovering QSA myself, right? So yeah, if you've ever done PCI you know it's a 12-step program and--and only people who've done PCI could get that joke. So they've just pulled this server out. Ah, it's an old server. Let's get rid of it. Well, it was the polling server between the point of sale service and the backend system. And so everything went from the point of sale server. It was pulled from--from that into the polling server and then sent to the database in the back end. And suddenly in one minute, 3,000 some odd stores went down
Timothy: Yikes.
John: And that was when I learned map the transaction flows. That's probably the biggest failure. The biggest success probably that I've seen is at a manufacturing organization seeing them really struggle because manufacturing facilities are run by computers, right? And inside the computer, there's a board called a PLC. That's a controller and that runs software. And so there was this big fight between everybody because it was running on Windows XP. There's no longer going to be any patches and you couldn't patch anyway 'cause it's 24 by 7 system. Even if we upgraded, the software wouldn't run on it and the manufacturer isn't going to make drivers for it, right? So suddenly it's this rock and a hard place situation and I'm with the global head of manufacturing and I'm explaining how I can build a Zero Trust environment around this to put a micro perimeter around that system and protect it without having to upgrade it and do anything. And it'll still run 24 hours a day and he got it. I mean, he got it like this and suddenly he became the champion for the entire company's digital transformation around Zero Trust because where did they make their revenue? From manufacturing a product. And so yeah, he's going to be that guy. In fact, a lot of times, the CEOs, members of boards of directors, people who run the business get Zero Trust easier than the technology people who are tied into some legacy thinking. So that's probably been the biggest success.
Timothy: That makes sense.
Anton: I've heard that from you before by the way. This is one quote that I recall from you when you were at Forrester actually where the CIOs sometimes get it faster than engineers which struck me as strange and illogical, but now I actually get it.
John: Well, 'cause I'm asking somebody to do something different than they've always done it. And I'm asking them to change and no one likes change, right? I'm enabling secure business operations and your ability to make money. And so I did this primary research at Forrester on business alignment 'cause everybody in it says, "We want to align with the business." Well, if you look at what the priorities are if you have a bunch of people stack rank them and you say, "Business leaders, what are your three top priorities?" And the outcome was increase revenue, increased profitability, and stop data exfiltration of our staff. The IT people were, you know, catch viruses and stop threats. And the last things that they were thinking about was increasing revenue, increasing profitability. So how do we align? Zero Trust aligns us to the business thinking. So I'm going to enable the business to do things securely. Instead of being the department of no, that's what security is, I'm going to be the department of yes, let me do that for you in a secure way.
Anton: And so funny enough, we're kind of getting into time and I had this simple, short, crisp question I want to ask you. What do you think will happen in the next ten years? I'm sure I can handle it in, like, I don't know, three, five seconds perhaps because it's so simple.
Timothy: His answer to what's the first step was actually very pithy. So maybe this one's going to be good too.
Anton: You've lived this life for ten years. 11 to be precise and then plus two more when you did the research. So what do you see happening in the world of Zero Trust in the next ten years?
John: I'm going to answer it in less time than it took you to ask the question. We're going to see near-universal adoption of Zero Trust concepts in these systems starting real soon. There's going to be huge amounts of external gravity whether they're compliance or business things to adopt a Zero Trust mindset. And you're seeing that when you're starting to see lots of people talking about it. I mean, you know, the CIO of Google was one of the first guys who said do Zero Trust. And in fact, a lot of things that I started to do came from somebody called Google and Google said, "Call me. Call John," right? So I got a lot of business from you guys so thanks. But you're starting to see the top military officers who run DISA starting to talk about Zero Trust. You're starting to see a lot of that stuff. NSA releasing their guidance which was pretty darn aligned with authentic Zero Trust versus, you know, multi-factor authentication equals Zero Trust that other people do.
Anton: Oh yeah.
John: So you're going to see a near adoption of it at a strategic level. And then there's lots of different ways you're going to deploy it based upon what the protect surface is. So there's room for everybody to play this but just don't say that you have a Zero Trust product. Say, "My product works well in Zero Trust environments."
Timothy: I like that. So I think with that, we're just about at time. John, I can't thank you enough for joining us on the podcast today. We had a conversation that ranged from Kipling to--on trusting trust to a nine-step program which was really four plus five, which does add up to nine. It's been a--it's been a great show today. You can find this podcast, wherever podcasts are sold. And follow us on Twitter, Myself
I'm on Twitter at _TimPeacock. You can follow my co-host, Anton at anton_chuvakin. So please as always, tweet at us, argue with us. If you like or hate what you hear, we might just invite you to the next episode, which I promise is more fun than arguing with us on Twitter. And so with that, John, thank you once again. And to our listeners, we'll see you next time on the next "Cloud Security" podcast.

View more episodes