EP69 Cloud Threats and How to Observe Them

[CHILL MUSIC] - Hi, there. Welcome to the "Cloud Security Podcast" by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of our Cloud Security Team here at Google. You can find and subscribe to this podcast wherever you get your podcasts as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast.

If you like our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Today, Anton, we are talking to some of our friends in the industry who do really interesting work in both the posture and threat detection side of cloud security. But really, we have a fun opportunity to dive into one of my favorite topics, which is malware in the cloud.

- Right. And to me, this episode is also an exploration of the same ongoing theme about threats in the cloud, threat detection in the cloud, how do we approach threats in the cloud.

- What are threats in the cloud? What are the threats in the cloud? Nobody knows.

- Nobody knows they're in the cloud, right? Well, wait a second. We're in the cloud too.

- I think I know. I think some people at Google know. But I don't think anybody knows the whole thing. I think we're all like the blind men in the forest with the elephant. We're still working for somebody to come along and tell us what we're really looking at.

- In a very, possibly nasty to our audience because it's a little self-serving, we did get a fairly recent internal presentation from somebody who protects the Google's use of cloud, and they had a fascinating list of threats. Of course we can say absolutely nothing [LAUGHS] about what they observed, but I can tell you right away that some of the things were surprising and some of the things were very much not surprising.

- Well, one of the things I'm looking forward to in today's episode is the conversation about Linux malware. I think we're in one of those evolutionary stages in the industry where there was a long time where people didn't think there was macOS malware. Some people don't think there's Linux malware. But the truth is, as platforms become more popular and more economically viable for adversaries to create tooling for, we see that tooling getting used more and more. So perhaps with that, let's turn things over to today's guest.

- And our guest today is James Condon, Director of Security Research at Lacework. Welcome to the podcast.

- Hey, thanks for having me on.

- So you guys published this fascinating report and it also hit on something that's been my obsession lately. Namely, what are the realistic and observed threats in the cloud? So in your report you do highlight if you type, so could you give me a bit of an informal intro. What are the threats you're observing, and also, maybe how did you observe them?

- Yeah. For sure. So one of the key points here is within our team, we really like to focus on what are the in-the-wild attacks. So we don't do a lot of hypothetical and threat modeling. We really want to get empirical evidence of what attackers are doing because attackers always do things a little bit differently than we predict.

So for example, when I first started in cloud security, I really wanted to know, Kubernetes is very popular, everyone's adopting it, everyone's talking about security, but are attackers actually targeting it? So essentially what we've seen over the time is there are a lot of different threats. I would say that the threat from the criminal organizations with financial gain is really one of the most predominant ones that we see.

We don't hear a lot of public reporting about APT threats, although we do know that those are occurring and managed service providers could be vulnerable to this. But I think my big takeaway and the thing that really drew me into cloud security in general is we're at the snowball on the tip of the iceberg of really understanding the threat landscape there. And I think that this is something that, with how fast the cloud moves, it's always going to be evolving.

- So that sort of means that you focus on what the attacker's actually doing. That's very useful, and I think that I do share your sentiment about threat modeling, but I feel like it's going to be a balance for many big companies. You do threat model around your app, but you also observe what's actually happening.

- Yeah, definitely. And so we have folks who do threat modeling from a security perspective, and then we'll focus on the intelligence gathering of, OK, here are the techniques. This is what they're doing. Are the majority of attackers trying to escape a container, for example?

- OK. Now that makes sense. And so this brings us to the next point, which is actually, funny enough, is a long-running argument between me and Tim about are cloud threats predominantly like regular threats that just threaten cloud assets, or are those novel things that were invented just for the cloud? So your point about Kubernetes, for example, sure, people deploy containers, but are attackers focusing on containers?

So sometimes we say-- well, both of us on this podcast hate the line, cloud is just somebody else's computer because we don't believe it's just somebody else's computer. But attackers seem to think so because many of the attacks seem to imply that they're after VMs, they're after CPU resources, they're after disk space. So how do you think of this? Are these new threats or are these old threats against new assets?

- Yeah. Definitely. So I think about it in a couple different ways. The funny thing is I share your sentiment around the cloud is just somebody else's computer because when I hear that line, I tend to think of someone else's computer as like a MacBook or a Windows laptop, not a cluster of servers running a bunch of containers, spinning up and down.

But as far as the original question about the on-prem threats, are these very similar or not to what we're seeing in the cloud, it's really yes and no. So for yes, high-level tactics to accomplish objectives are generally the same. So if we think about the kill chain, there's the idea of initial access, there's lateral movement, privilege escalation, and end objectives.

But the techniques between the two and the end goals can differ quite a bit. So for example, early in my career, I spent a lot of time investigating APT threats against Windows enterprise networks, and predominantly phishing and social engineering was the way to get initial access.

Now, the end goal in these cases usually was intellectual property theft. Now, in the cloud, I mostly see exploiting a web-facing application as an initial access or maybe a misconfiguration, and then in most cases, we're seeing financial gain as kind of that end goal. So yeah, that's how I think about it.

- So what about the argument that sometimes it's not so much that threats are different in nature, but maybe the importance is different? So for example, if I have a regular website, of course people can attack it, even if I don't use cloud. But you just pointed out that for cloud environments, initial access is very often through a web app.

So same with crypto miners. As when we did our threat research using our threat horizons reports, we found out that there's a lot of crypto mining in the cloud. Of course, you can have crypto mining not in the cloud, but the importance of crypto mining in a typical data center is probably low.

- Yep.

- But the importance of crypto mining in the cloud environment is probably high. So how do you think of this reranking of the threats, if it makes sense?

- Yeah. I think that makes a lot of sense. So if I zoom out a little bit, the way I think about it is, in general, attackers are going to follow the path of least resistance. And so if we look at a typical botnet, the idea of a botnet, really, is you can, at scale, run some sort of operation, and the fundamentals are kind of the same.

- At scale sounds like cloud, right?

[LAUGHTER]

- Exactly. So it's kind of fundamentally the same in that you have targets and you're trying to compromise them. And then you can deliver whatever payload that you want and you could deliver traditional ransomware or other types of malware. But we see crypto mining deployed because that's where they're having success.

That's where they're getting a lot of the money from. And then there's this idea of having a lot more available compute. So going back to your personal computer, yeah, you can crypto mine on it, but if you want to do this at scale, you need to do that in the cloud.

- So it sounds like the reranking of threats, certain threats becoming more important in the cloud, are very much connected to the cloud in here in properties, like scalability, being API driven, being public, all that stuff?

- Yeah. I think so. And then I also think part of it, too, is within the cloud you have a lot more options for disaster recovery and things like that, so that nature changes a bit of the threat landscape as well. And then the other variable in there is, OK, how much success are these different threat actors having in the traditional enterprise networks or the OT networks that they're going after? So the security of a different space can also affect the space of cloud security, and then how that goes is going to change the ranking of volume and priority.

- That makes sense, and to me, this is also interesting as kind of a highlight for what matters today. Now, here is the maybe painfully obvious question. Everybody obsesses about how configuration mistakes, configuration weaknesses is the bane of cloud computing.

There are whole categories of vendors being created for it, the whole CSPM. And so, what is the second most dangerous cloud issue after configuration mistakes? Can you name something that's right after? So imagine that somebody maybe took care of all configuration mistakes, what's their top priority?

- Yeah. So if we go back to the idea of, in general, attackers following the path of least resistance, the configuration issues are really that low hanging fruit. The second big part that I would say is very important and needs a lot of focus is really vulnerability management. So we really rely heavily on a lot of third party code to build our SaaS applications, and this makes vulnerability management a bit more trickier.

So on-prem you might have an appliance, and then you learn about CVE, and then you download the update. And you know whether that appliance is internet accessible or not. In this new world we live in, it's chasing down where a vulnerability might be and whether or not it's accessible can be extremely difficult. So as the cloud moves very fast, the layers of complexity grow in volume, and then being able to assess if you are actually vulnerable to a given CVE can be really tough.

So for example, if there's a CVE in an application that is bundled with a bunch of different Linux distros like curl or something like that, each different distro might end up giving a different score. And then if you're using containers, the way you build the containers it may or may not be accessible.

So it really creates this tough situation of, OK, we've got all these vulnerabilities. How do we prioritize it? Because we know that after configuration issues, scanning for vulnerabilities, or propagating once they get on cloud workloads, a lot of times involves looking for a vulnerable application to exploit.

- And that definitely brings back memories of my Gartner tenure where I dealt with VM a lot, and people have been still struggling with VM, Vulnerability Management, on-premise. And suddenly we have this whole VM in VMs and VM in containers and VM in modern cloud. So I can see how it can be tricky.

- Yeah.

- Thank you very much for this.

- Yep. Definitely.

- So going back to the main issue, configuration weaknesses. So of course, we can make jokes about how somebody may give security faults. Like, say a certain storage bucket is private, but people still unlock it and make it public and so the data leaks. So there are many more scenarios for how mistakes get made in the cloud.

So why is it so common for organizations to have insecure configurations in their clouds? Does your report reveal anything, or maybe your research reveal anything as to why it happens? Cloud is, what? 10, 15 years old at this point? So I would think in 10 years, we would learn to not make mistakes in configurations. Right?

- Yeah.

- Or am I naive here?

- Yeah, so let me dive into it. I think that's a really interesting one, and I think it's a theme of security in general. So I think it really loops back to this is simple in theory, but hard in practice.

- Yeah.

- And so if we take the view of looking from the outside looking in at security, security seems really simple. It's like there's an analogy that if security is your house and you're defending it from burglars, you add locks and alarm system. It can't be that hard. But in reality, especially as teams are moving really fast to build very large, complex applications, it's more like securing a large military base where you've got all kinds of different people with different levels of access, you got all kinds of different threats.

And ultimately, it gets complicated very quickly. So the reason I think it's hard is because it quickly becomes difficult to manage. And then as our products are developing extremely fast, our engineering teams grow. Before you know it, there's dozens of accounts across multiple cloud providers, you don't know who's spinning up what, and trying to put guardrails in, at that point, becomes really difficult.

So if you think about building a SaaS application, if you're on an engineering team, you kind of have this trade-off from the beginning of, do we invest the time now to build our application for a very large scale in the future, or do we build it to work now so we can get to market and then scale as we go? I think when it comes to security, what we need to do is get in the mindset of building a scalable security practice way before you need it.

So anticipating that this is going to grow very complex, very quick. So sometimes I see there might be a startup, and it's growing fast and there's 300 people and it's half sales, half engineering, and then they're like, OK, we need a security engineer. I think when you start thinking about the question, oh, now's the time to get a security engineer, you're already pretty far behind the ball.

- There's a thought that's stuck in my head now after this. Namely that if you are in this type of rapid cloud development, rapid cloud adoption, you need a scalable security team before you think you need it.

- Yes.

- That, to me, is a good headliner for the episode because I think that it sounds like a lot of people are not seeing this reality and they're thinking, we'll need it, we'll get to it, we'll do it, and then there's a delay. But ultimately, it's almost like when you think you need it, you're late.

- Yep. Yep. Exactly. And that's the way you need to plan for it because if I spin up an account in a cloud provider and I have a simple app on it, it's really easy to manage. But over time, it becomes more and more difficult. And then once you realize, oh, hey, we need some sort of enforcement, we need automation, we can't manually do this all the time, at that point, you're going to have a really hard time catching up.

- Yeah. No, that makes sense, and I think I like that line of thinking. I was meaning to ask you for a few common examples, but why don't I ask you for maybe the most ridiculous examples of configuration mistakes you've seen. You don't have to go refer to clients, but just in general, what are some of the really bad mistakes people make with configurations in the cloud?

- Yeah. I think overall I'm very optimistic where we're heading with this. Over the three and a half years I've been at Lacework, I've seen a significant downward trend of just, like, oh my gosh, how did that happen type mistakes. But there was a point three years ago where you could rely on, every week, reading an article about a large database with sensitive PII in it just simply being exposed to the world without authentication. It was like--

- Yeah.

- It was always Elasticsearch. It was always--

- Elasticsearch, always. Yeah.

- Redis was always a very popular one. And then there would be exploits for those as well, so that one happens pretty commonly. Another one is making storage buckets public that shouldn't be. It's easier to make that mistake than you think because sometimes there are certain things and that becomes a little bit more difficult.

But I think fundamentally where we're at today, there's three core areas, and that's identity, access management, that's storage in compute. And that's usually where these issues happen, and I think most of the issues are artifacts. As things become more complex, the vulnerabilities or the misconfiguration becomes a little less clear. It starts to become non-obvious, so to speak.

- Yeah. And I think I agree with that, and I think that people who make those mistakes, it's not like they are stupid. It's not like they can't really decide on one particular storage bucket, private or public. But if they have 27,000 buckets managed by 300 teams, that's when complexity rushes in. They don't know what to do.

- Yeah. Thinking back to another interesting example, I have two more for you. One is when I was doing some research about attack techniques with Kubernetes a couple years ago. I was very surprised to find how many administrative dashboards were just publicly accessible.

- Yeah.

- And that's kind of one of those ones where it's like, oh, man, this is tough because sometimes people are spinning this up and they might not even know that they have this service. In most cases that we saw, they weren't critical assets, but in some, you have this issue of, OK, you have cloud provider keys within your data store and now that becomes accessible.

- Yep.

- And then the other one that's funny to see is-- container escapes is really interesting to me because most container escapes we see are actually a container misconfiguration. It's overprivileged and things like that, as opposed--

- It's not the nasty zero day for Kubernetes.

- Yeah.

- For sure. For sure.

- Exactly.

- Yeah, yeah, yeah. Yeah.

- One of the funniest ones I saw is a report from in Azure, and there was a docker API that was open to the world. And what the attacker did is they spun up a overprivileged container that they could escape out of. So it was like they designed the container for the escape as opposed to kind of these-- we have these thoughts about these crazy ways that people are escaping containers.

- Yep. Yep. No, that actually makes sense. If you can't modify the environment around you, you should build an escapable container as opposed to create amazing ways to escape from hardcoded, secure containers.

- Yep.

- Makes sense.

- Exactly.

- So the other fun area that your report explores is, of course, malware. And while some people obsess about ransomware and ransom ops, we do see some examples of ransom operations in the cloud but your report seemed to highlight that it's a risk, but I didn't get a sense of how bad is it. So cloud malware, in general, ransomware, ransom ops, they're a bit of a Venn diagram here. Are these real risks in the cloud today?

- So yeah, let's dive into that. I think let's talk about malware first. So malware is definitely a big risk and I think it's really bubbling up underneath a lot of people's radar, and there's a number of reasons for that. So one is the collective security knowledge on malware is predominantly Windows.

We have a good body of research and it's understood a lot of the techniques and how to detect it and things like that. When we get into Linux, it's much less commonly understood. And one of the reasons is it's very difficult to automate dynamic analysis because there's just so many different devices and architectures that an ELF binary can run on.

Additionally, it's hard for people to notice that, oh, Linux malware variants are on the rise because if you took a fixed set of ELF files and a fixed set of PE files and then half were malicious and half were not and then you threw it at every AV, it would look like there was a lot less Linux malware in those samples because--

- Yep. Linux malware. Correct.

- It's just the detection rates are going to be quite a bit lower. But our team has really dove into this area quite a bit because we think it's fascinating and we think it doesn't get as much coverage as it should. But one of the things-- the two major trends that we see is we're seeing popular Windows malware being ported to Linux, which is a very interesting trend.

- Huh. But for the cloud specifically, right? They're not hacking somebody's quote, unquote "Linux desktops," mythical though they may be.

- Yeah. Yeah. And sometimes we have the assumption, a lot of these cases we believe this is targeting cloud. It could be that there's some other targets that are Linux based that maybe they're running into, but we find that to be pretty interesting. And then we just see the malware growing quite a bit.

So it's like three years ago, most everything was kind of a simple cryptojacker, and now we're seeing more interesting persistence and rootkit techniques. And so I think that's evolving quite a bit. As far as cloud ransomware, I think this is a really interesting topic. It's really on a lot of people's minds. The way I like to think of it is two categories instead of just putting it into one. So the first is traditional ransomware and then the second would be ransom and extortion in the cloud.

- Yes. Correct. Cloud extortion?

- Yep. Exactly. And so for several years, we've actually seen a lot of traditional ransomware samples for Linux, and they're spread by botnets. And it's similar to what cryptojacking is today, but we primarily see cryptojacking malware being dropped instead of ransomware because in the cloud, locking up a single server that might be on a container, it's a lot easier to mitigate, and so I don't think they're getting paid off very well. But the threat that I'm more concerned about is the ransom and extortion in the cloud.

And so I see this manifesting in a couple different ways, and I think we're starting to see this in public reporting a little bit more, but there's the scenario of a workload gets compromised and it has some sort of credentials or access to the cloud control plane. And then an attacker pivots into that control plane, and instead of just locking files and demanding a ransom, since disaster recovery plans and backups are a lot easier to do in the cloud, you can expect that sensitive data is going to get downloaded and then held for extortion.

So we saw this shortly after Log4j was released and happened to a crypto firm out of Vietnam. I believe the name was ONUS. But I think that this is the kind of key area to pay attention to. It's less of a threat, in my view today than what we're seeing in the OT space and enterprise networks. But as those defenses get better and better, I would expect to see more of this activity in the future.

- So if you are judging this not as a defender but as an aspiring criminal businessman, you probably wouldn't go into this now, but you'd be thinking maybe in two, three years I'll be doing this. But as of today, it's still not a good investment of your time.

- Yeah. I think so, and I think it's--

- How it feels to me, right?

- Yeah. That's kind of how I feel. I would say the one thing on the timeline that I do find a little concerning is some of the research that we've done in dark web forums is showing an increase in sale for access to different cloud accounts. And then as they go up for sale, the price is dictated by things like how many buckets do they have and maybe what type of data is in it and things like that.

As the ecosystem gets a little bit easier to access these things, I think that could come up a lot and I think the extortion piece could potentially be pretty powerful. But today, being able to take a company and set it to a grinding halt by freezing their OT network, I think that that's going to give them a higher impact, higher dollar figure.

- That makes sense to me. Yeah. Yeah. Perfect. So let me ask you the multi-cloud question. And of course, Lacework being a security vendor covering multiple clouds, I'm sure you see threat actors hit all three major providers, right?

- Mhm.

- So to me, I don't think I recall seeing it in the report, but my curiosity was piqued around, are attackers doing things differently for each cloud? So are there things that are Amazon specific, Google specific, Microsoft specific? Are you seeing any threats that are customized on the per cloud basis?

- Yeah. So we definitely are, and I think part of it comes from the general knowledge of how some of these attacks could occur. So if we think about some of the details that were released around Capital One and this idea of pivoting from the infrastructure into the control plane, I feel like after that, we started to see a little bit more of that style of tech.

So specifically, some of the stuff that we came across is we found some interesting scripts from this threat actor that we're following, and it had embedded in it Shodan queries that were looking for a specific vulnerability to exploit in an application. But then it was constrained just to AWS, and so we thought, oh, that's kind of interesting. They could do this in any cloud, but they're going for AWS.

Later, from the same group of people, what we ended up seeing is malware that would also drop a lot of AWS tools for querying the metadata service IP and attempting to automate getting into the cloud, damaging buckets, and all kinds of things like that. So I think there is quite a bit of cloud-specific things depending on the attackers knowledge. And then the larger body is really, what's the low-hanging fruit? What opportunities do I have?

- Right. So it makes sense if the particular team is writing Linux malware, they would still need to know something about specific cloud provider to get in, to download, to deploy, to achieve their goal.

- Yep.

- So I can see that it's much more skills driven rather than anything else.

- Yeah.

- It's not like they hate some specific cloud providers only. So we are close to our time, so I wanted to ask you our traditional closing question. Any recommended reading? Of course, we'll link to your report, but anything else? Any blog posts, anything else that you have in mind? And of course, the other half of our traditional closing question is, any particular one-- your favorite tip on securing cloud.

- Yeah. That's a great question. So recommended reading, I always would love folks to come to the lacework.com blog and check out what we're doing from a threat research standpoint. We really want to give people actionable threat intelligence and let defenders know what type of techniques to have. I would say my favorite reading on cloud security, in general, is the book "Securing DevOps."

I think that that's a phenomenal book of looking at the whole life cycle. A lot of times people see it in two halves, like, oh, there's security and they handle runtime threats, and then there's developers and they handle the developer lifecycle. And so I think this one does-- that book does a really good job tying that together.

And then as far as a tip, I would say for people who are born in the cloud, you're doing a startup, you're taking all this stuff on, I think you really need to be getting things in place security-wise way before you need it. So I think the common trend-- and it's hard when you're in a startup because you're trying to build the plane you're flying on. But the typical thing is, OK, you get SOC 2 because you need to sell your product and you can't without SOC 2. And then you start getting into GRC.

But at the end of the day, this really means overinvesting in a GRC team very early on because there's going to be significant cost considerations with how you store and handle data down the road if you don't think about that architecture. And then the other is trying to get some great security engineers at the beginning who can think about, how do we scale our controls and wrangle the big mess that this will become in one or two years, potentially?

- That does make sense, and I think-- thank you for reiterating that tip about getting the security team before you need it.

- Yep.

- So thank you, James. This was really fun and enlightening. Hopefully it will make a really good episode. Thank you very much for this and for the links.

- Yeah. Thanks for having me on. I enjoyed it.

- And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast.

Please subscribe so that you don't miss episodes. You can follow us on Twitter at twitter.com/cloudsecpodcast. Your hosts are also on Twitter @Anton_Chuvakin and @_TimPeacock. Tweet at us, email us, argue with us, and if you like or hate what you hear, we can invite you to the next episode. See you on the next "Cloud Security Podcast" episode.

[CHILL MUSIC]