19. Securing Containerized Deployments (Part 2)

Rich: In episode 19 of Mobycast, Chris leads part two of a discussion about securing containerized deployments. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Welcome, Chris and Rich. It’s another episode of Mobycast.

Rich: Hey.

Chris: Hi Rich. Hi Jon.

Jon: How you doing?

Rich: Doing good.

Jon: All right, what have you been up to this week, Rich?

Rich: Tomorrow, I leave for vacation. The last week has been just finishing up some stuff for the first time and probably my career actually planned accordingly and don’t have a whole bunch of stuff happening next week. It’s been really calm, which is unusual, the week before and I’m just waiting for that one thing to come up that I forgot about. As of now, it’s going real good. Tomorrow, I’ll be on a plane to Jersey and then I’ll be on the beach at my aunt’s beach house for seven days, so looking forward to that.

Jon: That sounds great. So you just relax by the beach or do you have any activities you do?

Rich: I don’t really ever turn off, so I typically listen to podcast, read business books, and have my notebook out for brainstorming.

Jon: Sounds pretty good.

Rich: Yeah.

Jon: How about you Chris?

Chris: It’s kind of business as usual for me. No traveling, just doing the usual, nothing really, just normal day-to-day stuff.

Jon: Well, then I have an assignment for you. By next week you need to do something incredible to tell us about. As for me, I’ve been just kind of watching the fire situation here in Douglas––it’s up here in the mountains of Colorado. About 20 miles away there’s a big fire, but they’ve called in hundreds of firefighters from places as far-flung as Idaho and big tankers that drop the red stuff all over the ground last couple of days and the smoke is in the air. It’s close enough where we’ve had ash fall on our porch. Not great. Not ideal. It’s been too hot and dry and windy this summer. More than usual. Hopefully, everybody just lays off the fireworks because this will come out in a couple of weeks, but this is recorded during fourth of July week. Hopefully, everybody lays off the fireworks, lays off smoking in and out of doors, flicking cigarettes out of their cars, and things like that so we don’t have any more fires.

Anyway, last week we talked about securing containerized Docker deployments or containerized deployments. As we expected, security is one of those things that you can talk about for a long time. We cut it off after about 20 minutes and we had gotten through two of six phases of the development life cycle that Chris had learned about during one of the breakout sessions at DockerCon. It was just such an interesting information that I think we should just continue this week and talk about the next four. Hopefully, we’ll get through all four but if not, security is important, so we’ll keep talking about it until we get through it.

Chris, maybe recap just in a few sentences and under a minute if you can. The first two, so that people can remember and then we’ll jump into number three. But we don’t need to get into the details of that, who, what, where, when, and how of the first three. Just remind us what they were.

Chris: Sure, yeah, you bet. Again, this was a talk about trying to match up software development life cycle to each one of those phases to what you can do from a security standpoint in each one. They talked about six different phases. The first two that we talked about last week, the first phase was code, what you can do there as you’re developing the code? After that, we went into testing phase and what you can do as you test that code before it’s then goes onto the future stages.

That brings us to the third step in that process, which is building the actual code. This is where typically code has been done, you’ve now tested it and verified it. You now need to build your artifacts and what you can do during that phase of the process. The big thing that you can do there in this phase is scanning of your artifacts to look for any kind of known vulnerabilities. Then also make sure that you are in compliance as well with any particular standards that you may need to develop against, for example, PCI compliance was one of the examples that they brought up.

To be PCI-compliant, critical security patches must be installed within a month of release. This is something you can definitely do as you build your artifacts. Have your artifacts scanned by these various services that are out there. There’s quite a few of them now that are available. Have them scanned and that’s just one more way you can just verify the integrity and the security of the code that you’re building.

Jon: Chris, this framing and breakdown of how to do security during the development life cycle is not original thought by Kelsus, do you happen to have just available who it was that gave this talk and what the name of the talk was just for people listening so we that we can attribute it correctly?

Chris: Sure. This is a breakout session again at DockerCon 2018. It was titled, Don’t Have A Meltdown! Six Practical Tips For Securing Your Container-Based Deployment. Two speakers were Justin Cormack, he’s an engineer at Docker, and Liz Rice, the technology evangelist at Aqua Security.

Jon: Great. Thank you. What’s number three? Where did they go next?

Chris: That was phase three. Security is important, but sometimes can be maybe boring and easy to get lose track of. Just to mention, some of the security tools out there, the scanning tools that you can use, Docker itself has their own Docker Trust Registry feature that’s part of Docker EE and that will do scanning of your images as you push them up to your Docker Hub Repo.

There’s also some open source tools like microscanner which is actually an open source tool that comes from Aqua Security which Liz Rice, one of the speakers, works for. That’s something to check out. There’s also another one called Clair. That’s also open source as well. Again, lots of great tools out there for doing this stuff. Really no reason not to do it, so I encourage everyone to get there and check those out.

Moving on, the next phase they broke out is called basically the runtime environment host, if you will. Basically, thinking about the actual machines that you’re going to be running your containers on, what can you do there from a security standpoint.

Jon: Chris, since we’re fans here of AWS and ECS, I’d like to bring it back to something that we think about. In this case, this would be EC2 instances and an ECS cluster, right?

Chris: Correct.

Jon: Right.

Chris: It’s whatever virtual machine or bare metal, whatever it is you’re running your code on, in our case it’s containers, wherever you run Docker and that engine is running, how do you harden that actual host. Really, the principle here is you want to reduce your attack surface and basically just don’t make it easy for anyone that might be trying to gain access into your system.

They did talk about some of the issues of host versus container vulnerabilities. Basically, files in a container versus files in the host itself and what ramifications that has there. There are some pretty important considerations on just like the host configuration and how you set that up for running your Docker containers.

Some of this stuff is not as applicable to folks especially if you run in something like we do, ECS inside Amazon, some of the stuff is taken care of for us,but nevertheless, the overarching theme here is to minimize your attack surface.

Jon: I wanted to talk about that word, “attack surface,” because that’s a bit jargony and it’s something that, unless you really understand what it means, it doesn’t put a picture in your head. I guess, I’ll try my best to define it. If you imagine you’re running thing being like a shape out there in internet land, some parts of that shape are visible to the rest of the internet and some parts of it are hidden from the rest of the internet.

It’s those parts that the rest of the internet can see. Maybe it’s open ports, maybe it’s machines that have IP addresses that are public. Maybe it’s some other things that are visible because of random security vulnerabilities in software such as software that reaches out and creates a reverse proxy to some machine out there in the public.

It’s things like that, things that are visible to the rest of the internet that would be your attack surface. Is that a decent description of it? Is there anything I left out?

Chris: No, that’s it. It’s definitely a very good overall description. I think like in this particular case, what probably they were focused on is just attack surface basically means software. The more software you have running on your host, the broader the attack surface is, and the more potential ways someone has to get it to gain access, to find a hole to get into it. Typically, that relates to a lot of the stuff that you were talking about. Ports, they’re opened usually by software. They are needed by various pieces of software and whatnot.

Again, the theme here is only have software only installed on that host that you actually are using and need. A lot of times, we’ll err on the side of convenience for us and we’ll kind of use a full-featured image or AMI that contains all the software that you need and then some. It’s just all right there out of the box. While that’s really convenient and makes life easier for us, as DevOps folks it also means that there’s a lot of stuff running on there that we may not even know about or how to configure it properly and maybe it does pose a security risk.

Jon: Can you give examples of a few things that might be convenient to have on a host that maybe shouldn’t be on a running host? I was thinking maybe compilers, not sure, various networking tools.

Chris: I think that’s where it boils down to. It’s going to be services that are running, that are listening for request that represent a way to kind of get in. You have network protocol, daemons, it could be things like something as simple as a DNS service or it could be NTP, it could be software package updating service type thing that comes as a convenience from some other distro or something like that.

I think just anything that closes a potential door into the system is something to be concerned with for sure. When someone is in the system, then you have to worry about those other things that maybe don’t listen. Like a compiler, but maybe having a compiler on there or some of the tools like that, then gives additional capabilities that makes it easier for somebody to cause mayhem.

Jon: We’re diving into this a little bit despite the fact that you said maybe this isn’t that important if you’re using a service like ECS, or EKS, or Google’s managed Kubernetes. It’s true that it’s not that important when you’re using those because the engineers that created those services on those cloud engines thought about these stuffs in advance for you.

But I guess the thing that’s come into my mind is that, what if you’re talking to a CSO, Chief Security Officer, or you’re talking to a client that’s concerned about security and they start asking hard questions about what you’re doing to make sure that things are secure or even if you’re responding to an RFP and you’re talking about all the things in your software that you’re going to do for security, it’s nice to be able to say, “These are the things that we’re thinking about.”

One of the things we’re thinking about is host security and these are the types of systems that we’re not going to be running on our hosts in production. The reason why is because we’re reducing our attack surface area. That makes it sound smart.

Chris: Yeah.

Jon: You’ve got to sound smart in this business.

Chris: There’s so much to keep on top of and we’ve tried to talk about this in previous podcasts as well. Even if you don’t have to do all the work, you should at least be aware of what the requirements are to play the game and to be able to talk about that stuff.

Jon: Exactly, yeah.

Chris: Part of this phase as well, they pointed out there’s various tools and benchmarks out there that will help you have better insight into just how secure you are from a host standpoint. There is a CIS Docker Community Edition benchmark that you can go and point at your installation and see how it’s doing against this, so the CIS is Center for Internet Security, so they published a benchmark for the Docker CE edition. That’s something you can run against your host to see how you do. There’s also one for Kubernetes and pretty soon, it looks like we’ll have one for Docker EE and then also Windows host operating systems as well.

Jon: That makes me laugh that CIS has something available for Docker CE, but not for Docker EE and yet, Docker EE is the one that maybe big enterprises would be more ready to trust. It’s just funny, but go ahead.

Chris: I suspect that again it might be like surface area type thing, that it’s so much easier. Docker CE is so much smaller and so much fewer parts, and also it’s been out there for the longest time as well versus Docker EE, it’s changing pretty rapidly, at least that’s where most of the innovation that Docker’s doing now is going to the energy. It maybe that Docker is doing more of the work there to get that mark in place than it is the CIS.

Jon: Interesting. It’s funny just because the types of companies that would be most interested in an Enterprise Edition are the type of companies that have actual CSOs–Chief Security Officers, that think about this stuff 24×7 and they have whole teams that work on this stuff.

Chris: Indeed.

Rich: Hey, this is Rich. You might recognize me as the guy who introduced the show, but is pretty much silent during the meat of the podcast. The truth is, these topics are often times incredibly complex and I’m just too inexperienced to provide much value. What you might not know is that Jon and Chris created the training product to help developers of all skill sets get caught up to speed on AWS and Docker. If you’re like me, and feel underwater in these conversations, head on over to prodockertraining.com and get on the mailing list for the inaugural course. Okay, let’s dive back in.

Jon: Is there anything else in this phase of building or is it just those tools that are available to take a look at your surface area and to make sure that you’re not running things on your host that you shouldn’t be?

Chris: I think that about sums it up for this. Minimize your attack surface area, take advantage of some of the benchmarks that are out there for evaluating, just how you’re doing in that regard and just begin minimal amount of software.

Jon: Great. Moving on, what do we have next?

Chris: That brings us to phase five, which is the run phase. Now we’ve done our coding, we’ve done our testing, we’ve built it, we’ve now hardened the host on which it’s going to run on, and now it’s time to run it.

Here, the theme was least privilege. You only want to get the least amount of privilege to your code that it needs in order to run. So definitely, here I’m being conservative as opposed to running as root per se.

Again, this comes back to everyone knows that this is best practice, this is what you should do. It does make life more difficult for folks as DevOps to employ this principle. It’s one of those things where I think in practice it doesn’t get implemented as rigorously as it should, and we all know that.

Maybe it’s a little bit of a dirty secret but this is definitely one of those things where you really should just roll up your sleeves, do the extra work, and deal with some of that additional inconvenience because it’s just so important from a security standpoint.

They gave a lot of examples here, like what are some of the things that maybe typically happens that are examples of having more privileges than you need or exposing the holes with your runtime environment.

One of those would be bind mounts. I think we’ve talked about this a few times on previous episodes with Docker being able to share file system access between the host and the container itself through volume mounts. We’ve talked about how this is definitely one of those things that it starts tampering with that purity of container isolation, basically poking a bubble inside that isolation container.

There’s perhaps some subtle things that can happen that you might not be aware that could happen. One example they gave was if you did a volume mount, and you mounted the /etc directory from your container to your host, by default your Docker container is going to run as root. When inside the container, if you run the command user add to create a new user from inside the container, what will happen is the user will actually be created on the host as well.

It’s because it’s just going in and modifying the /etc users, I believe it’s the /etc users file, and to add that user to it. So because we now have the volume mount, that’s where it gets written to, it’s on that. Now you do have your new user inside your container, but it’s actually on your host as well. That’s obviously really big “uh-oh” moment. So definitely be very, very careful with your bind mounts. You want to very much minimize the area of the file system on the host that you’re allowing them to have access to.

Jon: That’s super interesting. I love that example. I don’t know anybody that spend a lot of time working with computers that hasn’t kind of fallen back on the, “Hmm, I don’t understand that compiler error or that run error. Let me just try again as root. Oh look, it went away.” It is probably not the best way of doing things, but it sure is a common practice.

Chris: Yes so related to that, also recommended that in your Dockerfile set a user, actually switch from the default root to run a named user that has the appropriate level of permissions. Again, pretty much known best practice. We all know we should do it.

The bigger, more popular images will do this like the Node.js images, PostgreSQL, all the big ones that have a lot of activity behind them, they do this as best practice but when it comes to actually building custom software and building your own images with Dockerfiles, I don’t see a lot of people taking advantage of this and doing this, and even really understanding what’s going on.

Jon:Actually, help me understand this because I was thinking while you were talking about this that containers don’t really know anything about their host system, so how can a container tell the host, “Don’t run me as root?” It shouldn’t even know anything about its host, so how would it have any control over how it’s run?

Chis: This is the user context inside the container when it’s running.

Jon: So the containers own what is running its own stuff as.

Chris: Yeah. By default, it needs a user. By default, it’s going to be root if you don’t specify something otherwise.

Jon: Running of the containers themselves also should not probably happen as root, but it is likely that it is happening as whatever user you are or your runtime scripts run as. If you’re running, for example, on your local host, your containers are going to run under your own account most likely. But if you’re running it in some EC2 instance, you’ll be running as some user you’ve created. Do you know whether for example, ECS creates a special user like an ECS user on its EC2 instances?

Chris: I think the way it works is it’s the Docker image that is instantiating the containers. Say you’re running the Docker image and you’re using the command line, so you’re doing Docker run or what now, that’s sending a command to the Docker server. The Docker server is running on a particular user context.

I believe there may be a way to configure the Docker server engine to let it know under what context to run these things. It may very well be root. That’s an open question for the details on that. I’m not exactly sure how that works, but with ECS, it’s running the core Docker engine, the same, obviously client-server model. It has an agent that runs on there as well. It also has the ECS-optimized AMI that packages all that up, which is based upon Amazon and Linux as well.

Jon: That is so settled that I just need to really clarify it. You did a very gentle job of correcting me. I said you’re running on your local host. Let’s say I’m running as Jon C and I type Docker run, what you just said tells me, “No, my container is not going to be running as a process on my machine owned by Jon C. It’s going to be running as a process on my machine owned by whatever that Docker server is configured to run Docker containers as.”

Chris: Correct. That’s what’s actually creating that job. It’s the Docker server.

Jon: Okay, there will be a short process that is owned by Jon C that says, “Hey Docker server, do something.” And then Docker server is going to go run as whatever it should. Okay, cool. I did not know that. Very cool.

Chris: Yeah and maybe to wrap up this section, there is the “dash dash” privileged flag that you can give to Docker when you’re running a container. The recommendation there is unless you really need it, don’t do that, but again this maybe one of those situations where someone is trying to get something to work and they find out that, “Oh, I can’t do this because of permissions,” and search online, Stack Overflow or something like that and like, “Oh, I decided this is “dash dash” privileged and boom it works and now I’m happy.”

It’s important to understand that, that the full ramifications of that. It may have been all you needed to do is give access to a certain directory or something like that to a user, but instead you’ve now opened yourself up to so much more than that, so again, least privilege.

Jon: Great. Let’s wrap it up there. Again, just so much stuff to talk about in terms of security and I think that we’ve hit that. The thing that I love about this conversation is that it’s not just about, “Oh, you’ve got software running. How are you going to secure it?” It’s about how are you going to do security from beginning to end from all the way through your design and development, testing, build, and run pieces of yourself for the development life cycle. So, thank you very much Chris, and thanks for joining us, Rich. Have a good week.

Chris: Thanks guys.

Rich: Well, dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with show notes and other valuable resources, is available at mobycast.fm/19. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.