26. Docker Tips & Tricks (Part 3)

Rich: In episode 26 of Mobycast, we conclude our micro series on Docker tips and tricks. In particular, we discuss security and pruning. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Welcome, Rich, Chris. It’s another episode of Mobycast.

Rich: Hey.

Jon: Hey. What have you been up to this week, Chris?

Chris: Exciting times here at Kelsus. The team is growing and we’ve been busy trying to hire for a new position for a product manager. I have been busy doing interviews and meeting a lot of interesting folks and trying to find the perfect person to join the team.

Jon: Yeah. We’ve also been hiring mobile software developers and back-end software developers. My days are peppered with interviews but it’s fun to meet so many great people. How about you, Rich? Are you growing your team?

Rich: Yeah. We have a five part-time developers. They’re only part time because I can’t seem to get away and out of the way. Right now we’re really focusing on handing off all three development to two people to get them up to full-time and then we’ll start hiring junior developers. It’s been really a difficult process to me to get away from billed hours and I’m really just painfully doing it the wrong way until I figure it out.

Jon: I’ve definitely been there. You and I have talked a lot about that so I think you’re doing a lot of the right thing.

Rich: Yeah. Just a learning process. I’ll get there eventually.

Jon: Right. Last week we did another talk on tips and tricks from Adrian Mouat’s talk at DockerCon and this week we have, I think, just a couple more to go through. They’ve just been providing such fun conversation that we think it’s worth going all the way through the list. Chris, can you give us just a high-level quick overview of what the talk was about?

Chris: Sure, you bet. This was again a grab bag collection of tips and tricks, just common pitfalls that folks have when using Docker. Adrian is part of the Docker Captains Program, which is a collection of just Docker SMEs experts, working with the community, helping folks in the community adopt Docker, visit some of the common patterns, pitfalls that folks run into, and talk with some of the other Docker captains to see what their top things, top issues that they saw and along with his and put together a presentation of, “Hey, here are some of the top things we see out there and hopefully these are helpful for folks.”

Jon: Cool. The last one we talked about was shutting down gracefully and then it looks like we’re about to be able to get into some security tips.

Chris: Yes, indeed. It’s interesting this security, it just keeps coming up. If we go back and over all the Mobycast podcast, every time we mention security, I’m sure it’s, I don’t know what the percentage is, but it’s definitely a pretty large number.

Security should be top of mind. It’s a wide open area. There’s so many things to consider and in this particular break-out sessions and tips, just at the container level, what are a few things that you can do that help in the area’s security and to harden your image a little bit more, your container a little bit more.

One of the first things they talked about was you can use this read-only flag for Docker to say that when you’re mounting a file system, you want that volume mount to be read-only. That limits the ability for the mutations. It doesn’t allow for mutations to happen and it just increases your security posture there a little bit. If you don’t need to do writing, use the read-only flag on your volume mounts.

Jon: You have a volume mount that’s going to be making some host file system available to your Docker image. Your Docker image is stateless so it doesn’t need to be writing anything to that host file system. Mounted as read-only and the container can read from it and not write to it. Then when the container becomes compromised by some external force, then that external force cannot write to the host file system or whatever other file system got mounted.

Chris: Yeah and maybe to clarify just a little bit, I believe it may be a little bit more involved where the root file system of the container is read-only.

Jon: Is that what’s considered a volume mount?

Chris: In an update, that wouldn’t be. This is full disclosure. I have not used this flag myself so I haven’t had the chance to play around with it. The important point here is you can definitely limit the surface area of mutations. I’m thinking through this why I believe this is more in line with used in making your actual container itself read-only, the file system of the container, it’s not necessarily the volume mount. It’s just the file system of the container itself make it read-only.

It goes along with the fact that, for the most part, a container should be stateless rather than just everything is there in the image that it needs, it instantiates itself, it’s running, and any state it needs is dome outside the container, like making calls to a database as another service or what not.

Jon: I almost like that. It’s not just a security feature but as a best practices forcing function, like, “Hey, guess what? It’s read-only so don’t write to this thing. Go figure out another way to store stuff if you need to store stuff than on the local file system.”

Rich: Yup, absolutely. I think you can use this in concert with another flag for tmpfs. That would be the one case where there’s a potentially real use case for writing to a container file system as if you need just some temp files to work with or to do work.

Jon: But haven’t we learned that could be really dangerous? If you open up any writing at all, all of a sudden you can have a situation where you got a container filling up the entire host hard drive or whatever – without realizing it.

Rich: Absolutely. That opens you up to these programming bugs. You’re just not doing the right thing cleaning up after yourself. There are very real used cases where you need to have a temporary spot to do things. You can have a pipeline for doing image manipulations or video encodings, something like that, where you have intermediate files that need to be output and it just makes sense to write them to disk, passing it through.

Having that capability, it may be required but you can still have this read-only file system that uses this tmpfs parameter as well for Docker to basically mount a temp file system that you can use for those kinds of manipulations while still keeping the root file system read-only.

Jon: Just to argue, just for the sake of argument. That may hinder some of the security benefit that you get from just having everything read-only because I can just mount an attack vector. If I can get access somehow and I can start filling up a file system, then I can take down a whole service even if it’s tmpfs. I can take down the whole service by filling up that tmpfs and then all of a sudden, all the other containers on the host start breaking and things fall apart.

From a security perspective, not being able to write at all is more secure than having a tmpfs. But yeah, I get it. Sometimes, you just need files to write to to do some work. But yeah, any sort of writing that you allow to your mounted file system opens up this security hole that they’re trying to plug with read-only, right?

Chris: One thing would be basically just like the denial-of-service or instability from throwing up a disk. This is why we have tmpfs. Nothing important is in the temp file system, like /etc/passwd file is not there. That’s where this read-only comes into play. The ability to like, “Can you change the password file? Can you add a user account? What if there is a volume mount that’s established?” like somehow, somebody accidentally does it. By having this read-only flag, it would allow it so you can’t make those writes. That’s the real security benefit for doing the read-only option with your Docker container, is to make sure that things like that can’t happen.

Jon: Cool. What we have next for security?

Chris: The second one is don’t run as root. This is a biggy and I would say it feels like most images out there don’t really do anything in this space and so by default they are running as root. Obviously, security best practices is to run the least privilege level that you need. Give that by default, when Docker instantiates a container it’s going to be running as root, that’s something you need to do in your actual Docker image itself, to change that. This is definitely very much encouraged. Set a user in your Docker file. Switch to a specific user so that you’re not running as root.

You’ve seen some of the big major distributions out there make these changes over the last few years. Node.js is definitely very popular. They do this inside their startup, inside their Docker image, they’ll switch to a specific user so that they’re not running as root.

Jon: That’s awesome to hear because that’s exactly where I was going to go. I was just going to make a call-to-action for anybody that’s working on this type of software to not make it the default because that’s where it becomes a problem, it’s like, if you make software developers out in the world, to work harder to do things the right way, guess what? They’re going to do them the wrong way.

As open source developers and as people that work on Unix tools, just making it easier to not do it the wrong way is going to help the whole world. I’m glad to see that Node.js is already taking that on themselves.

Chris: This is very much the responsibility of whoever’s writing the image, who’s creating that container. It’s not a Docker problem. Your image can be any operating system in the world. They could have any kind of configuration. You have two users defined in them, you can have 100 users in it. Docker has no idea who to run it so it just runs. It just says, “I’m going to run this as PID 1,” and it’s going to get root. It’s really up to you to know, you as the application developer, you know, it’s up to you and your responsibility to say, “Okay, no. I’m not going to do it like that. I’ll create this new user account with the right level of privilege and run it as that.” That’s why they give you the user command. That’s actually baked in as a Docker image command so you couldn’t do that.

Jon: Right, makes sense. I think about other things like PostgreSQL, like how, when you install it, it makes you make another user. I get that at an operating system level, Docker is just delivering operating systems. It’s not really their job but I also think that, as a group of people that make software, we should also not leave it in the business application developer’s hands. Someone lower level than that should be thinking about this. It sounds like with Node, they have, but some other base image creators, there are people thinking about this to make sure that base images don’t get out in the wild that have this flaw.

Chris: Yeah, absolutely. It’s definitely one of those things where it’s like, “Hey, we know that this is what we should be doing as best practice,” and when you have these images that are getting million and millions of pulls, and they’re big open source projects with lots of eyeballs on them, you should definitely expect it that they should be implementing these best practices.

Jon: Right. Totally.

Chris: Yes. It’s really for when the case where you are putting together something custom, definitely keep this in mind.

Jon: Cool. All right, what’s next? I think we’re moving on from security.

Chris: We are. After that, there’s just a few more random tips just to make life a little bit easier. Another thing talked is if you use Docker on a regular basis, one of the typical commands you’d be doing is docker ps. That’s a way for you to see, “Here’s all my containers that are currently running and information about them.”

It points out that the default output for this is pretty ugly, it’s not too terribly readable, lots of times it will line wrap as well. There’s just a lot of information that it makes it hard to read and parse. There is a –format command-line argument for docker ps. With that, it’s very flexible and allows you to have complete control over the format and of that output. Given an example, if you just want to have a tabular format with the name of the container along with what image it’s using and it’s status, they’re easy to do something like that and you can do it as a command-line argument for docker ps when you’re running it.

Even better is to change your Docker configuration file to say, “This is what I want my ps format string to be.” For most people that’s in your home directory and in the .docker directory, underneath that, config.json, go in there, add an entry for ps format and put in the value for that command-line argument, that formatting string that you prefer, put that in there and now whenever you do a docker ps, you’ll now see this nice format. A little tip that can end up being quite useful.

Jon: I think it’s stuff like that, that if you take time to just know some of that stuff, it’s just going to set you apart. I just read a tweet three or four days ago that said something like, “I’ve been using Emacs for 10 years and I’m still pretty awful at it,” and it’s like, don’t be that person. Learn these things especially if you’re committed to something like it.

If you’re going to use Docker for more than a year, learn some of these stuff because that’s what really makes the mythical 10x developer, somebody who has taken the time to learn little things that shave minutes and seconds and just make life easier, as opposed to somebody who just learned a couple of things and reuses those same couple of things over and over and over and over.

Chris: I don’t know about you but I don’t know if I’m very sensitive to the messaging now or it’s just become more popular, but this whole idea of compounding interest and if you just get 1% better today or this week and you do that on a continual basis, that all adds up. You end up becoming that 10x improvement after five years, as opposed to not improving.

This is probably definitely one of those things where it’s a little tweak, it’s a little improvement but if you do, if you have a regular stream of these things, they all add up and they do have a compounding effect.

Jon: Right, exactly. Cool. Are we done or was there anything else you wanted to cover?

Chris: In this particular talk, talked a little bit about cleaning it up and how you can get rid of old images, containers, volumes, and networks, we’ve talked about this on previous episodes. Docker now has the prune command, so take good use of that. docker system prune will clean up everything. Run that every once in a while and you will free up tons of space on your machine. docker system prune is your friend.

Jon: Nice.

Chris: Yes. I think that wraps it up. It was definitely a big session with lots of great tips and also sparked quite a bit of conversation with this. We’ve got three episodes out of this, three Mobycast episodes out of it but it was fun and I think it was very worthwhile. Hopefully, folks will take away some favorites from this.

Jon: Great. Thanks, Chris and thanks, Rich for putting this together.

Chris: Yeah. Thanks guys. See you later.

Rich: Take care.

Jon: Bye-bye.

Rich: Well dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with show notes and other valuable resources is available at mobycast.fm/26. If you have any question or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.