24. Docker Tips & Tricks (Part 1)

Rich: In episode 24 of Mobycast, we dive into a new micro series on Docker tips and tricks. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Welcome, Chris, Rich. This is another episode of Mobycast.

Rich: Hey.

Chris: Hey Jon. Hi Rich.

Jon: Good to have you with us today. Chris, you’re back from your vacation. Are you feeling rested?

Chris: That’s a simple question with a complicated answer. TLDR is a fiddy-fiddy.

Jon: Nice. Did you get robbed or anything horrible happen on your vacation or was it a pretty smooth sailing?

Chris: No, not at all. This is the happiest place on earth. We were at Disneyland, so it’s all good.

Jon: Good deal. How about you, Rich? What have you been up to this week?

Rich: I played a round of golf yesterday, or was it yesterday, or two days ago? Two days ago, I think now. Shot a 116. For someone who never record scores, that was a goal of mine for the summer.

Jon: That is just awful.

Chris: You said 116?

Rich: Yeah.

Chris: Was that a basketball score?

Rich: Yeah. Funny thing about––we use the phrase like, “Oh, that’s par for the course,” so literally, par for the course is pretty impossible. Par for the course is like 72 and I shot 116 just to give you a perspective of how bad it was.

Jon: Yeah, I know. Irrespective of how bad it was and how much room for improvement you have.

Rich: I’ve been thinking a lot about it. This is the happiest I’ll be. I’m not any good so my expectations are not really all that high, but I’m such a competitive person that once I get to about 100, I have a feeling that I’m just not going to be happy playing golf anymore because at that point I’m going to have expectations that are probably either unrealistic or unsustainable.

Jon: That’s a guarantee.

Rich: Yeah, that’s what I hear.

Jon: Yeah, and over here, I already mentioned this to you, Rich. It’s been interesting fighting Facebook on behalf of one of our clients. Because of Cambridge Analytica, they are doing some new business verification stuff in order to get certain permissions for their apps. We’re just trying to get a list of the user’s friends, but users can see their friends that are also members of the app that we’ve created—nothing nefarious there—but just getting that permission requires a business verification and Facebook has been the worst.

They just deny every document we send over and we’re sending our real copies of articles of incorporation, business certificates, tax ID numbers, everything, and they just flat out deny everything with no indication of why. I have no idea. Maybe the documents are not even coming through on there and I no idea. No, no, no idea, and I’ve been talking to other people about it, everybody is saying the same thing. “Yup, that’s been happening to us, too.” So, that’s super frustrating. So if anybody is listening and having this same trouble with Facebook, I feel your pain.

But today, we’re going to talk about not Facebook. We’re going to talk about Docker and we’re going to go back to the well of DockerCon talks because these have been fun to review and go deep in. We’re going to talk about one of the tips and tricks. Chris, what was the title of this talk and who gave it?

Chris: The title of this talk was Tips and Tricks of the Docker Captains. It was presented by Adrian Mouat.

Jon: Cool. I have not heard of Adrian. Do you know anything about him?

Chris: Neither have I. He’s a chief scientist at Container Solutions and if memory serves me, he might be from England. So, across the pond, I believe. I could be wrong, too, so I apologize Adrian.

Jon: Yeah, and the tips and tricks format I imagine, was there an overall message to his talk or was it just more of a list of, “Here’s some things that’s been working well for me.”

Chris: Yes. This is absolutely a grab bag session. It was literally just, “Here are a bunch of things that in day-to-day use of Docker that are common things that trip folks up and these may be very useful to you.”

The Docker Captain Program is a program by Docker where they kind of officially recognize experts in Docker within the community, and then they kind of serve as evangelists for Docker and for helping folks out. They’re very active with blogging, giving presentations, speaking, and what not, and really just being the folks that are just experts in this space helping other people learn about Docker and to take better advantage of it.

Adrian is a Docker Captain in that program and this particular discussion was just, I think, based upon his own things that he sees in his work with other folks as well. He also obviously spoke within a group of Docker Captains and pooled their particular experiences and what they saw as like tips that would be particularly useful. Gathering that information, put together a list of grab bag of topics and that was what this talk was based on.

Jon: Great. That’s interesting. I was unaware of the Docker Captains. By virtue of this podcast, we are now assistant Docker Captains. Well, we’re assisting here the Docker Captain.

Chris: Exactly yes, so maybe we can assist. Maybe this might be an audition. This is our audition tape or can we join the Docker Captain Program?

Jon: Let’s just jump right into some tips and tricks.

Chris: Absolutely. As I mentioned, not really an overall unifying theme here. Just a grab bag collection. Starting off with one of the first things that Adrian talked about was volume mounts.

Jon: Our favorite topic.

Chris: Indeed. It’s definitely a very popular common issue where we talked about this before, where it breaks the container purity, we are now sharing information outside. You’re punching that hole inside that container bubble and sharing information with the host. There’s lots of good reasons for doing it, but you also just have to understand what the ramifications are when you do that.

In particular, he pointed out a particular “gotcha” with volume mounts. It’s a little obscure, probably most people won’t run into it, but it is pretty interesting. What is was, was he went through kind of an example of showing this in action.

In his example, he starts a container, and in that container, he specifies a volume mount and the volume mount, what’s being mounted is actually a file and so in this case, he was starting a container running Nginx webserver and the volume mount was pointing to index.html, which is the default home page, if you will, for Nginx when it starts up the display. That volume mount was being shared between the container and the host.

Once that was running, it shows you can go and cat the file on the host that says something like, say “Moby rules.” Now you can curl to your container and when you do that, it returns back the HTML it says, “Moby rules,” and that’s exactly what we expect.

The next thing he did was outside the container on the host, use your favorite text editor to go in, change that index.html, now change it to something like, “Gordon the turtle rules,” save it, and now curl to your container, your Nginx instance on your container again, and what do we expect that’s going to say?

Jon: In the perfect world, we have Gordon.

Chris: Right and just a gotcha. What it does say is it actually just says, “Moby rules.” We changed the file on the host that the volume mount was pointing to, but the container does not see it whatsoever. Any ideas why this happens?

Jon: I just assume that the container is holding onto the file. When it’s a directory that’s fine, but when it’s a file, that’s not fine. There is going to be a cached version of it inside of it?

Chris: Essentially what’s going on is with Docker, the volumes are mounted at the inode level. It literally has to do with the guts of the file system and how it works. It’s mounting at the inode level and text editors are saving to a new inode. That original inode still exists, it’s pointing to the old one when the text editor creates a new version of file, that’s a new inode and it doesn’t see it. That’s why this happen.

Jon: That feels like it’s the kind of thing where it’s like, “Hey, open source developers. Here’s a bug for you. Go get your name on the project. Go fix this.” Like return an error when somebody tries to play with your master file because it doesn’t work, at least warn people.

Chris: Sure. Yeah. Absolutely. It is probably one of those things where like, when it does happen, most people are going to just bang their heads against the wall like, “What is going on here?” just really not understanding it and flaming around a little bit

Rich: Hey, this is Rich. You might recognize me as the guy who introduced the show, but is pretty much silent during the meat of the podcast. The truth is, these topics are oftentimes incredibly complex, and I’m just too inexperienced to provide much value. What you might not know is that Jon and Chris created a training pod to help developers with skill sets get caught up to speed on AWS and Docker. If you’re like me and feel underwater in these conversations, head on over to prodockertraining.com and get on the mailing list for the inaugural course. Okay, let’s dive back in.

Chris: So that’s something to be aware of. Lots of ways to get around this, like obviously you can mount it out at the file level. You can mount it at the parent directory level and then you’ll be fine. You can also change the file other ways.

Again, most text editors are saving to a new inode, but if you do something like just overwrite the file, do an echo or something like that. You can do echo ‘Gordon the turtle rules’ and then redirect it to the index.html to overwrite it. That would update the file as well because the inode is not changing.

Definitely different ways around it, but again, I think the real lesson learned here is just understanding that these volumes are mounted at the inode level. So be aware of that.

Jon: Cool.

Chris: The next thing he talked about would be, he pointed out, just be aware of latest as a tag and there’s nothing special about the latest tag. When you push or pull or build an image, you can specify the name of the image and then you can do a colon, put the tag name. A lot of people will just say, “Oh, I want the latest so I’m going to go get myimage:latest.

The point here is that there’s nothing special about that term latest. It is not guaranteed to be new and it’s not guaranteed to exist even. It’s just the default tag name when you don’t specify a tag.

Jon: If you never specify a tag, then it maybe it works the way you would expect?

Chris: Kind of. This is where one of the more typical gotchas I see with this is when caching comes into play. When you pull an image, the Docker engine on your local machine is going to look to see whether or not that image and the tag that you’re asking for exist, and it’s a local Docker cache. If it does, if there’s a hit, then it says, “Hey, I got it. This is cached. I’m just going to go ahead and use that.”

The problem is that if you’re using something like myimage:latest, you go and build the Docker image on your CI server, publish it as latest, and then you go and pull it onto another machine, if that machine already has myimage:latest because it did a previous build on your pull, it’s going to see it as, “I don’t need to go get it because it already exists.” Even though the one that the CI server built is much newer, latest bits, you’re not going to get it.

That’s why be really wary of using the latest tag and really understand what it means. It’s much better to use meaningful tags. Here at Kelsus, we’ll use semver on our tags so whenever we publish a new Docker image in our CI server, we’re going to use major-minor patch semver as the tag name on that image. You could also use a git hash, a commit hash code as your tag name, but just something that’s unique, so you’re not going to get these collisions that lead to unpredictable results when you run your Docker images.

Jon: It sounds like another opportunity for an open source developer Docker Captain to do something better, just get in there and put in a pull request for better warnings and then maybe the Docker team eventually can look at deprecating the idea of latest because life’s too short to have to remember things like this. You know what I mean?

Chris: I’m not sure why it works this way because it feels it would be a pretty simple for Docker to do like just a hash comparison, so like go and…

Jon: Implement the feature.

Chris: Yeah. Just on the repository when you’re asking for latest, find out, timestamp, or hash code, or something like that and use that to see whether or not they’re different. So truly implement the full caching thing and do the caching at the file level not at the name level.

Jon: Rather than have people not rely on latest because it’s broken, let them rely on latest because it does what they assume that it does.

Chris: Yeah, because it useful.

Jon: Yeah, and as somebody who’s been developing software for a long time, it’s just so frustrating when things like this that are gotchas have to be disseminated through the community as opposed to just fix them as a software. I guess it happens once people start using the software and depending on features, if you change what those features mean or do underneath them, it requires a committee of people that approve and think about it, and all that stuff.

Unfortunately, we have to stop now because we are just up against the wall in terms of other things that we have going on today. Maybe we can come back to some more of these tips and tricks next week.

We’ll leave everybody on the edge of their seats wondering what the next thing is that is going to make their lives easier when they use Docker.

Thanks producer Rich, thanks Chris again for another fun episode.

Chris: Thanks guys.

Rich: Well dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with the show notes and other valuable resources is available at mobycast.fm/24. If you have any question or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.