15. All About Docker Compose

Rich: In episode 15 of Mobycast, Jon and Chris teach me all about Docker Compose. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Chris, Rich, welcome to Mobycast.

Rich: Hey.

Chris: Hey, everyone.

Jon: Good to hear your voices again. This week, we’re going to do a bit more of a technical Mobycast, talk about something technical for 20 minutes, but before we get started I just thought I’d see what you’ve been up to this week. Chris, what have you been up to?

Chris: This week, I’m fighting off a summer cold. Summer colds, they’re the worst.

Jon: Those are the worst. Have you ever had a summer cold and allergies at the same time?

Chris: Maybe. It just feels so unnatural to have a cold thing during June, right? I got the whole winter, not even a sniffle and here it is, June and it’s a full-blown cold.

Jon: That’s pretty brutal. I don’t know. I guess all I could say is I’m glad I don’t have one, but I’m fighting off some allergies. How about you Rich? What have you been up to?

Rich: Work has been good, but the weather in Denver has been pretty incredible. On Sunday, I played my first round of golf ever. Nine holes at a par three. I am at some public course in Denver and on my second shot, I laid it up like seven feet from the hole to putt for a birdie, the second time of ever playing and I still think I scored a six.

For me, I thought I had missed out on this career of golf had and I started, and then I realized that golf is incredibly hard and every hole thereafter was pretty terrible, so I’m back on par with everybody else.

Chris: I’m sorry. Did you say that this is the mini-golf, the miniature golf course and you are on the windmill hole, is that right?

Rich: It was a par three course, so it’s only one step removed from mini-golf, I suppose.

Jon: As for me this weekend, for the second time, I used the new ice cream maker that we got and made some cookie dough ice cream, and really it’s stuck with me in more ways than one, it was so good and it also has just stuck right with me––all that ice cream.

Anyway, let’s get started. This week we’re going to talk about Docker Compose. We talked a few episodes ago about how to setup Dockerfiles for just the best practices when setting things up for development and for use in staging our production and of the tools that we used for that is called Docker Compose.

I was thinking about it a little bit and I realized that while I’ve seen these configuration files and used this tool called Docker Compose, I think that I could use to understand it a little bit better, and fortunately, we have an expert with us here today that can give me a better education, help me understand. I guess the first question I’ll ask Chris is, what is it? What’s Docker Compose?

Chris: Docker Compose is part of the standard Docker tool set that comes with the ecosystem, if you will, and really what it is, you can kind of think of as orchestration for your laptop or your PC and that kind of environment.

It’s a tool that is a wrapper around the various Docker APIs and commands to help make that stuff easier. Instead of manually creating these Docker run commands or figuring out how to do volumes or how to inject aliases or DNS names, they use Docker Compose.

Docker Compose is that tooling around the core Docker runtime that just automates a lot of the stuff for you. It makes it a lot easier, so convenience functions type thing.

Jon: Let me try to unpack that a little bit because I’m not sure I followed entirely. Docker itself comes with a Docker command-line utility that lets you say, “docker start,” “docker stop,” “docker run,” and then it also has this other capability where you can create a Dockerfile that has some stuff in it, that when you do Docker create, it knows, “Oh, let me go and read what is in the Dockerfile. I’m going to make a Docker image based on what’s in that Dockerfile and do stuff based on what that Dockerfile tells me to do.”

Some of the things that it does—this is the question—say for example, there’s a volume mount that’s specified in a Dockerfile. Is it possible to specify a volume mount, both either as part of the Dockerfile or as a Docker command? Say you have a running Docker container, can you say, “Hey container, right now I want you to add a new volume amount and here’s the command for that?”

Chris: After the fact that you started mount the volume? No.

Jon: Mounting volumes happen at creation time. From my understanding, it happens through the Dockerfile specification, right?

Chris: No.

Jon: No? Okay. Help me understand exactly when mounting a volume occurs. I swear I saw it specified in the Dockerfile as, “Hey, point this directory to this directory on the host machine.”

Chris: This is where it probably gets a little confusing. The Dockerfile itself, that is the sequence of commands that tells Docker how to build your image. The advantage is something that is completely portable that will run in any environment running rocker. There is no dependencies there or anything like that.

You’re not specifying that in the actual image itself because it doesn’t even know what host it’s going to run on. Does that corresponding file system even exist on the thing that I am going to be running on? That volume is specified as part of the runtime. It can be this compile time versus runtime. The Docker image is compile time and volume mount is done runtime. That’s done with the Docker run command.

When you do Docker run, you say, “I’m going to instantiate this image as a container.” When you do that, you can specify all sorts of run time settings whether they be networking, or file system through volume mounts, or DNS, IP addressing, and whatnot. All that is specified at runtime. Those are some of the things again that Docker Compose helps makes a lot easier, that it helps with that.

Jon: I think I’m clear on this now. These runtime things are things that probably typically depend on the environment that the container is running in, the host, and they could be sent as flags or as arguments to a run command, but they can also be specified in another way.

You’re saying that Docker Compose is that other way. I can say in a Docker Compose file, I can create that. I can say, “Okay, well whenever I run this container, I want to setup these additional things, so that it runs the way that I want to.”

Chris: Absolutely. Underneath the cover is essentially Docker Compose that is going to be running a sequence of Docker commands, it may be things like––it could do a Docker build, it’s going to do a Docker run, it could do some other commands as well.

At the end of the day, underneath the cover, that is what it’s doing, but then again, instead of you having to manage it purely via the command-line and you need having Bash scripts to capture all these arguments correctly and set them up right, instead you can just have your Docker Compose file much more readable YML configuration and not worry about the specific syntax. It’s just much more efficient and effect.

If Docker didn’t create Docker Compose, the open source community would have.

Jon: Yeah, that makes sense and that’s exactly where my mind was going. I was starting to picture Docker commands with six arguments afterwards, each of the six arguments might have some long value, like a deep directory file structure, a deep pointer to some files down deep in the file system and just incredibly impossible to read.

One Docker run command could be four or five lines long on your terminal. Docker Compose just makes a way for you to specify all that in a readable YML way. Okay, so I get it. I understand what it is now.

Chris: By the way, that’s only part of what Docker Compose does as well. It’s orchestration for your local environment. It really gets interesting when you have multiple containers that you want to be running as part of your environment.

That’s where it would really start getting to be quite cumbersome if you were just doing Docker run commands for each one of these things, what sequence to start them up, and whatnot. Again, this is a problem space that Docker Compose solves for you.

Just about every kind of environment, it’s almost going to have more than one container that’s interesting to use. Even if you just make a standard microservice, chances are you you’ve got got an acting database for that, so you have two containers.

Jon: How is a Docker Compose file organized? What would it have typically from top to bottom? Say, we have exactly that type of situation where we have a container that’s got some application in it and then another container that we just want to have around because it got a database in it that we want to test against.

Is it organized by environment? Like here’s a section for dev, a section for stage, a section for production, or is it organized by life cycle? Like here’s what you do first, then second, the third. How is it organized typically for that type of situation?

Chris: For the most part, you can think of it as there’s sections for every container that could possibly be instantiated. For the most part, that is the base element that composes of these Docker Compose files is what’s description of the container.

You will have things like what’s the name I want for the container? What image is it going to use. You can say either use a pre-built image or go build one. You can specify things like environment variables and those can come either explicitly specified in the Docker Compose file or you can have them in a dock file that it can pull from.

You can specify things like networking settings for that container. We’ve quite talked about it now, obviously volume mounts. We specify that there as well for each one of these containers.

That’s the basic core building block of the Docker Compose file. Basically these container descriptions and say you have one to N containers in a Docker Compose file. There are other associated high-level sections that you have in there.

You might define a custom network for Docker, for your containers to run on, so that might be a top-level element there that you want to tell Docker how to create the network for your container. You might have containers just for data volumes that you want to use for something that’s more persistent across container instantiations, so things like that. But for the most part, you’re basically defining containers and there’s one to N of them.

Sometimes, you’ll want multiples of these running at the same time and startup sequence will matter. Other times, you might define very specialized versions of containers for doing things like running tests. You may very well have a separate container definition just for doing test of the standard container, so there might be some different settings there or whatnot. It makes sense doing that.

Jon: I’m still trying to get my head around––I understand that Docker Compose is going to be running Docker commands for me to instantiate my containers based on my actual post and the way that containers interact with the host is going to be defined there in the Docker Compose file.

What I can’t quite understand is, you had said that it can build them, it can run them, and I’m like, “But where and when?” I got all these container top-level elements. Where am I saying, “Build it,” or where am I saying, “Run it?” How does that work?

Chris: That comes down to just how you invoke Docker Compose itself. Just like the Docker command-line, the executable will take various action flags, arguments, and what not. Docker Compose has that as well.

You can tell Docker Compose to do things like build or you can say, “docker-compose system up” or “down.” Up means “I want to bring up the certain named containers.” I could say, “docker-compose up MyRestfulAPI,” or you could do, “docker-compose up,” and that would say basically putting up all containers that are specified in my Docker Compose file.

When you invoke Docker Compose, you tell it what it is you want to do. Likewise, you can say, “Hey, I want you to stop all my containers,” so “docker-compose down.” Maybe you want to kill all the volumes and remove images when you do that as well, so you, “docker-compose down -v -r MyAll” type thing.

You’re getting away from the Docker command-line interface and its flags, but the Docker Compose itself is very powerful. It has many different options and digging into that is a worthwhile effort.

Jon: Right, so I think a natural place for probably a lot of people’s minds goes, they are are going through this process of learning Docker and after hearing what you just said, it’s like, “Wait a minute. So Docker was going to relieve me of all of my dependencies on various people’s environments and get me away from there.”

Well, it works on my machine problem. But since we introduced Docker Compose, it’s sort of like, “Okay well, the files need to be where they’re supposed to be, the networking needs to work the way it’s supposed to work, and basically Docker Compose is where we put a whole bunch of dependencies on your host.” And it’s like, “Wait, is this breaking the philosophy because we have to share these Docker Compose files.”

The Docker Compose files are checked into GitHub, I think, and since they’re checked into the source code repository, all the developers share the same Docker Compose file which means don’t they all have to have the same essential things on their host machines and the same basic structure of where their files are and stuff?

Chris: Kind of. But at the end of the day, it’s not really any missing dependencies. Usually, things like a volume mount in your Docker Compose file, it may very well be for things like exposing log files, which is a convenience-type thing. You may be exposing artifacts of a gold process like desk coverage reports.

Typically, you specify that stuff as locations that are relative to the home location for your entire project. If you’re working on something like Git and you have the root level of that project in that repo, then in your Docker Compose file you might say like, “My host volume mount is the current directory and then /volume––whatever the name is that you want to write.”

You can build that up and tear it down in your scripts itself. You can definitely define all these stuff but really, it literally should be that there’s nothing anyone really has to do other than install Docker, install Git, do a clone, docker-compose up, and it just works.

Jon: Interesting. I guess it’s still the thing that boggles my mind a little bit. I may have to spend some time thinking about it. It’s just the fact that just where the line is. You just explain, “Well, yes, yes, it does. It does give you the opportunity to introduce some host system types of dependencies, but a smart developer would make sure to make those dependencies relative so that it works on anybody’s machine.”

That jarred my ‘the world is perfect’ kind of happiness towards Docker. Just a little bit, or it’s just like a little bit of, “Uh, you’ve got to make sure you do this right.” It also is sort of like, “Well, if you can do it in there, then why can’t you do it in the Dockerfile?” Why couldn’t I just say, “Hey, Docker’s got an ability to do a volume mount, but it’s got to be relative because it can’t have any dependencies on external systems. But I can at least assume that an external system has a file system and that can point to things on that file system based on my own directory from which I am running in.” I don’t know. There’s just this little bit of cognitive dissonance after hearing this whole story.

Chris: Some things to keep in mind is that, one, volume mounts—I think we talked about this in previous episodes—really, it kind of goes against Docker and the whole isolated, hermetically-sealed container.

We’re punching a hole—if we’re using volume mounts, we’re punching holes—we’re escaping the purity of containerism. Once you open that box, I’m sorry that promise is now gone. You broken the seal. If you now start having issues like, “Oh, it doesn’t work on my machine anymore. It’s not deterministic anymore.” Well, that’s because we broke that hermetic seal.

Another native line is that, definitely in the past, older versions of Docker, they did have a way to specify volume mounts in the actual image creation. It was confusing and I’m pretty sure that this was before Docker Compose existed. It may very well still exist, so maybe some other, physically, for doing this. I just personally have not any use case for it in the last three years, but I just don’t see how it makes sense, but it definitely was in the past and it was very confusing. Once Docker Compose came around and the Docker command-line itself allowed you to specify volume mounts, I believe that’s when it was deprecated.

Jon: Okay, interesting. I can live with this cognitive dissonance. I can understand this build time versus runtime type analogy that we’re using for Docker commands versus what we’d specify and our Docker Compose configuration files.

I think the one last piece that I’m missing is that I know that at Kelsus, we do different things based on what environment that we’re in. We have things that run in test mode or things that we do special for staging, sometimes the things that we do for production. Do we manage any of that in Docker Compose or is that in another system outside of Docker Compose?

Chris: Personally, us at Kelsus, Docker Compose is really for local development only. You can think of it as almost like an orchestrator for your laptop environment. Once we, all right, we’re using other things––We use ECS, other people might use Kubernetes, so you wouldn’t use Docker Compose there.

In that sense, yeah, you’re not managing different environments in your compose file. A related note to this, too, is that, you typically won’t create separate images for each one of your environments either.

But with image purity thing, you have a single image and then you use runtime information to dictate what the per-environment type stuff is, if it’s in staging versus prod, that’s all these environment runtime types.

Jon: Got it. Yeah, this is super helpful. I think in my muddling way of doubting and being confused, I may have gotten to a key point for Docker Compose, which is that it helps you in the dev environment on your machine. That’s a great place to use it so it can save you a bunch of typing or a bunch of weird scripts that have lots of Docker commands in them, so that you can bring up and bring down your environment, build it, and do those things with just quick little one-line commands.

Chris: Yup, absolutely. It’s also super useful as well like in the CI environments, when you’re running tests, those tests require more than one container to be running, Compose is a must-have type thing.

Jon: Right, so your CI environment, it maybe something that way your containers are just running on a bare machine instead of an orchestrated environment like ECS or some other built-up orchestrated environment that has its own way of dealing with containers and images.

Chris: Yup, exactly.

Jon: Got it. Cool. All right. Well, anything else to add before we close off for today?

Chris: No. Other than Docker Compose, it’s a good thing and do it.

Jon: Right on. Thanks for joining us, Chris and Rich. Talk you next time.

Rich: Yeah, thank you.

Chris: See you. Bye.

Rich: Well, dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with show notes and other valuable resources, is available at mobycast.fm/15. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.