18. Securing Containerized Deployments (Part One)

Rich: In episode 18 of Mobycast, Chris leads a DockerCon session roll up. In particular, we discussed securing container-based deployments. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: What have you been up to this week, Chris?

Chris: This week, I’m back on the ground after a couple of weeks of travelling, going back and forth between Seattle and San Francisco. Back in the office and getting it back to a bit more of a normal schedule.

Jon: Excellent. Nice to have you back and available for my everyday questions and queries over Slack. How about you, Rich? What have you been up to this week?

Rich: I’m leaving for vacation a week from today. Heading back home to spend some time on the beach. I’ve been thinking about how this is probably the first time that I’ve been a week away from vacation and haven’t been scrambling to get all the things done. I guess it’s just like a hat set to putting process together. I’m actually seeing them work out so it’s a weird feeling because I’m sort of like, “What am I missing?” I’m so used to things just being not in line with this type of thing. I’m waiting for the thing to pop on.

Jon: You don’t think we don’t have a list a mile long and things we’ve been waiting on for you?

Rich: Well that’s probably true.

Jon: A lot of vacation.

Rich: That’s probably true.

Chris: I’m going to call you five days from now, Rich, and see if you have that same sense of tranquillity.

Rich: I don’t think I will. Things have already started to pop up where it’s like, “Okay, now we’re getting back into the more chaotic life that I’m used to.” But it doesn’t feel as bad and I don’t know if that’s because of process or because of the calluses that you develop over time and you just handle stress a little bit better. But one way or the other, a week away from vacation, I don’t feel like everything’s going to implode on top of me so that’s a good feeling.

As we’re having this podcast, we’re launching one of our largest sites and that’s not stressing me out either. I guess things are good, whatever they are, they’re good.

Jon: Excellent. And then I have a brand new mountain bike, it’s so good, it’s called Ripmo, from a company called Ibis. I’ve been on it every single day since I got it except for one, I’ve been loving that. Also, I wanted to say, this is for you Chris actually, I just discovered that the coffee shop that I’ve been pooh-poohing because I felt like it was too urban for mountainous Eagle, it turns out they have just such good coffee, just so good. It’s called Color Coffee in Eagle here. I’ve been looking the other way because we have another coffee shop called Yeti’s that I’m pretty loyal too. They have great coffee too.

I just didn’t think that color coffee had a real mountain vibe but I’ve been going there to meet a friend before bike rides. Two things happened. One, I noticed oh, I recognize that ultra endurance runner and a huge picture on the wall. That’s kind of cool and local. And then the other thing that happened was the barista guy. He was like, “Oh hey, didn’t I see you on Abraham’s ranch running your bike yesterday?” It turned out yes because he was bombing down the hill, so fast running. I guess maybe it’s legitimately pretty mountain-y. Now, I think I’m going to end up going there maybe even more often than Yeti. It is very good. There’s my commercial for color coffee.

But, here we are, already spent five minutes talking about other stuff. Let’s talk about Docker. Let’s talk about containers. Last week, we talked about DockerCon. Chris was there. The tone last week was like Docker is a changed company. It’s not the source of all new innovation in the container world anymore. That’s not the only thing that happened to DockerCon. Some of the sessions Chris has said were very good. This week, we’re going to dive in. We’re going to go into a topic that we’ve been avoiding at all cost because it’s just, I’ll be honest, it’s not our main focus at Kelsus We definitely like to write secure software but we are not a security company per se.

But yeah, Chris is going to talk about in one of the sessions related to security and Docker so maybe he’s going to launch us off somehow. Chris.

Chris: Sure. One of the things I really do like about the DockerCon conference is that the actual breakout sessions, the technical content is across the board. It’s top notch stuff. The speakers that they have there, doing these breakout sessions are usually just absolute experts. They just really know what they’re talking about and they are able to go very, very deep technically, ranging from pragmatic information to almost academic level treatment of the underlying technology. This DockerCon didn’t disappoint with that.

I will say one of the hallmarks of DockerCon is that each one of these sessions is about 40 minutes long and they probably cover three hours worth of material in that 40 minutes. And so, it literally feels like you are drinking from the fire hose of expert knowledge that’s just coming at you. It’s pretty fast and furious for the most part. I did my best to jot some notes down as I was attending these sessions.

As Jon mentioned, one of the sessions that are pretty interesting was around security, as it relates to containers and how that might fit into your overall software development life cycle. This was a session called, Don’t Have a Meltdown: Six Practical Tips for Securing Your Container Based Deployment.

Obviously, a little bit of a nod there to the meltdown inspector, vulnerability and issues that happened last year, I believe it was, at the end of last year.

Jon: Good memory, I would not have caught that but maybe they mentioned it during the talk.

Chris: They didn’t, pretty interesting there, definitely very timely. Obviously, security is at top of mind for anyone in the industry, it’s an ongoing concern and whatnot. This was an interesting one. It was given by Justin Cormack, who is an engineer there at Docker, also, Liz Rice, she’s a technology evangelist at Aqua Security, one internal Docker person and one external partner giving this talk.

I’ve seen talks from Justin before. Justin is super hardcore technical guy. I believe he came from a company that Docker required that ended up becoming Docker desktop for Mac. I’ve been to sessions where he’s gotten really deep into networking and actually, security as well, storage, device drivers and whatnot—definitely very impressive on his breath of technical ability.

This particular talk, what they did is they approached the topic of security, trying to make it analogous to a typical DevOps pipeline and trying to line up basically what you can do at every one of these stages of your pipeline from a security standpoint. They specifically talked about six stages of that software development and life cycle and how security applies to it.

The six stages are code, test, build, host, run, and observe. These all align with what we typically see in a life cycle. First stage, you’re coding up your application, you’re writing the code, what security aspects do you want to take into account when you’re doing that. After you’ve coded, you’re now testing. What can you do during that testing phase. Then, you need to build your actual artifacts, your binaries, what can you do there. Then, you need to run your code.

On host, how can you secure and harden those hosts. And then there’s the actual execution environment itself. What kind of options or configuration you’re going to do for actually running that container on that harden host. And then finally, the observers is basically that ongoing run time analysis of what’s going on with your code and what you can do as far as things like intrusion detection or anomalous activity type thing or whatnot.

Rich: Hey this is Rich. You might recognize me as the guy who introduced this show but is pretty much silent during the meeting and the podcast. The truth is, these topics are often times incredibly complex and I’m just too inexperienced to provide much value. What you might not know is that Jon and Chris created the training product to help developer’s well skill sets get caught up to speed on AWS and Docker. If you’re like me, and feel underwater in these conversations, head on over to prodockertraining.com and get on the mailing list for the inaugural course. Okay, let’s dive back in.

Jon: Question, Chris. Did they throw around the word DevSec at all?

Chris: In this particular session, I don’t think they referred to that in particular. I think I heard it periodically throughout the conference. It’s one of those things that kind of feels like it’s still a bit new and people are still trying to figure out exactly what that means as opposed to something like DevOps, it’s something that’s pretty straight forward now. We’ve been living with that for a while and just about everyone agrees what that means.

Jon: Right. To just take this shorthanded, I like the term because it puts the responsibility of security back on developers and it ties in to what I’ve said at the beginning of this episode, which is, “We’re not a security company per se, but that doesn’t mean we’re not responsible for security,” and it doesn’t mean that our feet aren’t held to the fire for making sure that our software that we run is secure.

The talk itself seems to be about teaching developers how to do things all the way through in a secure way.

Chris: Absolutely. Again, these are the practical tips that you can do in every one of these stages of your software development life cycle. Maybe with that, we can dive into that first step, which is the coding phase, if you will, and what kind of things you can do there. They specifically pointed out the two main things you could do there. One is you could take advantage of code reviews. We can talk a little bit more about that.

The other thing they mentioned was taking advantage of static analysis tools. The static analysis tools are definitely something that just about everyone building software is using. Whether it be in the form of a linter or sometimes, doing metrics like code complexity metrics or whatnot.

Jon: GitHub is suddenly starting to email everybody. They’re doing their own static analysis on your code and letting you know about it, which is a service to the world, really. Have you not been getting those from GitHub?

Chris: I have not received any emails about that.

Jon: I’ve been getting this. Once a week, I get a security update from GitHub that just goes to reach in my repositories on GitHub and tells me what libraries are in there that have security vulnerability and what I should do about it, it’s pretty awesome.

Chris: In one of these later phases, we’ll talk about probably that in a bit more because there’s a difference between the static analysis tools that are actually looking at the code that you’re writing, interpreting it and trying to find patterns or idioms that may make it more vulnerable from a security standpoint. And then, you have the module package scanning or even binary scanning type of thing where there is published list of known vulnerabilities like CVE List and BILCO. You can do scans against that type of thing. There are lots of options in that space as well from a lot of different vendors.

Most of the tools out there were used to are more along the lines of linting or enforcing code style, maybe sometimes finding simple bugs and whatnot. They talked about a particular one called GAS, which is a static analysis tool for GOAPS. This one was interesting and that it was actually looking for problematic code dealing with security issues, whether they be things like SQL injection, being vulnerable to SQL injection.

It was definitely much more focused on security issues with your actual code that you’re writing, a pretty interesting tool. It is for GO. We are not really ourselves writing any GOAPS but it’s something that’s definitely on my to-do-list to see if there’s analogous tools out there for things like Node and perhaps Rails as well—pretty interesting tool. Definitely go check that out.

Jon: Very cool.

Chris: The other thing they talked about was code reviews. I’m definitely a firm believer in this. This is something that we do here at Kelsus, we’re pretty consistent with our code reviews and have other folks on the team look at the code that’s being committed and always keep an eye towards the security posture of that code and making sure that we are doing the right things. They just reinforced this point. Definitely, that’s a good place to do that and to make that part of the culture, that when reviewing the codes, it’s not just for the—is it conforming to what the feature’s supposed to do or is it robust but also, how secure is it.

Jon: Great. I have a logic sandwich that I always keep in my mind for code reviews, which is either on one side, “Oh, you’re so smart that you don’t need code reviews. Wouldn’t somebody else benefit from your intelligence?” That’s the one side. The other side is, “Oh well, you still have stuff to learn. Would you benefit from somebody looking at your code?” There’s no winning. You can’t be too smart or too dumb for code reviews.

Chris: Absolutely.

Jon: Okay. Cool. What’s next?

Chris: We’ve got through the coding phase. The next step would be testing. In addition to doing maybe your unit testing or integration testing, actually think of security testing as its own discipline. One example they gave was there’s a tool out there called test SSL. This is a tool out there that helps automate, does a bunch of test against code making request over SSL, TLS. It’s looking to see what versions of TLS that your code is allowing. Does it allow TLS 1.0, which is definitely vulnerable and your code shouldn’t be using.

This is just one of those examples of why not integrate this with your entire process. It’s like these tools exist out there and treat security testing just as important as other kinds of testing, whether they be unit or integration testing.

Jon: What’s the name of that tool again, Chris?

Chris: It’s called test SSL.

Jon: Is it also for GO or is it a drop in for any environment?

Chris: No. It’s a collection of bash scripts, it’s on GitHub, it’s open source. It’s actually been around for a long time. It’s got a long robust commit history to it. It’s stand alone and it’s totally language agnostic. This particular tool is testing the SSL and TLS protocols type of thing. It’s just making connections over those protocols to power you, whatever is it that you want to go and test—very much agnostic to platform.

Jon: Super interesting. Of course, my mind is immediately going, “Can you wrap that into your CircleCI testing pipeline or is it something you have to do separately?” It would certainly be nice if it could just run automatically.

Chris: It’s just a tool. You can absolutely wrap it with a script of your own to kick it off and to configure it and whatnot so there’s absolutely no reason why this can’t be a part of your automated testing done by your CI platform and just make it part of that thing. Just as we automatically run unit test and integration test, as we’re doing our builds in CircleCI, you could also have another phase for—now, we’re going to do our security testing, and so let’s go run test SSL against our Node backend service type of thing.

We do have a little bit of issues there. For us, it’s interesting because we do TLS termination at the ELB since we’re on AWS. We’re actually not doing TLS at our microservice level. Although, it’s on the to-do list for us to change. Instead of terminating the ELB, we want to go towards being encrypted in flight throughout the entire data scenario and use private certs on our microservices. Once we do that, then this would be very applicable.

Jon: Yes. Okay. That’s interesting. After just having spent a little bit of time arguing with somebody that is, “Okay, it’s okay. We can terminate at the ELB.” I’m interested to see how much actual work it is for us to not do that and whether we can streamline that and make it an easy thing to do for all of our clients because we certainly love being able to use AWS certs and putting those in. You can’t give them out of AWS so putting them into your own code onto your own container could be tricky.

Chris: True. As we’ve talked about before, AWS is constantly innovating. Their certificate manager now does private certs.

Jon: We had this conversation four or five months ago. Maybe it was even longer ago than that because time seems to have compressed for me but that certainly wasn’t the case always. I’m stoked. That’s cool.

Chris: Of course, we still have the problem. If you’re running on CircleCI, you’re not running inside AWS so how do you get that private cert?

Jon: Right.

Chris: There are some issues there but all those things are definitely doable. I think there’s a straightforward path for doing it without going through too many hoops. It’s all good.

Jon: It is more secure to terminate at the service instead of at the ELB. As much as you try to make a nice argument that terminating at the ELB is just fine, it’s nice to have everything all the way to the code be encrypted.

Chris: Yup.

Jon: Maybe we can just stop it there. We just finished talking about coding and testing, from doing it the secure way. Next time, we can move on to the other four on the list.

Chris: Sounds good.

Jon: Thank you for your time, Chris. Thanks for your time, Rich. We’ll talk to you next week.

Chris: Alright. See you guys. Bye.

Rich: Bye.

Dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with the show notes and other valuable resources is available at mobycast.fm/18. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.