93. Psst… Secrets handling for cloud-native apps – Part 1

Stevie Rose: Applications frequently need access to sensitive data, such as database credentials, API keys, passwords, and tokens. Of course, we can’t just store these secrets in plain text or hard-code it into our applications. Rather, we need to securely protect this sensitive information to ensure that only those with a need to know basis can access it. In this episode of Mobycast, Jon and Chris kick off a two-part series on handling secrets for your cloud-native applications. We discuss various approaches to secrets management ranging from basic roll your own techniques to fully managed solutions. We explore some of the most popular options out there and help you decide which one is best for you.
Welcome to Mobycast, a show about the techniques and technologies used by the best cloud-native software teams. Each week your hosts Jon Christensen and Chris Hickman pick a software concept and dive deep to figure it out.

Jon Christensen: Welcome, Chris. It’s another episode of Mobycast.

Chris Hickman: Hey, Jon. It’s good to be back.

Jon Christensen: Good to have you back. Happy New Year to everybody listening. This episode, we’re recording it just before New Year’s but we know this episode is going to drop on New Year’s Day. So happy 2020.

Chris Hickman: Yeah, where did 2019 go?

Jon Christensen: Things are going pretty fast in Mobycastland. I mean, we don’t know how your New Year’s Eve was, Chris, but did you have any plans that were going to make you feel tired today?

Chris Hickman: Yeah. So we have a family tradition now. We used to go ice skating on New Year’s Eve, and we’ve kind of since changed that after we kind of realized that the ice skating at the fake rink at the shopping mall was … The skates are so painful, so we switched. So it’s now bowling, which we kind of discovered it’s a heck of a lot of fun. We do the bowling thing, and then we have a favorite movie that we watch that’s three and a half hours long. It’s called Mad, Mad, Mad, Mad World and it’s like a total encore cast of just the biggest comedian names of the ’50s and ’60s, just a really fun movie. So we watch that once a year, New Year’s Eve, three and a half hours, which lasts us right around until midnight. We stay awake till midnight, and then it’s beddy-bye.

Jon Christensen: Right on. Wow, that was really interesting, I did not know you had those traditions. I thought you were going to say something like, “Oh yeah, probably be going to go to bed early.” That kind of thing, which is, unfortunately, I think that that is my story for this year. Traditionally I like to go see live music on New Year’s Eve, but I just didn’t get around to getting some tickets to a show. Nothing happening in Denver or up here in the mountains that really caught my attention. I’ve often gone to see the String Cheese Incident, which is a kind of jammy bluegrass-y band, but I’ve done it enough times that I just didn’t need to this year. So who knows, maybe I’ll be watching Mad, Mad, Mad, Mad World.

Chris Hickman: It’s a good one, it’s a lot of just classic lines in it. It’s just a great … Jonathon Winters and Spencer Tracy and Peter Falk. I mean, there’s just so many folks in that. Mr. Howell from Gilligan’s Island.

Jon Christensen: Nice.

Chris Hickman: Yeah. It’s great. And then on New Year’s Day, the tradition continues with making homemade bolognese sauce.

Jon Christensen: Oh yeah?

Chris Hickman: Yeah. So takes like four hours to cook, but it’s so good.

Jon Christensen: We’ll be over.

Chris Hickman: Yeah.

Jon Christensen: Hop on a plan to have some of that.

Chris Hickman: It’s only two and a half hours.

Jon Christensen: Yeah. All right, Chris. Let’s talk about software. So today we’re going to talk about secrets, and we’re going to talk about how to manage those secrets so that only your app can see them, do you want to kick this off?

Chris Hickman: Sure. So secrets, sensitive data, this is something that our apps typically almost always have a need for, right? Whether it be a connection string to connect to a database, API key credentials for talking to upstream dependencies or whatnot. So in general, just about any kind of application will have this need, so there’s secrets, right? There’s sensitive data, and we don’t want to just store those secrets in plain text, we certainly don’t want them hard-coded. We’ve all seen the big faux pas of someone accidentally committing something to GIT, and then it’s like, “Okay, that’s now gone forever, we need to rotate that credential because it was accidentally committed to source control.” So, we need to have a good, proper way of handling these things, and so-

Jon Christensen: And Chris, I wanted to also add, I mean, just for people listening, if you were developing software in the late aughts, early teens, things have changed in terms of how you’re supposed to handle secrets. So this is an important episode for you if you’re doing things in that sort of style which tended to use a lot of environment variables. And then, I guess, the other thing is maybe you’re listening and you have a startup and you went really fast, and just built a bunch of software super fast, not thinking so much about security, and it’s time to do things right, and you’re kind of looking around for best practices, this episode is for you too. Or just anybody else that wants to know how to do this right in a repeatable way, that once you set it up once, all your apps can use the same way and it’s not a lot of overhead. Okay, let’s keep going.

Chris Hickman: Yeah, absolutely. So okay, we kind of agree, we have sensitive data, we really do need to be responsible in how we protect that, we don’t want it in plain text, but there’s lots of different techniques for dealing with this, ranging from really simple roll your own, to more advanced solutions. So we’ll talk a little bit about that, you mentioned environment variables, that has definitely been a tried and true way of dealing with secrets, part of the variation in there becomes how those environment variables are declared, and when, and that kind of gives rise to what is the surface area of exposure for the secrets, as well. There are potentially some issues with environment variables with containers, just by virtue of how containers are implemented. We talked a bit about this on the VM versus containers mini-series. In general, though it’s not a huge issue, right? Because at the end of the day, with containers, environment variables do end up … You can see them as a file on the actual disk of the host, but you’re going to need root access in order to see that.
So it’s potentially a leak and you’ll hear about environment variables as potentially being leaky, but there’s some pretty severe concerns on that. So in general, the consensus is that you can use environment variables for your secrets with containers, you just need to do it in a measured way.

Jon Christensen: Cool.

Chris Hickman: Yeah, so, kind of maybe going forward a little bit with just secrets. So, we definitely want to have some comprehensive secrets management solution, we want to think it through and think about, we need to support different access patterns across container environments. So, you may have different environments, you may have to look at like dev or staging, and then you have your prod, right? So you want to be able to deal with those kinds of access patterns, you want to make sure that you do have isolated access to secrets so that it’s done on a per container basis, or per application basis and not at the host level, right? So you want to limit, again, the surface area, in that principle of the least privilege, so these are all things we want to think about as we come up with, “Okay, how are we going to deal with secrets, and what’s our management solution to that?”

Jon Christensen: Right, and it goes even deeper than that. I was just working on a proposal for a company where we would be building them some software, and the software we use would need some access to secrets. But essentially the team wanted to be able to keep those to themselves and not give another team the access to the secrets. So they came up a with a fairly clever way to give us access to something that wouldn’t reveal those secrets, but essentially give us the means of giving a pointer so that we could pass that pointer on and say, “Hey, this is the thing that is going to turn into the secret at run time.” So yeah, it’s such a big thing, like can everybody see them in terms of people? And then can everybody see them in terms of systems? And the answer to both should be, generally no. Unless you have a small team, then sure, yeah, probably everybody can see them. And then, yeah, definitely in terms of systems, it’s better if not all the systems can see all the secrets all the time. Just too much surface area.

Chris Hickman: Yeah, again, principle of least privilege definitely applies here.

Jon Christensen: Exactly.

Chris Hickman: Yeah. So some other guidelines we should be thinking about as we put together our secrets management solution is, we’re talking about the plain text, not a good thing, right? So we want to encrypt the secrets, so when they’re stored they’re encrypted, we also want to think about using multiple encryption keys so that we can limit the blast radius as well, in case an encryption key is compromised somehow. I mean, it’s a bit of extra credit, but it’s not too hard to do, so why not do it? So, consider having it probably on a per application basis, right? Different encryption keys.

Jon Christensen: So you’re saying, just to make sure I understand, a single secret is encrypted multiple times with different keys and stored someplace, and then you get to pick your key when you want it, and any of those keys will work?

Chris Hickman: Just think of it as you can have an encryption key on a per application basis, right? So-

Jon Christensen: Oh, okay, got it.

Chris Hickman: And maybe it’s a per application/per environment basis, right? So you have an encryption key for your login service in dev, and you-

Jon Christensen: Did you understand what I meant when my confusion, it sounded like … Did what I said make sense?

Chris Hickman: Yeah, I think you were thinking that it was, “Hey, we’re going to encrypt a value N times for N different consumers, and each one of those encryptions will be done with their own personal encryption key.”

Jon Christensen: Yeah that or even, “We’re just going to do it N times because we can and then we can kill any of those keys, and then the encrypted data is already sitting there ready to go with another key.”

Chris Hickman: Yeah, I mean, so that kind of gets into a little bit, I think the use case around something like that is access control in monitoring and auditing. So we’ll get into that more in the actual talking like what’s the implementation landscape look like and the various tools and services you can use? And there’s the other ways to get those kind of benefits as well, but I mean, just in general on principle, so we want to encrypt and then when we encrypt, think about having more than one, don’t just have one key for your entire organization, right? Have multiple keys, right?

Jon Christensen: Right.

Chris Hickman: So if there is some sort of breach or if you do need to revoke that, then the scope is limited to what it effects, right?

Jon Christensen: So how are we doing on the blast radius drinking game I think every time we say blast radius people have to take a drink?

Chris Hickman: So, yeah, it’s blast radius and then definitely undifferentiated heavy lifting, and I think maybe the acceleration rate of technology is changing very rapidly, right? Those three things, and then of course one, just drink if we say virtualization container, but-

Jon Christensen: There we go.

Chris Hickman: Yeah. Some other things, some are like just think about rotating your secrets often, right? So, this one is kind of interesting because here you can make a distinction between passwords and secrets. So, for user passwords, which we’re not really talking about here, right? We’re talking about, this is more for services and applications that they’re using. But just maybe as kind of an aside point is that we have these policies where like, “Change your password every 30 days.” Or something like that, and when the user has to pick the password, they’ve kind of shown that that actually ends up being less secure by having them change their password frequently because they end up choosing really bad passwords. So, instead, you want to pick something that’s just … If you pick a really strong password, then the only reason to change it would be if it’s compromised to you suspect it to be compromised. But with secrets in general, if you are generating the secrets automatically using high-quality techniques for making sure that you have a very strong secret, then rotation of that makes a lot of sense, right? And that’s what we’re talking about here, so-

Jon Christensen: Yeah, but where’s that’s difficult, Chris, is API keys and other things that you get from third parties, they may have an API for getting a new secret from them, but maybe not.

Chris Hickman: Yeah, I mean, so you do what you can, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So if you have the ability to rotate, then do it, if you don’t really have an ability to rotate, then, I mean, your hands are going to be a little bit tied there. But looking into it, you’ll probably find there are, especially for the most sensitive secrets, I mean, in general, API keys, they really shouldn’t be used for authentication. There should be something else in there as well that’s helping out with it, like for a really robust system. But then you have things like database credentials, right? I mean, that’s a pretty sensitive secret, and something like that should definitely be rotated and then you should have the ability to do that with whatever system you’re using.

Jon Christensen: Cool, yeah.

Chris Hickman: Yeah, and then just some other-

Jon Christensen: How often … Sorry, but how often is often? Does it kind of depend on how big your team is, and the sort of target size of your company? So, you might have a small team but you’re working on health records, so maybe you should do it once a quarter, or you might have a small team and you’re doing fruit quality. You have a fruit quality app, and not that secret of stuff, so maybe don’t rotate that often at all.

Chris Hickman: So, I think just couple points there would be one, that rotation of the secrets should be automated, right? So if it’s a manual process, then it’s probably a big problem. So you should be, from the get-go thinking about, “Okay, what is the automated solution look like?” So once you have that in place, the frequency ends up becoming almost it doesn’t really matter, you just pick something that, it really doesn’t matter at that point. So pick every 30 days, right? Or every week, or every two weeks, once you have that automated system in place for doing that rotation, it ends up being what is the window of vulnerability, right? Is how often you rotate. So if you’re rotating once a year, and it’s some sort of sensitive data key for API access, right? If you’re only rotating that once a year, and it gets compromised, and if you don’t have the auditing in place to even know that it’s been compromised, well then it’s going to be a year until it gets rotated, right?
So, rotating more often, that’s one of the big reasons for doing that, and again if you have the process in place and if you have put in the automation for this, then rotating more frequently should be just fine. There are systems that essentially rotate every second, I mean literally thousands of times a day type thing.

Jon Christensen: I want no more than one second worth of potentially fraudulent, fruit quality data.

Chris Hickman: Yeah, exactly, yeah. So I’d say yeah just really think it through like, “Okay, how am I going to do my secrets generation and then the rotation? And then what automation can I put behind that so that I don’t make this a manual process?” Because if it’s a manual process, you’re just not going to do it, especially if it’s more than a handful of things, right?

Jon Christensen: Yup. Well, and there’s so many things here in setting up an app the right way and thinking about things like automatic secret rotation. It can be hard, especially with a greenfield project or a startup to say, “Yeah, we’re going to start in on this, and we’re just kind of getting to a log in screen that does something because we put all of the non-functional requirements upfront, it’s going to take us six weeks.” That would be really hard for most business people to handle, so some of this stuff has to get rolled in over time depending on the type of company that you have.

Chris Hickman: Yeah, absolutely, and I mean, you don’t have to do everything all at once, right? But at least you should know what it is that you should be doing and you should plan for that, right?

Jon Christensen: Right.

Chris Hickman: And just know that you need to do it, right?

Jon Christensen: Yeah.

Chris Hickman: Don’t get two years down the road and then find out, “Oh yeah, secrets are just distributed throughout the entire system, they’re using some proprietary hashing algorithm and hard-coded into CONFIG files.” Or something like that, “And it’s going to take us weeks, and weeks, and weeks.” Right? To-

Jon Christensen: Yeah, solving that problem is way more expensive than doing what we’re about to explain.

Chris Hickman: Yeah.

Jon Christensen: Yeah, doing what we’re talking about in these episodes here.

Chris Hickman: Mm-hmm (affirmative), yeah, so I mean, you could just say, “Okay, I don’t have time to do it right now, we’re just starting up with a project. But at the very least we can have a proxy API for just, we know we’re going to have centralized secret management, right? So we’ll just kind of basically mock it out, sub it out with some place holders, and that’s just how we do the secrets to start off with.” And it gives that path, right? To go forward.

Jon Christensen: Yeah.

Chris Hickman: Yeah. Okay, so we’ve talked about secrets, we’ve talked about okay, what are some of the responsible ways of doing this and some of the key characteristics of a solution? I thought it would be kind of interesting to first talk about, well given that we just talked about this and what we’d like to have, what if we talked about just a really simple method? Like if we wanted to roll our own because there’s a lot of really good solutions out there that are managed that you can just wire up and start using, and definitely we’ll be talking about those. But I thought it would be kind of interesting to first talk about like, “Well, what if we just kind of rolled our own? Given some of these characteristics, how would we go about doing that?”

Jon Christensen: Cool.

Chris Hickman: And this is based upon something that we’ve done in the past before these other systems were available, and we can kind of talk through and reasons like, “Okay, how secure is this? What requirements does this meet that we’ve talked about? And then also what’s the potential attack surface that we have to protect? And what’s the mediation if there is a breach?” So given that, let’s talk about a simple and we’re just going to use basically encryption, decryption, and someplace to store. So, because we’re big fans of AWS, right? And we use it, we’re going to use S3 for storage, and we’re going to use KMS for encryption and decryption, right? So this sense of being a really straight forward, simple way of handling secrets, and what we’ll do is whatever our secret is, we’ll use KMS to encrypt it, the output of that is some ciphertext. And that ciphertext can then be stored in an S3 key, right? So we have some bucket somewhere and we can have keys, and we can partition our namespace there with prefixes to break it up on a per environment basis. So we have dev stage prod buckets, part of the prefix underneath their bucket.
And then for each application, maybe there’s another that we have, that’s part of the prefix, and then finally the key itself, right? Where that is the actual ciphertext, so what we’re storing on S3 is ciphertext, right? It has been encrypted using KMS, and so if anyone were able to read that file, it would just be gibberish.

Jon Christensen: Right, and the amount of effort to do that is you’ve got to make an S3 client, you’ve got to make a KMS client, you’ve got to get a key from KMS and then take your text that you’re going to put in S3 and encrypt it. So that’s just a couple API calls that you’ve made so far, and then once you encrypt it, you make another API call put your object into S3. So a total of four API calls, and then, of course, the air handling around each of those API calls. So, not too bad?

Chris Hickman: No, not at all. In fact, it basically ends up you can do it in Bash in about six lines of code type thing, right? Or, I mean, from the command line you can do it-

Jon Christensen: So in Bash, you’d be using the CLI to do it?

Chris Hickman: Mm-hmm (affirmative), yup.

Jon Christensen: Okay, cool.

Chris Hickman: Yeah, you bet. So, we have our secret, it’s encrypted, stored in separate text in S3 and then so now whenever anything needs to access the secret, right? It’s going to retrieve that ciphertext from S3, so it was going to be doing an S3 read, get item, and then once it gets that, then it can use KMS to now decrypt that ciphertext to recover the secret, right? So, again, just really pretty simple, but we’re using KMS for that strong encryption, for that symmetric encryption, which we talked all about in our encryption mini-series back in episodes, what was it? Wow, it was 73 through 76.

Jon Christensen: Okay, wow, you must have that window open because-

Chris Hickman: Yeah. I had to go through the RAM database in my brain but had to do some page invalidation’s to make room for it.

Jon Christensen: So that’s all pretty straight forward, Chris. But when it doesn’t work the first time it’s going to be because you didn’t have the right rolls available for whatever the code is that’s doing the encryption or decryption. So, got to make sure that there are roles in place that allow it to get access to S3, the bucket that you have, and allow it to get access to the key that you need, so some rolls have to be set up for this to work too.

Chris Hickman: Yeah, and that’s one of the strong advantages of this technique, right? Because this does give us a couple of different vectors for really locking this stuff down. So there’s really two resources here, right? One is S3, and one is the KMS, the CMK, so the customer master key that we’re going to use for encrypting and decrypting. So we can have policies on the S3 bucket key to lock that down with IAM on which rolls or users are able to even read that information, or read and write that, right? So you can lock that down, we can also use things like VPC endpoints to restrict access to come from within the VPC itself. So if the traffic isn’t originated from inside the VPC, then it’s denied, so we can really lock S3 down and just make-

Jon Christensen: You can make per environment, per application rolls?

Chris Hickman: Mm-hmm (affirmative), yup.

Jon Christensen: Yeah?

Chris Hickman: Mm-hmm (affirmative). So we have that capability of just being able to read and write ciphertext, we can lock it down, and then, of course, ciphertext, you can’t really do anything with that, right? You need to be able to decrypt it, and so to decrypt it you need to have access to the CMK, the customer master key that was used within KMS for encrypting it. So again, that’s another IAM policy, and so specifying users rolls that have permission to do that, and you can further lock it down by what the actual keys are. So if you have multiple keys, multiple customer master keys, like we said, “Hey, you shouldn’t have just a single key for encryption to your secrets, you should limit that blast radius and have multiple-”

Jon Christensen: Drink.

Chris Hickman: So you may have a key-

Jon Christensen: Sorry about that everybody, it just came out, it just came out, it what you do when you’re playing a drinking game is you say drink, but we’re not actually advocating any kind of drinking [inaudible 00:26:31].

Chris Hickman: Well, and it’s New Year’s Eve, right?

Jon Christensen: Right.

Chris Hickman: So there you go. So, slam your Cristal. Yeah, so we have the ability to just really lock it down on the KMS side as well with IAM policies, so it’s two vectors to really control access to these secrets, and if we just spend a little bit of time getting those policies set up correctly and just thinking through who needs access to it, we have a pretty secure solution, right? And then if we do need to revoke, or an application doesn’t need access to it anymore, then that’s just updating the IAM policies. So this has been a technique we’ve used in the past, it’s worked really, really well, again, it’s simple and it does take off some of the key guidelines or requirements that we need for securely storing our secrets and using our secrets. But it’s not full-featured either, right? So there’s nothing here with support for automatic rotation of secrets, how do you do that?

Jon Christensen: Right.

Chris Hickman: This is using KMS directly for the encryption/decryption, so that means we’re going to be limited to 4K in size for secrets, which probably is not going to be a probably for you, but it might. And so-

Jon Christensen: One thing that goes unsaid, that’s good about this and good about all of the secrets management options that we’re advocating for is that if you kind of get the secret as part of your code initialization. So it starts up and at run time it goes and gets the secret out of S3 and decrypts it with a key. That’s all just in memory and as long as you don’t write it anywhere to any kind of file, you just have your secret in memory and it is only in there for as long as your application is alive. Then, when your application goes to bed or gets shut off, the secret is not anywhere, it’s just very temporal.

Chris Hickman: Yeah, absolutely, so I mean, that’s a great point, I mean, this type of technique, you don’t have to use things like environment variables, so it’s part of that application startup, it can go issue these API calls, as you said, it’s just all just in memory. So it’s never in a disk, it’s never exposed as an environment variable and it’s very isolated, which is a very good thing. So that’s kind of the additional control that you have by doing this roll your own technique.

Jon Christensen: Yes, and it’s kind of the point, it like the point of all of these techniques is like, how can we get the secret to be as briefly anywhere, just have it available when you need it kind of thing, without doing anything to substantially slow down your application?

Chris Hickman: Yup, absolutely.

Jon Christensen: We cover a lot of information here on Mobycast, and if you’ve ever wanted to go back and remind yourself of something we’ve talked about in a previous episode, it can be hard to search through our website and transcripts to find exactly what you’re looking for. Well, now it’s a lot easier, all you have to do is go to mobycast.fm/show-notes and sign up, we’ll send you our weekly, super detailed outline that we use to actually record the show, and a lot of times, this outline contains more information than we get to during our hour on the are. So sign up, and get weekly Mobycast cheat sheets to all of our episodes delivered right to your inbox.

Chris Hickman: So that’s a roll your own technique if you will, just kind of wanted to walk through that just for completeness, to kind of illustrate the full gamut of what your options are. As we mentioned, there are full op systems out there for just dealing secrets management, and there’s some good one, and you really should be looking long and hard at these because they deal with a lot of the undifferentiated heavy lifting, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: That you don’t have to do, so why reinvent the wheel? Especially if you don’t need that full customization and control over the solution. So, in general, secrets management tools, some of the features that you really want to think about when you consider these are obviously they should allow you to securely store and tightly control access to your secrets. You want to think about protecting the data both at rest and in transit, you want this to be centrally managed. So you can go to one place and see all your secrets and know who’s able to access it and ideally you’d be able to see things like version history for when secrets are changed and whatnot. So, having that central management’s important, you obviously want secure authorization and authentication mechanisms for these secrets, and encryption obviously, and key management integration with those types of providers is important. Auditing, being able to audit who’s accessing secrets, using secrets, that would be an important feature, and then-

Jon Christensen: Yeah, that right there is probably a requirement of certain regulatory agencies, that auditing piece, yeah.

Chris Hickman: Yup, yup, absolutely, and then, of course, rotation of secrets and revocation as well, all important things to have. So these are definitely some of the things that you should expect from a secrets management tool.

Jon Christensen: Right, and I can’t imagine that there are any commercially successful ones that don’t just take care of all of these things for you. So, it’s not like a list of things to go make sure that they do, they do it, they all do it. It’s more just sort of an awareness of these are all the things these do, and if you’re rolling your own there’s probably a few of these things that you aren’t doing with your own that you would want to do, and it’s like, well if you’re going to write that, you might as well just buy it.

Chris Hickman: Yup, yup, absolutely. So, given that, let’s talk about some of the popular solutions out there. So first thought we’d mention, so HashiCorp, they have a product called Vault and this is very well known, very popular, it’s been around for years now, and that’s their secrets management system. So it works well for all types of environments, so it works if you’re on Prim, if you’re hybrid, Enterprise, it’s going to work well for that. It’s also multi-cloud, right? This is one of the great features that HashiCorp has for almost all of its product lines, they distinguish by being multi-cloud, by being cloud-agnostic.

Jon Christensen: That’s right.

Chris Hickman: But, some of the downsides of this is it’s not managed at all, this is self-hosted. So, you have to go spin up the boxes to run the software, to run Vault on it, right?

Jon Christensen: Uh-huh (affirmative).

Chris Hickman: And then you have the licensing fees as well for the software itself, so you’re paying for the machines to run it on, the licensing fees, and so if you are not a big company or you have lots of secrets to manage, then it can be really kind of expensive. Also, not just from a pure cost standpoint, but just from a human standpoint of how much overhead it is to learn this, spin it up, to maintain it and manage it, all that kind of stuff, right? So that’s one of the downsides to something like Vault.

Jon Christensen: Right, and then just in terms of all the requirements that a secrets manager has to have from auditing to rotation or revocation, protecting data at rest and in transit, unfortunately, HashiCorp Vault does not protect data in transit, just kidding, of course, it does, they all do, they all do all of these things, I’m just making sure everybody was listening.

Chris Hickman: I was like, “Oh man, how am I going to [inaudible 00:34:59].” Like, “Jon, did you fall and hit your head?”

Jon Christensen: No. It is wild to me though that you say it can be expensive because it’s not managed and it’s like wow, that’s the hard part of any of this, managing this stuff, why would it be expensive if you’re not going to manage it for me? Pretty wild, I mean, maybe part of what you’re saying is the management of your own boxes?

Chris Hickman: Yeah, so the total cost, right?

Jon Christensen: Okay, TTO.

Chris Hickman: Well, you’re going to pay to spin up instances, right?

Jon Christensen: Mm-hmm (affirmative), mm-hmm (affirmative).

Chris Hickman: So let’s just say you run on EC2, or maybe you have to go buy a 1U server from Dell or whatever, right?

Jon Christensen: Yup.

Chris Hickman: You’ve got to have the hardware to run it on, so there’s that cost. Then you have the cost that you pay to Hashi for the licensing fees for the software itself, and then you have the cost of just what it takes to run this from just maintenance and management standpoint, right?

Jon Christensen: Right, right.

Chris Hickman: So there’s three vectors there of cost versus a managed solution, really it’s just like, okay what’s the pricing structure? And that’s what I’d pick, right?

Jon Christensen: Mm-hmm (affirmative), yeah.

Chris Hickman: Yeah. So that’s Vault, just again, wanted to kind of throw that out there as this is one of the older kids on the block, it’s very established, it’s popular, whenever you talk about secrets management you’re going to hear about Vault, so just wanted to talk about that for completeness.

Jon Christensen: Cool.

Chris Hickman: So, moving on to AWS and managed solutions, and AWS offers two. So we have Systems Manager Parameter Store.

Jon Christensen: Favorite name, love that name

Chris Hickman: Yes. Yeah, and then they have AWS Secrets Manager, which is a standalone service that is really hyper-focused on just dealing with secrets, so two different services to use, pros and cons to each so I thought maybe we’d just kind of go through and just talk about each one of these. First, what they are and their characteristics, and then because they are so similar and there is a bit of confusion of like, “Well, which one should I use?” Let’s compare these two and try to tease that out, and understand what are the similarities, what are the key differences, and what are the reason you would choose one versus the other? So, given that, let’s talk a little bit about Systems Manager Parameter Store, so what this is, so Systems Manager is a huge offering, right? I mean, there’s at least 10 or 11 sub-services underneath Systems Manager, right? So there’s things like Patch Managers, there’s Session Manager, there’s the Run Command.
I mean, there’s just so many pieces to it. So Parameter Store is one of those sub-services underneath Systems Manager, and so it’s a service that allows you to store key-value pairs, and the values can either be unencrypted, so plain text, or they can be encrypted. So, some flexibility there, right? So it’s not just for secrets, right? It’s basically for just any kind of key-value pairs you want to store in a managed services, is what Parameters Store’s given you. So, as far as the encrypted values that it supports, there’s two tiers of support it has there, so one is standard and then the other is advanced. So, the difference between these two is with standard … So both of these, it’s going to be using KMS for encryption,

Jon Christensen: Okay.

Chris Hickman: With standard, you’re going to be limited to a maximum of 4K bytes on your values.

Jon Christensen: Okay.

Chris Hickman: And the reason for this is because, with the standard secure strength parameters, it’s going to do encryption and decryption directly using KMS with the CMK that you specify, okay? So KMS, it has limitation, right? It will encrypt up to 4K bytes, right? You can’t send it any more than that, KMS, right?

Jon Christensen: Mm-hmm (affirmative), mm-hmm (affirmative).

Chris Hickman: That’s just the limit there, so that’s why-

Jon Christensen: I mean, it feels like Systems Manager Parameter Store with standard encryption is downright similar to what we already described in terms of a roll your own solution.

Chris Hickman: It’s really similar because I mean-

Jon Christensen: Who knows what they use under the hood, yeah.

Chris Hickman: Underneath the covers, I would think they’re using something like Dynamo for the store, but you never know.

Jon Christensen: Could be S32, S3 has huge reliability and data retention and data trustworthiness.

Chris Hickman: Mm-hmm (affirmative), a lot of nines.

Jon Christensen: Yup.

Chris Hickman: Yup. Yeah, so that’s standard, so you can use that as long as your values are going to be less than 4K, and then at the advanced tier, what that is, any guesses what advanced it?

Jon Christensen: No, I’m sorry, unfortunately. I’m just waking up here.

Chris Hickman: So, with advanced, it’s if you have more than 4K bytes, right? It supports more than 4K bytes.

Jon Christensen: Oh, I thought you were asking me how it was implemented, like whether it was just … I was thinking standard uses KMS, uses the direct encrypt/decrypt, so how does advanced work? I actually don’t know, maybe they line up a bunch of KMSs and a link list, I have no idea.

Chris Hickman: Yeah, yeah no it’s just kind of interesting, you can see how, I don’t know what the history is for Parameter Store-

Jon Christensen: Oh, I see what you’re getting at.

Chris Hickman: But, they probably came out of the gate and they’re like=

Jon Christensen: Customer-focused, yes.

Chris Hickman: Well, they just-

Jon Christensen: “We need more than 4K.”, “Okay, we give you advanced.”

Chris Hickman: Yeah, exactly.

Jon Christensen: “We hereby introduce advanced.”

Chris Hickman: “And we called it advanced.” Right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, I bet that’s kind of how it evolved, I wouldn’t be surprised.

Jon Christensen: Mm-hmm (affirmative). Do you need to pay more for advanced?

Chris Hickman: Yeah. So that’s what advanced is, advanced is just it supports more than 4K bytes, and it does that by using envelope encryption, hopefully, you remember what envelope encryption is. That was back, again, in those encryption episodes, the four-part mini-series. So if folks haven’t listened to those, I would strongly encourage you to go back and to listen to some of those, especially episode 76, which was kind of a dive into the AWS encryption techniques, and really kind of a deep dive into KMS. So we talk a lot there about just what envelope encryption is, but just as a quick rehash, envelope encryption is you use KMS, and your customer master key to generate a data key. So this is a new AES 256 symmetric encryption key that KMS gives you, and then you now do the decryption outside of KMS, and you now you can do whatever size you want, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So if you have a megabyte that you want to encrypt, right? You go ahead and do that because you’re doing it outside the KMS, and then once you produce that ciphertext, you then have the data key that was used to encrypt it. You then encrypt the data key and you store it alongside the encrypted secret, right? So that’s the envelope, so you can think of inside the envelope is the encrypted data, and then on the outside of the envelope, or however you want to think of it, that’s the encrypted data key. So, in order to decrypt the value, you first have to have access to the customer master key to decrypt the data key, and then once you decrypt the data key, and you’re decrypting the data key using KMS, but once you’ve got the unencrypted data key, you can now use that to decrypt the secret cipher text outside of KMS, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, that’s-

Jon Christensen: The one thing just on all that, that I’d like to repeat that’s sort of easy for me to remember is just that the handshake style encryption that used public keys, so asymmetric encryption is slow and uses a lot of math, and the symmetric encryption that uses these data keys, the AES style encryption, is fast and uses a lot of very fast operations moving bits around, XOR type operations inside of a chip so you can do lots and lots of it really, really fast.

Chris Hickman: Yup, yeah. So that’s why we use asymmetric encryption for sharing of secrets because that’s a good way to share secrets, it’s secure when you don’t have a trusted intermediary, and then once you’ve exchanged the shared secret, then you switch over to the symmetric for the bulk of your encryption/decryption, you can do it very, very efficiently. And that’s how things like TLS works, so again, if all that’s new to you, go back to the encryption mini-series, pretty fascinating information. So that’s Parameter Store, yeah, so-

Jon Christensen: Yeah, Systems Manager Parameter Store.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: Advance, now we know, it’s not a linked list of KMS master key decryption, it could have been, but they instead use envelope encryption.

Chris Hickman: Yup, indeed. You could do that right? You could actually-

Jon Christensen: Yeah, totally.

Chris Hickman: You just splice it up into 4K chunks and encrypt and when you wanted to go read it you’d have to keep track of-

Jon Christensen: You put me on the spot in the interview, and that’s what came to my mind, right?

Chris Hickman: Mm-hmm (affirmative), yeah.

Jon Christensen: I did not get a job at Google today.

Chris Hickman: Well, it’s a new year, you never know, right?

Jon Christensen: Yeah.

Chris Hickman: That’s the great thing about New Years, is its hope and opportunity.

Jon Christensen: Yeah.

Chris Hickman: So don’t give up. Okay, so let’s talk about the other one Secrets Manager, and so Secrets Manager is fairly newish, it was first released in April of 2018. So, a little over 18 months old now, and so this is a service that, it encrypts and stores secrets only, and transparently it decrypts them and returns them to in plain text when you request them, right? So one of the big differences here is that with Secrets Manager, everything is encrypted, you can’t store anything unencrypted. So it’s been designed, really from the ground up, to be dealing purely with secrets and everything that goes around with it, so you can rotate and manage and retrieve sensitive data, it does support periodic rotation of secrets with RDS databases, it’s just built right into it.

Jon Christensen: Oh, cool.

Chris Hickman: So it knows exactly how to rotate secrets with RDS and does that on your behalf.

Jon Christensen: Nice.

Chris Hickman: So, the secret values that you’re storing in Secrets Manager, they can be any string or binary data up to 10,240 bytes, so quite a bit bigger than the standard tier for Parameter Store. So it does this by, it’s actually using envelope encryption, so Secrets Manager is using envelope encryption. So it’s kind of interesting, it creates a data key for every secret value, okay? So, if you’re storing 100 different secrets using Secrets Manager, then it’s going to have 100 different data keys, encryption keys, so that’ll generally be after using that CMK, that customer master key with KMS. Whenever value changes, then a new data key is generated, so it’s not just one per value, it’s actually one per value version is generated. So that data key, it’s encrypted using the CMK that you specify and because it envelope encryption, that encrypted data key is going to be stored alongside the metadata of the secret. One of the good things here as well is that because it is using KMS and because it’s using a CMK, and you specify the CMK for each secret. In fact, so you can be using multiple CMKs here with the secrets that you’re using with Secrets Manager.
So, again, you could have 100 secrets that you’re storing, and maybe the way you partition them up, you’re going to be using five different CMKs across that, you totally can do that. You can have one CMK, you can have five, you can have as many as you want. But because it’s using KMS and because it’s using a CMK, again, we can control access further through IAM, and we also have the auditing and all the other good stuff that comes along with it. So both Secrets Manager and Parameter Store, we kind of get that benefit of piggybacking on top of KMS and all of the goodness that comes from that.

Jon Christensen: Okay.

Chris Hickman: So maybe just quickly some just overall, the workflows, what’s happening with encryption and decryption. So this kind of helps a little bit I think just for the reinforcing that envelop encryption. So, when Secrets Manager, when you want to encrypt something, the first thing it’s going to do is going to call KMS’s generate data key API using the CMK that you specified for that secret. So, KMS will then generate this new 256 bit AES symmetric key and it’s going to return that both as a plain text value as well as the ciphertext version of it. So you’re going to get both the encrypted and the unencrypted version of that new data key. After that, now that is has a data key, Secrets Manager can then take the data that you gave it and it’s going to use AES 256 using the plain text data key, and then once it now has the ciphertext of your secret, it’s now going to store that and alongside of it, it’s going to take that ciphertext version of the date key and put that in the metadata of the secret, right? So that’s what encryption looks like, so pretty straight forward, kind of what we talked about.
And then decryption is really kind of like the reverse of that, so when you go to decrypt something with Secrets Manager it’s going to go and call KMS’s decrypt API operation and it’s going to pass in that encrypted data key that it gets from the metadata of secret. So it’s going to be using the CMK for that to decrypt it, and so now it has back the plain text version of the data key, and once it has the plain text data key, it can now do the decrypt of the ciphertext of the secret value, and that’s the decryption operation for Secrets Manager.

Jon Christensen: I wonder about one thing with both of these managed services, there’s the spy movie test like, “Okay, there’s a bomb about to hit the white house and you have 30 seconds to get the secret out of the system.” And one of the easiest ways when you only have 30 seconds and you have the entire chain of command standing around your desk waiting for you to get it done within that 30 seconds is if, within the console, you can kind of click through and get to secrets. Instead of having to use things that are hard to memorize like CLI commands, and API commands in order to get to anything. Do they let you get to stuff through the console if you have the right level of access or no?

Chris Hickman: Yeah, so both Parameter Store and Secrets Manager have robust console applications that you can go in and you can see what parameters are being stored, what secrets are being stored, you can-

Jon Christensen: I imagine they have a little reveal kind of thing, some kind of UX that says, “Show me this.” And now just like, “Let me puke all these secrets in plain text out onto the screen.” But reveal one at a time what I need to see with all my bosses behind me in order to save the world within 30 seconds.

Chris Hickman: Yup, yeah.

Jon Christensen: Okay, cool.

Chris Hickman: And so those are the absolutes that you get with a managed system for managing your secrets is, you want that full, robust capability and yeah, you don’t want to have to be relying on, “Oh, I have to only do this with a command line, and what’s that API command again?” And yeah.

Jon Christensen: Mm-hmm (affirmative), mm-hmm (affirmative), cool.

Chris Hickman: And then it’s too late, yeah. All right, well I think that is quite a bit for today, I think next time-

Jon Christensen: Yeah, especially with the hangover, or, I actually don’t know if this is coming … We talked at the beginning that this was coming out on New Years Day, but then you said later in the episode that it was New Year’s Eve, and so now I’m confused, and our listeners are obviously not because they know what day it is today, but I’m confused.

Chris Hickman: Yes, they should be listening on New Years Day.

Jon Christensen: Okay, cool.

Chris Hickman: Yes, yes.

Jon Christensen: So with a hangover. So, what better medicine, right? Talking about the details of encrypting and decrypting secrets, that is absolutely what you want to hear.

Chris Hickman: Well, I think you have the hangover because you were playing the drinking game on whenever we said blast radius. So, yeah.

Jon Christensen: Right.

Chris Hickman: Me, I had to follow the agenda here and I watched my Mad, Mad, Mad, Mad World, yeah. All right-

Jon Christensen: [crosstalk 00:53:55] thanks so much again, Chris, this is huge.

Chris Hickman: Yeah.

Jon Christensen: It’s a good episode, and looking forward to talking about it in a bit more detail and getting into how certain things work with actually implementing this next week.

Chris Hickman: Yeah, so next time we’ll go into like, “Okay, what is the difference between Parameter Store versus Secrets Manager? How do we choose?” Because they seem to be so similar, so there’s a lot of confusion there. So let’s figure that out, and then after that, let’s like, “Okay, we have a container, the containerized app, it’s running on ACS, how do we integrate with Secrets Manager or Parameter Store? How easy is it and just, what’s the mechanics of that? How do we do that?” So we’ll walk through that.

Jon Christensen: All right, thanks so much, catch you next week, Chris.

Chris Hickman: Great, thanks, Jon.

Jon Christensen: Bye.

Chris Hickman: Bye.

Stevie Rose: Thanks for being aboard with us on this week’s episode of Mobycast, also thanks to our producer, Roy England, and I’m our announcer, Stevie Rose. Come talk to us on Mobycast.fm, or on Reddit at r/mobycast.