94. Psst… Secrets Handling for Cloud-Native Apps – Part 2

Stevie Rose: In episode 93 of Mobycast, we discussed secrets management for our Cloud-native applications. We learned why we need secrets management, and some of the possible solutions available to us. Now that we know the theory, it’s time to put that knowledge into practice. In this episode of Mobycast, Jon and Chris finish their two-part series on handling secrets with Cloud-native apps. We show you how to easily implement secrets management for a containerized application running on Amazon Elastic Container Service, or ECS. After this episode, you’ll be a pro at keeping a secret.
Welcome to Mobycast, a show about the techniques and technologies used by the best Cloud-native software teams. Each week, your hosts Jon Christensen and Chris Hickman pick a software concept and dive deep to figure it out.

Jon Christensen: Welcome, Chris. It’s another episode of Mobycast.

Chris Hickman: Hey, Jon. It’s good to be back.

Jon Christensen: Hey, good to have you back. All right. So, here we are in the middle of a two-parter. We’re talking about managing secrets, and keeping those things safe, and only in the hands of the code that needs it to run. So, in this second part, what are we going to talk about Chris? Well, in the second part, we’re going to get more into detail of comparing and contrasting the AWS offerings, so I’ll say that part. And then, you can tell us, what did we talk about last time, Chris?

Chris Hickman: Sure, yeah. So, last time we kicked this off with, “What is a secrets management solution, and why do we need it?” So, our applications always have a need for access to some sort of sensitive data, usually, whether it be a database connection string, or passwords, or API keys, or trying to communicate with upstream dependencies.
So, managing our secrets is something that we just need to do, and we want to do it the right way. So, we talked about, “What are some guidelines for how do you do secrets management correctly?” And then, we walked through what the options are for doing this. So, first, just for completeness, we kicked off by talking about just a simple method for doing secrets management, that’s served us well in the past, where it’s basically just using our standard encryption and decryption, and just storing those secrets encrypted until we do need them.
So, and that’s just, we walked through a solution doing that with just using KMS and S3. So, very, very simple, but also pretty robust, but not having nearly the feature set that we can now get with off-the-shelf solutions that are out there. So, we talked about the landscape out there with various different secrets management solutions that do exist. And we talked about Vault from HashiCorp, which is a very, very popular, mature platform. It’s multi-Cloud, it’s hybrid. You run it yourself, it’s self-hosted. So, that gives it some pros and cons.
And then, we also talked about, there’s two different systems from AWS. So, there’s Systems Manager Parameter Store, and there’s also AWS Secrets Manager. And so, we talked about just broadly, “What are the cababilities, the features, of those three?” And then, so, today, now, we’re going to pick that up, and first start off, like, “Okay. Well, what’s the difference between Parameter Store and Secrets Manager? They sound really similar.”
And then, once we get past that, then we’ll talk about, “Well, how do we actually do this with our application? And how do we integrate with these, in particular, for our ECS-hosted containers?”

Jon Christensen: So, at the end of today’s episode, if you have some secrets that are improperly managed, and code that you’re working on, I can imagine what you’ll be able to do is, you’ll be able to go to your boss, say, “I have a new solution,” and then, you can copy-paste one line of code out of Mobycast.fm, and be the hero of your team. Right?

Chris Hickman: It won’t be that straightforward, but, believe it or not, I mean, actually, if you do decide to use one of the AWS secrets management solutions, it’s not a lot of effort.

Jon Christensen: Right.

Chris Hickman: So, there is a lot of bang for the buck here. And it may not be one line of code, but it may only be three or four.

Jon Christensen: Right. And I think it’s important to also point out that that’s a joke, because that’s not who we are, we’re not the “copy-paste one line of code” people. That’s not what we believe in, in terms of development.

Chris Hickman: Oh, come on, Jon. Go to Stack Overflow, right?

Jon Christensen: Just copy-paste-

Chris Hickman: Just Google it.

Jon Christensen: … Your way to the top.

Chris Hickman: Yep.

Jon Christensen: But, seriously, if it is only three or four lines of code, into those three or four lines of code are multiple hours worth of thought, and understanding, and foundational knowledge that everybody should have.

Chris Hickman: Sure.

Jon Christensen: So, sure. A couple lines of code, but let’s actually unpack it into lots of conversation and thought.

Chris Hickman: Or even just getting to that point, right, where you know, like, “Okay, this is the solution, this is best practice, this is the right solution for me.” And then, now the … So, sometimes it’s like the implementation ends up being the easy part, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: What really is the hard part is knowing what your solution should be.

Jon Christensen: So, the implementation might be the easy part, and there’s lots to understand. So, let’s get into it, let’s talk about some of the differences between Systems Manager Parameter Store, and Secrets Manager.

Chris Hickman: Yeah, absolutely. So, as we talked about the landscape, both of these systems, both Parameter Store and Secrets Manager, they allow you to store secrets, and they both support encryption of those secrets, decryption, they integrate with KMS. And so, it’s one of those things where it’s confusing. It’s like, “Well, how do I pick between one or the other?”

Jon Christensen: Yeah.

Chris Hickman: Right?

Jon Christensen: Yeah.

Chris Hickman: There’s these two separate things that look really similar. Secrets Manager in its name, obviously, it’s dedicated purely to secrets. Parameter Store gives it away, saying, like, “This seems to be a bit more generic.” And so, parameters in general, and those parameters, some of those could be secrets as well. So, let’s talk about, okay, how they are similar. So, they’re both managed, key-value-store services. Right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, you can have something with a key name, and there’s a value associated with it, and this is a managed service for storing and retrieving those. Likewise, the keys, they can have prefixes, right? So, just like we have prefixes with S3, so that we can have this pseudo-directory structure to things, right, and name-spacing, you can do the same thing with both of these, right?

Jon Christensen: Okay.

Chris Hickman: So, you can have namespaces for different environments, right? So, maybe it’s, one is for development, and one’s for staging, and one’s for production. And then, you can go drill down, maybe, into application, and then, maybe even sub-application, or something like that, right? So, you can have a namespace buildup that makes it easy to have some organization around these key-value pairs.

Jon Christensen: I love how we talk about that so technically, because it feels like, “We can have folders!” And then, everyone’s like, “Oh yeah. Cool, folders.” Do you know what I mean?

Chris Hickman: Right.

Jon Christensen: “Oh, we can have namespaces and prefixes,” it just feels a little geekier.

Chris Hickman: Yeah. Yep.

Jon Christensen: Yep.

Chris Hickman: Absolutely, yeah. Well, and, at the end of the day, it ends up, this is the stuff you really have to do, right? If you have more than one environment, and more than one application, you’re going to quickly find out, “Hey, I’ve got to be organized here.”

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: Otherwise, before you know it, you’re using secrets for your development environment in prod, or vice versa. Or, you’re now … just, nothing’s working. Right?

Jon Christensen: Right.

Chris Hickman: So, you’ve got to be organized with this stuff.

Jon Christensen: Well, and the business side of the companies started to figure out what folders meant, so we’ve got to make sure that it still feels like we’re talking a foreign language when we talk to each other about what we’re working on, so we’re going to switch to using words like prefixes.

Chris Hickman: Sure.

Jon Christensen: That way they will have no idea that we’re just implementing folders.

Chris Hickman: Sure, absolutely. Let’s just call them partition keys and be done with it.

Jon Christensen: Yes.

Chris Hickman: So, as far as similarities go, as I said, they both do encryption and decryption. They are using, of course, KMS to do that encryption and decryption.

Jon Christensen: Yep.

Chris Hickman: And both of these stores allow the values to be reference-able in Cloud formation templates, which is really nice. And, not just Cloud formation, but there’s a bunch of services that now have integration in with services, so that you can reference values of secrets inside your scripts, templates, and in other places as well. So, without any lines of code, really, right, you can now reach out, have those secrets retrieved, decrypted, and made available to some other application.

Jon Christensen: Is that reference something, just an ARN, and then colon, and then the prefix, and then the secret name? Something as easy as that?

Chris Hickman: Yeah. I mean, at the end of the day, the ARN includes all that information, right?

Jon Christensen: Okay.

Chris Hickman: So the ARN is fully describing, it’s basically saying it has account, region, service, and then the actual name of the key for the secret as well, right? Or the parameter, as it may be. So that ARN is the full thing, and sometimes you don’t have to use the full ARN, if, depending on the service, and depending on whether it’s within the same account, you can use the alias, or the short-

Jon Christensen: A token-ized ARN?

Chris Hickman: … Or just the name of the key, right?

Jon Christensen: Uh-huh.

Chris Hickman: So, you can keep it really simple. So, if your namespace was /Moby/development/myapp you can just do that, instead of the full ARN.

Jon Christensen: Cool.

Chris Hickman: Yeah. So, that’s how they’re similar. So, let’s talk about some of the key differences between them. So, one of the ways they differ is just the types of values that they store. So, Systems Manager Parameter Store, so you can store either encrypted or unencrypted plain text data into it, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, you have that flexibility. So, this is kind of the complete solution for all sorts of just parameters, configuration data, just any kind of key-value pairs that you want to store, Parameter Store will allow you to do that. And it has multiple different types of values. So, you can have strings, you can have lists of strings, and then you can have secure strings, which are encrypted.
Secrets Manager on the other hand, the only thing it supports are encrypted values. So, which makes it, I mean, again, the name gives it away, right? It’s a secrets manager. So, everything you store in Secrets Manager’s going to be encrypted.

Jon Christensen: I mean, I guess that’s like, “Okay. This let’s you do three things, and this let’s you do one thing, but the one thing is the same across both of them.” So, unless I’m nervous that I’m going to forget to store my secrets encrypted, like, “Well, I could forget and accidentally store them unencrypted,” because Parameter Store will let me do that, I mean, that’s potentially something to worry about. But otherwise, it still feels like there’s not a clear winner here.

Chris Hickman: Mm-hmm (affirmative). Again, other than, if you want one solution to manage all things, and some of that doesn’t need to be encrypted, then-

Jon Christensen: Oh, there you go. So then it-

Chris Hickman: … Parameter Store gives you that flexibility, right? So.

Jon Christensen: … It might be nice to not use two different services-

Chris Hickman: Yes.

Jon Christensen: … If you know … Okay. Got it.

Chris Hickman: Mm-hmm (affirmative). Yep. So, let’s talk about cost. So, cost is actually one area where these differ dramatically. So, with Systems Manager Parameter Store, there is no cost for parameters at the standard tier. And you can have up to 10,000 of these parameters per account, I believe that is, right?

Jon Christensen: Okay.

Chris Hickman: So, remember, Parameter Store has those two-

Jon Christensen: No matter how often you access them? So, you’re accessing these parameters just non-stop?

Chris Hickman: … Mm-hmm (affirmative).

Jon Christensen: They charge by the amount of parameters?

Chris Hickman: So, yeah. So, remember, Parameter Store has those two tiers, right? The standard and advanced. And so we say, with the standard, that is limited to a maximum size of 4K bytes.

Jon Christensen: 4K, yep.

Chris Hickman: And it is doing the encryption/decryption directly with KMS, using the CMK, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, and that’s one of the reasons why it has that 4K limitation. But so, if you are using standard tier parameters, again, you can have up to 10,000 of those. There’s no cost. It doesn’t matter how many times you access them. I mean, you will be subject to throttling quotas, if you go above it. So, like KMS, has quotas, right? Across your account. So, you can exceed that, and then you would be throttled.

Jon Christensen: Okay.

Chris Hickman: But there’s no cost at all for standard tier.

Jon Christensen: Okay. Okay.

Chris Hickman: Now, advanced tier parameters, they will incur charges. So, there is some charging there. But this is a big win for Parameter Store. So, basically it’s free, as long your values are 4K or less, and you stay at the standard tier, this is free.

Jon Christensen: And you remember to store them encrypted.

Chris Hickman: Yes. But unencrypted secrets cost the same price, they’re free as well. Just, you may have some other costs, right?

Jon Christensen: Right.

Chris Hickman: In handling a security breach. So, Secrets Manager, on the other hand, it does cost. And so, the cost here is pretty straightforward: $0.40 a secret, per month. And looking at $0.05 for every 10,000 API calls.

Jon Christensen: Okay.

Chris Hickman: So, if you have 100 secrets that you’re managing, that you have, that’s $40 a month for just keeping those secrets in Secrets Manager. And then, if you have lots of API calls, then that may add to the cost as well.

Jon Christensen: Yep.

Chris Hickman: So, but you do get a lot of additional functionality through Secrets Manager, so, again, it depends on your use case what features you can take advantage of, that your require, will help decide which one you’ll choose.
And so, given that, so let’s talk about, “So what are the additional features then, that Secrets Manager has that Parameter Store doesn’t?” So, again, Parameter Store is basically pretty straightforward and simple, right? You can have these strings, whether they be encrypted or unencrypted, you can store them, you can retrieve them. That’s about it. With the advanced tier there are some additional features you can have. You can have some policies in place and whatnot. Parameter Store does give you version history, you can see things that have changed and whatnot. But that’s basically it.
Secrets Manager does have quite a few additional features that may suit your use case a lot better, and may provide a reason why you’d want to use it over Parameter Store. So, one of those is, it does support autorotation of credentials. So, this is a really nice feature, especially some of the integrations that it has for services like RDS.

Jon Christensen: Yep.

Chris Hickman: So, if you have RDS databases, when you create a secret in Secrets Manager, it’s going to ask you what type of secret you want to create. One of those types is going to be, this is RDS credentials. And so, you basically just give it the username password for the RDS databases. That or store it as a secret. You can then define a rotation policy on that, and Secrets Manager will handle everything else for you. So, you can say how often you want that rotated. Secrets Manager will be responsible for generating a secure random secret for the password, and you don’t have to do anything.
So, really, pretty nice, complete end-to-end solution there for just making sure that your credentials are rotated with your databases. So, it’s integrated with RDS, and it’s also integrated with DocumentDB, and some other databases, Redshift, I believe.

Jon Christensen: Okay.

Chris Hickman: So, you also, if you have services that are not directly integrated in with Secrets Manager, it does give you a hook to manage your own rotation of keys, and it does this via lambda functions. And so, that’s something that it’s, they give you templates, and it’s pretty straightforward to do. So, again, if you want to have autorotation of credentials for other types of services, so it could be upstream services, it could be other microservices in your organization, it could be service-to-service, secrets. We’ve talked about in a previous episode of Mobycast, “How do you have microservices call each other securely?”

Jon Christensen: Right.

Chris Hickman: So, rotation of those kinds of secrets. I mean, all that would be supported with Secrets Manager, if you wanted to do that, and, again, a nice integration hook through lambda.

Jon Christensen: That is pretty cool. I like it.

Chris Hickman: It is, especially for RDS, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: RDS is one of those no-brainers. It’s just so seamless. The only thing you have to do is just deal with the making sure whatever’s using those secrets is re-reading them when they do change, right?

Jon Christensen: Right. Yes.

Chris Hickman: Which, from a detail standpoint, that’s actually … It could be a little tricky, right?

Jon Christensen: Yep.

Chris Hickman: To get it done right.

Jon Christensen: Yeah.

Chris Hickman: So, autorotation of credentials is one of the big value adds that Secrets Manager gives you. Another one is, it does do password generation by basically generating random secrets. So, it has that capability to generate these random secrets for you. So, I don’t know about you, Jon, but, for me, whenever I have to go generate some random secret, it’s like, “Man, oh, what do I use?” And a lot of times I’ll just use something like LastPass, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: To go generate a nice, secure random secret, with the kind of criteria I may have. Well, Secrets Manager will do that for you as well.

Jon Christensen: Right. I just copy-paste code off of Stock Overflow and run it.

Chris Hickman: There you go. Yeah. Or just use password:password, right? That always works, right? If you want to be really tricky, maybe admin123. Yeah. So, no, you don’t have to do that anymore with Secrets Manager. So, with Secrets Manager it will do that for you.

Jon Christensen: Cool.

Chris Hickman: And then, also, secrets can now be shared across accounts with Secrets Manager as well. So it gives you some cross-account cababilities there as well. So, those are the key differences. I think, for me-

Jon Christensen: Just, I want to comment on that cross-account thing-

Chris Hickman: … Mm-hmm (affirmative).

Jon Christensen: … Because, I just think it’s sort of funny, just the sort of side note, that, “Best practices: Separate all of your environments by having separate accounts for each one of them.” And, “Best practices: Give every developer their own AWS account.” And then, as soon as everybody started doing that, all these features, to be able to move data around, and move it across accounts, started coming into play.
And it’s like, “Ha ha, you guys, AWS, you’re totally going to ruin this nice fence that you built between accounts, eventually, here.” Just a side, unrelated to this whole conversation, thought that I’ve been having lately. More and more features coming around that make it easier and easier to blur the lines between accounts. “Wait a minute. The whole point was to have those lines be pretty strict.”

Chris Hickman: They are. This is kind of a security feature, right, because you imagine you would set up an account that is a very secured account, with few people that have access to it, really locked down, and that’s where you keep all your credentials. And so, you only have a few people that can use that account to enter credentials. But then, those credentials need to be retrieved by other folks that are separated, isolated, in other accounts.
So, I think it’s similar to having an account purely for logging, where all your logs go to a single account. This is kind of similar, where it’s like you have an account where this is where all secrets are managed, and other people can basically subscribe to them and retrieve them, but the actual management of them is done by this one account. So, that might be one of the use cases here for Secrets Manager in having that cross-account capability.

Jon Christensen: Okay. That makes sense. But I still want you to let me win this.

Chris Hickman: So, there’s no doubt, right? The permutations that you can have with number of regions, accounts, roles, users, policies, VPCs, endpoints, everything else that goes along, I mean, it’s super complicated. AWS Organizations, you can use that, and set that up. I mean, it can get really, really complicated, very, very quickly.

Jon Christensen: Right.

Chris Hickman: And there is so much variation here, and there’s so many different permutations, that, yeah, pretty quickly you can be up to your neck in just configuration hell, right?

Jon Christensen: Yes. But the other thing I’m saying is true. I saw some new cross-account EBS features, to make it easier to share data from an EBS drive across accounts. I saw new cross-account features around VPC communication, I saw new cross-account features around S3. And it’s like, these are breaking down the barriers that people put up on purpose, and then didn’t like. It’s just funny.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: It’s funny to me that that, it’s the forever trade-off between security, and ease of use.

Chris Hickman: Yeah, yeah. And the practical solution is somewhere in between.

Jon Christensen: Yes.

Chris Hickman: Right?

Jon Christensen: Yeah.

Chris Hickman: Is really what it is. And so, it’s like trying to find that right mix. And it depends, right, on different users. So, big companies, their inflection point is going to be different than startup companies, or small businesses.

Jon Christensen: Right.

Chris Hickman: And so, being able to try and support all of that-

Jon Christensen: Right.

Chris Hickman: … Ends up becoming … It just adds to the complication.

Jon Christensen: Right. This is a real big tangent, and we’ll come back to ECS secrets management in just a second, but one of the best things I’ve seen around cross-account, why to have many accounts for your different environments, for example, or different developers, is because it’s really, AWS service limits are done at the account level, so you can only have a certain amount of VPCs, or you can only spin-up a certain number of load balancers, that kind of stuff, and it’s easy for some developer to accidentally write a script that spins up the wrong number of something, or too many of something, and then, by doing that, they’ve suddenly prevented production from being able to auto-scale as needed. And then, that could hamper production, right?

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: That’d be terrible. And that is a very, very good reason to do multiple accounts. But I can so imagine a new feature coming out, like, “Introducing AWS service limits across organizations.” Or, “Introducing a way to share AWS service limits across accounts,” so now you can give some other account the unused services that this one account … You know? Can’t you imagine that?

Chris Hickman: Yes. Service limit bursting.

Jon Christensen: Yes, yes.

Chris Hickman: Burstability, right?

Jon Christensen: Yeah, totally.

Chris Hickman: Yeah.

Jon Christensen: Yeah.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: Anyway, we’ve got to go back to it. But, yeah, it’s like, those barriers are weird, and fuzzy.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: We cover a lot of information here on Mobycast, and if you’ve ever wanted to go back and remind yourself of something we’ve talked about in a previous episode, it could be hard to search through our website and transcripts to find exactly what you’re looking for. Well, now it’s a lot easier. All you have to do is go to Mobycast.fm/show-notes and sign up. We’ll send you our weekly, super detailed outline that we use to actually record the show. And a lot of times, this outline contains more information than we get to during our hour on the air. So, sign up and get weekly Mobycast cheat sheets to all of our episodes delivered right to your inbox.
Okay. ECS integration of Secrets Manager.

Chris Hickman: Yeah, okay. So, we’ve talked about the various, different services that we can use. Yeah, I think, for me, the way I’m wrapping my head around this is that, I really Secrets Manager for database credentials, and so I think that totally makes a lot of sense. And then, for application-specific sensitive data, so things like API keys, or maybe some credentials for upstream services, I think I like Parameter Store for that, just for its simplicity.

Jon Christensen: Yeah.

Chris Hickman: So, just something to keep in mind. Parameter Store does provide some integration with Secrets Manager, so I believe you can go through Parameter Store to access Secrets Manager, so you can have the one way of doing it in your code. So, something to consider as well.
So, let’s talk about, “How do you actually use this with a containerized application that running on ECS?” So, as we talked about, there’s pretty strong integration with these services, with other AWS services like Cloud Formation, so ECS is one of those as well. And it works the same way, regardless of whether you choose Secrets Manager or Systems Manager Parameter Store. So, let’s walk through that.
This does apply to both types of launch types with ECS, so both EC2 and Fargate. And the way it works is-

Jon Christensen: And I imagine this works the same way for ECS and EKS, although we haven’t really played with EKS that much.

Chris Hickman: … Yeah, EKS might be a little bit different. It’s definitely going to be different-

Jon Christensen: Oh, okay.

Chris Hickman: … And I’m not sure exactly what the integration story is there. It’ll be clearer once we get into this on how it’s implemented.

Jon Christensen: I see, because we’re going to talk about tasks, which is obviously not something that EKS even has. Okay. Cool.

Chris Hickman: Yeah, yeah. Right. So, the way this works with ECS is, at the end of the day, you’re going to be injecting that sensitive data into your containers as environment variables. So, we talked about this last time, about environment variables being one of those common ways for having secrets available to your containerized application. And, again, for the most, that is a very secure way to do it. There is a small chance of leakage, but someone would have to be able to have root access in order to see that, and to get access to that.
So, the way you do this is, you’re going to be specifying parameters in the task definition file for your task, for your containers, as name-value pairs. And so, it’s going to go in a very specific area in your task definition file. And it’s maybe a little unfortunate, or just something you need to be aware of, that there’s actually two areas of your task definition file that you can have secrets.
One is for the application itself, for your containers. And for that, you’re going to use the Secrets parameter. So, that’s the actual name of the parameter, it’s called Secrets. If you need to access secrets in your logging configuration, then that parameter is called Secret Options. I’m not sure why they have to be different names, and the disparity being so asymmetric. But it is what it is.
So, for the most part, people, I think, your secrets are going to be on your container bases, so you’ll probably just be using that Secrets parameter. But if you do have a logging configuration where you do need secrets because of the integration you’re using for whatever log provider you have, then you would be referencing to it as the Secret Options parameter. So, it’s just something to keep in mind.
And so, what you do to define these, right, is, so, inside that task definition file, you go to this Secrets key in the JSON, and then, for that, you now have an array of secrets, specified as objects. And so, for each object, it’s just a key-value pair. The key, it’s going to have a name, and then a value. Sorry. So, the name is basically specifying the environment variable name that will be exposed to the container. And then, the value, that’s going to be the ARN of the secret that you want to reference.
And so, that ARN can either be referencing a Systems Manager Parameter Store secret, or and AWS Secrets Manager value. Right?

Jon Christensen: Mm-hmm (affirmative). Makes sense. So, you’re defining it there, and then, once your code is running, then you can just refer to the environment variable, the names of the secrets as environment variable names, right?

Chris Hickman: Right.

Jon Christensen: Like-

Chris Hickman: Yeah, so, I mean, it’ll just be an environment variable that’s now available to your application running inside that container. So, you can use whatever stack-specific ways of doing it. So, if you’re using Node, right, it’s the process.env object to go and access that. So, whatever you use for the name of it, right, it’s like, maybe it’s called, “My Secret,” right? So, process.env.mysecret and now you have access to that inside your application.

Jon Christensen: … Well, that’s pretty cool. I imagine this has changed by now, but earlier this year, when I was doing stuff with Secrets Manager with Lambda, I actually had to go make Secrets Manager API calls from Lambda to go get my secret by name, and then make it available to the rest of the lambda functions. So, I had wrote a little code to go do that, and this is way easier than that.

Chris Hickman: Right. And so, you probably still have to do that with Lambda. But so, this is a nice integration point for ECS, right? Because it just makes it really, really easy to do, right? No code, you’re just updating the task definition file. And you’re choosing like, “Hey, what’s the environment variable name?” And then, “Here’s the ARN for my secret.”

Jon Christensen: I’m guessing that in EKS it’s probably similar, but just with the whole concept of pods, instead of task definitions.

Chris Hickman: Yep.

Jon Christensen: Yeah.

Chris Hickman: Yeah, it would be a no-brainer there.

Jon Christensen: Yep.

Chris Hickman: Yep. So, something to keep in mind. So, obviously, this data is going to be injected when the container is started, right?

Jon Christensen: Right.

Chris Hickman: So-

Jon Christensen: If you want a different secret value, you’ve got to restart your container.

Chris Hickman: … Exactly, right? So, any subsequent updates to that sensitive data after the container is started, such as credential rotation, your container’s not going to see that, right? It’s not going to change. So, you’re going to have to recreate that container. It’s going to have to just, yeah, restart it, right? So, quit that one and spin up a new one, because it’s only going to read it when they’re on init.
So, one thing you can do is, if you’re using ECS with the service, so if you actually have a service defined, right, you can then just go and update your service. You don’t have to change anything, you can just click the Force New Deployment option. And then, that will cause ECS, basically, just to terminate the existing task, and spin-up new ones with the same task definition file, which doesn’t change, right? Just because the value has changed inside Secrets Manager, or inside Parameter Store.

Jon Christensen: I’m imagining making a little thing that, whenever you update a secret, it would send a little notification that then would do that. Right? Just an auto-magic, “Oh, you changed the secret? Go blast all the containers away, and make new ones.”

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: Yeah. You could do that.

Chris Hickman: It’s really pretty straightforward, right? So, again, if you’re doing it as an ECS service, it’s literally just making an API call to say, “Update service with Force New Deployment.”

Jon Christensen: Right. I was thinking though, whether you could it with no code. I don’t know if you can fire it, have an alert, a Cloud watch alert, get fired when a secret gets changed. But if you could, and then you could … Aah, you’d probably need to write a lambda at the very least. A lambda that would go tell that service to restart.

Chris Hickman: Mm-hmm (affirmative). You know, I wouldn’t be surprised if you could do all that with no code using EventBridge.

Jon Christensen: Very neat.

Chris Hickman: So, we’ll link that as an exercise for the reader. Or maybe-

Jon Christensen: A little extra credit.

Chris Hickman: … [inaudible 00:33:29].

Jon Christensen: Let us know if you figure out how to do that.

Chris Hickman: Yeah.

Jon Christensen: Let us know.

Chris Hickman: Yeah.

Jon Christensen: That’d be cool. I’d like to read that.

Chris Hickman: Yeah. So, guess what? We’re not done.

Jon Christensen: Oh.

Chris Hickman: So, if we just did that, it wouldn’t work, right? Because you know what we’re missing: Our good old friend IAM permissions, and roles, and policies, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: Because, otherwise, this is a big, gaping security hole, right? If any ECS task is able to go read secrets.

Jon Christensen: Yes.

Chris Hickman: That’s not good, right? So, there’s some things that we have to do from a security posture in order to make this happen.

Jon Christensen: And I can read, but I remember that we were doing roles by task, instead of roles by the EC2 instance that runs all of our containers, because roles by task helps us have different tasks with different roles, and different cababilities. So, one task might need secrets, another might not. One task might need to put stuff in S3, another task might not. So, we’ll do a task execution role to give it access to the Secrets Manager, and to the particular secret within Secrets Manager, right?

Chris Hickman: Yeah, absolutely. So, this one of those ones, too, where it may help, go back to episode 86, when we were talking about server-less containers with Fargate, where we really went into detail on the various roles used by ECS. And so, there’s two roles to keep in mind here. One is, there’s task-based IM roles, and so that is the IM role that’s used by your application when your application makes calls to AWS services. And so, that we definitely recommend, “Hey, instead of using the default instance role,” right, ” and basically doing it at the host level, do your permissions at the container level using a task-based IM role.”
The other role is this task execution role-

Jon Christensen: Oh yeah.

Chris Hickman: … Which kind of sounds the same.

Jon Christensen: Yes. I just confused it, right? I just totally mixed it up. Yep.

Chris Hickman: But it is. It really is confusing with all of these names. But the task-

Jon Christensen: Right. And we’ve put out an AWS wishlist item to rename this to Task Management Role. That would be so much clearer.

Chris Hickman: … Mm-hmm (affirmative).

Jon Christensen: Ugh. Okay. I’m just … I mean, just name things in a way that makes it so that somebody reading it can be like, “Oh, that must do this.” And if I read Task Execution Role, it’s like, “Oh, that must be the role that is used when the task is executing.” Ugh.

Chris Hickman: Yep.

Jon Christensen: So, I’m just bummed by that poor choice.

Chris Hickman: Right. Yeah, so, but that’s what we’re here for, right, is to help clear that up, and to point that stuff out.

Jon Christensen: I was involved in the making of episode 86.

Chris Hickman: Yeah. You get a C- on the quiz.

Jon Christensen: Anyway. And I think I complained about it then, too. So, I’ll never, ever remember this.

Chris Hickman: Yes. And that’s okay. So, yeah. So, task execution role. So, that is the role that’s used by the ECS agent to make AWS API calls on your behalf, right? So, we used that back for Fargate, because we needed that in order for the Fargate task to be able to pull a container image from ECR. And then, now we need it so that our container, the agent, can go make these calls to either Secrets Manager or Parameter Store on our behalf, and create those environment variables, and inject them into our container, right? Which, that is not done by our actual application, right?

Jon Christensen: Right.

Chris Hickman: It’s outside of our application, right? So, that’s why it’s a separate role. So, we need to set up our task execution role. So, one, we have to make sure … if our task definition is not specifying a task execution role, we’re going to need to do that, right? So, to pick a role to be used for this particular purpose.
And so, if we go in, and we try to update our task definition file, right, to add these secrets into it, if we don’t specify the task execution role in that, you’re going to get an error inside.

Jon Christensen: For sure.

Chris Hickman: Even in the console. The console, actually which is nice, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: It was actually a pleasant surprise.

Jon Christensen: Yes.

Chris Hickman: Because I could have seen, instead, this working, “Whoa. I just made my update. I added the secrets section to it. Now I go and update my service with this new task definition file.” And, then it’s just maybe something just doesn’t work, right?

Jon Christensen: Yeah, yeah.

Chris Hickman: And it silently fails.

Jon Christensen: Right.

Chris Hickman: So, at least there’s some validation there to let you know, “Hey, you’re missing something.”

Jon Christensen: Yeah, and the classic would be, “undefined environment variable,” right?

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: “Huh. Why isn’t this environment variable defined?”

Chris Hickman: Yeah. “Why isn’t it working?” Exactly.

Jon Christensen: Yeah.

Chris Hickman: So, we’ll have to specify the task execution role inside the task definition itself, when we specify the secrets. And then, in addition to that, obviously we’re going to have to make sure that that role has access to the required APIs that it’s going to need in order to do this. So, if you’re using Systems Manager Parameter Store, then you want to make sure that it has the ssm get-parameters action, is one of the operations that it can do.
And then, if you’re using Secrets Manager, then it’s going to be secretsmanager:GetSecretValue, is going to be the action. So, you want to put that as part of the … you add that to a policy, IM policy, and attach it to that role, the task execution role, so that it has access to secretsmanager, or ssm. And then, a best practice would obviously be to further lock that down and specify resources that it can do this on.

Jon Christensen: Right, right.

Chris Hickman: So, the actual ARNs for the secrets, if it makes sense. If it’s a limited scope, or a handful, then it makes sense to go ahead and lock it down there.

Jon Christensen: You mentioned just now two specific policies that you need to add to the role. And they are not in the show notes that are going to be on the webpage.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: Which I know, probably, already the mid role went, where I’m like, “Hey, yeah. Get our show notes. Have them sent to your email.” But, let’s talk about that more specifically. Those are probably handy. Knowing that, and being able to refer back to that would be nice, if you’re listening to this and your memory is anything like mine is, it’d be nice to be able to go and see one place where it was like, “Oh, what was the detail of what he said there,” without having to go back and listen again.
Yeah. So, that’s an example of the detail that’s in the show notes that come to your email, versus the detail that’s on our webpage. So, little plug for that.

Chris Hickman: Yeah. And you’ll find out pretty quickly, “This is a problem,” right? So, because if you don’t have these permissions applied to that task execution role, then you’ll be able to update your task definition file, you’ll be able to update your service and roll-out, spin-up your containers, where they’re going to go out and try to fetch these secrets. And, actually, what will happen, is that your container will be stopped. It’ll actually fail.
And if you then go in and see what the results are, and why it stopped, you’ll see an error message that says something like, “The stopped reason was fetching secret data from Secrets Manager, access denied, exception to the user,” which is interesting to see, because the users going to show you an assumed role, using STS from the task execution role. And then, it’s just going to say it’s not authorized to perform GetSecretValue on this particular resource. So, you’ll find out really quickly, right?

Jon Christensen: Yeah.

Chris Hickman: If you’re missing this. And it actually-

Jon Christensen: That’s interesting that it just-

Chris Hickman: … It tells you-

Jon Christensen: … Shuts down your container. That is pretty fascinating.

Chris Hickman: … It does, yeah.

Jon Christensen: Yeah.

Chris Hickman: So, it will fail to spin up because of this.

Jon Christensen: Uh-huh. Right, right.

Chris Hickman: So, it is a very hard fail, right? It’s not a silent, soft fail. It is a in-your-face, you-are-now-broken-type thing.

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: So, very important to update your task execution role, to have that capability.

Jon Christensen: Well, super cool. We did it. We talked about secrets.

Chris Hickman: We did. We did. And hopefully, I mean, for your containerized applications on ECS, managing your secrets now using either Secrets Manager or Parameter Store, it is just really pretty easy. Once you get over these, like, “Okay, you’ve got to make sure that you have the roles, and policies attached to them,” but after that, it’s really just, you can have some way of responsibly creating these secrets inside one of those systems. But then, after that, just fetching that information is just so simple. It’s just updating your task definition file.

Jon Christensen: Right. And the two hard parts that remain are still rotating secrets for anything outside of RDS, or the integrated services. That could be a little tricky still.

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: And then, reacting to secrets that changed, if you are rotating those secrets. Dealing with that in the rest of your code.

Chris Hickman: Yep.

Jon Christensen: That could be a little tricky still.

Chris Hickman: Yep. The good news here, there is a new service with Systems Manager that basically deals with that. If you do have updates, how do you notify other subscribers that it’s using that information? How do they get notified? So, there is a really very new service that AWS just recently announced, and that is called AWS AppConfig.

Jon Christensen: Okay.

Chris Hickman: And it’s now, this is a new capability of Systems Manager. So, definitely something to look into there as well.

Jon Christensen: Awesome. Sounds like everybody’s going to stay employed if they just keep using AWS stuff.

Chris Hickman: Yeah. And, I mean, it’s a natural extent, right?

Jon Christensen: Yep.

Chris Hickman: We could just talk about, “Oh, these are the hard parts,” right? So, I mean, it totally makes sense.

Jon Christensen: Yeah, whenever there’s hard parts-

Chris Hickman: Because they’re hard parts for everyone, right?

Jon Christensen: … Yep.

Chris Hickman: So, it’s funny, because we, for one of our clients, we actually built this. Right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: Because we had the same problem, right? So, we basically built this configuration service, right, so that we could make updates, it could be managed centrally, and then, when updates were made, notifications went out to all subscribing applications that they could then reload. That information [crosstalk 00:44:37].

Jon Christensen: Yes, yes. Yeah, that’s such a common problem.

Chris Hickman: Yeah. So, now we can fold that down, and use something like AppConfig instead.

Jon Christensen: Super cool.

Chris Hickman: Yes.

Jon Christensen: All right. Well, thanks so much, Chris. I’ve got to run.

Chris Hickman: Great.

Jon Christensen: I’ve got things to do.

Chris Hickman: All right.

Jon Christensen: So do you, listeners.

Chris Hickman: Thanks, Jon. All right.

Jon Christensen: Talk to you next week.

Chris Hickman: Yeah.

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: Bye.

Jon Christensen: Bye.

Stevie Rose: Thanks for being aboard with us on this week’s episode of Mobycast. Also, thanks to our producer, Roy England. And I’m our announcer, Stevie Rose. Come talk to us on Mobycast.fm, or on Reddit, at r/Mobycast.