102. Automate all the things – Updating container secrets using CloudWatch Events + Lambda


Back in episode #94 of Mobycast, we showed how Amazon Elastic Container Service (or ECS) makes it easy to inject sensitive data into your containers.

However, ECS only injects secrets at container startup. It’s up to you to ensure that the container is restarted if secrets are updated. But who wants to manually restart containers?

In this episode of Mobycast, Jon and Chris are back to provide an automated solution to this problem. We show you step-by-step how to leverage CloudWatch Events and Lambda to automatically update your container secrets. After listening, you’ll be able to “automate all things”. Well… at least for updating container secrets :-).

Show Details

In this episode, we cover the following topics:

  • Developing a system for automatically updating containers when secrets are updated is a two-part solution. First, we need to be notified when secrets are updated. Then, we need to trigger an action to update the ECS service.
  • CloudWatch Events can be used to receive notifications when secrets are updated. We explain CloudWatch Events and its primary components: events, rules and targets.
  • Event patterns are used to filter for the specific events that the rule cares about. We discuss how to write event patterns and the rules of matching events.
  • The event data structure will be different for each type of emitter. We detail a handy tip for determining the event structure of an emitter.
  • We discuss EventBridge and how it relates to CloudWatch Events.
  • We explain how to create CloudWatch Event rules for capturing update events emitted by both Systems Manager Parameter Store and AWS Secrets Manager.
  • AWS Lambda can be leveraged as a trigger of CloudWatch Events. We explain how to develop a Lambda function that invokes the ECS API to recycle all containers.
  • We finish up by showing how this works for a common use case: using the automatic credential rotation feature of AWS Secrets Manager with a containerized app running on ECS that connects to a RDS database.


End Song

Night Sea Journey by Derek Russo

More Info

We’d love to hear from you! You can reach us at:

Coming soon…

Show Buttons
Hide Buttons