90. VPC Ninja – Part 2 – Private subnets with VPN (continued)


In episode #89 of Mobycast, we introduced using private subnets for your cloud network. We learned about the differences between public and private subnets, as well as some of the key technologies they depend upon such as NAT, or network address translation.

We also learned that using private subnets comes with a new problem – how to access these private resources? We discussed three primary approaches, before settling on VPN as our choice.

In this episode of Mobycast, Jon and Chris continue their three-part series on using private subnets with your cloud network. We finish our network design by guiding you step-by-step in setting up a software-based VPN and building out private subnets. We also share some inside tips that will make you look like a cloud networking pro.

Show Details

In this episode, we cover the following topics:

  • Before we get started, a CAVEAT. There are other (potentially BETTER) ways of accessing resources on private subnets.
    • We’ll talk about these (such as AWS Client VPN or AWS Systems Manager Session Manager) in future episodes.
    • But a great choice (with the most flexibility/power) remains our current choice: a third-party software-only VPN solution.
  • There are many options for third-party software VPNs, both commercial and open source. Some of the options we considered include:
    • SoftEther
    • Openswan
    • OpenVPN (* our choice)
  • Discussion of the different flavors and pricing models for OpenVPN Access Server.
  • Step-by-step walkthrough of installing OpenVPN Access Server via the AWS Marketplace.
    • Including how to setup TLS for your VPN server.
  • We detail the process of how to create private subnets within a VPC.
    • Create new subnets to be used as private subnets, keeping in mind a multi-AZ design.
    • Routing table considerations.
    • Setting up a NAT gateway to forward Internet traffic for private subnets.
  • Some pro tips to keep in mind when building out your cloud network.
    • CIDR block considerations (the “Goldilocks” approach to sizing).
    • Did you know that NAT gateways are SPOFs? We discuss how to improve availability.


End Song

Tachyon, by Roy England

More Info

For a full transcription of this episode, please visit the episode webpage.

We’d love to hear from you! You can reach us at:

Strevie Rose: In episode 89 of Mobycast, we introduced using private subnets for your cloud network. We learned about the differences between public and private subnets as well as some of the key technologies they depend upon, such at NAT, or network address translation. We also learned that using private subnets comes with a new problem: how to access these private resources. We discussed three primary approaches before settling on VPN as our choice.
In this episode of Mobycast, Jon and Chris continue their three part series on using private subnets with your cloud network. We finish our network design by guiding you step-by-step in setting up a software based VPN and building out private subnets. We also share some inside tips that will make you look like a cloud networking pro.
Welcome to Mobycast, a show about the techniques and technologies used by the best cloud native software teams. Each week your hosts, Jon Christensen and Chris Hickman, pick a software concept and dive deep to figure it out.

Jon Christensen: Welcome. Chris, it’s another episode of Mobycast.

Chris Hickman: Hey, Jon. It’s good to be back.

Jon Christensen: Yeah, good to have you back. Today we’ll jump right into it, because we’re part two of a series. This series is the VPC Ninja Series. In part two we’re going to continue on talking about connecting to your private subnets via a VPN. Before we get started, Chris, do you want to just recap a little bit, what we talked about in part one?

Chris Hickman: Sure. You bet. Yeah, in part one went through just subnet 101 and when you’re setting up your VPC and this concept of public subnets versus private subnets. In particular, private subnets and what they are and why we care about them. Things that are in your public subnets are resources that are available to the open internet. And so we only really put resources on our public subnets that really need that direct internet access. Everything else should on private subnets, which are not reachable by the open internet.

Jon Christensen: Right. And then just jump out even a little from there, the whole point here is a lot of times we’ll look at tutorials or we’ll be learning AWS stuff and we know in the back of our minds that there is some best practices that we should be using for security and just general setup or just what everybody does when they know what they’re doing. And this is part of it, putting things on private subnets. Everything, everything, everything on private subnets is part of being a good cloud native developer.

Chris Hickman: Absolutely. Yeah, and we talked a little bit about just how internet access works for resources on private subnets, in particular NAT or network address translation. We went really deep into NAP, understanding what it is and how it works. That was interesting. NAT is something that everyone hears the term, they know that I need a NAT entrance or NAT gateway, and NATs going on. But do you really understand what NAT is? If you don’t, if you can’t really explain it, go back and listen to that previous episode. You may … you’re going to learn some things I think and you’re going to find it interesting. I know I did.
And then we also talked a bit about IP addressing and a physical versus logical addressing.

Jon Christensen: Yeah.

Chris Hickman: Basically Mac addresses verses … which are tied to the hardware … versus IP addresses, which are logical, which are not necessarily tied to the hardware. This gets rise to like, “Well, how can a single host have more than one IP address?”

Jon Christensen: Right.

Chris Hickman: And so we talked a bit about that. And then we said okay, now we’ve said hey this is best practices, this is a good thing to have these private subnets and put our resources on them. Well, this opens up some new challenges of, “How do we access these things?” Because we can no longer … it’s no longer just a simple SSH and do it from your laptop at home, right? Because they’re not publicly accessible. So how do you get to those? And so we talked about three different ways of doing that. Basically broke it down into physical, the physical method, the leapfrog method, and the vitual method.
Physical was that I need a direct pipe between where I am and my cloud.

Jon Christensen: Got to share a data center.

Chris Hickman: Yes.

Jon Christensen: Are a data center? I’m not a data center.

Chris Hickman: I’m not a data center. Also, got lots of money.

Jon Christensen: Yes.

Chris Hickman: It is an expensive option. We talked about the leapfrog option, which is bastion host or jump box, jump host. And that’s where you basically have one machine that has the internet access that first connect to. And then from there you can now jump to the private resources. Talked about the pros and cons of that. And then finally the virtual option, which is essentially VPNs, virtual private networks. And that is the best of all worlds, right? It’s very convenient. It’s very flexible. It’s got lots of different options there, ranging from very expensive to essentially free. And you have options that require hardware and then ones that don’t require any hardware at all.
This week we’re going to talk about okay, now that we’ve said we want private subnets, we’ve talked through the various options of how you connect to those things, and we’ve decided let’s go ahead and focus on, we’re going to go with that virtual option, that VPN. Let’s go ahead and finish our network by setting up a VPN and building out private subnets inside our VPC. We now have this really robust VPC network design in the cloud that is built on best practices. That’s going to give us the best mix of security as well as able to do things out on the public internet.

Jon Christensen: This is like a how-to. How are we going to do this, Chris? This is a podcast. Are you going to crack open your machine and show us? How are we going to do it?

Chris Hickman: We’re going to use this brand new technology where we’re streaming video to your ears and pictures to your ears. No. This is … we’re going to walk through the steps. I think it’ll be pretty graspable. Is that a word?

Jon Christensen: Yes. It is today.

Chris Hickman: It is.

Jon Christensen: Language is in flex always. And today it just took on a new word.

Chris Hickman: Absolutely. I think it’s going to be something that it will be pretty easy to follow through on. Obviously we’re not going to go into all the specific details, but the broad steps will be there. And I think it’ll … after listening to that, anyone would have the recipe, if you will, to go do this on their own. I think it’s definitely worthwhile to walk through that.

Jon Christensen: Great. Great. Yeah, should we get started?

Chris Hickman: Yeah. Before we do get started, I just want to throw some … a copy out there. One, this is … I’m not saying this is the only way to do it, or even the best way to do it, right? Because there are actually other potentially better ways to access your resources on private subnets. We mentioned in the last episode one of the new VPN options that AWS offers. It is AWS Client VPN. That is definitely something to look that. It’s newish. It was announced back in December 2018. It is software only for remote access users. Kind of really right up our alley of the solution that we’re going for, that we’ll talk about today, but this is something that AWS provides and it’s managed.
There are some limitations with it. But we’re going to save that for a future episode. We’re not going to talk about that today. We’re going to … Just keep that in mind that that’s out there, but we know that isn’t us. But we’re not going to talk about that.

Jon Christensen: Okay.

Chris Hickman: And then the other option that is also potentially a really great option for this scenario is AWS Systems Manager Session Manager. This is a tool, technology that AWS provides through Systems Manager. Systems Manager installs agent software at all the machines that you want to manage. That agent then gives rise to the ability to connect to it.

Jon Christensen: Okay.

Chris Hickman: You don’t need to have a VPN to do this. You don’t have to have an SSH server installed. It’s one of those things that gives you a lot of what you need without having to have a VPN or without having a direct connect connection into your VPC. Again, there’s some limitations there and there’s some things you can do, some thing you can’t. That again, we’re going to talk about this in a future episode of Mobycast. It’s such a big subject.

Jon Christensen: Yeah, I could ask a single question and blow up this entire episode.

Chris Hickman: Yeah, you could. So don’t. We’re not even going to … You don’t even get that option. Right? I just wanted to throw out those caveats there that we do know about that and definitely can be very viable, but we’re going to save those discussions for future episodes. Otherwise, we would never get through anything today. We would-

Jon Christensen: Right. Right.

Chris Hickman: As Jon said, would blow it up. Then the other thing just wanted to point out is a great choice for doing this, again this third party software only VPN solution, which we teased at at the end of the last episode, but it gives you a lot of flexibility, a lot of power. It’s a great choice. This, what we’re going to talk about today, is still a very valid, good choice for you to know about and for you to consider.

Jon Christensen: Let’s throw out another benefit of it. It’s works in AWS and it works in not AWS, right?

Chris Hickman: Absolutely.

Jon Christensen: Yeah.

Chris Hickman: It’s … yes. There’s nothing AWS specific about this choice.

Jon Christensen: Cool.

Chris Hickman: Yeah. You can use this in Azure. You can use it in Google. You can use it connecting two on-prems together if you wanted to. Right? Yeah.

Jon Christensen: We also did leave out another way of connecting to a private subnet, which is to make a phone call to somebody in the data center and then tell them to tell you what they see on the screen.

Chris Hickman: Yeah.

Jon Christensen: Have I ever done that? Yes, I have done that.

Chris Hickman: Yeah, that won’t work with AWS, because there are very few people allowed inside those data centers.

Jon Christensen: Right. Right.

Chris Hickman: I do remember back in the day when I was at Microsoft, with our data centers, and it was much more accessible. It was locked down and you had to go through several layers of security, but I spent a lot of time inside of those data centers as just deploying services and debugging it because we didn’t have the tools actually to do the remote access.

Jon Christensen: Yeah.

Chris Hickman: This was really the only way. We would joke that the operators there would have to put on roller skates to replace failed drives, right? Because it’s such a big data center, all these rows and racks and racks and racks of servers and drives. You have hundreds of thousands of machines, right? Therefore, hundreds of thousands of drives. They’re going to be failing constantly, so put roller skates on the operators so that they can get to those racks quickly-

Jon Christensen: Yes.

Chris Hickman: And swap out the drives. Yes.

Jon Christensen: Totally.

Chris Hickman: But different story now. This third party software on our VPN solution that we’re going to be going over today. The rationale for choosing that is it is very cost effective, doesn’t cost a lot of money. It’s going to work for remote access users. You don’t need to have an on premise location, install a switch or a rod or what not. And then this also … it’s pretty sophisticated that’s easy to manage. Really for broad strokes, it’s a good solution for a lot of you out there that don’t have dedicated network engineers, that don’t have a huge budget. We’re going to go with that.

Jon Christensen: Cool. All right, let’s jump right in.

Chris Hickman: Yeah. And maybe just a personal story too. We were … Jon, you and I were talking about this a little bit earlier, right?

Jon Christensen: Yeah.

Chris Hickman: Where I have my own personal AWS account. That’s where I host my world famous personal blog that we’ve talked about. I think three page views in the past month. But that’s another story. But I have my own-

Jon Christensen: Shows you’re keeping track there.

Chris Hickman: Yeah.

Jon Christensen: That’s good.

Chris Hickman: Google Analytics.
Yeah, I have my own personal AWS account. I’m running things like ECS and these … My personal blog is hosted as a Node.js application and using load balancers it’s using TLS and all that good stuff, right, that we would expect for best practices. But the one that it hasn’t had is it hasn’t had private subnets. Everything has actually just been public subnets. And this is true for … I created this probably about four years ago. Up until recently, it has been all public subnet, even though I know best practices. Hey, my database server should be on private subnet, right? And my ECS host should on private subnets.

Jon Christensen: I think it’s really common when you’re getting started on something that you just don’t want to fiddle with that, right? It’s just easier to just throw open all the privileges, you’re using Root, you’re having wide open security groups, you’re using public subnets. All the things that they tell you not to do actually make life a lot easier when you’re getting started on something.

Chris Hickman: Oh, absolutely. Absolutely. Right? It’s-

Jon Christensen: Then … But I guess to add to that, it’s still not a good idea because you never do get around to fixing all those in a lot of cases. So maybe it’s better to just make it easier on yourself to do the right thing so that when you get started on something it’s not a seven day journey of just following a few best practices that should have been fairly easy if you had taken the time to learn them and get them into muscle memory. That’s what this is about.

Chris Hickman: Yeah. Absolutely. For me personally, it was just this nuance that comes up of okay once you do have private subnets, how do you connect to those things? That was, for me, the thing that prevented me from creating private subnets, right? Because, for me, there’s two big issues. One, was just the time involved with going and figuring out okay what do I want to … what solution do I want to use? And then, once I figured that out, then going getting it installed and configured. There was just some amount of effort to do that. The second thing that was actually a pretty big consideration too was cost.
This is my personal AWS account, it’s coming out of the personal credit card. It’s really just for my own learning and playing around with things and then hosting my own personal blog. How much money can I justify to spend on that per month? For me to go to private subnets, that means things like I’m going to have to now pay for a NAT gateway. I’m going to have to pay for a VPN in some form or another, whether it be hosting on an EC2 or pay license fees or what not. It’s going to dramatically jump the cost.

Jon Christensen: Which would really be darn close to free, right? If you’re doing hobbyist development and you’re paying monthly fees, it feels a little bit wrong because all the big companies, in order to attract developers, need to make sure that they’re not forcing developers that are hobbyists to pay a lot of money out of pocket.

Chris Hickman: Yeah. Yeah. There is a lot available on the free tier at AWS, but unfortunately a NAT gateway is not one of them. Now I’m looking at instead spending about $30 a month to have ECS fronting by ELB and TLS, a pretty nice robust solution, but I want public subnets to go to private subnets ended up probably changes that bill to go from $30 a month to about, could of actually about $50 more, probably at about $80 a month now. It’s pretty significant, right?

Jon Christensen: Yeah, that is. You’re already paying for cable and Netflix and HBO Plus, not yet Disney Plus, but that’s going around next year.

Chris Hickman: Yeah. Yeah.

Jon Christensen: Yeah.

Chris Hickman: So for the personal hobbyist, for the individual developer, this is a real concern, a real issue. If you’re business, right, this is not a concern at all.

Jon Christensen: No. No.

Chris Hickman: We’re talking to folks out there that are doing this for work, for sure probably. If you are doing this for a hobby, go ahead and do all these steps, create it all up, and then just tear it down. Right?

Jon Christensen: Right.

Chris Hickman: So you know how to do it, you can keep that working. That way it’s only going to cost you a few bucks. And you get to go through this great process of just knowing exactly how to do it. I think we’ve teased the start of this three times now I think. Now we can finally get-

Jon Christensen: Shall we get started?

Chris Hickman: Yeah. Okay, we’ve decided we want a VPN solution. We’re going to go with a software only VPN solution. We’re going to go look at what are the third party options out there. There are tons of options out there in this space ranging from paid commercial products to free open source products. You can go to the AWS Marketplace and do a search for VPN and you will see just a bunch of options there for manufacturers like Checkpoint, Microsoft, Sysco, just lots of options. But again, you’re going to pay for it as well. There’s open source options out there like SoftEther, Openswan, and there is the community version of OpenVPN. There’s probably even other ones out there.

Jon Christensen: I’m sure you could run through your spam folder and find four or five different options.

Chris Hickman: Probably. Indeed, along with-

Jon Christensen: Yeah, don’t choose those options.

Chris Hickman: Give us remote access to your machine and we’ll set it up for you because your Windows server needs to be updated.

Jon Christensen: Right. Right. Exactly. That’s what I was getting at.

Chris Hickman: Yeah, please don’t do that. Let’s … We want to keep cost down and still have an easy to use, well supported version. I looked at SoftEther, it is a viable option out there but it’s very much one of those things where it’s going require a lot of work. It’s not particularly well supported. In the past someone put together an AMI and put it on the marketplace that you could go and use. That’s no longer even there.

Jon Christensen: Boo.

Chris Hickman: It’s just suffered technical theft and rot. You really have to go build your own now. Again, possible but a lot of work. Same deal with Openswan. OpenVPN though, there’s a company behind it. It is open source and you have the community version, which you can go and doesn’t cost a dime and you can go configure all that and set it up and run it yourself. But then they also have the paid versions, and those are very well supported. They have all the software around them that does the administration and just the set up, right? Just makes it so much easier. OpenVPN is very well supported, lots of folks use it. It is one of the ones that AWS would recommend I think for a VPN service in AWS.
This comes in two flavors. One is hosted, where you just pay by the hour through Amazon, through AWS. When you do that it works out to call it about $2 a day or $60 a month to do that way. But that includes the EC2 as well-

Jon Christensen: Right.

Chris Hickman: Behind it. And then they also have a BYOL version, which is bring your own license. This one you only pay for the EC2 resources that you use. For me, in my particular case, I can host this on a t3 micro and that ends up being about $7.50 a month to host.

Jon Christensen: Okay.

Chris Hickman: Then I just have to take care of the licenses. You can buy licenses from OpenVPN at $15 per user per year with a minimum of 10 users. It’s $150, one time fee. Now you have up to 10 users that could use it and you’re good for a year. You’re looking at $7.50 a month plus $150 a year.

Jon Christensen: That seems fairly startup friendly on pricing.

Chris Hickman: Yeah, it is.

Jon Christensen: Yeah.

Chris Hickman: Yeah. Yeah. It gives you 10 users. That’s again-

Jon Christensen: For a whole team for a year.

Chris Hickman: Yeah, absolutely. But the little loophole, or really nice thing here is that if you choose the BYOL of it, you don’t actually have to give it a license key to use it. All functions are available without any time limit. The only limitation is that you can only have two simultaneous VPN connections to it.

Jon Christensen: Ah.

Chris Hickman: For me-

Jon Christensen: Yeah, that’s great.

Chris Hickman: This is perfect, right?

Jon Christensen: Yeah. Hobbyist level.

Chris Hickman: Yeah. This is what I did. I went to the AWS Marketplace and searched for OpenVPN Access Server. I chose the BYOL version and then I’m good to go. Now the only I have to pay for for this VPN really is just the $7.50 a month for hosting the t3 micro on AWS.

Jon Christensen: That is not an expensive VPN. That’s great.

Chris Hickman: Yeah. Yeah. Yeah. That’s what we’re going to do, right? Let’s go and install OpenVPN Access Server. What are the steps that we need to do? What we’re going to … The very first step is we’re going to go launch a new EC2. We’re going to go into the AWS console, go to the EC2 dashboard and launch a new EC2. When we do that, we’re going to go to the marketplace, right? The marketplace would be an option there for it. We’ll click on that and we will be able to now search. You just type in OpenVPN Access Server. You’ll see those pre-built AMIs. Choose the BYOL version.

Jon Christensen: Okay. You’re essentially looking for an AMI that’s provided by OpenVPN?

Chris Hickman: Yes.

Jon Christensen: Yeah.

Chris Hickman: Yes. Now you’re just going to go through the normal process, right, of whenever you spin up a new EC2, right? It’s just this is the way the AMI … instead of choosing Amazon Linux 2 or something like that, you’re actually just choosing this AMI through the marketplace that’s already pre-built for you. You go through and you’ll tell it what VPC you want it all on, what kind of instance type you want. The one thing that you need to make sure that you do when you install this is just make sure that you’re selecting a public subnet for the instance, right?
Your VPN server has to be on a public subnet, not a private one. Otherwise you’re not going to be able to connect to it.

Jon Christensen: Sure. Makes sense.

Chris Hickman: Yeah. Once we’ve now spun that up, another thing we’re going to want to do is going to assign an elastic IP to it. That’s so that we can have this dedicated IP address that is well known, that we’ll be able to use to connect to it.

Jon Christensen: Cool.

Chris Hickman: Another step that I did is I went and I created a Route 53 A record for this particular EC2. I can give it a friendly name so I can remember it. For me, it’s VPN.ChrisHick.com. That is now assigned to that elastic IP. Now I can just whenever I need to connect my VPN, VPN.ChrisHick.com.

Jon Christensen: If you want to try to break into his VPN, it’s VPN.ChrisHick.com.

Chris Hickman: Yes. Indeed. I have it pretty well locked down so I welcome the challenge.

Jon Christensen: DDoS only is what’s going to work on that, because you can only have two active connections as once so it should be pretty easy everybody.

Chris Hickman: Well you have to successfully connect to it, right?

Jon Christensen: But can’t you-

Chris Hickman: Who’s going to-

Jon Christensen: Doesn’t a pattern of failing try-

Chris Hickman: Know the password’s password123? Yes. Now we’ve spun this up. We’ve got the IP address. And now we’ve got the DNS record for it as well. And then there’s just some general just setup, installation, admin stuff to do, right? It’s you’re going to SSH the instance. There’s an Install Wizard. You just run that when SSH into it. You can then go and definitely change the default password, right? That’s … this is when you install a route-

Jon Christensen: Probably should do it on your VPN server, yes.

Chris Hickman: Yeah. Yeah. Admin, admin, not so good. Change that, update the OS, those kind of things. Then at that point you can then log into the web based UI. Now you can have this really nice way of configuring your OpenVPN server. One of the things to point out here is that how do you manage users? OpenVPN gives you a range of options. If you have LDAP, it can use that. If you have RADIUS, you can use that. It also … but it supports this really, really straightforward easy thing called local authentication. What that is is it’s just keeping … it has basically a user database that’s local to the machine.

Jon Christensen: Okay.

Chris Hickman: You don’t need any integration. It’s really easy to do so you can just configure users right there. And just here’s the username and a password, right? And just set them up. It makes it just really easy. This is one of the things that AWS Client VPN does not have you … you can’t do with it.

Jon Christensen: Oh, in Client.

Chris Hickman: With Client VPNs, setting up users is much more challenging.

Jon Christensen: I imagine you have to pull in other AWS services, which is kind of typical of AWS. You might have to connect to IAM or Cognito or something like that.

Chris Hickman: You do. It actually uses certificates.

Jon Christensen: Okay. AWS Certificate Manager.

Chris Hickman: Unfortunately-

Jon Christensen: There you go.

Chris Hickman: Kind of. You actually have to create the certificates yourself-

Jon Christensen: What?

Chris Hickman: Using open source tools that you then import into Certificate Manager.

Jon Christensen: Remember what I said about blowing up the episode?

Chris Hickman: There it goes. No, we’re going to keep moving on.

Jon Christensen: Okay.

Chris Hickman: Another great feature that OpenVPN has is it supports MFA through Google Authenticator.

Jon Christensen: Oo, cool.

Chris Hickman: You can lock it down even more that way. Then the last thing you need to do is you want to set up TLS for VPN, right? You want to be able to access it securely, encrypted, encryption of the wire. It does come with a self signed certificate, right? But that’s not going to work for if I want VPN.ChrisHick.com. It’s not going to work with that, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: This was one of the challenges. It was like how do I want to do this?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: I thought of … There’s two possible ways I thought, right?

Jon Christensen: I’m imagining you were thinking about putting it behind a load balancer?

Chris Hickman: Yeah. Absolutely. That was my first … it was like boy, it would be so nice to use AWS Certificate Manager-

Jon Christensen: Yeah.

Chris Hickman: To … I already have a wild card search for my domain in AWS Certificate Manager so wouldn’t it be nice to use that? What if I fronted this with an ALB and then use that certificate, right? Because with the certificates that are created with Certificate Manager you have use those with AWS Resources and specifically you can really use them with ELBs, Cloudfront, and API Gateway. Not going to use Cloudfront or API Gateway for this, but ELB, yeah. What if I go ahead and create a target group for my EC2, that OpenVPN Access Server EC2, add that to an ALB and then have the ALB perform the TLS termination? That was my first thought, right?

Jon Christensen: Yeah.

Chris Hickman: Very quickly discovered, unfortunately, this is just not going to work for a bunch of different reasons. OpenVPN itself does work behind load balancer. ALBs, in general, are really designed for HTTP and HTTPS traffic, so port 80, port 443. OpenVPN uses some other ports as well.

Jon Christensen: Okay.

Chris Hickman: That makes it problematic. And then getting health checks to work and just dealing with these other port issues, right, just for all of these reasons it’s like okay, this is just not going to work all that great. Had to fallback to the option I know was going to work all along, but I didn’t want to do it, which is okay we have to go basically create SSL cert, right, and install it on the EC2 itself. That’s the route that I went. This is pretty-

Jon Christensen: Yeah. It sucks. It’s rolling back time to 2015. There you are cracking open-

Chris Hickman: It really is.

Jon Christensen: SSL and doing it.

Chris Hickman: Yeah.

Jon Christensen: Yeah. Yeah.

Chris Hickman: The good news is is in the past it used to be wow, this costs $400, right, to go get an SSL certificate. And it used to take days and days for it to be made available because they had to do various checks, right, and what not. That’s now changed with now you can do things like email validation or domain DMS validation. Then now there’s companies that do this for free as well. Let’s Encrypt is definitely one of those free ones. I ended up using ZeroSSL, which is like a really nice wrapper around the process for using Let’s Encrypt.

Jon Christensen: Okay.

Chris Hickman: Very quickly, literally within a matter of 60 seconds using ZeroSSL had a Let’s Encrypt SSL cert for my VPN. Then I just installed that cert using that web based UI. Then I was off and running.

Jon Christensen: Oh, that’s handy. That’s nice that they did that. You didn’t have to go dig around in the file system of EC2 to figure out where to put it.

Chris Hickman: Nope.

Jon Christensen: And then change its mod to make it executable or something.

Chris Hickman: Nope. Just-

Jon Christensen: That’s good.

Chris Hickman: Prepare your cert and upload through the web based UI and away you go.

Jon Christensen: That’s very good.

Chris Hickman: Did that and now I have my TLS. And then the last thing for me was just to lock down the security group for the EC2 that was hosting the VPN software, right? Understanding exactly what ports, the minimal number of ports that I wanted open, and where I was allowing that traffic from, right? Things like I still want to be able to SSH into this, but I made it so that SSH is only from inside the VPC. Only once you actually have a VPN connection can you SSH to that particular machine.

Jon Christensen: We cover a lot of information here on Mobycast. And if you’ve ever wanted to go back and remind yourself of something we talked about in a previous episode, it can be hard to search through our website and transcripts to find exactly what you’re looking for. Well now it’s a lot easier. All you have to do is go to Mobycast.fm/show-notes and sign up. We’ll send you our weekly super detailed outline that we use to actually record the show. A lot of times this outline contains more information than we get to during our hour on the air. So sign up and get weekly Mobycast cheat sheets to all of our episodes delivered right to your inbox.
Let me just make sure that I know where we are here. We chose OpenVPN. You started an EC2 instance with OpenVPN’s AMI. Then you installed … I’m sorry, then you configured your, as part of configuring OpenVPN, you set up user … or wait a minute. Let’s back up a little bit.

Chris Hickman: Sure.

Jon Christensen: After having the EC2 running, you got in there and did you have to do anything with the command line? Or-

Chris Hickman: Yeah.

Jon Christensen: Other than just-

Chris Hickman: Just access … after it’s been spun up, we assign the EIP to it, we created the DMS name for it-

Jon Christensen: Yep. Yep. Yep.

Chris Hickman: We can then SSH into it. You’re going to be presented with okay, hey run this Install Wizard. You’ll go through that. It’ll ask you some-

Jon Christensen: Oh, it automatically does that? It’s like hey, user, do this.

Chris Hickman: You’ll see that. The message of the day is configured to show you this, right?

Jon Christensen: Cool. Sweet.

Chris Hickman: Yeah. You go and run that. Then once you run that Install Wizard, basically you’re telling it what ports to actually start up the VPN service.

Jon Christensen: Okay.

Chris Hickman: At that point you can then exit out, disconnect from that, and then that’s when you now log into the web based UI, right?

Jon Christensen: And then the web based UI is where you’re setting up your user accounts using, in your case, the local authentication?

Chris Hickman: Exactly. Yep.

Jon Christensen: Okay, cool.

Chris Hickman: Then you go and change things like the host name so that it’s now VPN.ChrisHick.com and you can set up MFA-

Jon Christensen: Cool.

Chris Hickman: And those kind of options and what not.

Jon Christensen: And then lastly, you set up TLS and you get that by using not the … what was the name of the TLS service or the certificate service that you used?

Chris Hickman: Use ZeroSSL.

Jon Christensen: ZeroSSL.

Chris Hickman: Which is a wrapper-

Jon Christensen: And that gave me a-

Chris Hickman: Around Let’s Encrypt.

Jon Christensen: Let’s Encrypt. That’s right.

Chris Hickman: Yeah.

Jon Christensen: Okay, cool. All right, now you’ve got the whole thing running and it’s behind TLS. What are we going to do next?

Chris Hickman: Yeah, now at this point we have our VPN access to our VPC, right? This is, we now have a way to access private subnets. Let’s go ahead and now we need to create some private subnets.

Jon Christensen: Okay.

Chris Hickman: Let’s walk through that real quickly.

Jon Christensen: There’s a really subtle point you just made. It’s like set up your VPN before you set up your private subnets and then your life will be easier when you have those private subnets. You don’t have to then … because you have your VPN and you make sure it works. And then you can set up the private subnets because then you can get into them. You don’t have to go … It’s like I’m imagining unit testing. The other way you could have done it is to set up private subnets that you wouldn’t be able to access and then set up your VPN and keep banding on it until you can access them?

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: That seems like more the unit testing approach, but the approach that you just mentioned actually seems easier, seems like less possible things could go wrong if you get that VPN set up first and you know that the VPN not going to be the problem. It’s set up. It lets you get into your VPC.

Chris Hickman: Yeah. It’s less risky too, right? It’s like imagine you say oh, go create my private subnets first. I put my database on there. And then now I’ll go set up my VPN. Well, in the meantime you can’t access your database, right? Or maybe it’s your ECS cluster or something like that. Maybe you’re having problems doing that and now all of a sudden a crisis comes up and you actually need to access those machines. Well, sorry. Not going to happen. You have no way to get there unless you maybe you have a friend that works at the data center you can call, right, or something like that. It’s just not going to happen.
Anyhow, if we create the VPN first and we know that it works, then proceeding with building out our private subnets is just takes away that risk. And again, now you know it’s another problem, it’s not the VPN. If you do have problems, right?

Jon Christensen: Right.

Chris Hickman: We talked about private subnets are just subnets and it’s just their routing is different, right? They’re not accessible from the internet is what makes them different from public subnets. What we’re going to do is we’re going to create some new subnets that will act as our private subnets. One thing to consider with this, right, is okay you want to … a subnet is tied to a specific AZ. All right, you can’t have a subnet that’s spread over multiple AZs. We know that we want a multi AZ architecture, right, so that we have more resilience to failures and have higher availability. At the very least you want at least subnets in two AZs. In general, if you can go to three and if your region supports that then that’s even better, right?
Again, a good example would be if you had two AZs with 50% of your load in each AZ, if an AZ fails, right, then that means 100% of your load is not being brunted by the single AZ, which may be a problem.

Jon Christensen: Sure.

Chris Hickman: Versus if you have three AZs and a third of your load is spread across each AZ, if one fails then it’s now half the load is on each of the remaining two AZs.

Jon Christensen: Right.

Chris Hickman: Which is a lot more doable. We’re going to choose … I’m running in us-west-2, Oregon, and that currently has four AZs available. We’re going to choose three. Let’s go create three new subnets, one per AZ. We’re going to go into the console there, go to the VPC area, go to subnets, and just create new subnet. We can pick our CIDR ranges. For keeping things simple, for me personally, especially since this is for my own personal site, I can use a /24 range, which gives me up to basically 256 addresses, right?

Jon Christensen: It feels perfect, right? It’s more than you’ll ever need, but a lot.

Chris Hickman: Definitely, for a personal account for sure, /24 is a good choice. Create three subnets. I’ll put them in each one obviously to make sure each one is in a different AZ. So I’ll name them something that makes sense to me. It would be like private-A for the private subnet that goes into the us-west-2a, and private-B that goes in us-west-2b, and private-C for us-west-2c type thing. We now have our three subnets spread across three separate AZs.
The next thing to do is you need to create a new routing table. This is going to be our private route table. There in the VPC section there’ll be a route table option over on the left side. We’ll click that. We’ll now say we want to create a new route table. In this route table, subnets get a route table associated with them, right?

Jon Christensen: Let’s just define what a route table is real quick. It’s a thing that’s going to in your mind, as you imagine a network, you can imagine it living on the routers that sit between where all your computers are and the rest of the world, the rest of the internet. That’s maybe not exactly where they are, or what they are, but for all intents and purposes it seems like a fair enough visual to have in your mind that there’s this little machine that sits between your computers and the internet that your machines can talk to to figure out what they’re going to do when they do stuff.

Chris Hickman: Yeah.

Jon Christensen: Yeah?

Chris Hickman: You can think of it as it’s you have a password … you have some-

Jon Christensen: It’s a router, right? And on the router is going live this route table. That’s what I’m getting at. There’s … I imagine a physical device, but it probably isn’t. It’s in charge of deciding where things are going to go, where messages or packets essentially are going to go. And the reason that I like to picture it between my subnet and the rest of the world is because it can talk to … every single one of the computers on my subnet can talk to it directly without talking to any other machine. That’s the main thing I’m getting at. They all have direct wired up access to this thing, whatever this is essentially.

Chris Hickman: Yeah, maybe it’s-

Jon Christensen: I don’t know what really is happening, but … Sorry to interrupt, but I don’t know really is happening on the floor and the AWS data centers and it could be that my entire subnet is just in two actual computers. And even though there’s several VMs I don’t know what’s really going on and the router could be just some software running somewhere. And it could be that there’s even multiple hops to get from those machines to that router, but in my mind, as I think about this, the machines that are running, that I’m setting up a route for it, they have to be able to know where to sends packets. So they need to ask, they need to query a route table. Basically, hey, route table, tell me what to do with this.

Chris Hickman: That’s exactly what a route table is. Basically just a lookup table, right, that says okay … By the way, the route table lives on every machine, right? When you assign a route table to a subnet, basically any resource spun up in that subnet gets that route table loaded onto its machine itself. When the computer … when your EC2 host needs, says my networking, I need to send a packet to this other particular machine, it’s going to look at its route table to figure out where it should send that packet to, right?

Jon Christensen: Oh, yeah.

Chris Hickman: It’s the instructions for-

Jon Christensen: That’s a little bit of a … yes, you’re right about that, but also you can change the route table locally. You can actually break your routing by just going onto your route table inside Linux on specific machine that’s in that subnet and break its connection or break its networking by just changing the route table there, or augment it, not just necessarily just break it, but augment it or change it. What I think you’re getting at is the whole protocol of the way route tables … they have … like the units themselves, they have a hierarchy to them. I think what you’re essentially saying is at the router level that gets copied down to the local level, but you can still change it locally, too.

Chris Hickman: A router has its own route table. Just like each computer has their own route table. And they may get them from different places, right? But it’s just again, it’s just a lookup table that say okay, I need to send a packet to this IP address, the route table tells me how to do that. If there’s no range in that route table for that particular IP address, then it has no idea where to send it, so it can’t, right? It just gets dropped on the floor.

Jon Christensen: Yep. Yep.

Chris Hickman: Actual physical networking equipment, like a router, it just has its own route table, which that’s going to be perhaps even different, right, than-

Jon Christensen: Yeah, I guess I really do want to understand, because this has always been one of things that’s confused me about using AWS. And maybe if you can help me through this, other people will learn from this too. I was trying to say that the route table lives somewhere else. And the reason I was trying to say that is you literally don’t configure it on every machine. You literally are not in each machine setting up a route table when you set up a private subnet. In fact, you might not even have any machines in your private subnet. In fact, when you create a private subnet, you don’t. There’s nothing in there. But you’re setting up a route table. So where is that thing? And I guess my question is, is it living on a router and getting copied to every machine, or is it living somewhere in an S3 bucket or something? And as machines get spun up there’s some software that knows to go and place that route table into each machine?

Chris Hickman: Yeah, I think for AWS, in their case, this is just a service, right, so it’s all software. It’s just somewhere so that when you spin up an EC2 inside of a subnet, right, that’s all software and it’s part of its initialization and configuration. It’s okay, what’s the route table associated with the subnet and now configure that EC2. Its local … its route table now has that information, right?

Jon Christensen: Mm-hmm (affirmative).

Chris Hickman: That’s its route table now. And if you make changes to that, right, if you make changes to that in the VPC console, right, then those changes get pushed to-

Jon Christensen: Right. Right.

Chris Hickman: Other machines over there as well. But it’s all there local to the EC2, so that when it wants to send-

Jon Christensen: Right.

Chris Hickman: Network packets, it knows where to go.

Jon Christensen: I think something that I said was incorrect then. Let’s correct something I just said, which was I was suggested that this route table concept is hierarchical, like DMS is, and maybe there’s a master route table and individual route tables. That’s not true, is it?

Chris Hickman: No.

Jon Christensen: Not true. Okay. That was incorrect. But the reason I got to that conclusion is because you can make a route table without any machines your network. And so I was like oh, well it must be for something. Maybe it’s in the router, right? See how I deduced that? The route table’s got to be somewhere, maybe the router is owning the route table and the machines as they go on using the router’s route table. But no, no, no, no. That’s not it either. It has its own route table. The machines have their own route table. And this route table that you make in AWS is just like, I guess it’s just a file that it’s storing somewhere and then applying to each machine as it loads up.

Chris Hickman: Yep. Yeah.

Jon Christensen: Yeah, okay. Cool. Cool.

Chris Hickman: The subnet’s not complete without a route table, right? You couldn’t … If you didn’t define a route table for VPC, even though there is no … for a subnet, even though there’s no machines in it, well once you did put a machine in then its networking’s broken, right? It’s the subnet definition was incomplete without the route table.

Jon Christensen: It’s so wildly confusing for me because so much of what you do on an AWS Linux machine is you’ve got to do it. You’re in there and you’ve got to actually set it up yourself. If you want to update the operating system, go update the operating system. Log in and update the operating system. But other stuff is handled in the console separately, like this route table thing. Yes, it is something that the operating system actually needs to get into itself and has to be installed in there. You could, like I said, you actually really can go change it.

Chris Hickman: Oh, absolutely. This is a whole other topic. We’ll talk about it in a future episode about how Docker works, but when-

Jon Christensen: Okay.

Chris Hickman: You’re running containers, they’re manipulating route tables, right-

Jon Christensen: Yeah.

Chris Hickman: To set up these other networks, right, so that there’s a private docker network that gets created, right? That’s all done through route tables. How does container networking work on a particular machine inside Docker? This is all route tables-

Jon Christensen: Right.

Chris Hickman: IP tables and what not.

Jon Christensen: Knowing the difference between the stuff in the AWS console that applies to well this is how the hardware is configured and is over … infrastructure, I guess with a capital I versus the stuff that’s actually configuring the operating system of the machines that you fire up is tricky. There’s not a line in the console that’s like everything above here is wires and machines, and everything below here is your operating system stuff. That would be kind of nice, from a user experience perspective?

Chris Hickman: Yeah. I don’t think the line would be so clean. It would … I think it would end up being a circle that looked like a blob. It just wouldn’t be clean. There’s just so much going on and things have blurred between things that used to be in hardware and are now in software, things that used to be in software are now in hardware. Sometimes now there’s special chips being made for certain … It’s just always-

Jon Christensen: Yeah. Yeah.

Chris Hickman: Changing, right?

Jon Christensen: Yep.

Chris Hickman: Yeah. But the principles are there, right? That’s not changing.

Jon Christensen: Right.

Chris Hickman: That’s why if you understand networking 101 then setting up a VPC ends up being like yeah, this makes sense. I understand this. I can use some of the same concepts and techniques. These all apply. How they actually get implemented might be different, but the concepts apply.

Jon Christensen: Cool.

Chris Hickman: Yeah. Cool. We have our private route table now that we’ve created. We’ll now assign that to our private subnets. Another thing we’ll want to do too is we’ll just want for the VPC that we’re putting these subnets in, we’ll now want to make sure that this new private route table is the default, the main route table for the whole VPC. That way when we’re creating subnets, by default they’re going to be private. We have to manually change the route table to be a public route table to make them a public subnet. It’s just again a best practice type thing to err on the side of being more conservative with the access.
After we have the route table, then we need to actually to leverage to actually get out to the internet. That’s where the NAT gateway comes in. We need to create a NAT gateway. We talked all about NAT in the previous episode.

Jon Christensen: Right.

Chris Hickman: AWS has this service. NAT gateway is managed. It’s scalable. It’s up to 45 gigabit bandwidth. This is definitely the way they go to do NAT in your VPC. We just create a NAT gateway. We put it into a public subnet. It’s very important, again, it needs to be in a public subnet.

Jon Christensen: Next to our VPN machine?

Chris Hickman: It doesn’t have to be in the same subnet as our VPN server, but it does need to be in a public subnet.

Jon Christensen: Okay.

Chris Hickman: Then assign an elastic IP to it. Once we’ve done that, now we can go update that private route table to say okay, for all traffic that’s not internal to the VPC, we’ll add a catchall route that gets … and that forwards to the NAT gateway. Basically it says hey, if this network packet is going to somewhere other than inside the VPC, okay it’s going out to the internet, then here’s the catchall route for that, which says everything else and that goes to the NAT gateway. Then the NAT gateway will then make sure that that now goes out through the internet gateway. Now this way machines on the private subnets can now make outbound connections to the internet through NAT and through the internet gateway.

Jon Christensen: Right on. It’s backwards from that, right? The first statement in the route table is hey, everything in this route, using this /24 for your center block, everything inside this subnet, whatever the numbers are, and then /24 basically just say this range of IP address says inside. And then you say 0.0.0, everything else, goes out to here.

Chris Hickman: I think that’s the it gets displayed to you. It doesn’t matter what order you create it in because at the end of the day, it’s the most-

Jon Christensen: Well, wouldn’t it though?

Chris Hickman: No, because with route tables-

Jon Christensen: Because you’re looking at it and you’re like hey, so everything’s-

Chris Hickman: The most specific route wins. They’re not applied in order-

Jon Christensen: Oh, okay.

Chris Hickman: Like NACLs

Jon Christensen: Oh, okay.

Chris Hickman: No, they don’t get applied in order like NACLs, right? It’s the most specific route wins.

Jon Christensen: Okay. Secrets of the pros right here.

Chris Hickman: Well, it would be chaos, right? This would … everyone … could you imagine-

Jon Christensen: Yeah, it would be, right?

Chris Hickman: The uproar-

Jon Christensen: Good point.

Chris Hickman: Because everyone would be cut out by this and they’d be like, “Why do you make it this” …

Jon Christensen: Cool.

Chris Hickman: Yeah, and that’s it. The last step is just … then we can just test and verify that now our VPN’s working and we can access something on our private subnet. Me personally, I just spun up an EC2 on one of those private subnets. It didn’t have a public IP, it’s just a private IP. I then enabled a VPN connection. And then I did SSH from my laptop into that private IP. First time was the charm. There you go. We now at this point, we have it set up. We have our VPN. And then we have our private subnets. And we have machines on those private subnets that are not accessible from the open internet. You have to have the VPN connection in order to connect to them.

Jon Christensen: Wait, I missed … okay, I missed the part where the VPN on the public subnet was configured to be able to get access into the private subnets.

Chris Hickman: It gets that by default, right? Because it’s inside the VPC. It’s using the public route table.

Jon Christensen: Okay.

Chris Hickman: And the public route table has a route in there that says okay, for anything … it can talk to anything inside the VPC.

Jon Christensen: The entire VPC can see itself?

Chris Hickman: Mm-hmm (affirmative). Yeah.

Jon Christensen: Yes.

Chris Hickman: You’ll see this by default. This is what route tables will add for you when you have a route table in your VPC. In fact, I believe you can’t even change it; you can’t delete it. It’s just there.

Jon Christensen: Oh, seriously? Because I was just about to say I bet if you wanted to you could actually lock off … you could create a public subnet that was in the same VPC as a private subnet, locks the private subnet away from the public subnet, even through they’re all in the same VPC. But what you’re saying is, “Nope. Nope. The very first thing that happens when you make a VPC is it gets a route table where everything inside the VPC gets to see everything else”?

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: Okay. Interesting.

Chris Hickman: Yep.

Jon Christensen: Ah, this stuff. It doesn’t stick. It’s weird how certain code stuff sticks in my head, but networking stuff, for whatever reason, does not stick in my head the same way.

Chris Hickman: Yeah, well-

Jon Christensen: There you go.

Chris Hickman: It’s a big open topic, right? There’s lots of things to know. This is why AWS in their certification offerings, they have one just for the advanced networking specialty, right?

Jon Christensen: Yeah. Totally.

Chris Hickman: There’s enough information there that it warrants its own certification exam.

Jon Christensen: Right. I think … I guess a part of it is just experience and miles traveled. Before the advent of AWS, you needed to travel a lot of miles or have a very specific job type to deal with big networks, to deal with … right?

Chris Hickman: Mm-hmm (affirmative).

Jon Christensen: You had to be an operations person, or a network person in order to even think about this stuff. Otherwise it was set up for you and here’s your command to get to the thing you need. These days it’s like hey you can set up 100,000 computer network in an afternoon.

Chris Hickman: Yeah, absolutely.

Jon Christensen: But you don’t know what you’re doing, because you never had that job before.

Chris Hickman: Yeah. But there’s only one way to get that knowledge, and that’s to experiment and to use it and to play around with it.

Jon Christensen: Yeah. Yeah.

Chris Hickman: I want to just … maybe just a few pro tips that are related to this. One is CIDR block consideration, right? When we’re creating our VPC, when we’re creating our subnets we’re going to specify CIDR ranges, right? Those are going to say what is the range for IP addresses being used by the VPC in general or the particular partitions inside the subnets? And then how big is it, right, with he mask? We talked about /24 represents 256 addresses. I think it’s /16 is 64K addresses. And then the smallest one, I think it’s /28, which is 16, if I remember correctly.
A couple things to keep in mind, right? You want to make sure when you’re setting up networks, whether it be if you want to be able at some point have multiple VPCs, you want to have more than one subnet, you want to make sure that it’s non-overlapping IP ranges, right? Otherwise you’re going to run into some big, big problems. You want to make sure … and we can break this down into VCP and subnet. At the VCP level, you definitely … you want to allocate a single continuous CIDR block that’s not overlapping with any other VPCs that you may have. It’s tempting to make it big, right? Just do /16 or whatever like that and now you’re like, hey I’ve got all the flexibility in the world. But you need to be careful of that, because if you do make it too big, you’re going to run into problems potentially with overlapping IP ranges.
But if it’s too small, right, you’re going to run out of IP addresses. Really it-

Jon Christensen: You wouldn’t run into that …. At the VCP level, you wouldn’t run into that unless you tried to do something like peering your VCPs, right?

Chris Hickman: Oh-

Jon Christensen: Well, if its’ too big, right, if you let in hundreds and hundreds of thousands or millions of computers in your range, the only problem that would happen is if you tried to have two VPCs and then peer them and then all of a sudden your VPCs could potentially have computers with the IPs because the ranges are overlapping, right?

Chris Hickman: Yeah. Well it’s … Don’t think it so much as VPCs, just other networks. Just by virtue of having a VPN connection, we actually have another network, right?

Jon Christensen: Yeah.

Chris Hickman: For me, accessing the VPN from my home office on my laptop, my particular router, I’m setting up my own private network, right, with its own IP range. If that overlaps with the private IP range of my VPC, I’m going to have problems when I establish a VPN connection.

Jon Christensen: If your VPC’s going to talk to any other networks at all via direct connect, via VPN, or via VCP peering, or any other way, then that’s when you could have these-

Chris Hickman: Yeah.

Jon Christensen: These collisions-

Chris Hickman: Yeah.

Jon Christensen: At the VPC level.

Chris Hickman: And that’s why you want to be mindful of your size. Don’t go necessarily with the biggest one that’s there because that’s going to again, maximize the chances that you’re going to have this problem with overlapping CIDR blocks. Be very mindful of that. Think about it. Like okay, how many IP addresses do I really need here? Estimate above, go above that, but not drastically above that. Me and my personal account, something like 1K addresses would probably be more than enough effort. If I had more than that, then it’s like hey, something’s going really-

Jon Christensen: It’s not a personal blog anymore. Yeah.

Chris Hickman: Yeah, it’s going really well, right-

Jon Christensen: Right.

Chris Hickman: Which means we just burn it down and start over. Yeah, that’s something to keep in mind is just your CIDR block considerations when you’re creating your VPC and your subnets. Again, it’s not just for your AWS networks, it’s for everything else that may connect to it, whether it be VPN or on site or what not. Then one other thing I wanted just point out is that NAT gateways are single points of failure, unlike the internet gateway. Internet gateway is not per subnet, it’s multi AZ. NAT gateways are single AZ. That means if that particular AZ goes down, that the NAT gateway is hosted in, well you have no NAT anymore, right?
For best resilience, you really want to have multiple NAT gateways. Ideally one per AZ. That way if an AZ does go down, it’s only effecting the resources in that AZ. What you do is you’d, for every single one of your AZs you’d create a NAT gateway and then you would have separate private route tables, one per AZ. Each one of those private route tables would then forward to its particular NAT gateway. That way if AZ-A goes down, and it was using the NAT gateway and AZ-A, the other two, B and C, are not effected whatsoever.
A typical thing that people do is they’ll just create a single NAT gateway and think they’re good, but they’ll find out that there’s an outage in the AZ that that NAT gateway’s hosted in the now all of their private subnets have lost internet connectivity across the board, right?

Jon Christensen: Yeah, that’s terrible.

Chris Hickman: Yeah.

Jon Christensen: It’s also the kind of limitation when you say managed service, it’s like mostly. Not quite.

Chris Hickman: Well, it’s managed, it’s just single AZ unfortunately.

Jon Christensen: All right.

Chris Hickman: Yeah. It’s not highly available.

Jon Christensen: Yeah. Cool.

Chris Hickman: All right. Yeah, a quick summary. Now we have our VPC. It’s got public and private subnets. We only have our public facing resources on the public subnets. Everything else is protected on those private subnets. And we now have secure access to those private subnets via a VPN connection. Now we can start moving applications onto our private subnets. But I think that’s going to be in the next future episode of Mobycast.

Jon Christensen: Sounds good. Well, this was pretty interesting. I learned a lot. Just about how this stuff fits together. Stuff that I’ve used and clicked on and had in my own systems. Yeah, thanks for educating me and getting me over a couple humps, a couple of learning hurdles that I had. Thanks so much, Chris.

Chris Hickman: Awesome. Thanks, Jon.

Jon Christensen: Yeah. Talk to you next week.

Chris Hickman: See you.

Jon Christensen: Bye.

Stevie Rose: Nobody listens to podcast outros. Why are you still here? Oh, that’s right. It’s the outro song. Come talk to us at Mobycast.fm or on Reddit at R/Mobycast.


Show Buttons
Hide Buttons