89. VPC Ninja – Part 1 – Private subnets with VPN

Stevie Rose: You may know that it is best practice to use private subnets for your cloud network, but have you actually implemented them? They can be challenging to set up, especially if you have an existing VPC, and using private subnets creates a new dilemma. How do you even connect to these resources? John sometimes complains that Mobycast makes him eat his security broccoli. Well, it’s time we add network and cauliflower to the mix, to ensure that he and you have a well balanced cloud native diet.
In this episode of Mobycast, we kick off a three part series detailing step by step, how to incorporate private subnets for your cloud network. After listening to these episodes, you’ll be able to set up your VPC like a true networking ninja. Welcome to Mobycast, a show about the techniques and technologies used by the best cloud-native software teams. Each week your hosts, Jon Christensen and Chris Hickman pick a software concept, and dive deep to figure it out.

Jon Christensen: Welcome Chris. It’s another episode of Mobycast.

Chris Hickman: Hey John, it’s good to be back.

Jon Christensen: Good to have you. So we’re recording today, on the day after Halloween. I don’t know if that’s a… I think we’ve talked about this before, and Chris, that’s not a big holiday in your family. Did you do anything yesterday?

Chris Hickman: my kids are getting a bit older. One is off to college, I think we’ve mentioned before, and daughter’s in high school, so past that age for trick or treating. This was, I think, the first year we didn’t have a kid go out trick or treating. So we just stayed at home and the doorbell fired like every 30 seconds, and drove my dog nuts.

Jon Christensen: That’s fun.

Chris Hickman: It was a busy night for the dog.

Jon Christensen: How funny. Yeah. And then, my kids are a little bit younger so we’re in the throws of it. Alana was Harry Potter, and that was pretty cute. Fun that she decided to be Harry and not Hermione, like several of her friends the same age. And then Jonah was Darth Vader, which was awesome, because he had like the full helmet, not just a mask.

Chris Hickman: Nice.

Jon Christensen: Yeah, it was really good. And the thing is, just because of his four year old proportions, he looked a little bit more like Spaceballs version.

Chris Hickman: Very big head, Darth. Yes. Head Vader.

Jon Christensen: So yeah, pretty fun. Anyway. Yeah, so we’re not doing another episode in a series today. We’re starting something new. And so people that are listening, this is about VPC, so it’s an AWS topic. Turn it off if you use Google Cloud or Azure. Please turn it off now. Okay.

Chris Hickman: No. Not so quick.

Jon Christensen: It’s just us now.

Chris Hickman: Just about everything we talk about today is, probably equally applicable to all the cloud providers, and it’s actually applicable to on-prem as well. There are some AWS specific stuff, but also it’s really at the core, just networking as well, and useful stuff.

Jon Christensen: Right. And so stick around. I think when I was first learning AWS, and I think when anybody comes to AWS, the thing that you got to AWS for, usually is not like, “I want to go create a big old network, and I want to make it fancy.” Like, that’s not why people sign up for AWS. They sign up for AWS because they want to run some software, typically, or they want to have a database in AWS, typically, right? But then you go and you’re going to make your database, and one of the questions that it asks you is, “which VPC would you like to send?” And then there’s like this little default VPC that’s named something like AFG123, and you’re like, “I don’t know. yeah, that one.”
And I think for a lot of people, and maybe this is you, now you Chris, but you, the listener, it’s like you put that off like, “Ah, let me worry about that later.” Or you realize you can’t put it off so you do a tutorial or something online, and you get a VPC setup, and you just do it, and you follow the steps. But maybe you haven’t really fully understood what you’ve done. And maybe you’ve made some selections in the console that the tutorial told you to do, but you’re not sure they’re best way to do it. And you ask around and it turns out that everybody else is in the same boat. Nobody really knows that they did it the best way, except for like a few people that give talks and they say that they work at Netflix, and of course Netflix did do it the best for Netflix, not for you.
So yeah, let’s dive into this a little bit and help people sort this out. And I’m not saying that everybody’s dumb about VPCs, but they’re this big part of AWS, and it’s not really what people arrived there for. So I think there are so many people that try to gloss over it and get to the things that they care about more, like running software, running a database. Wouldn’t you say it Chris?

Chris Hickman: Yeah, absolutely. We’ve talked about this so many times where there’s just so much to know. There’s so much foundational knowledge to just come to speed on, and really have a good understanding of it in order to consume the higher level services. So even something as simple as just consuming an RDS database, you got to know about things like security groups, you’ve got to know about networking, VPCs, sub-netting, you need to know about maybe NAT goals. There’s just a lot of foundational knowledge there. And so it’s pretty easy to skip but not have a full understanding of that, and have some holes in it. So that’s what we’re trying to do here, is to help fill in some of those holes.
And the spirit behind today’s topic is really this concept of, “Okay, I have a VPC. I understand public-facing stuff, and maybe that’s like the default way I have my network set up.” But as a best practice, things that need to talk, to have the open internet talk to them, or that are public, those should be on public subnets. But everything else really, should be on private subnets, that are not publicly accessible. So how do you go about configuring your VPC to have that best practice configuration where you have a combination of private and public subnets? And then once you do that, there’s some issues that come up, where it becomes less convenient now to connect or interface with those machines that are now private because they’re not accessible over the open internet. So how do you do that? We’re going to explore those options.
So we’re tackling some topics in VPC, but it’s going to be pretty targeted here, and really wrapped around that concept of best practice of having a VPC with both public and private subnets, and then establishing a connection to those private machines so that we can do whatever it is that we need to do, whether it be troubleshooting, debugging maybe some patching or whatever like that. We don’t like to do that, because servers are like cattle, not pets.

Jon Christensen: Right, right. So it sounds like this may be a bit of a Trojan horse Chris. I thought this was going to be fu networking stuff, and now I’m eating my security broccoli.

Chris Hickman: I will try to put the chocolate sauce on security topics as much as I can, because as much as we don’t like it, or as much as the concept of is like… They’ll want to talk about security. It’s so important. So we’ll always find ways to sneak that in.

Jon Christensen: Right on. And this is network-level security, or infrastructure-level security, where you might have a wide open app, and it’s not protecting itself, and that whole level of security we’re not talking about today. But that app is not even going to be available to the wide internet because it’s on a network that isn’t.

Chris Hickman: Exactly. Yeah. So why don’t we get started with subnet 101, and know what that is. So at the very top, you have a VPC. It’s a virtual private cloud. You can think of that as your overall network, and then you can divide that network up into sub-networks, and that’s what a sub-net is. So it’s a smaller subset, and it’s got its own network address space. And when we talk about subnets, we talk about them as being either public or private. So public subnets, what distinguishes them from a private subnet is that on a public subnet, there is a route directly to the internet on that subnet, through its route table. So it means that machines out on the open internet, they are able to route directly to a machine that is located on that public subnet, and vice-versa. And that’s really what a public subnet is. A public subnet just means that there is a route to the internet directly, for that particular subnet.
And so conversely, a private subnet is a subnet that doesn’t have a direct route to the internet. So there is nothing in the route table that allows for that connection from the open internet to go directly to a machine that’s on that private subnet. In order to enable the reverse direction, so to have machines that are on the private subnet to talk to the internet, we have to do some other things that we’ll get into.

Jon Christensen: Right. And I would say, Chris, that it feels to me like configuring a VPC in the AWS console is analogous to, “I’ve got a server room and I don’t know how many computers I’m going to put in it, but I definitely have some routers, and they’re all set and ready to go, except they’re not configured.” And so I’m going to go start configuring routers, and I’m going to say, “Well this one, configure this way. This one is configured that way.” So I’m setting up the way the network will work, but I’m not putting any computers into the network. Which is maybe a little different than, if you’ve set up a small network before you do computers first. You set them all up and get them talking to each other, and then worry about your router. But it’s more routers first. It’s more like, let’s get into the console. Tell AWS how you want the network to be set up, and then you put in computers later. Yeah?

Chris Hickman: With something like AWS, or Google, or Azure, you can’t do anything until you have a network to put stuff in. So absolutely, you’re going to have to create that network first. As you mentioned previously, there is a default you get. When you create an account, you’re going to get a default network, with just the default settings. And so that you don’t have to [crosstalk 00:10:59].

Jon Christensen: Production software on it, default VPC. Let me just use that. Here we go.

Chris Hickman: So it would be interesting to see the statistics for the amount of production workloads out there that are actually running on the default VPC. And I think it’s north of 50%.

Jon Christensen: Yeah, I bet it is. Yeah. So here we go. We’re going to change that with one podcast episode.

Chris Hickman: Absolutely. So maybe just a little bit more about private subnets. So why have them? And it really is like, it’s limiting your exposure. So servers, or machines, or resources that don’t need to be directly accessible from the internet, put them on the private subnet so that they can’t be. And so you’re reducing that surface area of attack. It’s just good hygiene. It’s good practice. If you’ve ever looked at… Go ahead, stand up a web server, and put it on any server, put it on publicly accessible and internet and look at its log file. It’s like immediately, the bots, the port scanners, they immediately find it and just start hitting it, and trying to find ways in and vulnerabilities and whatnot. A lot of those things are not too terribly smart, but still, they’re probing.
And so, if you don’t need that direct connection to the internet, then don’t do it. So that’s why we have these private subnets, and basically that should be your defacto location for putting resources is, put them on your private subnets. And you need to have a reason why something should go onto a public subnet.
So we talked about private subnets. So, resources that are on these private subnets, they do need to talk to the internet, but we said that they don’t have a direct connection to the internet. So how do they do that? And so that’s done through something called NAT, which stands for network address translation. Really what that is, is it ends up becoming a proxy to the internet for the machines that are on that private subnet. So the NAT essentially ends up becoming the public interface for everything behind it. So you have these devices acting as your NAT router, that is fronting all of your private network.
For this discussion, we need to say it’s fronting the private subnet. And so it hides everything behind it, in that private subnet, and as packets go out, it then translates it so that it looks like those packets are coming from the NAT device, instead of actually the private. And then when replies come back from the internet, it has some smarts that knows how to do the mapping to say, “Okay, which one of the private machines should this reply be routed to?”

Jon Christensen: Right. And that’s just the magic, some Linux magic, or some router magic that is able to do that. And one of the things I remember Chris, is I think it was 1999, Linux was just out as a open source operating system. It was so new that it was pretty hard to install. Like you would buy a windows machine, and then you would try to install Linux on there, and it was at least a days’ worth of effort, if you hadn’t installed Linux on that particular flavor. So I was sitting in my new job, I was just out of college, and I was sitting there trying to install Linux on this machine. And the whole point of what I was doing is, I was setting up a network for our little office in this startup company that I was working for, and the computer that I was setting up at that time was going to be the DMZ computer. So it was going to do NAT for the entire office. That was how we set up NAT there. It wasn’t something… We didn’t have a hardware device for just doing that. We just set it up with putting two ethernet cards into that machine, and one ethernet card had a public internet address, and the other one had a private one, and I was able to set up routing between the two, and then that was the proxy, that was the NAT.
And so ever since then, like that hands on experience makes it easier for me to think through, what does this mean? I literally, I picture that computer that didn’t even have a case. It was just sitting on top of a piece of anti-static on the desk next to me. And I had it open because I was swapping out different pieces of hardware. Whatever piece I couldn’t get to work with the Linux operating system, I would swap one out to find one that did have a driver that would work with Linux. So that’s why it was open and had no case. But yeah, that’s… Did you learn about NAT in a similar way, just hacking and doing it and setting up a network, or did you come to NAT via the cloud?

Chris Hickman: So a couple of different ways. One, back in the late 90s, when I was at Microsoft, we did have the concept of a DMZ for basically pushing content to the MSN, to msn.com. DMZ doesn’t necessarily have to be NAT, right? You can have proxy servers, and sometimes people would use bastion host to separate these networks. But this concept of something that’s separate, you have public versus private, right?

Jon Christensen: Yup.

Chris Hickman: So there’s that, but as far as NAT goes, I think that really dove into the details, just setting up a home network with our router. Any router you have for your home network, it’s going to be doing that, whether you realize it or not, because you’ll see things like, “Okay, what’s the WAN address?” And then you also say like, “What’s the IP address range that you want to use?” So you’re specifying, basically, your public, which is the WAN address that’s assigned to you from wherever you’re getting your internet access from, and then you’re defining the private subnet space, essentially, by configuring that router to what IP addresses to use. And these routers, they’re all going to have defaults too. So some don’t really have to get into that. You’re set, when you’re configuring these things, because it’s going to, by default, use something like 192.168.1.0 or something like that, right?

Jon Christensen: Yup. That’s how you can know how to just go on someone’s wireless and go to that and then put an admin password and you’re in.

Chris Hickman: Yes. Admin Admin. So at any rate, yeah, I think that was my first exposure to NAT, was just setting up a home-based router.

Jon Christensen: Yeah. That makes sense that it would be there and not at a big giant company like Microsoft where you’ve got whole teams that do that stuff for you.

Chris Hickman: Yeah, absolutely. Yeah.

Jon Christensen: Yeah. Cool.

Chris Hickman: Yeah. And it-

Jon Christensen: So We know what it is. We know that essentially how it works, by having the two different IP addresses and routing between them, just sending everything forward to the private one, and receiving everything back, and sending it out.

Chris Hickman: Right. And the reason why it’s called NAT, network address translation is because what that NAT device is doing is it’s actually manipulating the IP headers. So the IP header is actually, there’s fields in there for the source address, the source port, and then also the destination address and the destination port. And so when you’re sending traffic from the private network to the public internet, that NAT device is now changing the source address, and source port values to no longer be the private space, but to be, actually, the IP address of the NAT device. And so then that gets forwarded on, and then now-

Jon Christensen: And it must also get tagged with the actual computer that it came from. Right? So if it was like 190.168.1.12, probably the NAT computer is going to remember that.

Chris Hickman: The NAT device would remember that. So what the NAT device does, is it will have an in memory lookup table that it builds. And so it’s going to keep track of these conversations. And what it does is it uses a combination of IP address and port to uniquely identify responses. So when an outgoing requests is initiated, the NAT device is going to say, “Okay, I know what private IP address you are.” And you’re going over, say port 80. It’s going to go in there and change that, so that now the source IP is itself, and then it may pick up a new port. So that’s what it’s going to use now, to uniquely identify it.
So now the port that it’s going to go back to when the reply comes back, it says, “Oh, this is for my IP address,” the NAT device’s IP address, because that’s all that the public knows about. But it’s like, “Oh, and the port is 49000, 212, or something.” Now it goes through its lookup table and says, “Okay, what is that port mapped to?” And then it’s, “Oh, I know that it’s actually this private… This is the private IP address that it goes to.” And now it knows who to forward it on to.

Jon Christensen: Pretty sweet. I didn’t know the details of that. Very cool.

Chris Hickman: Yeah. So it has this in memory, lookup table that allows it to know who it should be forwarding these replies onto. And that’s why in order for NAT to work, all conversations have to be initiated by the private network.

Jon Christensen: I think that, gosh, this is just so interesting and this is going to be hard to get all this content in. But the thing that I’m realizing is that I thought that maybe the actual packets would get tagged with something so that they would carry that private computer information with them, but somehow obfuscate it. But the fact that they don’t, feels like a security advantage, right? Even if it was obfuscated, there’s nothing about that private network that’s traveling with the packets, and that’s cool.

Chris Hickman: Yeah. Yeah. So it’s just abiding by the rules of TCP headers. What we’re talking about here too, like this applies specifically to TCP/UDP and also ICMP, right? Other protocols have to do other things. But because TCP and UDP, they have these source IP, source port, destination IP, destination port as part of their headers, then NAT devices can use that information to do this, and to build those lookup tables, and just to make NAT work without any modification whatsoever to the devices that are communicating across that, which is a really nice feature. So neither the machine on the private subnet, nor the machines on the public open internet know that NAT’s happening. It’s transparent to them.

Jon Christensen: Right. Very cool. Yeah. Alright. So where do we go from here? We know how it works.

Chris Hickman: Yeah. And maybe related to this, is also this concept of, you now have devices that have multiple IP addresses too. So the NAT has a public IP address. It has to, in order to be that proxy to the open internet. But then it also has to have private IP address as well, because it’s on the private network. So how does that happen? And there’s many different ways to have multiple IP addresses. We can talk a little bit about…
So when you’re dealing with computers on a network, basically there’s two types of address. There’s a physical address, and there’s a logical address. The physical address is known as the MAC address. It stands for media access control address. This is the physical address. It’s a unique ID. It’s assigned to the network interface controller itself, always, by the manufacturer. Some devices allow you to actually change it in software afterwards. But for the most part, think of it as like it’s a burned-in physical address, unique ID, that’s assigned to that network interface card. And that’s its MAC address.
And then you have the logical address, which is the IP address that’s now assigned to that. And there’s another protocol, it’s called address resolution protocol. And basically these are just tables used by the network itself that can map IP addresses to MAC addresses. This is what allows, when a packet comes in, because TCP, it has, to bring it home, “What’s the destination IP address that this thing should go to?” When that packet comes into the network, the network uses these address resolution protocol tables to then, given that IP address, what MAC address does this belong to?

Jon Christensen: Right. Because at some level, some electronics have to happen that literally switched to… They involve physical things, wires and things like that.

Chris Hickman: Yeah. And I think that it’s got to be routed to some physical device, right?

Jon Christensen: Yeah.

Chris Hickman: So there’s a table there that says, “Okay, given this IP address, what’s the MAC address?” And so with that, you can actually have multiple IP addresses mapped to the same MAC address. So that’s one way to have multiple IP addresses. So as long as your device supports that, and it’s making the proper ARP calls to register it, you can have multiple IP addresses for the same MAC address. You can also have multiple network interface controllers in a machine, right? You can have two, three, four, however many network interface controllers your particular piece of hardware can support. And so there’s that.

Jon Christensen: I’m thinking back in 1999, that thing that was sitting on my desk on top of a piece of anti-static that I was putting ethernet cards into. I’m guessing those ethernet cards didn’t support multiple IP addresses. Those physical cards, or the drivers didn’t, or something. But the fact that I had to have two of them tells me that there was a limitation there.

Chris Hickman: Yeah. So it could have been that. There’s also the networking considerations of like, “Okay, are you going to do this at the physical level, or are you going to have more sophisticated software that does the routing to do the networks?” Are you going to do VLANs and whatnot?

Jon Christensen: It could have been [crosstalk 00:26:05]. Literally like a cord plugging into something that goes out to the outer, wide world internet, and another cord plugging to the next computer on the private network. Yeah.

Chris Hickman: Yeah. So anyway, I just wanted to bring up that concept of IP addresses for machines. Like these machines like are… We’re dealing with both these private networks and these public networks. And so any EC2 that we spin up in AWS, and this again goes with Azure and Google as well, it’s going to have, definitely a private IP address. But then we can also give it a public IP address as well. So it has multiple IP addresses. And you can actually have even more than that. So with AWS, depending on your instance type, you can have, eight, 16 IP addresses per ENI. And then you can have multiple ENIs per EC2, right? So it’s possible that a single ECT can have scores of IP addresses that are assigned to it.

Jon Christensen: Scores. I like that.

Chris Hickman: Well, I don’t want to say 100, right?

Jon Christensen: Let’s go varying, in this conversation.

Chris Hickman: Yeah. So it’s like, “I need some unit of value here.” So I’m wearing my Gettysburg hat for that scores. It’s better than Fortnite, right? For addresses.

Jon Christensen: We cover a lot of information here on Mobycast, and if you’ve ever wanted to go back and remind yourself of something we talked about in a previous episode, it can be hard to search through our website and transcripts to find exactly what you’re looking for. Well now, it’s a lot easier. All you have to do is go to mobycast.fm/show-notes, and sign up. We’ll send you our weekly super detailed outline that we use to actually record the show. And a lot of times, this outline contains more information than we get to during our hour on the air. So signup and get weekly Mobycast cheat sheets to all of our offer episodes delivered right to your inbox.
Okay. So now we have multiple IP addresses. We understand that. And what we’re doing here, just everyone listening along, we’re going into detail around each fundamental pillar of networking that you need to understand in order to put together a VPC, and use a private subnet with it, and have workloads running on that private subnet that are safe from the wide world, the intranet.

Chris Hickman: Right.

Jon Christensen: So where do we go? Now that we know we can have the multiple IP addresses, we understand how networks, what’s next?

Chris Hickman: So now we have this, and we know this is the best practice. We want to have our VPC set up with private subnets. So let’s do that. What’s stopping us? And so just something to point out now, is that once you do start putting your resources onto a private subnet, you have to ask yourself, “Well, how am I going to connect to that?” Because right now, if you spin up something on your public subnet and you need to access that machine remotely, it’s going to have a public IP address. You can open up port 22, in a security group assigned to that instance, and now you can just SSH from your laptop at your office, into that machine, and do whatever you needs to do. Like look at a log file or run a software, OS update or something like that. Right? Pretty straight forward, and easy.

Jon Christensen: And Chris, I think one of the best ways to approach this that’ll resonate with a lot of people is just that common, common early AWS experience where you do create an EC2, and you’ve put the right security group on it. Like, “I should be able to SSH into the thing. And there it is. It says right there, but for whatever reason, it’s timing out on me. I just can’t get to it.” I think that’s something that has happened to so many people. It’s definitely happened to me. Has it ever happened to you? Because it feels like the security group is that, “Oh gosh, it must’ve been a security group.” Okay, “Now let me open up the security group to the whole world. There’s nothing about the security group now that could possibly be keeping me from this machine. Why can I not get to it?” That’s definitely up to me.

Chris Hickman: This is the fun of network troubleshooting, right there. There’s like a million things that could be blocking you from that. A typical one would be like, “Hey, port 22 is just not allowed with the security group that you’ve assigned to that EC2.” Is this a [crosstalk 00:30:37].

Jon Christensen: That’s the first one you learn, right? That’s the first thing that happens to people. And then this is number two. The second thing that happens to them is, “okay, I’ve learned never did do that anymore. I’ve always got up in the darn port. Two, if I open the port and I still can’t get to it. And I think this is number two because it’s unlikely that other reasons that this could happen would be like, “Oh you’ve got a network access rule that…” I can’t remember what they’re called, but they’re like network access, like overall rules that say you’re not allowed to get to this thing. But you don’t get there before you get to the whole, “I’m going to put things on a private subnet.” That’s the next level in the Mario Brothers of AWS, is starting to put things on private subnet.

Chris Hickman: Right? Yeah. So we’re going to have this issue of like, How do we connect to it?” And one of the ways you’re going to know that you’re going to have a problem, is if you’re going to go to the AWS console, you’re going to go look up this EC2, see what is IP addresses. You’re going to see, “Oh, it’s like this 10.10.1.13. And now if you do SSH, to EC2 user at 10.10.1.13, it’s just going to sit there forever. There’s no route to that. It’s not going to be able to connect, because that’s a private network address space that your laptop just does not, it’s just not going to find it. There’s nowhere to go. So that’s not the IP address that you can use directly in that particular instance.

Jon Christensen: Well now help me out there. Help me out here though, because I was imagining like, “Okay, so then let me slap a public IP address on it.” Oh gosh, I still can’t get to it.

Chris Hickman: You can assign a public IP address to your EC2. So that’ll be one of the options that you have when you launch an EC2 is whether or not to assign a public IP address to it. But if you’ve launched it into a private subnet, and if it truly is a private subnet, and again, we said a private subnet means that there is no route directly to the internet. Then even though it has a public IP address, there’s still no route from your remote machine, from your MacBook, from your desktop, from the office, to that machine. Even though it has a public IP address, there’s no route to it.

Jon Christensen: Now I feel seen, this is me. This is what happened to me at least, 25, 37 times.

Chris Hickman: Right. So just giving yourself a public IP address does not make it publicly accessible, right? You literally have to have the plumbing in place in order for it to make that connection. And this is, maybe, a subtle point to make as well, but you can have machines on the private subnets using public IP address spaces if you really wanted to. You shouldn’t do that, but there’s nothing preventing you from doing that. Again, especially with NAT, and NAT is hiding that whole address space. So if you wanted to, you could use… You just pick whatever address space you want. It doesn’t really matter.

Jon Christensen: The private address is not being allowed to be on the open internet, like from the point of view of internet service providers, is not a convention. But once you’re in your own home network, it’s like really all about like helping you realize, “Oh, that’s a private address.”

Chris Hickman: Mm-hmm (affirmative). One of the really great things about NAT is it extends, in a way, the IPV4 address space. It’s been one of the saving graces actually, because almost everyone is-

Jon Christensen: Yeah, I’m never going to learn IPV6. Forget it.

Chris Hickman: Yeah. Almost everyone is still on V4. V4, 32 bits gives us theoretically, little more than four billion IP addresses. And in practicality, because of the way that that cider works, it’s actually less than that. It’s somewhere around three billion or so. So three billion IP addresses, that’s not a lot, given how many devices that we have. There’s more smart phones, I think, right now, than that. So definitely have more devices than that. And so the NAT allows us to live with that limitation, because we only need one IP address, and we could have thousands of machines behind that, in duplicated address spaces. But that’s all hidden, because it’s this private network. And so NAT really deals with this concept of IP address exhaustion, because it is so limited.
And so this is what actually internet service providers will sometimes do as well. I know talking with some of the team members that are down in Argentina, some of the service providers they have down there, public IP addresses are very scarce, and they don’t have that many of them that have been allocated to them. So what they do is they use NAT with their internet connections.

Jon Christensen: I would not be happy about that.

Chris Hickman: Yeah. Well they don’t have a choice. And the truth is, without NAT, then they probably wouldn’t have internet access. Or it’d be super expensive. So it really deals with this problem of like, “Hey, we don’t have a lot of IP addresses to deal with here. So what do we do until we all switch over to IPV6?” Which is-

Jon Christensen: That’s going to take seven episodes of Mobycast.

Chris Hickman: Yeah, yeah, yeah. Cool. So we’ve talked about this concept of, private subnets are good. We want to do it, but how do we connect to them? And so really, it boils down to like… You can think of this as there’s three main solutions to this. So you have like a physical solution, you have a virtual solution, and then there’s this other one that’s… We’ll call it a leapfrog solution, right? So the physical solution, is I have a direct network connection from where I am, into my private network, right? And that is separate. Only the people that should be using that can. And it’s secure. And so that’s, that’s one way of doing this. So AWS, they have Direct Connect, right? That’s their technology for basically giving you this network pipe that is private, and it’s direct from your remote location into your VPC.

Jon Christensen: And it can go all the way into a private subnet of a VPC?

Chris Hickman: It will go into your VPC. And then from there, you can set up the route tables accordingly.

Jon Christensen: Okay.

Chris Hickman: So with our route tables, we usually set it up such that any subnet within our VPC can talk to any other subnet inside our VPC. And that’s all just with our route tables.

Jon Christensen: Cool.

Chris Hickman: So, there a fine solution. However, there are definitely some downsides.

Jon Christensen: One thing I’ve never known about Direct Connect since I’ve never used it, but it just comes up and up and up and up with enterprise type folks, is that a machine? Do they ship you a machine? Like what is that? Or is it like a guy or what is it?

Chris Hickman: It’s a network leprechaun. So yeah, what is it? Great question. So Direct Connect, what it requires is, you’re going to be dealing with some network provider that has worked with AWS, that provides that direct connection into AWS’s backbone, essentially. So you’re provisioning a circuit, essentially, with one of these partners, typically somewhere very near to where your machines are located, whether you’re co-hosted or wherever you’re co-located, whatnot. And on your side of it, you need to have a network device. So you’re going to have, basically, a router, a switch. And as part of setting up the Direct Connect, you’ll be updating the configuration of that, so that it knows how to talk to the other end of that circuit that is now inside the AWS backbone going into your VPC.
So on the AWS side, they’re setting up that end of the connection, and the networking, and the end point that’s going to happen. And then on your side, you’re dealing with someone that’s giving you an actual physical connection, and the routing necessary to get to that. And you have to have that physical network device to do that configuration.

Jon Christensen: Cool, it makes sense.

Chris Hickman: So these are some of the cons, right? This is expensive. It’s requiring dedicated hardware on your side, and this only applies if you have an actual physical location that needs to be connected to your VPC. You’re not going to get Direct Connect for your laptop. It’s not going to work. That would be an example of a physical solution to how to talk to our private subnets. Another possible solution is a bastion host, or a jump host, or a jump box. And this is what I was talking about, a leapfrog solution.
So really what this is, is this is that concept of, you first connect to a machine that is publicly accessible. And then from that machine, you can now jump to the private machines because of that bastion host, it has access to both the public internet as well as to the private machines. So typically, what this is, is your bastion host, it’s a public facing server that’s running the SSH Daemon so that you can shell into it. And then once you do SSH into it, you now can do another SSH session to go to something on the private subnet, because that machine, it’s inside your VPC now. So it has access to that.

Jon Christensen: That bastion host, I imagine if you went on there and you did something like a netstat, you’d see a couple of eth0, and eth1. And the eth0 would have the public IP address, and the eth1 would have the private IP address. Like you’d be able to see, “Oh yeah, this is on both networks.”

Chris Hickman: Yeah. And that’s true for any EC2. It doesn’t have to be a bastion host. So it could be anyone.

Jon Christensen: Sure, sure.

Chris Hickman: As we talked about, all of these were set up that way. So if it has a public IP address assigned to it, they always have a private IP address. So anything that has a public IP, you’re definitely going to see multiple. And chances are, with AWS and with EC2s there’s probably multiple IPs, regardless of whether or not you’ve been have a public IP. I wouldn’t be surprised.

Jon Christensen: Right. So then this bastion host, also, you might be able to look at the route table on that host itself, and see that it has a route out to the internet, and a route into the subnet, the private subnet?

Chris Hickman: Yeah. So bastion host, it has to be on a public subnet, because you need to be able to connect to this over the public internet. So it’s going to be on a public subnet, which means that the route table for that subnet is going to have a route for talking to both machines that are on the private subnets, as well as a route out to the internet. And so typically with AWS, the way that we do that, we’re going through an internet gateway. So there’ll be a route there for that public subnet that says, “okay, if it’s anything that’s not internal, everything else,” the catchall, “That’s going to get routed to the internet gateway.” And then the gateway will then get it in and out. So that bastion host, it’s on a public subnet. And so that’s how it has this public access. And so if you were to look at the routing on that machine, that’s what you would see, that there’d be a route for internal, the private subnets, and then you’d also see a route that shows it’s going through the internet gateway to do everything else.

Jon Christensen: Perfect. Cool.

Chris Hickman: And there’s nothing really special about a bastion host. Literally, all it is just a server that’s on a public subnet, that’s running SSH, and that’s it. It’s just how you use it. The advantage of this is that you’re minimizing the surface area to be protected. So instead of having SSH access, you can put these other things on the private subnets. They don’t have to be on the public subnet. You don’t have to manage this large surface area of machines that can be accessed from the open internet. You should have rules that say, “Only allow SSH traffic from this IP range, from the internet.”
So find out what your remote locations are, and have rules in there, as opposed to just letting any traffic from anywhere in the internet come in on port 22, on the SSH. So it minimizes that surface area. It does allow you to have your machines on the private subnets that you want to connect to. But obviously, some cons here. There is this extra layer of indirection that you have to deal with, right? So it’s just another hop to go through, and it’s not just from a performance standpoint, it’s also just from an actual usability standpoint. So if want to SSH into a private machine, I first have to do an SSH session into the bastion host, and then after that, once I’m on the bastion host, I now have to do another SSH session into the machine that I want to access.
So there’s that extra layer of indirection. Another thing that is a management complexity, is that SSH keys, and how those need to be managed. So for each one of those machines that you want to connect to on the private networks, you’re still going to need SSH keys. So how do you manage that? Where do those keys reside? It’s not great to just copy those machines and leave them permanently on the bastion host, because then you’re defeating the purpose of it, right? Because now it means anyone that can get into the bastion host, they have keys to the kingdom, right? And they can go access any other machines. So there’s that complication.
It’s also a single point of failure, typically. So you can have multiple bastion hosts, but you can’t necessarily put a load balancer in front of it, because it’s SSH, typically. So it’s a single point of failure. And then as I talked about, there’s just the security risk of protecting that bastion host. It becomes a pretty key, critical, potential hole in your perimeter.

Jon Christensen: It’s funny, because it’s like the easiest way, in some ways, like, “Oh yeah, just throw up this machine and then connect to it.” And then you’ve got your access to your private subnet. And some cases, it may be the first thought like, “Ah, okay, I can just do this, and then I don’t have to about setting up something complicated.” But it’s not. It ends up being, overtime, the amount of time you spend messing around with that stupid bastion host, or hosts, just eats into your day and your soul.

Chris Hickman: Yes, it does. And it’s a good point to say. For completeness, we’re mentioning these solutions, but it’s not really a great solution. So bastion host is definitely something a bit more of a legacy solution. There may be circumstances where it makes sense for you, but for the most part, you should be looking at other things. And so maybe that’s a good lead into the third category of solutions, which is the virtual one. And that is, basically, virtual private networks, VPNs. And so this is another way of getting that secure connection to my private VPC, but to do it over the internet, by creating this tunnel that’s secure and encrypted, and end-to-end. So VPN is that virtual solution, and it’s definitely a good option here. Because it’s balancing out. It doesn’t have to be expensive. It doesn’t require any dedicated equipment. It can, but it doesn’t have to.
So there’s many different options here for VPNs, and it [inaudible 00:48:12] branding, and the cost, and the equipment requirements. And the other thing with VPNs is, they’re good for both connecting and on-prem location to your VPC, as well as just doing just general remote user access. So this is the, you’re at the coffee shop on your MacBook working away, and you need to connect into your VPC. You’re not at an on-prem location. There is no hardware link like that, and you just need to be able to access it securely. So VPN gives you a solution there as well.

Jon Christensen: That’s right.

Chris Hickman: Yeah.

Jon Christensen: Yeah.

Chris Hickman: And there is, again, lots of options here, and just in the AWS world, some of the options include; there’s Managed VPN, that requires a network component on your side. So it’s really for on-prem to VPC connectivity. There’s a VPN CloudHub, which again is dealing more towards on-prem and being able to connect multiple on-prems together by using your VPC as a hub. And another category is just third-party software VPN solutions. This is interesting for me.
So maybe a little bit of a detour, story time. So in preparing for this episode, going and research, just making sure that it was using the right terminology that AWS is using and whatnot, and I have done a lot of… Just using of VPCs and networks and VPNs and really understood like what the options were, I studied for exams, and really feel like I know a lot of this material pretty well. And as I was looking at the specific VPN options, I came across a page on AWS where it was like, “Oh, there’s site-to-site VPN. there’s Client VPN. There’s VPN CloudHub, and there’s third-party.”
And like site-to-site VPN and client VPN, what is that? That’s sounds new. And so it was one of these things where it blew my mind a little bit, that, again, that that rate of innovation that’s happening is so rapid. It’s like I completely missed this. So it used to be called managed VPN, and that was where you had this dedicated piece of hardware that’s at your on-prem site connecting in your VPC. And that’s now been called site-to-site VPN. And they have this other option now called Client VPN. And Client VPN is the software-only solution. No hardware necessary. And so this is the one that’s going to work for your general remote user access. It was just so surprising to me to come across this, because I just had not seen any mention of it whatsoever. All exams that I’d taken and studied for, it was never mentioned. And it’s like one of those… It’s such a big new feature, that it just took me off guard, to find out that this existed, especially because I didn’t really see that this existed until after I went through the process of figuring out how to go set up a software VPN, and it ended up being quite a bit of work.
And that’s what we’re going to talk about more as we go through this is like, “Okay, we want to go with a software VPN solution.” “Okay, how do we do that? Let’s go see what the choices are.” And now we have to go implement that. And so I went through that whole process and as I was dotting I’s, crossing T’s doing some final research, I come across this AWS Client VPN, and I’m like, “Oh my goodness.” This is the AWS managed way of setting up this software-only VPN. And there’s some really, really great things about it that make it super easy and quick to set up. It does have some limitations that are not so nice as well.
So I think this is something just completely for a future episode, to go dive deep on, just Client VPN. And how do you go set up a software-only VPN using this? And what are some of those trade offs that it makes? But again, I was really surprised to see that this is… I had no idea that this was even an option. And it was released in December of 2018.

Jon Christensen: That’s really recent. Yeah.

Chris Hickman: Yeah.

Jon Christensen: So all of this is such a good teaser for a few future episode, because that’s exactly where we are right now. We’re at the end of the limits of my, and I imagine, my intention span matches, or is it a little bit smaller than you, our listener. So here we are at the end of it. We’re going to have to save everything else for next week. What do you think, Chris?

Chris Hickman: I think, absolutely. There’s a lot to cover here. And I think this is a good point to stop. We’ve covered this whole concept of VPC and split it on into public subnets, into private subnets. We went deep into NAT, and really understanding what’s happening there, and with IP address spaces. We’ve talked about networking, like how do you can have multiple IP addresses assigned to the same machines. So you can be these machines that are in-between, that can have access to both public and private networks. And we’ve talked about, “How do you connect to these things? That, once you do put them on these private networks, how do you connect to them? Because it gets to be a bit trickier. And we’ve gone through all the options, then we said, “Hey, VPN is a really good way to go.”
Well, next time we can talk about, “Okay, now that we’ve decided we want to VPN, what are our options there?” And a spoiler alert, or maybe not so much of a spoiler, we’re going to go with a software-only solution. So we’re going to walk through that. And then once we have our VPN set up, we’re now ready to go. We can now start creating our private subnets, and we’ll just walk through that. What does it mean to create a private subnet? And setup a NAT for that, and then be able to VPN into one of these private machines.

Jon Christensen: Right on. Well, I’m looking forward to that, Chris. Thanks so much for today’s stuff too. And I want to say something real quick. We got a comment today on Reddit, and it was so fun to get it. And one thing that was funny about it is that it talked about… The guy that asked the question seems to have done that before. And it’s like, “Hey, that’s us. We’re here. We are literally here on Reddit, and you can talk directly to us.” We’re not these people that are like talking heads. We’re actual people that will answer your questions. So come do. Come ask us questions. And I will answer them in a lighthearted way, and Chris will answer them in a very deep and technical way. Alright, so thanks so much Chris, and we’re looking forward to talking again next week.

Chris Hickman: Alright. Talk to you then, Jon.

Jon Christensen: Okay. Goodbye.

Stevie Rose: Nobody listens to podcasts outros. Why are you still here? Oh, that’s right, it’s the outro song. Come talk to us at mobycast.fm, or on Reddit at r/Mobycast. (singing)