75. An Encryption Deep Dive – Part Three

Rich: In Episode 75 of Mobycast, we continue with part three of our series on encryption. In particular, we discuss end-to-end encryption in practice. Welcome to Mobycast, a weekly conversation about cloud-native development, AWS, and building distributed systems. Let’s jump right in.

Jon: Welcome, Chris and Rich. It’s another episode of Mobycast.

Rich: Hey.

Chris: Hey, guys. It’s good to be back.

Jon: Good to have you back. We are recording on the day before the weekend, that would be a Friday. I’ve never asked this question ever before. Rich, do you have any plans this weekend? Anything fun?

Rich: I rarely have plans. I feel like I work all the time these days, but the weather holds out nice. I’ll definitely try to get outside and at least get some sun.

Jon: It’s just like where you’re working from. You’ll send me Instagrams of you working from your boat this weekend.

Rich: Yes, I wish it was […]

Jon: […] better in the office even if you’re working from the boat.

Rich: Yes, because it’s great WiFi from that boat.

Chris: By the way, your bathtub does not qualify as a boat.

Rich: Well then it’s art. That would be an art Instagram photo sent you.

Jon: I don’t want that one. Chris, how about you? Any plans for the weekend?

Chris: Yes. Lately this July–August, my weekend plans are usually long bike rides. So, get out and put the hammer down. Usually I bike with a good friend as well, that we like to make each suffer quite a bit. That’s on tap this weekend and just the usual native chores, yard work, and all that other fun stuff.

Jon: Cool. If you don’t get a Strava, PR, or Kalen, don’t bother coming in on Monday.

Chris: Kalens are next to impossible now. That’s the problem with Strava, the more you use it…

Jon: The harder it gets.

Chris: I’ve done a segment a hundred times for the last five years. The PR is just so hard.

Jon:For those that think we’re talking in codes, Strava is an app that tracks your bike ride. The Kalen is when on a certain segment of your bike ride that they track are the fastest ever to ride that segment because they keep track of everybody that rides it. And a PR is a personal record, so that’s the fastest you’ve ever ridden it. So fun. I love Strava. It’s both Chris’ and my favorite mobile app, I think. 

Chris: It’s used frequently for sure.

Jon: As for me, I’m going to go and be a tourist in my own hometown. My folks are going to watch our kids and my wife and I are going to stay in a hotel in our own hometown in Colorado Springs, which should be interesting. I’ve never done that before. […]

Chris: Colorado Springs?

Jon: Yup.

Chris: Interesting. What’s special about Colorado Springs?

Jon: It’s got Pikes Peak Fair, which is pretty cool because the town itself is at about 6000 feet and at the top of Pikes Peak is 10 miles away as the crow flies and it’s over 14,000 feet. It’s a pretty impressive mountain. I like the west side. The side that’s in the mountains, there’s a little town called Manitou Springs and it has its own Penny Arcade that’s really big. It’s been there since I was a kid and it hasn’t changed. I’ll go play that Penny Arcade games. There’s saltwater taffy store, ice cream shops, fun restaurant, and a little creek that flows through it. That’s probably what we’ll spend our time doing.

Chris: Cool. When you said […] on staying there in the area and you mentioned Colorado Springs and the first thing that came to mind for me is Aspen, Vail. It’s probably crazy, ridiculous, expensive right now, too.

Jon: Near Colorado Springs is a little bit more affordable than those places since it’s on the front range and it doesn’t have the hoity toity reputation as Aspen, or Vail, or Telluride. It’s definitely more affordable. The thing that it’s really famous for is military bases and right wing politics. That’s what it draws more than anything.

Chris: There you go. It’s a mix for everyone there.

Jon: Yup. That’ll be fun. Here we go, we’re going to talk about encryption some more. We thought we were going to do three episodes on encryption and guess what, I think we’re going to do four. There’s that much to it and these episodes live forever. A lot of people like to go back and learn some stuff from past episodes, and there’s just more to talk about with encryption. It’s such an important part of being a software developer. Knowing what you’re doing when you use these encryption libraries and what they’re doing, and how to apply them to your use cases is so critical.

If you thought last week was getting hardcore by understanding how TLS works, this week we’re going to get even more hardcore because TLS is for free and it’s a commodity. If you’re running a website, you’re using highly available libraries that use TLS. You don’t have to do much and it’s just easier, you just get it. You just press a button in AWS or you just put a little line of YAML into your cfn set version and out comes TLS. 

Now, we’re going to talk about stuff where it’s more on the bleeding edge of how people are using encryption but it’s so important. It’s about encryption in non website types of communications; person to person communication, group communication. This stuff is still very much in flux. Everybody knows that iMessage is encrypted end-to-end. Everybody knows that WhatsApp is encrypted end-to-end. We’re going to talk about the underlying things that are at play when you do end-to-end encryption and where that is in the software space right now. Chris, you’ll take my general introduction and get more specific with it. Go ahead.

Chris: We’re […] and then encryption. Specifically, this is where two parties are communicating and they’re going through intermediary but the intermediary just cannot see what the messages are. It’s only encrypted traffic going through it. Kind of like that guarantee that no one else can see this. No other company, no servers can actually decrypt these messages and see it in plain text. It’s a secure end-to-end communication channel between two parties but going through an intermediary.

Jon: I wanted to point out that this has two purposes. There’s the obvious purpose of I don’t want people reading my messages no matter what. I just want them to stay mine. I have a conversation with you, Chris, or you, Rich, that’s between us. That’s not for anybody else. That’s not for Facebook. That’s not for Google to read. That’s not for Apple to read. That’s the one part of it. 

There’s the other part of it too, which is that those companies, by limiting their access to the messages that we have with each other, they also limit their liability. They don’t have to be responsible for making sure that people are not breaking the law, because they can’t. There’s really two sides of this. Everybody gets protected when things are encrypted end-to-end.

Chris: I think that was definitely one of the big appeals for WhatsApp when they came out of the gate of doing this from the get go. It just did not want to have any of that liability that goes with,  “What are people saying using our software and is there illegal activity going on?”

Jon: NSA comes along and says, “Hand over the keys,” and they say, “I’m sorry. We don’t know.”

Chris: We don’t have them. Exactly. Let’s go through some examples of how this works. There’s a company called Virgil and they have an SDK for doing this end-to-end encryption. This is for people developing apps that want to have this secure messaging while going through a server. It’s a client-side SDK that you can implement and allow you to do this kind of messaging.

Let’s talk about, “How does this work?” It is based upon public key encryption. There are public and private keys for all the users that want to use this system. The public key needs to be published somewhere. We talked about this in the previous episode with SSL digital certificates for servers where you have a certificate authority. It is the one where you’re publishing the certificates to and that certificate has a public key associated with it. 

For Virgil, they have some infrastructure that you can use for publishing the public keys of all the end users that are using your apps, that are using this SDK, that you need to communicate securely. The private keys, those are kept only by the end user themself. Just like we talked about with TLS and public key encryption in general, the only one that can decrypt the message is the one with the private keys. It’s very sensitive, you need to keep that secret. That will be kept on the end user’s device. 

They do have a bit of a problem because with something like this, an end user can log in multiple different ways and use your app potentially multiple different ways. They could have a tablet, they could use an iPad, and also an iPhone. 

Jon: Web browser.

Chris: Web browser, right. What do you do there? What they do is, part of their infrastructure, they do allow for the storing of the private key in their cloud as well. This way, when you have a new device that you’re using that doesn’t have the private key already embedded in it, it can then go communicate to the cloud and say, “Hey, I need the private key.” How do we secure that?

Jon: I’m guessing it works very similarly to how TLS works in the first place. We’ll use public key encryption in order to authenticate yourself, and then once you have the public key encryption working, you can pass over the private key.

Chris: It doesn’t need to be that complex because at the end of day, the private key is essentially a password. For this particular case, the user, they have a password and they use that to encrypt the key essentially. It’s basically a symmetric key encryption but their password is the encryption key so only they know it. So, pretty straightforward, very much in line with what we’ve been talking about.

Jon: That makes sense. I just want to add more color to that because your password essentially is your private key means that you can’t lose it. Unless you lose your password you’re probably going to have your private key. That’s just the way that you get it. I’m just thinking about this from a more holistic approach, using different devices or you forgot your password maybe you can recover your key that way, by creating a new password. Therefore, getting back to your key through some other roundabout way of getting back to that key. Basically, the idea is if your password is your key then it’s something that’s in your head or something that’s in your password manager and that way, nobody else has it.

Chris: Yes, and it gets into some nuances like what happens if you do create your password. Do they have to basically create a new public key and private key pair. Those are the exercises left to the reader, if you will.

This public key and private key, when the user wants to send a message, it’s going to go look up that particular recipient in this server infrastructure. Say, I’m going to send a message to Sally. Go look up Sally in the server and get her public key. It’s going to go ahead and use this public key to encrypt the message and then it can now send the message to Sally. When Sally receives it, of course, she’s now going to use her private key to decrypt that message. The encryption and the decryption all happens client-side. None of this is happening server-side. Of course, it’s only encrypted data going through the server now. So, not readable by anyone on the server-side. It’s only readable by me and by Sally.

Jon: It’s really happening client-side. These are the keywords stored on the server but the client is going ahead, and grabbing that key, and downloading it.

Chris: Yes, all that encryption-decryption is definitely happening on the client-side. We talked about this past with TLS, same thing happens here. The message is also going to be signed to verify that, again, it hasn’t been tampered with and it’s coming from who it should be coming from, the authentication part of it. This is much more typical, at least the way that Virgil describes it, is much more typical to the traditional code signing that you talked about in the previous episode.

The message is going to be signed by the sender’s private key. So, when I created this message, I used Sally’s public key to encrypt it. After I’ve done that, I have now encrypted the message, essentially with my private key. When Sally receives it, she will look up my public key and use that to decrypt the message. She can use her private key to decrypt the inner envelope, the actual true message itself.

Jon: Yeah, the thing […] me is that signed message digest.

Chris: Yes. That’s how it basically works for one-to-one. It’s really pretty similar to how TLS works except now we have to figure out how to do public key and private key on both sides. That’s why there’s some intermediate service in the cloud. Service that’s a repository for storing, for having this user directory with their public keys available for the people to go look at. So, it’s like a certificate authority, if you will, but for this end-to-end encryption. How de we deal when there’s multiple recipients? I will send a message not to Sally, but Sally, Jon, and Rich all of the same time. Any ideas how can we do this?

Jon: The most obvious thing is if we’re using asymmetric encryption, we’re going to get Sally’s, and Jon’s, and Rich’s public key and encrypt our message with those public keys and send them over. Then they would use their private key to decrypt. It’s all the same message. The encrypted message would look different to each of them, but once they decrypted with their private key, it should look the same. 

Chris: Yes. We could duplicate the message. We can say, “Hello, world” and I’m going to send it to three recipients. Go look them up, get their public keys, take “Hello, world” encrypted with Sally’s public key, then take Jon’s public key encrypted with that key, and then the same thing with Rich’s public key. Now we’ve duplicated the message. There’s a copy for every single recipient, encode the message, once they received that blob they can go ahead and decrypt.

Jon: That might get a little hard if you’re going to send a message to a thousand people, might start to wear your phone down a little bit.

Chris: Yes, or the other networks here in Europe, your broadband, and your data allotment as well. Not just that, but also just the size of the messages. That’s a larger message, there’s issues duplicating it. It’s not very efficient. It would get the job done, but it’s not very efficient. 

There are techniques for doing this a bit more efficiently and this really builds again on what we’ve talked about with TLS. It boils down to using both symmetric encryption and public key encryption here and this is where we can get the win. We can encrypt the message once with the symmetric key and then we can use the public key encryption to share that symmetric key with all parties. So the duplicated part of it is really just the encoding of the symmetric key once per recipient, but that’s small because it’s just the key.

Jon: Is it just going to be like TLS where we go ahead and make up a new symmetric key for every group conversation that we do? Like, […] Jon, Rich, I’m going to have a conversation with them. I need to go use their public keys to send them a symmetric key. Let me make one. Now, let me encrypt it with each of their keys and send it on over. Then they can decrypt it and now they have the symmetric key?

Chris: Exactly. Literally, the steps are: first, you are going to generate a random symmetric key. Then after that, your message, you are going to encrypt it with that symmetric key and that will help create your ciphertext. Then what you’re going to do is you’re going to take that symmetric key and you’re going to encrypt it with the public key for every one of the recipients, say, “I’m going to have an encrypted versions of that symmetric key.” Then what you’ll do is you’ll actually create a secure hash of that symmetric key. We’ll talk about why that’s useful. 

Now, the information we have is the encrypted message, the ciphertext, we have an encrypted symmetric key, one per recipient. We have a secure hash of that encrypted symmetric key. So, now we just stitch all that together. The message that gets sent out is that ciphertext. It’s the secure hash of the symmetric key and it’s the encrypted symmetric key for each of the recipients. That gets sent to everyone. Now, whenever a recipient gets this message, what they’re going to do is they’re going to try to decrypt each one of those encrypted symmetric keys. The trick is how do they know when they decrypted it correctly.

Jon: We can’t be like, “This one’s Jon’s and this one’s Richard’s,” or something like that.

Chris: You could do something like that. You could just put some metadata in there or whatever and say “here’s my unique ID” or something like that came from the server and I’m associated with that. But you don’t need to.

Jon: But doing that might create a little opening like there’s now some data about the recipient that is knowable to other parties.

Chris: Well, now it’s possible that the recipients know something about identities, who else is on that conversation, which may not be good.

Jon: This thing that you’re about to say helps with that. You try to get us to all of them and then compare it to the hash, right? Hash it and then compare it to the hash.

Chris: Right, because we generated the hash of the symmetric key. That’s one way function that creates this hash value of it. If we take what we think it is and we apply the hash function to it and get the same value, then we know that we have the right key.

That’s what they do. Take your private key. You decrypt each one of those encrypted symmetric keys, one at a time. Once you’ve decrypted it, you then run it through the hash […] to see if you get the same hash value. If you get the same hash value, now you know that is your key. You get that key to decrypt the message. So it’s nice, it’s tight, it’s efficient. You’re not sharing anything about identity. There’s no way to know who else is on that. You just have these encrypted keys.

Jon: That could even obfuscate how many people are in that thing if they wanted to. Here’s a couple extra users for sure.

Chris: Yes, you can do extra, you can’t do less. That would be nice. That’s essentially the technique for doing this one-to-many encryption. We don’t need to make multiple copies of the message, we can encrypt the message once. But we’re going to use symmetric encryption to our advantage here so, we just do the encryption once. Then we can use the public key, private key, to make sure that only the parties should be able to read it, can do it. This is called envelope encryption. This is just a really common technique in encryption in practice and it’s really important for just scaling encryption to be able to go really fast.

We talked about this previously how symmetric encryption is so much faster than public encryption. Dealing with the math, you’re dealing with bit operations like XOR instead of dealing with prime numbers like with public key encryption. When you want to be fast but you don’t want to share a secret, you’re going to use this combination of public encryption to share the secret and then use the shared secret to do a symmetric encryption.

Jon: Right. There’s another thing that we’re going to talk about in today’s episode. As an intro to it, I just want to talk about an experience that we had with one of our clients.  We were building some direct messaging and I brought this up before building some direct messaging for Zipex. And just because of my awareness of how WhatsApp works and how iMessage works, I was like, “Oh yeah, this should be end-to-end. Let’s just do that. That should be easy and straightforward because other companies are doing it. Let’s do it.”

That was not my experience in the market. For one, I didn’t feel like it made sense for our team to build the entire messaging infrastructure and interfaces. To solve problem, direct messaging really isn’t the core of what Zipex, the company, is about. So, why build it from scratch? It’s like a buy-versus-build decision. This one felt like a buy. So, we started to look at all the options out there and we looked at Twilio. We looked at another one called, Pub Knob, and with that Comment Chat, and we looked at Sunbird, and some others that didn’t even make the cut. Those were the final four.

All four of them, when I said, “Yeah, we’re really interested in your end-to-end encryption capabilities. We’re definitely going to take advantage of that.” They all said, “Oh yeah. So we definitely support that mission. You should totally check out Virtual Security.” And I was like, “Oh. So you have to use this other thing.”

The effort to implement Virtual Security on top of one of those existing third parties was non-trivial. I mean, it’s definitely some code. Yeah, maybe you can do it a couple of months, but not a weekend project because you have to think about all the things. We mentioned one earlier, we talked about what happens if you lose your key. Well, Virtual has some ways of handling that, but you have to create UI for it. Virtual doesn’t have any UI for that. What happens if you have two person-conversation that turns into a three-person conversation. You have to make a product decision around that. 

So, let’s just corner cases and things you have to think about in order to create a proper end-to-end encryption direct messaging platform and it is not something that you can buy today. I won’t name the company, but one of those four, I did. I got all the way into the conversation of how to do this with their product. And they said, “If you do this, you’ll be the first company that’s ever done it, that uses us.”

Now is the segue into the last part of the episode. They also said, “By the way, you should check out this other protocol if you’re going to do this because some companies that are looking into this are landing on this protocol.” Maybe you can talk about that protocol.

Chris: Yes. We talked about how this secure end-to-end encryption is becoming more and more prevalent and necessary. TLS is well established, 20–30 however how many years it’s been out there when most of the communication was just point-to-point, it’s between a client and a server. Now, we’re so used to things like chat, instant messaging, and group messaging. In applications, like Slack, video calls, and conference calls. Just so much in the way of these multiple parties and want to have secure messaging, so there’s a lot of work out there.

There’s a lot of folks trying to build apps to do this. Then there are groups that are getting together to put together standards and specs for doing this. One of those is Signal. Signal is an open source effort to spec out a protocol for what does it look like for having secure messaging and then for end-to-end encryption. It all came out from the efforts of building an app to actually do this, this messaging between two users that was end-to-end encrypted so that the server didn’t know about it.

So Signal was created in 2013 by Open Whisper. It’s codification of some of the techniques we just talked about. Instead of leaving it to everyone just to roll their own, it’s there. They’re trying to actually spec it out and say, “Okay, here is the actual protocol.” Not only dealing with just the encryption, but also dealing with things like how do you do authentication, message integrity, destination validation, how do you repudiate messages and participants.

Jon: What does that mean? Repudiate.

Chris: Basically say, “Hey, this participant shouldn’t be part of this anymore,” and later, “This message shouldn’t be part of it.” So, to remove it, it deals with order of messages. So dealing with message delivery, it deals with offline and asynchronous delivery of messages. 

Jon: So, it’s just a ridiculous number of things, right? You listed off some things that if you were to go into your client meeting and they were like, “Yeah, we want to build a chat app.” You probably wouldn’t get into the, “Should we verify the order of the messages is absolutely trustworthy?” That’s hardcore. When you care about that stuff, you’re really building a serious application.

Chris: Yes. Like you said, your example of ‘attack at dawn,’ you don’t want ‘at attack dawn’ to be what shows up in your client. So, things like that become important in practice. They’ve also done, with the Signal protocol, some interesting things just making it more secure in case the secret is compromised. They have what’s called a Double Ratchet Algorithm. It’s essentially an algorithm that is constantly renewing and maintaining those short lived symmetric encryption keys. 

What happens is, if someone can actually get one of these short lived keys, if they actually do get it, they can only read the messages that are still using that key. But it’s constantly changing, so in order for someone to truly be able to see what’s been messaged they have to be able to intercept every single one of the messages. They can’t just do a few packets here and there. It has to be everything. It’s a key part too. It’s taking up the security a notch by having this negotiation, this ongoing renewal of those short-lived session keys.

It’s definitely gained a lot of traction. It was originally developed by the folks that built the Tech Secureapp. It was then incorporated in WhatsApp, the Instant Messenger app uses it, Skype uses it for private conversations, Facebook Messenger is using it for its secret conversations. So, a lot of traction out there for folks implement. It’s open source. There’s an SDK out there for things like Javascript, Java, and C. So really easy test to get going and using it. It’s definitely something to look at. 

Jon: For sure. I would say all of this is not commodity stuff. It’s open source and you can play with it, but you’ve got a lot of work to do to turn that open source stuff into a real application. I’m going to make a little forecast. I would say within two years, maybe three, this will be a commodity. This will be practically built into the work you do like, “Oh yeah, I need a secure communication between my users.” Just npm install and you got it. 

Chris: Just encryption, in general, is going to become more and more important. It’s one of the reasons for doing this series. There’s data breaches all the time, there’s hacks all the time. Who can you trust?  The responsibility is going to push back to us, the end users, to really care about encryption and we should be looking to make sure that our data is encrypted and it is stored encrypted by the servers so that when they do get hacked, they’re not seeing anything, they’re not getting anything. So, privacy encryption is really, really important. It’s going to come much more because the hacks are not stopping and they’re getting bigger. Just recently here, it was at Citibank.

Jon: Capital One.

Chris: Capital One, yeah, by an engineer here in Seattle. Someone just misconfigured a firewall on a web server and that was enough for the person to gain access. Millions and millions of people’s confidential financial information just boom, out there.

Jon: You said that it’s up to us and I do think that there’s users pushing for this and people that are responsible and making sure that they can use secure services. But one of the biggest market drivers is that there’s more and more liability on companies as each new breach happens. As different governments around the world make new laws, they’re starting to put a lot of onus on companies to not let this happen. 

So as […] developers that typically work for companies, not for ourselves, we are put in a position of what can we do for a company to make sure that the company doesn’t share any liability if something happens? Like if a terrorist attack happens or something, what are companies doing to make sure that they are not on the hook financially for any damage that gets done? That’s where this and encryption comes in. It’s like, “Okay, keep us out of it.” Companies are like, “What can we do to not even be able to see this information or have access to it?” So that, I think, is driving us almost more than anything.

Chris: You’re right, it cuts both ways. These breaches, it’s the end user side aspect and like, “Hey, I don’t want my information compromised.” It also cost these companies millions and millions of dollars and it’s incalculable how much damage it causes when it does happen. So, they would definitely like that to happen as well. Who knows? We might see more and more services that pop up that are basically the equivalent of HSM in the cloud and that’s used as an intermediary.

HSM—hardware security module—we’ll have to get into this in the next session. That is a black box where the keys are stored and the key itself never leaves that black box. Whenever you need to use that key to decrypt something, that decryption happens inside that black box and the only thing that comes out is the decrypted data, never the key.

Jon: Interesting. Well I’m looking forward to that conversation next week. Enjoy your weekend, both of you. Thanks, Rich. Thanks, Chris.

Chris: Thanks guys. All right. We’ll see you next week. 

Rich: Thank you.

Jon: Bye-bye.

Rich: Well, dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with show notes and other valuable resources is available at mobycast.fm/75. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.