44. AWS Launches DocumentDB – Should Mongo Be Worried?

Rich: Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Welcome, Chris, it’s another episode of Mobycast.

Chris: Hey, Jon. Good to be back.

Jon: Yep, and you’ll notice that I didn’t mention Rich. He’s not with us today, and it’s because we just recorded one with him yesterday and we’re doing another one today because of this timely thing that’s come up. The one that we recorded yesterday, those who’ll be listening, you’ll hear it next week because this week we’re going to be talking about AWS DocumentDB. Our decision to do this was like, “Oh, this is so interesting,” and we were starting to have a conversation about it. Two minutes into the conversation, we were like, “We should do this as a Mobycast,” and then we were like, “Let’s do it tomorrow,” so here we are.

Chris: Absolutely. Yeah, it’s was kind of like, “Okay, stop talking about this. We could wait until we’re on tape.”

Jon: Right.

Chris: “Save it for the tape.”

Jon: Yeah, AWS launched DocumentDB with Mongo compatibility this week. That’s January 9th of 2019. What was your reaction when you saw that, Chris?

Chris: My initial reaction was definitely not a surprise. The rumor mill has been speculating that this was going to happen for months now, and it makes a lot of sense, just given what AWS is really strong in. Obviously, they’re really strong in knowing how to run and operationalize distributed systems. They’ve done this for many open-source packages like things like Redis and Memcached, and recently, Kafka.

MongoDB is a very, very popular document database. It’s basically the grandfather of them all for the most part as far as the commercial marketplace goes, so very popular and lots of folks are using it. If you’re running inside AWS, it’s all on your own. You have to do the management of it, so that’s why we have DocumentDB. AWS looks across the landscape to see that they have quite a few of their customers that are using MongoDB but they have these problems. They’re struggling with how do they get the performance availability out of it that they require, just the management of it. That is stuff that customers don’t want to have to do. It’s really not a core differentiator for them. It’s undifferentiated heavy lifting.

Jon: You have personal experience with this at CalSys, right?

Chris: Absolutely. Yeah. We are users of MongoDB ourselves, and we know very much the real pain that goes along with managing that. We have MongoDB in multiple environments. In each one of those environments, depending on its availability requirements, there’s clusters of MongoDB and so we have to deal with having replica sets and how do we do backups and, more importantly, how do restores when we need to.

We have to test and verify that failover works the way that we expect it to and that applications respond accordingly. We have to continually do software updates and make sure that our agents are current and up-to-date. Also, it’s somewhat of a task for us to go upgrade Mongo versions as well on their own. There’s a lot of overhead and maintenance that comes along with our MongoDB.
Jon: Can you give me just a little more specificity around our setup? Are we installing it directly on our middle EC2s–that’s not the right term–but just directly onto EC2 instances?

Chris: There are middle EC2 instances. It is a thing.

Jon: I know. That’s why I commented.

Chris: We’re using standard EC2 instances and VMs, and we are directly installing MongoDB on those. For our replica sets, they’re typically three-node clusters, so that means three EC2s that we’ve spun up. On each one of those, we’ve manually installed them with MongoDB software. We do use Mongo’s monitoring service to help us manage these nodes. That’s MMS and so there’s an agent that gets installed on each one of those nodes that then we have to set up the communication path from our AWS datacenter over the internet to MMS so it can see those nodes, and then, from an MMS console, we can do things like initiate backups.

Jon: That’s so weird. MMS lives outside AWS and we have to basically put a hole in the firewall for it?

Chris: Yes, absolutely. There has to be some way for MMS to connect to it, whether that’s private versus public, or firewalls, or some way of providing that connectivity. MMS is run by Mongo, and so it’s absolutely not inside of VPC and there’s no VPC endpoints or anything like that.

Jon: What’s weird to me is that MongoDB is like, “Here’s the binary. Go install it wherever you want, but MMS? No, we’re keeping that. We’re managing that.” That’s just weird to me.

Chris: I think it goes with the evolution of Mongo. I’m sure we’ll get into this a bit more as we get into this podcast.

Jon: I know I’m cutting away from the outline a little bit, but what I’m doing here is I’m just trying to get a sense of what we do today and why we do what we do because I think understanding that from our real perspective of owning or operating a Mongo cluster or multiple Mongo clusters will help you, dear listeners, understand how powerful this new thing from AWS could potentially be.

Chris: Going back to your original question, what was my reaction to the announcement, another way of saying that is I was extremely ecstatic. I immediately started putting together the nodes, and I actually put it on our work management system. Yes, there were pivotal cards created just for this like let’s go ahead and start testing this out. How fast can we get off of our dedicated MongoDB onto DocumentDB so we don’t have to manage and I don’t have to worry about is restore going to work? Are the backup jobs running correctly? What is the level of granularity? We’ll get into all this as we talk through some of the features.

Jon: Before we push forward into the features, though, I still have one more burning question about how we ended up where we are today with multiple clusters of Mongo that we’re managing ourselves inside AWS. That last question is I know there’s managed Mongo services out there, and I think Mongo itself even offers one. Why are we doing it ourselves? Why DIY when those other options probably did exist for us?

Chris: Managed Mongo does indeed exist, and I think we’ll dive deeper into this, too, as we go along because it really sheds an interesting dynamic into the tension between the companies. Why we’re not using it today, probably two main reasons. One is the fact that this decision to use Mongo was quite a while back. It was over two years ago, is when we started using it. Atlas didn’t exist at that time, which is MongoDB’s managed service. There were other things like MongoHQ.

Jon: mLaB was another one.

Chris: Maybe the second reason why we don’t do it is that those are all outside AWS so you basically have to go out over the open internet from your service inside the VPC, your application servers, that need that data that are going out over the open internet for that. There’s a very real performance and dependency issue. You can mitigate the security with things like just being smart about firewalls, encryption and things like that, but it’s one more hurdle. To actually have the data co-located inside our own VPC is definitely an advantage.

Jon: Cool. There was no Atlas and we wanted to stay inside AWS. Anybody making that argument to me is like, “Yep. All right, let’s do it. Let’s fire up our own Mongos on EC2s.” Let’s move forward into this new world.

Chris: Sure. As I mentioned, DocumentDB was recently announced. It is fresh, hot off the presses, January 9th, and what is it? It’s a drop and replacement for MongoDB, which is really interesting. They’re not hosting MongoDB. Instead, it’s basically wire-level protocol-compatible with MongoDB. They specifically say they’re emulating the responses that a MongoDB client would expect from a MongoDB server. It’s impersonation.

They look at the MongoDB API and that’s what they implemented. They implemented the Mongo API. I think it’s important just to remember this is not hosted Mongo at all. As far as I know, there’s probably no Mongo code.

Jon: It’s like Kramer on Seinfeld decided that he could be the movie-phone.

Chris: Exactly, yeah. That’s kind of what this is, okay, what’s the API? We will just go and craft our own implementation of that that mimics that and performs that API. That’s how they had built it and delivered it. Along with that, it is a managed service offered by AWS, and all the great aspects that you would expect from a managed service from AWS applies here in areas of scalability, reliability, security. All the check marks are here. What’s interesting is that when they announced this, they talked about how the storage layer for this is SSD-based with 6x replication across three separate AZ zones.

Jon: That’s so good.

Chris: This to me sounds like this is the exact same storage platform like Aurora uses. I wouldn’t be surprised, actually, if they’ve done some–and DynamoDB might be doing the same thing as well, although with DynamoDB they definitely talk about 3x replication, not necessarily 6x. Definitely, the Aurora storage platform–it sounds like this is the exact same storage. That same technology, they’re using it here for DocumentDB.

This might be a good time to talk about this as an aside. There’s been just rumors and ponderings on, “Well, how would they go implement that? How do they implement this?” I know that you were looking on a Reddit thread and someone suggested that their vote was that this was Postgre. They implement this on top of PostgreSQL, and I think this is another hint in a puzzle. It’s probably the Aurora Postgres flavor, is how they’ve implemented it.

Jon: I’m just looking at that thread now, and there are little signatures to that that people were putting out to give it even more credibility, like identifiers are limited to 63 characters, the same characters that Postgre limits identifiers to and then a collection size limit of 32 terabytes coincidentally, which is the maximum PostgreSQL table size.

Chris: Think about how cool it is how much existing technology they probably leveraged for this. This is a pretty major new managed service that they’re offering.

Jon: I’m going to need some pizza-sized team in like a weekend.

Chris: Maybe not a weekend, but, yes, they were probably extremely efficient with this. It’s pretty interesting and pretty cool because I think a lot of these features, they got for free. The reliability, it’s designed for 99.99% availability, so four-nines availability. We talked about the storage layer with the 6x replication across three AZs. Database can scale from 10 gigabytes all the way up to 64 terabytes in size. You can have up to 15 read replicas so you can scale up quite to support just a tremendous amount of read traffic in these clusters.

Jon: Do you know how much read traffic a single instance can handle? It sounds like you could basically put whatever you want on this.

Chris: Yeah, it’s up to you. They offer this. I think you get your choice of six different instance sizes, ranging from–and these are the database-optimized instance families. I believe it goes from–the instances have starting at 16 gigabytes in RAM up to 488 gigabytes of RAM. I think it would be really hard to find a situation where you’re like, “This is not going to work for you from a performance standpoint.”

Jon: Yeah. If you got it to work on Mongo, then you can get it to work on DocumentDB. If not, you need more.

Chris: Massive scalability. The backup story here is awesome. They’re offering point-in-time restores with second-level resolution to any point within the last 35 days, which is awesome. This is exactly what they have in DynamoDB now. I’m pretty sure this is the same thing in Aurora as well. Again, this really goes back to–this is not so much DocumentDB doing this; it’s the underlying core technology that they’re leveraging that they just get this for free. It’s pretty impressive there. And then security-wise, great security starting with authentication is supported and you also have encryption both in-transit as well as encryption at rest. These are just easy checkboxes that you do when you go and spin up your DocumentDB cluster.

Jon: AWS gets some flak, and I think they deserve it for not having those checked by default where you’d have to uncheck them. I think they should change that.

Chris: I think the default now–encryption in-transit is by default with DocumentDB and encryption at rest may not be by default but it’s just right there in your face, easy enough just to check it. Unlike with DynamoDB, DynamoDB encryption at rest is now the default, and this actually happens for everything that’s hosted. Now, it’s just not new projects. It’s a really impressive new product dropping replacement for MongoDB, and we’ve talked about, “Well, why did AWS do this?” Really, they’re just listening to their customers. They’re giving them what they want. They see the broad use of MongoDB on AWS. It is a lot of pain and hassle to manage and run those databases. Hand that work over to AWS and let them do the management, and that’s exactly what this is.

Jon: The fact that it’s not Mongo. You were worried that maybe it’s not as stable or maybe it’s got some barnacles on it that aren’t so good that need to be smoothed off, or maybe there’s some concerns, but there’s enough evidence that it’s built on top of production quality, highly already used and burned-in stuff that the only parts that may be subject to a little bit of flakiness is just the API, the interface.

Chris: That’s where the code is and that’s where the bumps and the warts are going to be.

Jon: Your data is going to be fine; it’s just making sure that it’ll work for you in your situation.

Chris: It really just comes down to the fidelity of the API implementation, like does it do exactly what MongoDB is doing and also what is the level of coverage that you have? They don’t have 100% coverage right out the gate. Not every single API is implemented. Again, they probably looked at what’s the 99th percentile. Maybe there’s 300 API calls in general and maybe 80 of those represent the 99% of the usage, so that’s what they focused on out of the gate.
They do say that, obviously, they have the telemetry. They can see what calls clients are making against the platforming for those things that aren’t supported. They can track that stuff and they can use that directly as feedback into, “Hey, these are the most popular calls that we haven’t supported. Let’s go and implement it.”

Jon: I’m also thinking that even though the API might be the same, there’s just things about how databases work that could cause some trouble. MongoDB, I believe, is largely–people do a lot of eventually consistent programming with it so they expect things not to be there right away if there’s some replicas, correct?

Chris: Yes. We’ve talked about this with DynamoDB and the same thing with MongoDB. If you want–eventually consistent is the default. If you want strongly consistent, you can do that, but you’re going to pay a price.

Jon: I’m imagining that some systems, and especially anything that was created leaning heavily on magic timeouts might break when you switch out for DocumentDB, like those magic timeouts might not be the right length anymore, stuff like that. It could be faster, right?

Chris: That can cause some problems. You also might have scaffolding code around failovers, like there may be some extra stuff that you had to do yourself in your code to deal with failover cases. Now, you wouldn’t need that anymore.

Jon: There could be some code that you have to have to take in order to drop in?

Chris: Or perhaps even remove.

Jon: You can remove some code?

Chris: Yes.

Jon: Cool. Where are we here?

Chris: With that, maybe we can talk a little bit about going into more about okay, what does this mean? What’s Mongo’s position in this? What are they doing in this space and whatnot. We’ve hinted that MongoDB does have a hosted service for the MongoDB database that’s called Atlas. This is actually pretty important to Mongo. It’s something that they say it’s growing 300% annually.

Jon: I’m not surprised. It’s the query, like I’m using MongoDB. How do I have that managed? What’s my managed solution? Oh, look, Atlas. Cool. I can use that.

Chris: This last fiscal quarter, they reported and accounted for 22% of their revenue, so that’s huge. That’s almost a quarter of their revenue is coming from Atlas. This is a big important piece of their business.

Jon: Did you read that thing or were we talking about this thing the other day? I was reading something or watching something that talked about open-source code and where it’s come from and the big tech companies’ attitudes towards it. Essentially, that thing was saying, “Big tech companies have realized that their competitive advantage is not their source code; it’s their ability to operate it at massive scale.”
Once you can do that, you have a moat around your company that’s pretty un-overcome-able. That fact, I think I believe that. If you think about the big tech giants, one of the things that everybody says about them is like, “Google stuff,” like they know things that nobody knows because they know how to run a million computers at once or a hundred million computers at once. Same with AWS, and it’s not surprising that Mongo–we’re seeing their ability to do that at scale, become the engine of their growth.

Chris: There has been this undercurrent of tension between the open source community and the folks that are using that technology, specifically like the cloud companies, like Googles, Amazons and Microsofts of the world. There’s been this tension where it feels like these cloud companies are increasingly relying on open-source software to add new features to their offerings, but they’re not really contributing back to the projects.

It’s just taking and not giving. You’ll hear this referred to as strip mining where these cloud companies are basically just coming in and leveraging that work that was done by these other groups to build their businesses and to get them new features. When you’re in the cloud and you have to run applications, and code, and infrastructure, that actually–it’s not the procurement of this offering that’s the hard part or even sometimes even the development of the software; it’s really operationalizing it, the running of it.

Jon: Just think about hiring people that can do that. They can rack and stack machines all day, and I can think about where there’s going to be and how to deal with little micro-outages that happen across systems in a million computers. Those are super highly-specialized people, and they’re not just easy to hire so they tend to get sucked into one company, and then that company just keeps them forever, and they have a huge advantage having those people, having that skill, having that context, all in-house, and in my mind that’s harder than developing software for sure. That problem is a harder business problem.

Chris: It’s very much just reality. You take your average engineering team and say, “Hey, can you go write an application that leverages Kafka and uses the Kafka API to implement some event-driven?” or whatnot, part of your application, and any engineering team will be able to go and do that and implement that. To now actually run that as a service, to set it up correctly, to have your Kafka clusters and to have monitoring on it, to be able to handle failures and be able to just have alarms and monitoring, everything else that goes along with it, that’s the hard part.

Again, it’s this undifferentiated heavy lifting. Your customers don’t care that you’re really good at managing Kafka. Why spend a whole bunch of time on that? That’s why these cloud companies are so attractive because it allows you to not have to focus on those things that you really shouldn’t be spending time on, and you focus on what makes you different, like what is your core competency and how do you focus on that?

Jon: For the cloud companies, it literally is differentiated heavy lifting. It is their core competency.

Chris: Yeah, absolutely. Maybe this is a good time, then, to talk about what happened back in October that was definitely related to this whole undercurrent of tension of cloud companies being accused of basically ripping off open-source software. What MongoDB did is they came out with a new license, and they announced this last October 2018. It would be effective with the Mongo 4.0 release. It’s called the Server-Side Public License. It’s the SSPL.

The intent behind this was, basically, it forces–if anyone offers this as a cloud service, as a managed service, then they have to–any code that they write to help enable that as a managed service, that has to be published as open source itself. Any kind of value-added software they put on top of Mongo in order to run it as a service, that’s now open source.

Jon: That sounds like PPL just with additional server types of things added.

Chris: This is definitely MongoDB’s take on trying to protect themselves. They’re trying to protect Atlas. They would much prefer to be the de facto, ideally, probably the only out there, like if you need managed Mongo, you go to Atlas. I think that was definitely one of the motivations here. By having this, it definitely makes it harder for folks to do stuff without giving back. This is probably one of the big reasons why DocumentDB is supporting MongoDB 3.6 and not 4.0 because 3.6 is not covered by this new license. It’s only 4.0 and afterwards.

By mimicking MongoDB 3.6, it appears that SSPL does not apply to DocumentDB right now. Of course, this means big questions like, “Okay, what happens going forward?” Is DocumentDB going to be stuck on Mongo 3.6 forever? That’s an open question. Who knows what’s going to happen there? If they want to go beyond that, it’s either going to be a legal battle where their position may be, “Well, we’re not using the software. We’re mimicking the API. Because of that, we’re not subject to the SSPL.”

Jon: If you just see the thought experiment, too, of, “What can Mongo do?” like, “What can they do as they add to their product that would cause people to not want to use it?” I’m thinking Version 4.0, Version 4.1, Version 5. They’re going to add features and maybe people want those features, but who’s better at adding features to software than AWS? Nobody is. I can’t imagine AWS not being able to keep up via DocumentDB. If they can start to get people over at the 3.6 level, they’re going to hang onto those people for sure. Then, if it becomes an arms race of features, they’re going to win it.

Chris: It’ll be interesting to see if they do become basically a fork. If that happens, there’s going to be a lot of backlash. That would be a pretty big bridge for them to cross, I think, for them to say, “You know what? We’re going to start innovating here. We’re going to start adding new features that are not part of MongoDB 3.6. It’s just the stuff that we hear customers asking for,” and so now DocumentDB becomes no longer 100% MongoDB-compliant. Now, that’s just part of it and then it extends it. It becomes something new and something different.

Jon: I think the super side would be okay, but becoming incompatible with 3.6 would be a problem.

Chris: For sure, it would if it’s not compatible. Then, you start getting into the realm of well, why not just get people into Dynamo? That’s the core platform there. If you’re going to do any kind of innovation, doesn’t it make sense just to do that innovation on DynamoDB instead of DocumentDB?

Jon: Honestly, like I said, AWS knows to meet their customers where they are, and legacy is where the money is getting created. For companies, it’s making money or it’s dying. That’s why legacy exists. Yeah, they’re just going to the legacy and trying to pull it over because I think that they did a great job in 2018 of making everybody aware of DynamoDB and then convincing people that whoa, DynamoDB is pretty hot shit. Now, new startups are like, “Oh, we want to start inside AWS world, totally,” and maybe they don’t even sniff around Mongo anymore. For Blue Sky Development, AWS is starting to win that, I would guess, and so now they’re like, “All right, let’s pull in the legacy.”

Chris: I think that’s definitely where they’re at right now and, for the near future, it’s going to be like where do they go from here? Are they able to support MongoDB 4.0 API or are they stuck on 3.6 and it’s up to them? Where do you go from there? I can definitely see them–this is the stake and I’m forgetting to make it easier for people that are on Mongo. We support through 3.6 and then we’re going to provide paths for people that need more features and want to continue innovating for migrating you to DynamoDB. They have database migration service. This is really one of those things that’s a really easy migration to make other than–you do have the frontend coding, your clients, and updating them, but we’ll see how it plays out and where this goes from here. It’s definitely early days, and we can probably spend many, many days theorizing and guess what’s going to happen.

Jon: There was one thing that we talked about before that we didn’t bring up today that I just want to maybe close with, which is we know for sure that the people that run AWS and the people that run Mongo run in the same circles. They go to the same conferences. They have business meetings together. They know each other, so they have to have talked. You just know that that’s taking place. It’s like, “What happened there?” Did those business meetings start to fall apart? Is that the reason that AWS didn’t just work with Mongo to bring their own version of a managed service in-house that used real Mongo? I guess it’s got to be because of Atlas. It’s got to be because Mongo was like, “No, we’ve got to protect this Atlas thing and so we just can’t let you do that.” I can imagine.

Chris: For me, personally, I just don’t see any way for them to have a win-win scenario of partnering up. Hosted Mongo is such a huge strategic part of their business model for MongoDB so for them to actually make that transition into AWS and do some sort of revenue-sharing, I don’t see how that’s too terribly appealing to Mongo and then also for Amazon. They don’t have any other kind of relationship like that. For the most part, AWS is still a margin-based business. If you have to go pay licenses or do a rev split, that’s going to cut in significantly into your profitability model.

Jon: I could be wrong here but I think Mongo, at least some level of Mongo up to 3.6 is fully open source and they could have just been like, “Well, we’re going to do it. We’re just doing it,” but the decision to not do that must have to do with that thing we’ve talked about like difference in license. “If you’re going to handcuff us, we’re going to create a way where we’re not beholden to your new license scheme.”

Chris: I think whenever AWS did start looking at hosting Mongo, I’m sure one of the things they looked at was just well, what would it take for us to operationalize this and to do things that our customers have come to expect with point-in-time restore and automatic failover and whatnot, and I’m sure they looked at it and said you know what, we really want to leverage our core technology with what we have, and that’d be a much better way for us to get there than it would be just to host it as it is. I’m sure this path to go down, the mimicking, the API strategy, that was well before the SSPL was announced.

Jon: It had to have been.

Chris: Absolutely, it was.

Jon: There’s no way they developed all that. I’m sure it was fast, but it was not that fast.

Chris: It’s interesting, though, too. There’s very strong rumors and people that are familiar with what happened that they do say that AWS was very much planning to announce DocumentDB at re:Invent, but it was in mid-October when this SSPL was announced. That drew enough confusion into it where they said I would not be surprised if what they had at that point was MongoDB’s 4.0 emulation, and what they had to do over the last six weeks was scrub that out and bring it back down to 3.6. I’d bet you that was what happened in the last six weeks.

Jon: Wow, that would be wild.

Chris: Anyone from AWS that may be listening, if you want to send a random, anonymous message to either confirm or deny, then that would be great. It is interesting, too, to hear some of the responses from Mongo once DocumentDB was announced by AWS. Here’s a quote from the CEO of MongoDB. I’m not going to try to pronounce his last name, but his quote was, “Imitation is the sincerest form of flattery so it’s not surprising that Amazon would try to capitalize on the popularity and momentum of MongoDB. However, developers are savvy enough to distinguish between the real thing and a poor imitation.”

That, to me, is a pretty weak response, to say, “Hey, we’re the real thing and anyone else is obviously not good because it’s not us,” but the reality is it probably is really pretty good. It is something to be very worried about. I think this caught them back on their heels a little bit although they had to have known it was coming. I think it just doesn’t feel like they had a really–they thought it through, like what’s the position.

Jon: No, they probably should have focused on, “We’re going to do our best to create the best possible database that we can,” instead of digging in on the competitive front. Yeah, AWS is eating the world, and I love it and I’m worried about at the same time. That’s a whole other philosophical argument or conversation that we could have, but maybe we could leave it with snowflake, they’re coming for you next.

Chris: Yes, go public when you get the chance.

Jon: All right. Thank you so much for that conversation, Chris. It was super-fun.

Chris: Yeah, it was fun. Thanks, Jon.

Rich: Well, dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with show notes and other valuable resources, is available at mobycast.fm/44. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.