05. Making the Business Case for Docker

Rich:

In episode five of Mobycast, Jon and Chris make the business case for implementing Docker into your organization. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software development. Let’s jump right in.

Jon:

Here I am again with Rich and Chris. Let’s just start by talking about what we’re working on this week. What have you been up to this week, Rich?

Rich:

We’ve been working quite a bit lately on building a headless WordPress apps. We’re wrapping up a project that uses view on the frontend and WordPress in the backend to create a career builder which is pretty cool. Most of my time has been spent trying to push WordPress in odd and different directions. That’s pretty much it.

Jon:

That’s fine, that sounds good. How about you, Chris, what have you been up to?

Chris:

I have been super busy helping out the team push a big release to production. We did a lot of great work in a very short amount of time. Now we’re all working hard to stabilize that and keep it solid so that we can put it to production.

Jon:

We definitely need that at Kelsus. Thank you for that. I can’t even remember what I’ve been working on. I’m 100% getting ready just for this podcast right here. This week we thought we’d talk about building a business case for doing CI/CD with Docker, basically changing your entire software development process to put Docker in the middle of it and build a robust CI/CD pipeline disk around Docker.

We wanted to talk about this because a lot of the benefits that get thrown around about Docker are maybe a little bit too hard to understand for the people making financial decisions in companies. How can you get them to essentially release budget or let velocity slow down a little bit while you decide to implement this. How can you convince them that it’s a worthwhile thing without their eyes blazing over? That’s what we’d like to talk about this week.

The challenge for us is going to be to keep out of all the technical jargon and keep it focused on things that financial decision makers at a company could understand. In order to get started, I thought we just talked about one of the things that would be the easiest sell for any company.

If you can make a case to the people at our company that the problem that you’re having with the development team is that future velocity has gotten grinded to a halt. If we do this, if we put in Docker and we build this stuff but the CI/CD pipeline that future velocity has gonna go back to what it was and we’re gonna stop losing all these potential value that we’re losing right now because we have horrible feature velocity.

That would be the easiest business case in the world. If you can promise that, then no financial decision maker at any company would say, “No, no, we don’t want that.” They would all just immediately say yes and you don’t need to put some numbers to it. Let’s try that.

I thought maybe the way we could try that is by talking about our past and talking about sometimes in our past where we’ve been working on teams or even running teams that see feature velocity go to zero and think about with those times, would a CI/CD pipeline with Docker have fixed the problem. Chris, you think you could give an example of when this has happened in your past?

Chris:

Absolutely. One example that comes to mind is, this is definitely a few years back when the CI/CD tool set was definitely not nearly as mature as it is right now. There were some tools out there but I’m pretty primitive. I was building a website and various associated demon services to do various background tasks.

It was all written in .net and lots of different components and made new lines of code. Developing the code and getting the features in there was not necessarily too much of an issue. What was a very big issue was when it came time to deploy this from somewhere other than my development machine. That was very much in manual process.

Running tests very much a manual process, one side was the code was solid, then it meant I had to package up a release, .net compiling source code, building the binary assets, and copying files that were static assets, make sure everything is in the right locations, buildup the zip file then do things like FTP to the servers where the deploy is happening, stopping services, unzipping file. It was literally just packing lunch, this is gonna be an all day affair and also quite risky.

Doing the release is always one of those things where it’s like, “I know this is gonna take a while.” Sometimes you end up in a position where you just don’t have the time to do it, these [inaudible 00:05:56]. Definitely ran into that issue on that project if I had a CI/CD pipeline in place, obviously all this time and effort would not have been spent and it would’ve gone a lot faster and I would appreciate it.

Jon:

I think I’m pretty easy to convince the CIO or the CFO type of business case. That sounds like it would’ve been a really easy thing to convince, I guess you were your own boss. You could’ve convinced yourselves to do it in different way at that point.

Chris:

Interesting. It was kind of the catch-22 because that was a smaller company, it was a bootstrapped company. It did not have a lot of development resources. It was always one of these situations where can I actually carve out the time to make the investment to improve my tooling so these things go faster or do I just do it? It always came down to that calculation. Do I wanna spend 20, 30 hours building the tooling or do I wanna spend 4 hours to get this done? Usually the answer was, “Yeah, I’m just gonna remodel through it and get it done because I can’t afford the 20, 30 hours right now to do it.”

Jon:

It makes sense. I can definitely relate to that with some of the situations that I’ve been in as well especially like bootstrapped companies that I’ve done. I have another example, I worked for a company called StorePerform in the early 2000s, also Bret Swanson, another person from Kelsus worked there. That’s not where we met each other but we both worked at that company.

After about a year and a half of building a Java app, it was a big enterprise, Java app, enterprise Java [00:08:02] and the whole thing. After about a year and a half, it got to the point where the only features we could add were just tiny, just put a drop down menu here or redo the sort order with this stuff there. But adding a new big feature just became impossible because every time we try, we would end up breaking everything.

The people that have the capability to add a new feature were constantly fighting fires, they just had no time. The thing was deployed at major retailers each in their own in-house clusters. Each cluster would be five or six machines and then a database. This would be at Lowes, BestBuy, sometimes Sears. All the developers were constantly just trying to keep the thing alive.

When they did try to add new features, the whole thing would break. Similar to your story, we would also see release cycles take forever. You have a whole release manager who’s guiding builds and testing and everything from the point of being complete for another two weeks or even longer before you get–the technicals I guess would be hardened and ready for customers.

I am not sure with that company whether a CI/CD pipeline would’ve helped dockerize. I think it might’ve. A couple things that I think would’ve been useful are one, we did have unit test. We did have some test coverage, it was not in a continuous integration server. I think we lived a lot of our lives with broken tests and just put up with it.

That definitely would account for some of those bugs that will come out of nowhere, regression bugs and bugs making it out of the production that made our customers unhappy, thus the firefighting and the developers being everly busy.

The other thing, I think, that Docker forces people into, not necessarily but just the fact that it’s a componentize way of deploying is that if we were doing it with Docker, Docker wasn’t available but if we were, we probably would’ve tried to structure our services, our APIs, more modularly so that a whole big application didn’t need to get replicated across all the machines in the cluster and then only smaller parts of it needed replication.

I think more modularity would have created a situation where there was less opportunity for changes in one into the system breaking another into the systems. That could’ve also helped us. I think there were other problems at that company that were slowing in the feature velocity and I would’ve been a little uncomfortable, honestly, making the arguments with the CEO there that doing this because it was just solve the problem completely.

Chris:

That’s a good point too because there are various reasons why a project stall. The pattern that you just described is so typical. Basically, a team gets going, they had to create velocity, they add months of features, it grows and grows and grows. But over time, whether it’s a year or two, it becomes this big model. Essentially it’s a big ball of mud and it reaches a point where everything is so tightly coupled that making changes to it is very risky, it’s very scary and you get that feature velocity going way down.

If that’s the kind of problem that you have or sometimes there’s some process or culture issues going on as well to have this decreased velocity, if you say, “I’m gonna fix this by turning on CI/CD, that’s just adding fuel to the fire. You’re just not gonna go faster with your problems and it’s gonna make things worse because you’re gonna feel so bad because it’s gonna give you so much more insight into all of the stuff that’s broken, the deploys are gonna fail much more quickly and you’re gonna be so in a reactive mode. In that case, definitely probably not a great idea to throw CI/CD to that particular idea or that particular issue.

Jon:

Just for the people listening, I think we did solve that issue in the end and we did with something, I think that we call the strangle pattern which basically we took that big ball of mud and said, “Let’s move it over here, let’s not touch it. It seems to be working, let’s build other systems where we wanna build other features.” That was the way out of that feature velocity nightmare.

Chris:

Definitely. A typical path, the pattern of how do you move from a monolith to modular services and microservices type approach.

Jon:

I have one other story that I think I could tell that’s interesting but did you have any others?

Chris:

For me, a lot of the experience has been when feature velocity is an issue or it comes down to what are the manual steps that you’re doing that really could be automated by your tool sets.

Jon:

When we were talking earlier, you were talking about having too many priorities, getting lost trying to do too many things at once. I thought maybe you had a specific company that you worked at, or a team you worked with where that was the problem.

Chris:

One of the recent projects in the past, it was a company where we’re building a software in the healthcare wellness phase. A startup company trying to establish itself in a new market so very much had a need to go fast. We started off on this mission of building software very, very quickly to meet the goals that we had as a company for the features that we wanted to get.

There are various people in the company that that was their job, to come up with what the roadmap should be, what features they need to have and then basically really push for those things to happen quickly. That’s what happened. The engineering team builds the code, it’s done very quickly. We know that there is technical debt that’s been accumulated along the way but a very real pressure to deliver features based upon the needs of the company and where it’s at in its life cycle.

As that process continues, we ran into this issue, a natural issue of as you’re going fast and you know that you are accumulating technical debt, you still have the pressure to add additional features because now your product is now perhaps in the market place, you have really users using it, those new users are now exposing some of those real problems from your technical debt.

You have this problem where you have a bunch of technical debt that’s causing lots of problems and very real issues that are causing existing users to not be happy but the business itself has still not evolved to its full business plan. There are other people on the team who are pushing for new features. There’s this very, very real tension.

For me, in my particular experience, I had to make that trade off. Do I add a new feature to keep the project leadership team happy or do I fix the bugs that are causing our existing users to be unhappy? I couldn’t do both because of resources, I had to pick one or the other. That was a very common theme, you’re just basically getting into triage mode. That, for us, ended up being very defocusing just because we didn’t have the resources to really fully do what it is that we wanna wanna do.

We are spreading ourselves too thin, we didn’t know were we stabilizing the hardening our version 1.0 of our software platform or were we building out version 2.0? Because of that lack of focus, basically we did a mediocre job on both things and it’s very frustrating.

Jon:

I seem to recall from talking to you before that that particular project did actually have, it was using Docker, it did have pretty robust CI/CD pipeline or did it not?

Chris:

That was something that evolved over time. When I first started on that, [00:19:39] of it, it was dockerized and it was very rudimentary first steps. Over a year, 18 months, we had a very robust, very mature CI/CD pipeline fully automated and we had continuous deployment, we had support for actually multiple environments that could be built out automatically by automation software. We basically had infrastructures code, very advanced technical.

However, that doesn’t solve the problem. It’s a fundamental problem, lack of focus. We could go fast but because we were spread so thin on what new work we wanted to do, we weren’t creating enough new work to really take advantage of this great pipeline that we have.

Jon:

That’s several examples, one of them it was a no brainer. But in a CI/CD pipeline and Docker, save yourself a bunch of manual. Another one, maybe that’ll be helpful but there are some other fundamental problems in this triangulation pattern what was needed. In your final example, it was already there and you still ran into velocity, go to a halt. Obviously, as a business case, this is not a fantasy episode, it doesn’t fix everything.

Rich:

How about for new projects? Is there ever gonna be a case that you’re gonna consider not going ahead and putting these things into place for a brand new project when someone comes to the idea?

Jon:

This is something that I’d talk about, of course, a lot because I think I come from working with a lot of startups that it’s not clear what the ultimate value of the software we create is gonna be. We have very little time to produce some software that could get a sense of whether it’s valuable or not. The company can’t really afford to spend any time not related to doing that experiment, doing the experiment of do people want this, do I have product market fit?

I think in those cases, when you’re just building software to see if people wanna use it and not building software for the long term, it’s probably not a good idea to spend much time at all doing automated testing, putting together infrastructure as code, even using Docker may be overkill in that case.

Rich:

Maybe it’s a conversation with those founders or the money that’s supplying the resources that, “Let’s get to this point with the intention of rebuilding from scratch.”

Jon:

Absolutely. I can’t tell you how many times I’ve had that conversation. When you have increased your value well enough to attract investors for your Series A, use that Series A to build this right.

Rich:

Basically product market fit is that milestone that will typically get you to that Series A. That initial money should be spent on rebuilding the app that you just spent six months building.

Jon:

The thing is when you’re building an application to be a startup and you get the traction to get that product market fit, it’s not just a prototype, it’s not something that’s completely a throw away code. Real people are gonna be using and depending on it so that transition between using irresponsible development techniques to using responsible software engineering techniques is gonna be tricky. You have to make sure you don’t make your users hate you through that transition.

Rich:

I think for me, the difference is are you building the prototype or are you building the MVP?

Chris:

How do you differentiate them?

Rich:

For me, a prototype is something that is a very short duration, it’s something that has to be thrown together very quickly. You can think of it as a mockup of what the overall piece of softwares that you’re building and you just need to go really, really fast to flush it out. It’s a part of your spec or your requirements, if you will.

An MVP is something that you actually plan on running, you are gonna use it to determine your product market fit, that means that it has to be. As Jon said, it has to be reliable, it has to be robust. You’re not gonna find if their product market fit if that thing has fallen over daily, if users are getting the errors constantly. The answer to the question of do you do this whole? Do you have a robust CI/CD pipeline? Do you dockerize? When do you do that?

A big part of that is how much is that gonna cost you? If you’re a brand new team and you’ve never done this before and you have to learn along the way, that’s a huge investment. For us here at Kelsus, we’ve already gone through that investment phace. For us right now, start new projects, setting it up from scratch with the CI/CD pipeline, fully dockerized, it’s just the way we build things now. It’s very natural and it’s not very expensive for us to do. We can measure that like maybe it’s an extra day or two on a project.

Jon:

The fact that you start it that way really makes me wanna walk back a little bit what I said before because what I said was really coming from the point of view of not already being so familiar with that and having the tools and techniques just ready at hand.

Rich:

But you guys aren’t gonna be on every project. Some people listening are probably going to be in between..

Jon:

Don’t use your startup budget to learn how to do this.

Rich:

This does also serve a great point that it is an investment to learn this stuff and to build in these maturities into your process because once you’ve gone over that hump, then starting new things becomes so much easier and you get all the benefits of this. We’re now reaping the rewards of all this automation. The tools are working for you instead of you working for them.

Jon:

[inaudible 00:21:48] startups because your last startup didn’t work out? Use that time in between startups to learn this.

Rich:

I wonder if there’s a way further to be some calculation that you can do side by side? I’m curious how quickly you’re gonna get to market with your prototype? I’m curious how far you probably can get with that, maybe you can input some number of users using it before it starts to hit these problems along the other side where it’s the slower ramp. Over time, this one is gonna be more cost effective than this one.

If given something that’s predictable over 18 months and you’ll say, “18 months, we’re gonna be caught up regardless, we’re gonna be slow to build features early on on this side but on this side where we’re quick on the features but then at some point, they’re gonna catch up.”

Jon:

It’s funny that you say that, Rich, because we did do a little bit of thought around how much it costs to bring the team up to speed. You can’t see the spreadsheet that I’m about to look at so we’ll just talk about some of the numbers in it. These are just guesses but I wanted to get a sense of how expensive it is to take a team from zero to one when it comes to learning not just Docker but learning the whole front to back of how to make Docker the center of a robust CI/CD pipeline. That starts with you developing on your local machine and ends up automatically putting software into the cloud, inside the Docker container or multiple Docker containers.

Imagine that it cost you $100 an hour, imagine you have 20 on your team. On that team there are four projects, each project has five people on it. You can picture that theme, that’s four groups of five. Each of them are working on different but integrated software.

This is not a startup, this is just some company that’s out there trying to make a software. Maybe they’re currently stocking some 2006 to 2008 development practices where they’re using a platform as a server and they’re writing in Python or Rails and they’re just pushing to their platform as the service would get. It seems to be working okay, or maybe they have some tests but they’re not getting picked up by CircleCI.

The tests maybe are just so subtle that maybe the team needs to learn a little bit more about writing an effective test. There’s a lot that they need to learn to go from being a development organization to a software engineering organization.

First of all, maybe each of the people on the team, maybe they need about $1500, just plain old training. They just need to go through some training, to buy some online courses or books or whatever, each of the people need that. For a 20-person team, that costs about 30k.

You can imagine, we’re gonna have some lost time, we’re gonna have to pay for some of their time while they’re coming up to speed on this. I’m just thinking about Kelsus, I’m thinking about how long it took for some of our people to come up to speed on all of this because Kelsus was in exactly this situation a couple years ago. I would say each one of them needed around $200 of work to get this all figured out and to get it in their muscle memory.

We’re still adding to it so there’s still new things that we’re learning. Just a basic CI/CD set up with Docker, a couple hundred hours. That start to set up really add up because when you’re at a couple hundred hours at $100 an hour, that’s $20,000 per person. You’re talking about $21,500 per person, and for 20 people, that’s almost half a million dollars, that’s $430,000 just to do this. Maybe not what you wanna take to your CFO, we wanna do this and it’s gonna cost $430,000.

Chris, you were the one that did this for Kelsus. You joined Kelsus and you brought us into this world. I think that if you had said, “This is what we’re doing and it’s gonna cost $430,000,” I would’ve said, “Maybe if we can get just the right client to sort of sponsor this, I might think about it but I don’t know if I can afford it. I don’t know if we can go from this type of team we are. We can’t pay $430,000 for this.” We don’t have the cashflow for that at the time. We definitely didn’t. But we did do it. How did that work?

Chris:

Because I didn’t tell you it was gonna cost $430,000.

Jon:

Right, right.

Chris:

I think these are all very, very reasonable assumptions. It’s an investment and personal development and growth as engineers to learn these technologies and to bring this part of these things into tool set. Learning costs money. It’s not free. It’s gonna take some time to do it so it is an investment. I think if I did come to you and focus on like, “Hey, this is gonna be a $430,000 investment.” That would have been a pretty difficult conversation to have. I can also flip it around too and say, “Jon, do you think, in retrospect, this costs $430,000?”

Jon:

Yeah, I think it did, actually. I wanna just talk openly about our biggest client and our best partner, they’re a company called KBS. They’re all over our website. They’re I think a services company. We’ve been building software for them for about three years. I would say that in terms of our billing to them, there may have been a point at which the leader there sort of felt like he got for about $600,000 worth of spend, he got about $200,000 worth of features in terms of what he was used to getting. That sounds awful when I say that. That sounds just like the worst thing in the world. But all of a sudden, now we’re in a position where, I can’t say the name of the client, but they have a service company, it’s a very well-known company that we’re very excited to be building software for.

We would not be in the position to be able to deliver software to this big, well-known company if it weren’t for the fact that we’ve put in this investment and it can serve to be litmus test of are you software engineers, and do you build stable, big, scalable, real software? I think maybe Chris, you can give an example of what the client was used to, with the little project we did for them, and what that was really worth versus what they’re getting now.

Chris:

Yeah. I think there’s been a lot of evolution since that relationship started over the two or three years that we’ve been working with them. I think in addition to Kelsus, I’m doing some growth, they’ve also grown and matured in their capabilities. Three years ago, they were looking at technology in a much different way. They were looking at it as very business-driven, just in time development, like, “This week, what is the current business problem or what’s the feature that we want and we can just spin it up really quickly and time is more important than anything else.” Not really the sense of we’re engineering software.

We’re building software for the long term that’s gonna be robust, it’s gonna be maintainable, it’s going to be code that’s modulars so that we’re not duplicating ourselves. They have matured in kind of where they’re at and where they’re head is around, how they’re viewing software development and their technology capabilities.

Early on, the relationship, it was very much more of a pattern of hey, we need to get something out quickly, it doesn’t really matter so much about how it’s engineered, it just needs to happen quickly. That works as long as you don’t have very high demands. You don’t have too much usage or it’s just not mission critical to the business.

Jon:

Right. To bring that [inaudible 00:34:59] a little bit, we built an application for them. The prototype, we did in two weeks and it was one of their first ever customer facing applications. One that they didn’t use internally, one that they wanted to show to their own customers and it was just a hit. It’s just great the fact that we just built that thing so fast and the customers actually use it. I think they’re still using it today.

But what about that app, why is it working? Why we don’t just do everything like that?

Chris:

That particular app, one of the reasons why it was built so quickly is because we had actually been working on a bigger app that was a bit more general purpose and it was in design to be more mission critical, that our customer had one of their customer that needed this similar functionality and so the idea was, “Can we just duplicate that functionality for this other customer?”

One of the reason that it happened so fast and in the two-week time is it was basically a copy paste of a bunch of code that already existed. Then it was just hacked into it or modified to whatever slight differences it needed for the individual customer that that was gonna be deployed for. That worked pretty well.

It was a single customer, it wasn’t multi-tenant. There was no sensibility to it. It did the one thing and did it well but one thing we have struggled overtime now is how do you actually improve upon this, how do you add to it. Especially when we’re still putting so much effort into the main LAN system from which that was cleaved.

We have this problem where we can’t really update that application all that well without just duplicating all of our efforts and that becomes kind of a very much of a sore thumb that we know. It’s now coming back to cause problem. We now have a roadmap to basically bring that back into the full proper, now that we have much more of a platform in place.

Jon:

Right. This is something that just happens all the time where you deliver a prototype that has a lot of value and you deliver it very quickly; and then the customer points to that thing eight months or a year later and says, “Why are you able to do that, so fast and everything else takes so long?” And you just have the answer but it’s hard to also say, “Look, we can’t do anything with this anymore. We can’t add more features to it.”

We began that hour talking about all of these applications that we’ve worked on that have gone to a halt feature velocity wise, that thing we worked on for two weeks and immediately after the end of that two weeks it was already, like stuck in the mud, there is nothing more we can do with it because of the way we built it.

Chris:

Yeah.

Jon:

Yeah. Basically, looking at this, looking at KBS, and looking at our relationship with them; and learning this within, bringing this final development to them has been a huge win for them because now they are able to attract a caliber of customer that they were never able to attract before. It’s very easy to make a business case for that being worth it. For a company that’s absolutely not a technology company can now be respected as a technology company has been huge for them. I think that doing this kind of development and having this kind of capability leads to that, leads to those types of outcomes.

If you are in it for the long haul, if you are doing something where the software is supposed to live for more than a year, absolutely worth the investment. We also tried to figure out on the benefit side. Can we actually put members to any of the benefits? Are those numbers big enough that you can just take this to your CFO and say we’re gonna do this, here’s what it cost, and here’s the benefit.

Honestly, this is a little difficult, that’s why we have stumbled around a little bit talking about the business benefits of Docker because it’s hard to put numbers to them but I did get one out of all the ones we talked about. We talked about dealing with broken builds or works on my machine types stuff; we’ve talked about setting up new developers, we’ve talked about how Docker saves money on infrastructure because you can have multiple applications across one side of infrastructure now.

Of all of those things, the one that is maybe the easiest to quantify and comes out with the biggest numbers is actually the broken build or it works on my machine lost a [team con 00:40:14] because in a company or in a team where they’re hurting and have a lot of broken builds, whenever your team is not working on new stuff, you are just churning out money for nothing.

Imagine you have those four teams that we talked about earlier, and each of those teams loses four hours every time a build is broken, all of the people lose four hours which is very, very easy for that to happen. Imagine that each month, each of those teams has three builds break. The total cost of those–assuming $100 an hour per developer–is right around $300,000. It’s not quite as high as our $430,000 just to learn all this stuff so when you’re willing, just on broken builds alone, you don’t recoup your investment but you get awfully close.

Since you don’t have to retrain $430,000 in year two, if you’ve got software for the last two years, you’ve already paid for it. Maybe not the best business case for your CFO but it absolutely is a business case. Absolutely you can, with confidence, if you’re dealing with broken bugs, say, “Docker’s gonna fix this. Docker’s gonna save money. CI/CD pipeline is the way to go.” I wish I could do better. I wish I could say, “Look at this $2 million in savings that you’re gonna get.” This is not bad, two-year return on investment. It’s not bad.

Chris:

I think I might have a narrative that it’s a little bit meta. In tech, just like in life, some of the best investments are preventative, and it’s very, very hard to put a finger on any one of those granular benefits. In order to be healthy, you have to do a lot of work. If someone says, “How do feel three years after becoming vegan, a paleo, or whatever, or starting the gym?” You just say, “My life just feels better. I know it’s better but I can’t tell you why.”

Jon:         Right. Yep.

Chris:

But the end result is maybe a longer life. Those things are very hard to say but I think you can rationalize that with this too. Doing these things preventative and it’ll be hard to say that you prevented a problem because the problem never actually happened. But through experience, these problems always happen, and you’re having lots of them in our other clients and stuff like that.

Jon:

Yeah. It’s hard to prove a negative. These things are vitamins and exercise.

Rich:

Yeah, that’s exactly right.

Chris:

We’ve become blinded to what we’re used to. Using your example of exercising or going vegan or what not. If you’re not doing that, then there’s very real repercussions. Maybe it’s hard for you to go up the stairs, maybe you’re more tired, more lethargic, maybe you don’t sleep as well. But these are all things that you’re just living with. You’re not measuring the cost of that.

Jon:

Because it’s just life.

Chris:

You’re just used to this. I think it’s the same thing with development. Broken builds, that’s just part of life. That’s the way it goes. You become immune to it, you’re bonded to it. It’s factored into your daily routine.

Jon:

Another thing we looked at was what is that infrastructure setting? I think you’d have to have a pretty big company with a pretty huge installation to really wow your CFO with the infrastructure savings. Just sort of off-the-cuff I thought, “Well, what if each of your four applications run a cost of $1000 a month, if you do it each with its own environment, each with its own staging and production, with its own EC2 instances in AWS.”

Imagine you can have 60% reuse of those machines. You could put one in a third basically, application on each of the EC2 instances that you have. 60% savings would take an annual cost of $48,000 down to about $28,000 which is awesome. Save the money, use it, but again it’s like these incremental costs, incremental savings, they start to add up.

Another one, new development set-up. Imagine if it takes a developer about 20 hours with the old style of doing things to get up to speed, it’s like, “Welcome to the team. Yeah, I think there’s a read me summary. I don’t know if we’ve updated it in a while but go on there and check it out, and try to get your development up to speed, or your machine ready for development. Steve over there is the last one that did it. If you’re in trouble, talk to Steve.” Inevitably run into trouble and of course, the first thing you run into is installing the database in your machine that never works. And then 20 hours later, you’re ready to develop. If you have four developers a year on a team of about 20, that sounds about right, then that’s around $8,000 that you lose to development set-up. Again, another small but fixable loss of money.

Chris:

The one thing that’s not backed into that is the time it takes to repair all the documentation necessary in order for someone to do that, to set that up.

Jon:

Yeah, sure.

Chris:

There’s many hours there.

Jon:

[inaudible 00:40:16], remember? Because you did it once and nobody ever updated it?

Chris:

Yeah. Yup. The other thing too on the infrastructure is even though the actual, hard costs there, maybe it’s not so much of a big improvement but the consequence of having less DevOps personnel to handle that. It’s so much easier to maintain a cluster of five EC2 nodes and to harden those, lock them down, keep them secured, patch them, than it is to have five separate [inaudible 00:46:28] CAPs with two nodes each. It just requires a lot more maintenance and it can’t be [inaudible 00:46:34]. Those are very real hours that are ongoing. Those add up a lot.

Jon:

Yeah. We haven’t put numbers to those but you absolutely could. Another one that we didn’t put hours to but kind of occured to me because it relates back to a conversation we had before about how doing this lists your whole team’s skills. There’s that mythical concept of the 10X developer. If you have a team of 20, maybe you have one or two 10X developers on that team. Those people are always, in the old style of doing things, those people have tendency to get pulled into solving hard issues inside the infrastructure where the code is running because they’re the only ones that understand how to do that. As soon as you lift the rest of your team, now all of a sudden you have lost those one or two 10X developers’ productivity and you might go I wanna apply that in some ways. The rest of the team maybe they go from being 1x developers to 5X developers and those two 10X developers, they’ve 10Xed. If you could quantify that, I think you’d put numbers through it. You could add this to the growing list of benefits that you get from this.

Chris:

You might not even have to put numbers to it because HR departments try to build culture in a lot of different ways. If you’re creating an opportunity for employee growth within your company, that has to have a positive impact on culture. Maybe it’s your glassdoor view. Those types of things will elevate if the people are being challenged and recognizing that as a part of this organization I’m a better developer which means that either I can improve with a better position here. But also just my career in general and as a result of being in this company, that’s why I learned it because otherwise. We’ll brag that in time.

Jon:

Right.

Chris:

Absolutely. Who wants to work at a team that’s using 2008 processes? It’s 2018. This really is an investment. We’ve talked about the cost here, this is the initial investment. You’re not gonna be paying the same price for subsequent projects. You really pay that investment once and then you get to reap the benefits going forward. Again, it’s the whole elevate, it’s kind of the difference between making a choice to continually do something manually or buying a tool that will save you 20 hours a time. And yeah, it costs money to buy a tool, but once you do that, now you own it, now you get to take advantage of that going forward.

Jon:

Yup. Really, in any case where you have long term software that you’re gonna have in production maintain, and the desire to continue to build new software, to add to your collection, your suite of software, if that time horizon is at least a year but probably 18 months or longer, do this and convince anybody that might not wanna let the team spend this extra time, convince them that it’s worth it because it absolutely will be. The only time it’s not is when that time horizon is super short and you don’t already know how to do this.

Alright. I think we’re good, thank you.

Chris:

Great. Thank you.

Rich:

Thanks.