22. Adventures in AWS Certification

Rich: In episode 22 of Mobycast, Chris recaps his experience attending a rare, and privileged workshop at the AWS offices in San Francisco. Welcome to Mobycast, a weekly conversation about containerization, Docker, and modern software deployment. Let’s jump right in.

Jon: Alright, good morning and welcome Rich and Chris, it’s another Mobycast. Good to have you.

Chris: Hey guys.

Jon: What have you been up to this week, Rich?

Rich: I have been up to not a whole lot. I’ve been heads down for the last week just back from vacation. Playing catch up, so just a lot of house cleaning, and project management, and trying to deploy a few projects this week and next week. Much more heads down than anything else.

Jon: Right, how about you Chris?

Chris: Yeah, I’m going to copy paste Rich’s answer there, except instead of coming back from vacation, I’m looking forward to going on vacation here in about two weeks. Heads down kind of making sure everything is buttoned up and in good shape, and then have some time off and go to California.

Jon: Right on, I bet everybody listening is like, “I want to work at Kelsus, that guy’s on vacation all the time.”

Chris: There’s been a lot of travel recently, but not really vacation.

Jon: As far copy paste goes, I have to share this joke I just saw on Twitter, somebody that was speaking at a guru conference today and the thing that everybody has been tweeting is, “Good developers copy, great developers paste.” That’s so funny.

As for me, I’ll give a little plug for one of our clients, I’ve been thinking about one of our clients this week because we’ve been kicking off a marketing campaign with Rich’s help from Secret Stashe, just to help them grow and hopefully cross the chiasm on […] and they are the social media for pets, get your politics out of there, get your whining out of there, and just enjoy pictures of furry animals, and it’s pretty fun. It’s good to see that take off a little bit and it’s fun to play with the marketing and see what works and what doesn’t with Rich. That’s what I’ve been up to.

This week we’re going to take a little break from talking about DockerCon remixes during those DockerCon talks that we’ve been doing for a few weeks and get back over to AWS because Chris had an experience that not that many people in the world have had, and I think it’s worth talking about. Chris recently, I guess it was about a month ago now, you went to an AWS exam development workshop. What was that? What is its purpose?

Chris: Yeah exactly, it’s kind of a mysterious sounding thing. What it is, it’s just like many other tech companies Amazon and AWS in particular have certification exams that they have that cover the type of material content in various different areas, and these exams are taken by folks to kind of demonstrate and prove that they are experts in those areas and they can get their certification as an expert.

These exam workshops are basically writing content for these exams, and making sure that it’s proper content, it’s testing the right stuff, and making it up there on to the actual exams that these candidates are taking.

Jon: Surely, if you are writing on that content for all those exams, they must have given you a free pass and certifications on all of them just automatically right, Chris?

Chris: Not quite. I actually thought long and hard about whether or not to do this. It’s quite an investment in time, this was that this was a three-day workshop and involved travel. It’s actually pretty difficult work and you’re not compensated for it or whatnot. But there are some very much kind of more subtle intangible benefits and that’s the reason why I’m overwhelmingly glad that I did it.

As a token of appreciation, you do get a badge to take one exam, complementary. It can be any exam, so I’m saving that one for my Solution Architect Pro when I go take that one.

Jon: Nice. I’m excited for you to take that one. So how did you get involved in this? Did they just, anybody they’re like, “Hey, we have a hard time finding people to write these so please just come.” How did you get involved?

Chris: They have a pool of subject matter expert, SMEs. It’s basically a list of, here’s folks that have taken exams and passed them, kind of recognized as subject matter experts and that is their pool of people to pull from. They have actually quite a few of these workshops going on throughout the year across not only the US, but also in other parts of the world as well. Focusing on the different exam topics as well as different parts of the exam development process, it’s not just the—writing the questions is just one portion of that process. There’s other pieces to it.

You have to figure out what is the content that should be covered. Who is this geared towards? What should that person be able to do? There’s workshops just on that kind of stuff to figure that out.

Jon: Knowing you, and I’ve been thinking about this a little bit before, too, because I can’t imagine that just anybody that passes one of these exams is automatically like, “Oh yeah, you should write questions for It.” Knowing you and knowing your skills and strengths, I imagine you really nailed it, you probably got sort of over towards the end of the bell curve in terms of people’s performance. I wonder if they do have sort of a threshold that they keep to themselves that you’re above that threshold of people that they’re hoping to write questions.

Chris: That’s a good point, and now kind of like thinking back through the process, there is quite a vetting selection process that gets involved. the way that it worked was, I went to re:Invent last year in November 2017. I kind of decided the last minute to go ahead and take one of the exams there and I did so, passed it, and one of the things that they mentioned was like, “Hey, if you pass your exam later here and if you would like to participate in our SME, be considered for our SME program, just let us know, we’ll take your information.”

Kind of ironic it was a pad of paper and pencil to get added to the SME program. This re:Invent conference where just incredible technology where it’s like we’re putting facial recognition into a camera that you can put in your garage door and have it sent back to the Cloud, or jotting down names on paper. That kind of started the process.

I think a few months after that, they reached out with a questionnaire. It’s actually pretty extensive. If I remember right, it’s at least 15 to 20 minutes to go through as that many page questionnaire, just going through it all, just experience and background, and whatnot. Then after that was when I started, I think I got put on a mailing list and just whenever these workshops were happening, they would bounce in my inbox, let me know that, “Hey, there’s one in New York, there’s one in San Diego, there’s one in Dallas.” With various different––both the exams as well as types of material.

I’m not sure exactly how much more vetting they did on the backend, I know they obviously have things like LinkedIn, and they can go and look at your history and whatnot, and experience level, but that questionnaire that you have to fill out was pretty extensive.

Jon: Interesting. So for people listening, if it’s something that you want to do, if you want to get involved in one of these workshops, I imagine that you may have to have some experience in the industry as well as, I wouldn’t be surprised if you have to get a fairly good score on your exam.

Once you decided to sign up, then you travel down to lovely San Francisco, one of my favorite cities, and tell us a little bit about the experience.

Chris: This was an item development writing workshop. The essential items is the Amazon lingo for the exam question. This workshop was a three-day workshop.

Jon: Sorry to interrupt, that’s what happens when developers just type the names of things.

Chris: Actually, one of the things that was kind of eye opening for me, too, is that putting together certification programs and exams, that’s a career. They do this as a career. They’re the ones actually that probably comes up with this terminology, because there are a lot of specific terms to this.

So that was pretty interesting and eye opening to me. You have people that this is their specialty and they can make careers of it. There’s quite the demand for it. There’s so many companies out there that have these kinds of training programs, and certification programs, and whatnot, even ranging from very formal to more informal.

Jon: Another little side track to that, if you happen to be listening, and you’re like trying to think how you can climb the corporate ladder and get really high up in the corporate ladder, finding those things that are maybe not the highest demand types of jobs, a lot of times it’s a great way to climb the ladder quickly, find yourself in a VP role at a big company and get all the same benefits of the other VPs in the same company that fought tooth and nail for years and years to get to where they’ve got.

I think of a few tracks like that, but not to take anything away from the exam developers, but I can imagine that there’s a lot of people sitting in computer science class right now thinking, or even any other type of class that are imaging that they’re going to be corporate test writers.

Chris: Supply and demand. If the supply is low, then your economics are definitely in your favor. For anyone other than with the pickup COBOL, by all means. The opportunity is still there.

That was the focus of this workshop, it was basically like, let’s go and write new items, new questions for this exam, and it’s for one exam, it’s for the Solution Architect Associate level exam. Contents covering that material that the Solution Architect exam covers and maybe some of the things to point out.

They mentioned that this is a rather large group. I think we were probably 15-people big. That may not sound like a lot, but for a workshop like this, especially with how much review goes on, it was large, but also very productive and really interesting to get so many different perspectives and whatnot. Of that, I believe there was three of us that were not Amazon employees.

Most of these things I think are definitely sourced from Amazon. I almost felt definitely like even that much more of a privilege to be there as a non Amazon employee. They definitely had to go through a bunch of hoops just to accommodate us being there things like, building access was different, getting access to their communication tools like Chime and just silly stuff like scheduling and travel arrangements, and whatnot.

It’s definitely more of a hassle for them to have external SMEs participate in this, but from my part, I’m glad that they did and it was really a great experience being there.

There was several folks there that this was not the first workshop they had been to. I’ve never even known that we had external folks participating in this, so that was interesting. I don’t think it’s all too terribly common, and maybe they’re kind of branching out now and that’s going to become more leveraged work in the future, but I think in the past, it’s probably a bit more heavily internal only folks participating in this.

Jon: Interesting, you go through the actual workshop, you’re spending three days doing this, is it pretty much just heads down working, or you’re working with other people. What was that like? Were you just trying to sit there thinking of ideas for test questions or were you brainstorming with people? What was the interaction like?

Chris: I will say I was totally blown away by just how difficult of a process this was, like writing exam questions, it’s not an easy process and it forces you to use the knowledge and experience that you have with these things and kind of flip it on TED and think of it in a different way.

The overall process was individually––you would work, you would come up with ideas for exam questions. You would write them up, develop them and then once that happened, it would then go into a review process.

You’re in a room with a whole bunch of other SMEs that really know what they’re talking about, and you’re right in that question that, hopefully it’s going to end up on the actual final exam test.

It’s definitely a high bar for content, you want to make the question hard enough, but not too hard. Again, this was for the associate level, not for the professional level. I have to keep that in mind. Obviously, there’s guidelines and guardrails for the type of areas that your question can test.

Just spending the time of kind of like coming up with what would make a good exam question that would stay within those guardrails for content, be at the right level of difficulty and definitely be technically 100% correct with no ambiguity. This has to be very quantitative, not qualitative. There can’t be any subjectivity about it.

Jon: I just feel like I have a hot take from this, I want to extrapolate what you just went through that you said involves mostly AWS people and sort of an intense work exercise, and you said that it was largely individuals with a little bit of collaboration at the end, and it sounds like it was very polite, it sounds like there was nobody that was you know, it was polite and maybe a little bit formal, not super informal that people just kind of went like throwing ideas around type of stuff.

Also, the other thing I kind of picked up on was that you kept talking about a high bar and that when you all came back together to review questions, that there was maybe a sense that if your questions weren’t very good, you would be a little ashamed, and that sort of tells me that there’s maybe a lot of competition.

I just want to extrapolate this to the rest of AWS that it could be, you’ve got to see a little bit of what it’s like to be an employee there, it could it be that it’s fairly formal, and it’s fairly high level of competition among people, do you think that that’s possible?

Chris: I think some of those are definitely spot on. I’ll clarify this a little bit. The actual item development portion of actually writing the exam questions, that was definitely very much independent, but then this review process that happened was extremely collaborative and polite and this was made very clear at the very get go, it’s all about constructive criticism. At the end of the day, all that matters is the best material is created for these exams and that it’s not a judgment on the item development author.

This review process was extremely rigorous. I’m not convinced that this reflects the Amazon culture, because again, there were a few of us that were external. I think this just reflects folks that are experts in their fields, just having opinions and being very knowledgeable, and taking the task very seriously.

There were times where we would have quite animated discussions over a certain word like should versus can, slowest versus fastest versus optimal, that kind of stuff. The bulk of our time was spent doing review, you’d spend a certain amount of time individually, heads down people are writing these questions, once you finish your question that would go out to the editorial review where it’s just basically making sure that’s adhering to all the linguistic guidelines that they have, like you’re using proper case, you’re using proper tense, you’re using the proper third person language and syntax type of thing.

So the editorial review, once it went through that, there would be a technical reviewer that would go through it, and this technical reviewer wasn’t even a part of this group. They’re actually somewhere else off site, and that was just part of their job. They were very in depth technical expert that would go through and they would do their pass to verify that this was technically accurate.

One of the things you have to do when you’re writing your exam questions is that you actually have to give site proof of what you’re saying the answer is. It has to actually link to something that’s produced by Amazon that’s publicly available that definitively shows that your answer is indeed true.

Those are the kind of things they’re looking for in that baseline editorial and technical review process, and then after that, it would be group review. That’s where we spend the bulk of the time in these.

We need to split the two groups because it was so big. You have a group of six, seven, eight people together reviewing these exam questions like you were taking the exam, and then just going through them, and then debating is this a good question? Is it technically accurate? Is it concise? Is it covering the right material? All that kind of criteria.

Most of those reviews for an exam question, it was not a quick process, it’s not like 60 seconds done, some of them took a while. It was made that ideally you were not in the room in the same group that was for the questions being reviewed that was written by yourself.

We did get into a point where sometimes that did happen, that’s not very fun, to actually be there when your stuff is being reviewed, to kind of hear some of the comments. People were not there necessarily to be polite it was really just about like, let’s make sure that this reflects the best possible material and that it is indeed accurate. It’s testing the right stuff and just generating the best output.

Jon: When you were split in the groups of six to eight, did anybody talk about how you are two-pizza teams? And is that your first time ever as part of a two-pizza team at Amazon?

Chris: I’ve never worked at Amazon to consider that. I can say no pizza was served. They did a wonderful job with catering and food, they brought in breakfast and lunch every day, but no pizza.

Jon: We’ve got to wrap it up. Any last takeaway? I guess, I’ll say my takeaway from this is kind of interesting to hear about the rigor of how AWS puts together its test, and I do think that it’s interesting to think about whether that rigor is a reflection of something deeper in the Amazon culture, in the AWS culture. I’d like to believe that it maybe is. Other than that, any other takeaways you’d like to bring up?

Chris: I was just surprised by how difficult it was to do this. This was very mentally taxing a three-day workshop, but at the same time, it’s super rewarding being around such a great group of people that knew more than I did. To actually hear their ideas and feedback, and to learn some new nuances of the other technologies was a great experience.

The other thing, definitely some tips for those that are interested in planning to take these exams, I would say that it feels like in the past, the exams were more, I don’t know, I would say they were designed to trip you up, but they were definitely trickier like in more of down in the details level and kind of a bit more maybe esoteric.

It kind of feels like now, they are driving more towards making it just much more real world relevant, and things that normally an expert wouldn’t have to remember that they would actually just go look up maybe what’s the maximum throughput of a certain service or something like that, and then megabits per second.

Those absolute numbers, that’s not the stuff that they really want to verify and test on, it’s really more like just the practical concepts and how does it apply to you actually being an expert in these technologies to be able to develop the correct solutions for folks that need it.

It’s a great direction that it’s moving in, they’ll make the exam that much more relevant. For folks that do pass it they say, “Yeah, I can go out and hit the ground running and actually do the kind of things that this exam is supposed to be proving that I can do.”

Jon: That’s great to hear. I definitely think that these types of exams should be fairly passable for people that have spent a good deal of time in AWS without too much study of esoteric facts so good to hear.

Chris: Indeed.

Jon: Thank you, Chris and thank you producer, Rich. That’s another good discussion on Mobycast. Talk to you next week.

Chris: Thanks guys.

Rich: Well, dear listener, you made it to the end. We appreciate your time and invite you to continue the conversation with us online. This episode, along with the show notes and other valuable resources is available at mobycast.fm/22. If you have any questions or additional insights, we encourage you to leave us a comment there. Thank you and we’ll see you again next week.