DIK_VED4A

AWS-Supp_DC.doc

Home >Computer Science homework help >DIK_VED4A

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <head> <meta http-equiv=Content-Type content="text/html; charset=unicode"> <meta name=ProgId content=Word.Document> <meta name=Generator content="Microsoft Word 14"> <meta name=Originator content="Microsoft Word 14"> <link rel=File-List href="CS-250%20Fireside%20Chat_%20DevOps%20at%20Amazon%20with%20Ken%20Exner,%20GM%20of%20AWS%20Developer%20Tools%20-%20AWS%20Online%20Te_files/filelist.xml"> <title>CS-250 Fireside Chat_ DevOps at Amazon with Ken Exner, GM of AWS Developer Tools - AWS Online Te</title>  <link rel=themeData href="CS-250%20Fireside%20Chat_%20DevOps%20at%20Amazon%20with%20Ken%20Exner,%20GM%20of%20AWS%20Developer%20Tools%20-%20AWS%20Online%20Te_files/themedata.thmx"> <link rel=colorSchemeMapping href="CS-250%20Fireside%20Chat_%20DevOps%20at%20Amazon%20with%20Ken%20Exner,%20GM%20of%20AWS%20Developer%20Tools%20-%20AWS%20Online%20Te_files/colorschememapping.xml">  <style>  </style>  </head> <body lang=EN-US style='tab-interval:.5in'> <div class=WordSection1> <h1>CS-250 Fireside Chat: DevOps at Amazon with Ken Exner, GM of AWS Developer Tools - AWS Online Tech Talks CC </h1> [00:00:00.00] [MUSIC PLAYING] [00:00:10.90] AARON SCHWAM: Hi. Thank you so much for joining us today for our Fireside Chat on DevOps at Amazon. My name is Aaron Schwam. I'll be your host. I lead product marketing for the AWS developer tools. With us today is our special guest, Ken Exner, who is the GM of the AWS developer tools. Ken, thank you for joining us. [00:00:25.93] KEN EXNER: Thank you, Aaron. [00:00:26.63] AARON SCHWAM: Can you please introduce yourself for the audience. [00:00:29.21] KEN EXNER: Sure. Hi. I'm Ken Exner. I manage developer tools for Amazon, which includes both internal developer tools, so everything we use inside of Amazon for software development, as well as external developer tools, the developer tools that are available for AWS customers. [00:00:45.96] So on the internal side, it's all the things that software developers at Amazon use, from source code management systems, to testing tools, deployment tools, some of the operational tools that we use. And this is across all of Amazon. So not just AWS, but the retail business and AWS. [00:01:03.83] On the external side, it's many of the same tools that had been externalized for AWS customers, as well as tools like Cloud9 in IDE, the SDKs, Command Line tools, Elastic Beanstalk, X-Ray-- a lot of the developer-focused services in AWS. [00:01:21.83] AARON SCHWAM: Awesome. So let's actually start on the internal tools, the things that people don't see or hear about as much. So as the owner of the internal tools, how does that affect your day to day in terms of do you have any influence over the way Amazon delivers software, how we train people, et cetera? [00:01:39.23] KEN EXNER: OK. Sure. So internally, we refer to this as builder tools. So externally, we're known as developer tools. But inside of Amazon we're affectionately known as builder tools. Amazon likes to refer to people that work at Amazon as builders. So we are builder tools. [00:01:56.61] In addition to providing the tools that we use across Amazon for how we develop and release software, we're also responsible for how we onboard, train, communicate, and continue the education of developers at Amazon. So for example, we work with developers in onboarding, so that when they start at Amazon they go through an SDE boot camp, which is sort of an immersive training program that allows them to get up to speed with how to develop the Amazon way very quickly-- how to use our tools, how to understand what the Amazon processes are like. And this is sort of an immersive two-day boot camp that is meant to be intensive. [00:02:37.61] We're also responsible for a lot of the developer training and internal conferences. So we have an internal developer conference called DevCon, which is kind of like our internal reinvent. It's where we communicate with developers. We do it three times a year in different locations around the world, because we have multiple locations. But it allows us to continue the ongoing training and development and communication with developers at Amazon. [00:03:04.40] AARON SCHWAM: Awesome. OK. So keeping that builder tools hat on, I'd like to start with a story some of our customers are familiar with, but we at least want to start with kind of the Amazon story for how Amazon embraced DevOps. So kind of at a high level, can you walk through what that looked like, and then we'll dive in. [00:03:21.41] KEN EXNER: Sure, sure. So the story usually begins like in the early 2000s-- so 2001, 2002. Everyone sort of knows Amazon as being a DevOps culture. But this wasn't always true. Back in the early 2000s, we were more of a traditional organizational structure. [00:03:40.20] We had more of a functional hierarchy, a functional matrix organization. We had development teams. We had testing teams. We had operational teams. And they all reported up to their leadership, so different silos of an organization. [00:03:57.39] And we also had more of a monolithic architecture. So not like what we have today, but it was more of a monolithic style architecture. And what I mean by that is that it was one big deployable unit. So it was-- basically it was C-- it was a Perl Mason code on top of C++ on top of an Oracle database. [00:04:18.50] And we had a bunch of developers all clobbering away at this system-- it was called the Obidos system-- and trying to push out changes in a release train model. So if you're not familiar with a release train model, it's basically-- it's a manual release process where you get on the release train, you maybe do one a week, one every two weeks. You manually QA everything, and then you get out one deployment a week or every two weeks. [00:04:45.44] It was a very cumbersome, slow process. And one of things that we started to realize is that we were not able to move faster. We kept throwing more developers at the problem, and that didn't help. What we found is, you know, double the size of the development team, and we weren't getting double the productivity, because we were bottlenecked by the architecture and by the organizational structure. So that led to some of the changes that became the foundation for DevOps at Amazon. [00:05:14.81] AARON SCHWAM: OK. So that was kind of the impetus, was we just weren't seeing the return on investment of trying to move faster. So at a high level, what were the things that we did to change our structure and accelerate software delivery? [00:05:30.45] KEN EXNER: Well, the big thing was-- and I'll call it decomposing for agility. So we wanted to decompose not only the organizational structure, but also the architecture. If you're familiar with Conway's law, it states that the architecture will follow the organizational pattern. [00:05:50.26] So what we had was a monolithic organizational structure, and we had a monolithic architecture. So we wanted to decompose it and go to smaller pieces that could move in a more agile fashion. So we did two things. We broke apart the monolithic architecture, and we broke apart the monolithic organizational structure. So I'll talk about each of them. [00:06:12.48] For decomposing the monolithic architecture-- back then, service-oriented architecture, SLA, was the popular thing, the pattern de jure. Everyone was moving towards service-oriented architectures. And we said, we're going to do the same thing. Amazon wanted to move towards a service-oriented architecture. [00:06:35.64] Today we would call this a microservice, but essentially the same thing. We wanted to go from this big monolithic architecture to a bunch of smaller services that each had their own interfaces, and these services could communicate with these other services via standard interfaces. [00:06:53.10] The important part here was that not only was it trying to decompose the architecture into smaller pieces, but you had a decoupling that allowed them to communicate with each other via standard interfaces. So I could deploy some changes to my service, as long as I maintained a backwards compatibility with my interfaces-- interface was the contract-- I could allow other teams to communicate with me. I could be able to move as quickly as I wanted to. [00:07:20.97] So we broke apart the monolith into these smaller pieces. But we also realized we needed to change the organizational structure so that we could organize differently. Because we were, again, going from this big monolithic architecture to a bunch of smaller pieces. How were we going to organize around these hundreds or thousands of different services? [00:07:40.62] We could no longer have this big development team and a big testing team. So what we realized is we wanted to have more of a decentralized organizational structure-- small teams owning small services. And if we had these small teams owning small services, they could be fully decoupled and able to move independently. [00:08:00.78] And that's what took us down this path of DevOps, moving towards these small teams that owned everything end to end, owned these services end to end. We called them two-pizza teams at the time. And this has become part of Amazon folklore. The idea of a two-pizza team was simply that you had a team that was meant to be small-- small enough that you could feed them with two pizzas. [00:08:27.33] AARON SCHWAM: Clearly, I wasn't on these teams. [00:08:28.88] KEN EXNER: No, no. You'd be your own two-pizza team. But initially, we were focused on small. Again, we were going from this big architecture to these small agile pieces, so we wanted everything to be small. [00:08:41.86] But that wasn't as important as having these teams be autonomous and independent. We wanted them to be able to move independently, and you wanted them to have ownership. You wanted them to have end-to-end ownership of their service. [00:08:57.72] They were not just responsible for the development of that service, they were responsible for the operations. They were responsible for the testing. They were responsible for the backlog and the roadmap, talking to customers. They owned that service end to end. [00:09:11.43] So essentially, the transformation was big monolithic architecture to a bunch of smaller services, and more of a functional hierarchy organizational structure that got decomposed into a bunch of small, autonomous, two-pizza teams. [00:09:25.28] AARON SCHWAM: So clearly, one of the big lessons here is decomposed fragility and the two-pizza teams. So what are the other major learnings that we had in our own DevOps journey? [00:09:35.83] KEN EXNER: Sure. In addition to decomposing fragility, I think another big part of the DevOps story for Amazon is around tooling and automation. When we used to have a centralized ops team, they would do all the deployments. As we broke apart into a bunch of different teams owning their own development, they had to own their own release process as well. [00:09:59.35] So when that happened, this is essentially how the builder tools team at Amazon was formed. We were essentially trying to automate that release process, because we had a bunch of teams that now owned their own deployment and release for their own services. [00:10:13.37] So we wanted to make sure that they were following best practices, and they didn't have to rely on the best practices of these skilled experts. So how are we going to make sure that these teams can deploy experts, but across thousands of different teams? [00:10:27.70] And that took us down the path of building tools to automate the entire release process. It started with deployment. So how do you deploy to the retail site without any customer impact? But it slowly kept going further until we had full CICD. [00:10:43.48] AARON SCHWAM: Can I ask how you decided where to start, and how you kind of went through the process? [00:10:48.20] KEN EXNER: Well, I think like a lot of businesses, we had started doing CI. So we automated the source and build integration, and we just kept going beyond that. And we said, well, the entire release process is a manual workflow, where you go from source to build to a bunch of preproduction environments. You do budget testing that's manual, and then you deploy. [00:11:10.92] We said, let's just turn all of that into automation. Amazon loves automation. Everything we do is about, how do you automate a fulfillment center? How do you automate different processes? So this to us was just another process that could be fully automated. [00:11:26.55] Now at first it was a little bit scary. People were a little bit apprehensive about this idea that you would commit a change to a source code repository and then it would flow automatically to production. But then what we started to realize was that anything that we could manually do as part of a release process, that we could manually test or manually inspect, we could put them into automation. [00:11:52.34] And that could happen every single time, no matter how big or how small that deployment is, so that you know every single time you do a deployment we would do the same tests. And you would have this increasing battery of different tests that would happen every single time. [00:12:08.88] Initially, this started with a lot of the integration test suite-- you would start putting it as part of a release process. And then it started going across other types of testing-- browser and web-based testing, things like Selenium. Then it started going into load testing, automated load testing, as part of every single deployment. And we kept doing more and more. [00:12:29.97] And slowly over time we started to get fairly sophisticated with our release process, so that every single time we were deploying, we were going through a number of different types of testing. [00:12:39.18] AARON SCHWAM: So how did you actually know that you were kind of making progress and improving? [00:12:44.49] KEN EXNER: Well, we measure everything. So at Amazon we love to measure everything. We monitor everything. We measure everything. And one of the things that we saw was that not only were we able to push out changes more frequently, we were also having a higher quality product. [00:13:01.50] We were able to have fewer problems in production. So we had higher quality and faster release cycles. So we were able to release more and get higher quality product out of the release process. [00:13:14.70] AARON SCHWAM: Awesome. So you mentioned CI. You mentioned a lot of the sophisticated testing. It seems like there's a really scary kind of hump to get over there in terms of like continuous deployment, which I know we practice at Amazon. Can you talk a little bit about that and how we got over that fear? [00:13:31.29] KEN EXNER: Sure. So the entire release process is kind of a pessimistic process. We call them pessimistic deployments, because we're constantly trying to find reasons to fail the deployment. So if a test fails in a preproduction environment, it'll stop that, roll it back to its last known good state. And then the developer has to go figure out what had happened and we push that release again. [00:13:57.60] But if at any point in the release process there's something that causes the system to either-- alarms go off because CPU spikes or a test fails, it's going to stop that, automatically roll it back, and then go back to the last known good state. So it's constantly looking for reasons to fail this deployment, either in preproduction environments where it's running a bunch of tests, or as it slowly fans this out into production. [00:14:27.98] So when we deploy into production, you typically deploy to one box in one AZ in one region. Let it sit there for some period of time, bake there for some period of time. Also, run a bunch of transactional tests-- like has every single piece of that code been exercised in production? [00:14:47.67] If there are any problems, it's going to stop it, automatically roll it back, and go to its last known good state. If it succeeds, it's then going to start fanning out to the rest of that AZ slowly, and then it's going to go to the rest of that region slowly, and then it's going to fan out across the different regions. [00:15:03.54] But throughout this entire process, all the tests, all the automation, are always looking for reasons to stop it and throw it out. So it's a very pessimistic process. And if you can get through all that, you usually have a good release. [00:15:17.23] AARON SCHWAM: Very interesting. So that covers kind of the CI and the CD. But what about like some of the most sensitive, or what about like security? How do you maintain security in this model? [00:15:30.43] KEN EXNER: Sure. Security is sort of managed throughout the entire process. I think it starts first with the developers needing to think like security engineers. And this is an important part of Amazon's culture. We want these engineers to not only be developers, but we want them to be operators, and we want them to be security experts. [00:15:54.76] Someone doesn't come to Amazon knowing much about DevOps or whatever, they typically come to Amazon with a CS degree. They know engineering. And we end up having to teach them how to think like security engineers, teaching them how to think like architects, how to think like testers, because they're going to be doing all these different functions. [00:16:14.78] So when it comes to security, one of the first things a developer is going to do when they're starting a new project is they're going to work on a threat model. They're going to design their architecture, and they're going to work on a threat model. And they're going to review that threat model with a security engineer. And that security engineer is going to give them feedback, and it's going to-- they're going to own their own security. [00:16:36.07] So they're going to be responsible for thinking like a security engineer. Because they're closest to this, they are usually the people that are most apt and able to find problems with the software. So we want them to think like a security engineer. [00:16:51.22] Once they've done that, they're going to start doing their development, and they're going to submit their code for code review. And that code is going to go to one or more peers. Teams can configure how they want to do peer reviews. And their peers are going to give them code reviews that get done before it gets committed. [00:17:16.06] We're also going to do a bunch of automated checks as part of that code review process. So there's a bunch of static analysis that happens as part of that code review process. So again, we can look for security issues through static analysis. [00:17:28.15] Once it goes out to the build process, we have additional static analysis that happens as part of the build process. Then as we start to go out into the release pipeline, there's a bunch of other additional checks that happen. There's a bunch of canary monitors that run positive and negative security checks against that deployment before it goes out. And we have a bunch of checks that verify the integrity of that pipeline. [00:17:51.88] AARON SCHWAM: And these are all instituted centrally or by each team? [00:17:55.54] KEN EXNER: Both. So there are some things that we can institute centrally that affect all of Amazon, or at least all of AWS and all of retail, and there are some things that can be mandated or governed at a local level. [00:18:13.70] So one of the things that we realized along the way is that we could inspect a pipeline. And if we could inspect a pipeline, we could determine whether or not it was following best practices. And if we could do that, and we could describe those best practices, we could allow people to create rules that govern the shape and structure and contents of a pipeline. [00:18:38.08] So this allows me, as a organizational leader, to have rules for my teams. So I can create a rule that says that every new commit that gets deployed has to have at least 70% code coverage, for example. So if I want to have a rule that I'm not going to let anything deploy unless it's 70% code coverage, meaning unit test coverage, I can enforce that for my organization. [00:19:05.89] Or AWS-wide-- we have certain rules that are either because of compliance or governance or security, we want to have rules that cover every single deployment at Amazon. An example I like to use is we don't like people deploying to every region at the same time. [00:19:22.21] So if a team changes their pipeline so that they're deploying to every single region as a first step in a pipeline, that's a bad practice. We don't want them to do that. We can automatically stop that deployment because they've broken that policy, that rule. And this applies to every team in AWS. So we're able to create these rules that stop deployments, either at a team level, organizational level, or company-wide. [00:19:47.90] AARON SCHWAM: So there must be some sort of-- I mean, that's a lot for a team to take on. So there must be some sort of like templates or best practices that you're sharing or vending out to the organization. How does that work? [00:19:58.87] KEN EXNER: Well, in general, we want to give teams the best practice pipeline by default. So we have templates for how to get started. We model all of this in code so that you can set things up the right way by default. [00:20:12.89] So a lot of the internal scaffolding tools, and a lot of the templates that we use for setting up a pipeline in all the different resources that hang off that pipeline, we have best practice templates for getting started. Super important. That helps developers get off on the right foot. [00:20:32.09] But then by having this inspection capability where we can inspect changes to that, we can make sure that they don't change it in a way that's bad. People don't always know all the best practices. There's a lot of lessons learned through the history of Amazon through a lot of mistakes that we've made. [00:20:50.73] We can take all those learnings and bake them into policies, bake them into not only that the best practices that are a part of the template, but the best practices that are enforced on that pipeline as policy. And this way, developers don't have to feel bad that they didn't know that you shouldn't deploy to every region at the same time, because there's a policy that stops them from doing that. [00:21:12.11] So initially, this, again, sounded a bit draconian at first, like we were going to start blocking things. But it's actually been very liberating for us, because it allows us to make sure that we're maintaining best practices, and developers don't have to make mistakes and learn those the hard way. [00:21:31.43] AARON SCHWAM: So I can see how that would be very reassuring for engineers. They don't have to be the experts of everything. So you were you were talking about teaching. So I'd love to hear more about kind of the cultural aspects of this that we know is really important. [00:21:45.78] KEN EXNER: Yeah. So as I was saying, DevOps is not something you can get a degree in. You get a degree in CS. And if you're going to expect developers-- [00:21:56.18] AARON SCHWAM: Or marketing. [00:21:56.79] KEN EXNER: Or marketing. If you're going to expect developers or marketing to own not only development, but testing and operations, everything else, you need to teach them those other things. Some of that happens just through training and normal educational activities that we do. But we have found that the most effective way to teach people is to make them do it and review it with people. [00:22:21.51] So the example I gave before about security threat modeling, and then reviewing that with the security engineer, we do the same kind of thing for other things. Another example would be, when you begin a project, you're also going to do a design. That design is not going to be done by an architect. We call them principle engineers at Amazon. [00:22:41.78] But an architect is not going to do architecture. An engineer on the team is going to do that, and then they go review that with a principal engineer or with an architect. The role of a principal engineer or architect at Amazon is to review and teach. It is not to actually do the architecture. [00:22:58.28] Same thing with the security engineers. Their job is not to do the threat model, it is to review it. Because you want the engineers on the team to be doing this themselves. Same thing with testing. You might do a test plan at the beginning of a project, and you will view that with an expert in testing. [00:23:16.94] The tester is not going to do your testing or do your test plan. They're going to review it. Because we want the team to own their entire process-- the architecture, the security, the testing-- they own all of it. And you want them to develop expertise in all of it. [00:23:32.58] So we spend a lot of time-- it's very time-consuming-- but we spend a lot of time teaching, because we want the engineer to know the security of their system. We want them to know the testing. We want them to know the operations. So it's an important part of this. [00:23:48.00] I think the other thing that's pretty important at Amazon is if something is important to the business, it's important that the leadership model that and show that. A great example of this is security and operations. Operations is critically important at Amazon. [00:24:04.39] And you know this because the leadership demonstrate that. They spend a lot of time actively engaged in operational reviews, going through ops meetings, looking at pages of graphs and giving feedback on the status of operations for each team. [00:24:23.09] In a typical week, I will spend at least six hours in ops meetings. And it's not uncommon for directors and VPs to spend a lot of time doing operational reviews. Andy Jassy, Charlie Bell in database leadership-- they spend a lot of time reviewing operations and reviewing security. And when you come into Amazon, you see your leadership spending this much time focus on operations and security, you learn very quickly that it's important. So I think it's important that businesses understand that if you want everyone in your business to take something seriously, it helps that leadership does too. [00:25:00.42] AARON SCHWAM: Yeah. So Charlie Bell actually hosts a weekly meeting on this, does he not? [00:25:03.95] KEN EXNER: Yeah. So yeah. Every Wednesday we spend a couple hours-- all of the AWS leadership team goes into an ops meeting with Charlie Bell, and we review COEs, which are sort of a five whys, RCA-type process, where we look at issues that have happened, and we recause them and discuss what happened and learn from them. [00:25:28.52] But we also spin a wheel. We have a wheel spinning where it lands on a service, and then that service has to present their ops, which is presenting their dashboards. [00:25:40.70] AARON SCHWAM: So essentially, you have to be ready. [00:25:42.11] KEN EXNER: You have to be ready. So every team in AWS has to be ready. At any given week, you could be asked to present your dashboards. And you're expected to know everything that happened. [00:25:53.15] So we'll pour through all these different graphs. We'll look at every single blip and ask what happened. And I should be able to explain what had happened. So in order for that to happen, I have to have my ops meeting so that I am able to dive deep and understand what had happened for each of my services. [00:26:12.12] AARON SCHWAM: That must've been fun in the early days when there were only 10 services. [00:26:16.15] KEN EXNER: We used to be able to fit in a small room at a small table, but it's gotten to be a very big event. [00:26:22.79] AARON SCHWAM: Yeah. OK. So stepping back from the engineering and the operations, we've got all these distributed two-pizza teams all over Amazon. How do you handle kind of the planning and the prioritization when we've got such a decentralized org structure? [00:26:37.85] KEN EXNER: Is it some big master chess game where we plan it? No. What we've actually found is that the best way to plan is bottoms up. And the reason why is that the teams who are closest to product are also closest to the customer. [00:26:54.05] They're the ones who are working with customers on a day-to-day basis. They're the ones who are not only operating these services, but working with customers and fielding their questions. They know what the customer wants. So rather than have this big architecture where we sort of plan things in a grand way, what we do is a bottoms-up planning process, because we want the people who are closest to the customers to be telling us what we should be doing. [00:27:17.40] So we have an annual planning process. It's called OP1, OP2-- Operating Plan. And what it's-- it's fairly unique. I've not seen something like this in the industry. But at every level of the organization you write a six-page doc that talks about what you're going to do over the next year, what your plan is going to be if you have flat resources, and what you would do if you had incremental resources. And you present your business plan, and you have six pages to do it. And then this happens at every level of the organization. [00:27:47.82] So for my organization, I get a bunch of these six-page docs that I review with these teams. And then I take all of this, and then I turn it into a six-page doc that I then present to Andy Jassy and Andy Jassy's leadership team. Andy Jassy does the same thing. He turns all of AWS into one six-page document that he then presents to Jeff Bezos and the S-team, and then resources flow down. [00:28:12.54] But what is happening in this process is you're allowing everyone who is closest to the customer to have input into the planning process. And the layers of management are essentially acting as a-- rationalizing the different requests, and making judgment about what we should be focused on. [00:28:30.71] But you're still taking all the input. You're not silencing that. You're pulling the input from the customer, from the people closest to the customer. And then you present that all the way up, and then resources flow down. So that's how we plan things on an annual basis. [00:28:47.90] After that, because we want these different teams to still look and feel and operate like startups, we manage them like startups. We ask them to take goals, so they come up with goals given the resources that we've given them-- this is your headcount, these are different resources. [00:29:05.78] We ask them to sign up for goals, and then we track those goals over the course of the year. And the rule of management is essentially to act sort of as a board of directors, managing their different startups. [00:29:18.32] Each of these different startups has their goals and their metrics. And we meet once a month or once a week to review how those teams are doing against those goals and those metrics. But the decision-making, what the team is doing, is being decided on by that team. And our job is just to manage them like a board of directors. [00:29:38.99] AARON SCHWAM: Awesome. So you've talked about basically three things. We had decompose for agility. We had automation and CICD. And we talked about kind of a culture on the back end here. So what are the biggest challenges or downsides that come with kind of this model and the way we operate? [00:29:57.14] KEN EXNER: There's a few. Well, one of the problems that we've had is when you have a bunch of autonomous independent teams, communication and consistency can be hard. So even though our planning process tries to find duplicates-- like this team over here is going to be-- is working on the same thing as this team over here, through the planning process we usually try to find the most egregious cases of that. [00:30:24.11] It still happens. We still end up with two versions of the same thing in completely different parts of the company. Because you don't have as much visibility across the entire company. So that happens. [00:30:37.52] One of the quotes I love from Jeff Bezos is, "I'd rather have two of something than none of something." And the idea is that we accept that. We accept that it's going to be an imperfect model, because you're going to end up with duplication. [00:30:51.44] But we rationalize it afterwards. We fix it afterwards. We don't want to slow things down in order to get things completely aligned. So we accept a little bit of that. [00:31:01.07] The other problem with this model is it's hard to have consistency. You don't only have duplication, you have challenges of consistency. But what we've found is that that's usually just a factoring problem. You can usually solve that by creating another two-pizza team that factors out that problem and turns into another service or component that then drives consistency. So it is difficult, but we've found ways to cope with it. [00:31:28.76] AARON SCHWAM: What about cooperation across teams or organizations? [00:31:35.30] KEN EXNER: Yeah. If every team has agency, they own their own business decisions. There is often this problem of how you convince that team to do something that you need them to do. And it's essentially like a bunch of different startups, and you have to convince each of the team-- [00:31:52.70] AARON SCHWAM: Be compelling. [00:31:53.27] KEN EXNER: You have to make a compelling argument that this is something that we should be doing. Usually, what we do is anything that's going to be driven across all these teams, we try to do as part of the annual planning process. There are some things that are going to be sort of top-down. [00:32:06.77] We are going to decide as a business we are going to go into a new region. So we ask all teams to plan for that. There are sometimes things that originate in one team that they haven't asked of a different team. And if you can get that into the planning process, you allow the team to plan for it. Make the case-- get them to plan for it. [00:32:24.57] AARON SCHWAM: So one other kind of challenge I was curious about is scaling DevOps teams. So how does that work? Can every team be a DevOps team? Would love your thoughts. [00:32:34.58] KEN EXNER: Sure. Well, we have found that most teams that do software development can be DevOps teams. It gets a little harder when you get a little bit further from software development into things like IT and stuff like that. But if you're doing software development, we have found most teams can be DevOps teams. [00:32:51.11] We scale them by splitting them apart-- mitosis. We have a team that's owning something. It gets a little bit bigger. We split it apart. This team owns this service. This team owns this service. And we just keep splitting them apart. [00:33:06.08] EC2, for example, began as one two-pizza team. It's no longer one two-pizza team. It seems much bigger than that. And it just keeps breaking apart into a bunch of smaller services, and a bunch of smaller teams owning those services. So we just keep splitting things apart. [00:33:21.59] Usually, it reflects the architecture, because you don't want to have one-- EC2 is not one big monolithic service. It's a bunch of different services. So you have these smaller teams owning smaller services, and you keep splitting things apart. [00:33:34.67] AARON SCHWAM: All right. Ken, thank you so much for joining us today. We will reserve time for Q&A. So we're going to pause now and then take your questions from the chat. [00:33:50.03] So first up, we have a monolithic architecture. How should we think about moving to the microservices model? [00:33:57.17] KEN EXNER: Sure. So I've seen companies do this-- one of three models. So there is sort of the Big Bang approach. We're going to make a top-down decision that we're going to break apart the monolith and do it all at once. This is kind of what Amazon did. [00:34:14.45] Jeff Bezos and the leadership team decided we are going to do this. And it's going to take us a few years, but we're just going to do it. That works-- sometimes it doesn't, but that's one model. [00:34:25.86] Another model I've seen is businesses that basically just take anything that's net new, sort of anything that's green field they're going to do the new way. So leave everything behind that's monolithic, and anything that's new develop it as a microservice in the way you want it to be going forward. And then pull the business in that direction over time, so net new. [00:34:50.81] The third way I've seen this happen is carving it apart piece by piece. So slowly over time, saying we're going to carve this piece off and turn it into a microservice, and then slowly chip away at the monolith. [00:35:05.36] I don't know which is the best pattern, but I've seen companies succeed and fail in any of those three different models. It really is just about execution. You can do it all at once, you can sort of chip away at it, or you can do something net new. Any of those models can work. [00:35:21.67] AARON SCHWAM: OK. Next question. We release a big batch of features every couple of months. How would it look to do releases a couple of times a day? So I'm kind of interpreting this question as like long-running features. [00:35:36.49] KEN EXNER: Yeah. Yeah. I think this is-- so AWS does a lot of releases, but we don't do like one every second-- well, not yet. But we are actually pushing changes like every couple of seconds. [00:35:52.96] But what happens is not all those changes get surfaced as features. Sometimes if it's just a bug fix, we can just push that out. Sometimes if it's sort of an additive feature, we can push it behind a feature flag. We use feature flags a lot at Amazon. [00:36:10.08] AARON SCHWAM: What's a feature flag? [00:36:10.66] KEN EXNER: A feature flag is basically wrapping the change in code that has a flag that you flip when you're ready to release it. So the change is sort of pushed out to production, but it's not customer visible until you change the flag, until you flip the configuration. Then it exposes the feature. [00:36:29.74] AARON SCHWAM: So it's literally just not read in the code file? Or it is and [INAUDIBLE]-- [00:36:34.75] KEN EXNER: It is, but it's wrapped in a flag in a configuration file so that it doesn't become public until the configuration is changed. But it is pushed out to production. That is very common. [00:36:47.83] The other thing is sometimes it's an additive feature that doesn't get exposed immediately. But we can just sort of push that out, and then when we're ready to expose it, we'll just expose it. But the idea is we always want to be pushing things out regularly. [00:37:02.59] Typically, a team will push things-- do 5 to 10 different deployments a day, constantly releasing. But as long as you can wrap things in a feature flag, or if it's sort of a net new thing that doesn't change the existing experience, you can be pushing these things out very regularly. [00:37:20.98] AARON SCHWAM: OK. That was very interesting. So we've got a couple more here. Next one-- our security team is hesitant. What have you learned to help to get the security team onboard with this model? That's a good one. [00:37:32.02] KEN EXNER: Yeah. So I think security orgs, you should be a little bit concerned about developers having all the control. But I've seen a lot of that change. As security organizations start to realize that by having a lot of automation, you can ensure a lot more governance and a lot more controls. [00:37:52.19] So there's the whole DevSecOps movement right now. And what DevSecOps is is about injecting security into the release process. What I described earlier, how Amazon does things, we have a lot more security testing, a lot more security controls that are part of the release process. That can happen every single time. So we have a lot more controls in place because we have all this automation. [00:38:18.26] So I think as most security orgs start to realize this, that they can get more consistency by moving towards DevOps and towards this full automation, they actually get a lot more comfortable. Because they can actually ensure that processes are being followed every single time a deployment happens, that you're not relying on manual processes. [00:38:38.33] So if I can have a given positive/negative security test that runs every single time a change happens and that's part of every release process, I'm going to get a lot more comfortable with it. So I think security orgs are starting to naturally start to embrace DevOps, as they start to realize that DevOps is also DevSecOps, it's about injecting security into the release process, they're getting a lot more comfortable. [00:39:06.73] AARON SCHWAM: OK. Next up. Are these teams really only all just software developers? [00:39:13.99] KEN EXNER: That's a good question. No. So when we moved towards these two-pizza teams at Amazon, we actually said, OK, we're going to have just general SDEs. [00:39:25.21] AARON SCHWAM: Software Development Engineers. [00:39:26.08] KEN EXNER: Software Development Engineers. And we tried that for a while. And then we started to realize that most teams did have some other specialist requirements. A good example of this is front-end developers or webdevs. A good webdev is going to be much better than a well-intentioned back-end developer. [00:39:49.21] So we realized that there was room for specialization, and that these teams, like startups, should have a mix of different skills. There's going to be a lot of generalists SDEs-- Software Development Engineers. But there's typically going to also be a webdev or a front-end engineer, product manager, a doc writer, a systems engineer. [00:40:14.70] So we start having different functions within these two-pizza teams. Most of it tends to be software development engineers, but where there is specialization, we've learned that it helps to have specialists. So have a product manager. Don't have an engineer try to be a PM. Don't have an engineer trying to be a marketer. So we started in-- [00:40:36.35] AARON SCHWAM: And don't have a marketer try to be an engineer. [00:40:38.00] KEN EXNER: Don't have a marketer try to be an engineer. We started to realize that it actually helped to have a little bit of diversity in these teams. [00:40:45.62] AARON SCHWAM: Thank you so much. That's all the time we have. We really appreciate that you made the time to join us today. To learn more, we've got a couple different resources for you. The first is our DevOps page-- [email protected]/devops. It's got a lot of definitional stuff that we talked about today, but we're also pushing a lot more content to that page this year. [00:41:03.17] We also have lots of reinvent videos and things of that nature up on YouTube. And of course, this talk that you saw today will be up on YouTube in a couple of days. Thank you. </div> </body> </html>