Quotes About Production

A significant benefit of this is that when production incidents are shown in the same work systems as development work, it will be obvious when ongoing incidents should halt other work, especially when we have a kanban board.

Even high-profile product and feature releases become routine by using dark launch techniques. Long before the launch date, we put all the required code for the feature into production, invisible to everyone except internal employees and small cohorts of real users, allowing us to test and evolve the feature until it achieves the desired business goal.

Frequently, Development will take responsibility for responding to changes in the market, deploying features and changes into production as quickly as possible. IT Operations will take responsibility for providing customers with IT service that is stable, reliable, and secure, making it difficult or even impossible for anyone to introduce production changes that could jeopardize production. Configured this way, Development and IT Operations have diametrically opposed goals and incentives.

In addition to collecting telemetry from our production services and environments, we must also collect telemetry from our deployment pipeline when important events occur, such as when our automated tests pass or fail and when we perform deployments to any environment. We

Instead, code is only "done" when it has been fully tested and is operating in production as designed. (Note

It was important to us that for a developer, adding production telemetry didn't feel as difficult as doing a database schema change.

The latter pattern is what has become known as immutable infrastructure, where manual changes to the production environment are no longer allowed—the only way production changes can be made is to put the changes into version control and re-create the code and environments from scratch. By doing this, no variance is able to creep into production.

In Operations, we may deal with this problem with the following rule of thumb: When something goes wrong in production, we just reboot the server. If that doesn't work, reboot the server next to it. If that doesn't work, reboot all the servers. If that doesn't work, blame the developers, they're always causing outages.

I'm pretty sure we don't do any sort of analysis of capacity and demand before we accept work. Which means we're always scrambling, having to take shortcuts, which means more fragile applications in production.

a deployment pipeline. That's your entire value stream from code check-in to production. That's not an art. That's production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand.

virtualized environments from Bill's team work as expected, we can go into production one week from Friday." I gape at Chris. He just made up an arbitrary date to go into production, with complete disregard for all the things we need to do before deployment. I have a sudden flashback. In the Marines, we had a ritual for

How can we manage production if we don't know what the demand, priorities, status of work in process, and resource availability are?

Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management.

How can we manage production if we don't know what the demand, priorities, status of work in process, and resource availability are? Suddenly, I'm kicking myself that I didn't ask these questions on my first day.

Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck.

I have a sinking feeling in the pit of my stomach. How can we manage production if we don't know what the demand, priorities, status of work in process, and resource availability are? Suddenly, I'm kicking myself that I didn't ask these questions on my first day.

Next, we found numerous instances where developers have administrative access to production applications and databases. This violates the required segregation of duty required to prevent risk for fraud.

We also create pervasive telemetry so we can see how all our system components are operating in the production environment, so that we can quickly detect when they are not operating as expected. Telemetry

how we organize our teams has a powerful effect on the software we produce, as well as our resulting architectural and production outcomes. In

In the DevOps ideal, developers receive fast, constant feedback on their work, which enables them to quickly and independently implement, integrate, and validate their code, and have the code deployed into the production environment (either

The team quickly made a surprising discovery: only 50% of the source code in their development and test environments matched what was running in production.

work can bounce between teams endlessly due to incomplete information, or work can be passed onto downstream work centers with problems that remain completely invisible until we are late delivering what we promised to the customer or our application fails in the production environment.

Work is not done when Development completes the implementation of a feature—rather, it is only done when our application is running successfully in production, delivering value to the customer.

We cannot achieve deployments on-demand if we always have to wait weeks or months for production or test environments. The countermeasure is to create environments that are on demand and completely self-serviced, so that they are always available when we need them.