Staging Intent

People tend to define staging in relation to production. “Staging is where you deploy code before you deploy to prod.” “Staging is like prod but without customers.”“Staging is prod lite.” Staging can be all of these things, but let’s clarify its intent.

Staging is where you validate the known-unknowns of your systems.

These known-unknowns are the dependencies, interactions, and edge cases foreseeable by the humans in your company and the machines they tend. Staging is where you gain confidence in your systems by consensus.

Why staging?

Why have a staging environment? It’s easy to brush this off by saying “best practices,” but I think it’s good to examine best practices from time to time and make sure they actually fulfill your needs.

Let’s define our needs as establishing confidence in our code, infrastructure, product, and deployment pipeline to ensure better stability of our platform.

Can we fulfill these needs in other ways? Perhaps! To dig into this, let’s address the main argument against using a staging environment: tests.

Staging & Testing

“You don’t need staging when you have good tests.” I’ve heard this from small startups and from companies that are nearly household names. These companies have two or two and a half environments: local development, an elaborate testing framework, and production. The testing framework is impressively robust and is fixed fairly quickly if it breaks. Passing builds are deployed to production with all the confidence green Jenkins jobs can buy. This process balances the edginess of the Cowboy Coder with the warm fuzzies of We Did All We Could.

The Good, Bad Ugly about Testing?

But there’s an inherent problem with this model: It depends on the forethought of individual humans. Tests are good, but tests must be written. And who is writing them? Often, the person who wants to ship that section of code as soon as possible. Even if you write tests in pairs, that’s only two humans trying to account for every probable interaction of code in the wild. That’s not setting anyone up for success.

UI Testing

This model breaks down further when your product has a UI. People don’t write unit tests to ensure a sidebar is the proper shade of millennial pink. Mockups only go so far, and not everyone can make design meetings or stay focused in them. Running your UI in an environment where employees have to look at and interact with it smokes out issues from color mismatches to weird button behaviors.

Even without a UI, tests don’t account for all possibilities. Asking one or two humans to imagine the innumerable interactions of machines isn’t likely to produce good coverage. But an environment where those interactions exist and you can let the code run organically likely will. Enter staging environments.

When should you build a staging environment? Get your hands dirty

Ideally, before you have customers using your product, and it’s still a small, lightweight, and easily portable application. Realistically, however, you’ll probably build one about two months after the latest outage that made it to your board.

Who should build it?

Often the first instinct is to assign it toQA because it “makes sense,” but staging is a kissing cousin to production and should be constructed the same way. Have your infrastructure team create the platform and your application engineers fill it with their services.

Different technologies teams use

As a result, be prepared to discover that Team A uses DynamoDBagainst company best practices, and Team B uses custom Capistrano scripts because they think Jenkins is boring. You’ll want to take an inventory of the different technologies teams use, identify dominant players, phase out outliers, and make transitioning into a more homogenous pipeline a primary roadmap goal.

The Workflow