Everything Is an Experiment

I really like framing processes, activities, projects, etc. as experiments. We're introducing AI? It's an experiment. We're switching agile methodology? Experiment. Another reorg? Well, sounds like another experiment. An experiment can either succeed or fail - that's the beauty of it. We're doing the thing, we're committed to set it up for success, but we're also acknowledging what we don't know on the way there.

Here are some thoughts on how to frame the experiment and actually enjoying the results later.

1- Be Accountable

You said that using this new tooling is an experiment. So why are you insisting on throwing more people on it, when your existing team tells you the cost is too high?

Framing an activity as an experiment also means admitting defeat when it's the right time to do so; going back to the drawing board and taking responsibility for whatever worked and whatever failed. Of course, also make sure to celebrate when the experiment succeeds. Reflection and accountability are useful regardless of the outcome.

2- Articulate a Hypothesis

If you can't spell out your goal when approaching the experiment, why are we even here?

For example, you'd like to propose a bug bash. Why? Is it because you were neglecting the support queue? Is it because the product team is overwhelmed with the backlog? After figuring out the why (experiment goal), explain how your suggestion will support that goal. If you struggle with this part, you will struggle with executing the suggestion.

3- Define Success Metrics

How do you know you'll be successful? Connecting back to your 'why', a 'better feeling' is not a good success metric, but 'fewer support issues are open under this category' is. While this is a tough one to define (especially if you have somewhat of a consensus around an idea, folks will be more forgiving about not having it), it will help you in being more intentional when approaching the execution of the idea.

4- Identify the parameters

In one of my past jobs, we were a group of a few new, motivated directors. We all had at least 7 great ideas on how to change the engineering org for the better. Did it work? I don't know. Things got better, but it wasn't clear which idea supported those improvements and which ideas actually introduced noise and change-fatigue for the team.

Introducing a better incident management process? Great. Might want to hold off on forcing a new unit tests coverage quota. Both are excellent changes, but one might impact the other, and it will be a real challenge to identify which was successful and how to standardize that success.

--

All this to say: be clear about the basics. What is it you're proposing? Why? How will you approach it? Who are the people involved? When should you check-in and revisit the experiment? If your answer is 'Depends', congratulations - you have an engineer brain, but this is the time to actually break down that decision-tree and articulate the different options/combinations of parameters.