Thanks, Mr. Kay!

The private school I attended through the eighth grade taught a rigorous science curriculum. I didn’t appreciate it’s rigor until I transferred to a public high school. Actually, appreciate is a really strong word. I didn’t miss writing 10+ page science reports in high school. My appreciation came recently as a working adult. I appreciate that my initial exposure to science was inextricably linked to the scientific method. It’s burned into my brain. It has become extraordinarily helpful as an approachable, well-documented thought framework for my team.

How do we use it?

Observation: What is x? We observe it in the wild. We read about who is doing it well and who isn’t doing it well. We learn about the available tools, popular tactics, and game-changing strategies. We learn the vocabulary. We leverage those who came before us and learn from their lessons.  

Hypothesize: Now that we’re armed with enough information and research to be slightly dangerous, what do we want to learn? At this stage, the most important thing we can do is to resist the instinct to put a number of tactics into practice. Beware: as a rule, smart individuals tend to want to test a lot. Once we understood the tools and the various tactics, we wanted to try it all. Discipline, my friend. I am deeply indebted to Mr. Kay from my middle school science days for instilling a deep appreciation of controlled variables. If you test everything at once, you learn nothing. Define a test, define which variables to control and which to test, how to measure, the length of time.

Test: finally, we get to test something. We tend to use the minimum period of testing necessary. We’re not Facebook, so we can’t run a test and generate statistically significant results in mere minutes. Strong organic traffic during the testing period is a blessing. (Really, when is strong organic traffic not a blessing?) We’re diligent about defining our measurement system and metrics before we begin the test. Someone is in charge of reporting results on a regular schedule. Time is a constant variable. We don’t report results every couple of days, we report them every 3 days or every 7 days. Time is always included in the results.

Measure: what happened? Use the measurement tools and metrics we previously defined. Sure, go above and beyond, but only after following the instructions we created for ourselves.  

Learn: what did we learn? What worked, what didn’t work? We’re a team of debaters - in the best possible way - so someone will always be the devil’s advocate. What could we have done better? If the results are inconclusive, why? What do we want to try next?

Last, but most importantly: Document.

What has the organization learned? Have we learned something specific to the test (of course we have!)? Have we also learned something about how we run tests and how we might be more effective next time? I’m a big fan of immediate, formal reflection. What worked, what didn’t work, what should we do differently next time. We use Google Docs to document what we learn, but we rarely reference them. Just like in middle school, the act of writing, editing, and distilling learning into a written document is enough to sear the important lessons into your memory. These documents are incredibly useful during the onboarding process for new team members. It sets the tone for how we work, how we think, and how we learn from everything.

Thanks, Mr. Kay, for providing me with an appreciation for how science works and, more important, for equipping me with a thought framework that is applicable outside of the lab.