The A3 Experiment Sheet
Reporting on experiments is really really important. When you produce the results and thinking behind your tests, they serve as your PR, Marketing, Persuasion and Publicity tool in one.
By all means, produce a technical or geeky one for all your data science, analytics or testing buddies — but always make one for C level executives.
The best way to explain your experiment is with a story:
We saw this, We did that, This happened, It made us think something.
This is a sheet for capturing this story as you roll:
You can use this sheet to turn into Report Cards, feed into a Wiki or knowledgebase and generally optimise the intellectual property you are learning from tests.
Nobody at C level (or indeed most of the rest of your company) really needs to know the mind numbing detail of data that your test collected.
So what you should report starts with how the test is formed, how it is documented or described and what data gets filled in as the experiment progresses to fruition.
And if you accept that, then the very documentation involved in testing needs to support the outcome. The outcome is not the financial or lift percentage by the way — it’s what you usefully learned from the test.
If you’re in a fast paced testing environment, you need some way of quickly hacking documentation together. Maybe you need to print stuff and pin them up on the wall, along with heaps of pictures. Maybe you’re just juggling 50 tests this month so you don’t have time for ‘pretty’ documentation — just functional stuff.
I find it worthwhile to at least get people thinking — to fill out some form or chew over their hypothesis — BEFORE they ‘bring’ their tests to you. Having some preparation or screening filter can work wonders for deflecting stupid test requests.
Powerpoint, PDF and PNG versions here:
https://www.dropbox.com/sh/3m56vbwqa8jg0ue/AAC_ZenU0vYz08ghNiD7mzYCa?dl=0
Hope the sheet is useful — any mods, improvements, feedback welcomed!
Props
I’ve shamelessly adapted the Lean Stack sheets from http://leanstack.com/how-we-use-lean-stack-for-innovation-accounting/ and have created this one specifically for split testing, although it can by used to drive hypothesis led work generally.
