written by
Luke Szyrmer

How Many Experiments Do I Need To Run?

Unknown unknowns Assumptions Risk Innovation 2 min read

A teacher said to her student, "Billy, if both of your parents were born in 1967, how old are they now?"
After a few moments, Billy answered, "It depends."
"It depends on what?" she asked.
"It depends on whether you ask my father or my mother."

It's also like that with the number of experiments you need to run, in order to build a new product.

It depends.

On how much of a breakthrough you want your product to be.

Photographer: Rafael Pol | Source: Unsplash

Soviet inventor Genrich Altschuller created the "Theory of Inventive Problem Solving", or TIPS. In the original Russian, it's called TRIZ. TRIZ's a massive topic in its own right, providing a systematic methodology for solving problems creatively.

Apparently even creativity can be systematized.

Anyway.

While Altschuller reads quite heavily in the translated English version, he shares nuggets of wisdom--which are very relevant for product entrepreneurs.

For example, he pored over about 40,000 patent filings, mining for insight and inspiration. It turned out, that there were clear patterns in how inventors approached problem solving. Depending on the type of challenge being addressed, the number of experiments the inventors did made the technological breakthrough even greater.

Number of experiments

This shows exactly what's required if you want to create a breakthrough product. Most commercialized products which are "revolutionary" will require at least 10,000 well-thought out experiments.

You don't know which of your experiments will reveal insight. If you aren't doing thousands of them, then you ain't gettin' none.

The takeaway here is clear. Everyone wants quick results. Sometimes they do happen. (I'm all for being optimistic.)

More often than not, though, you've got a lot of experiments to run. Systematically. In order to really get beyond conventional wisdom. If you want to discover a counter-intuitive truth. So get started already!

Taking this into the Lean Startup realm, before you even start your technical experiments like Altschuller, you can run marketing experiments.

Figure out what your market wants. What problems they have. What keeps them up at night.

Then, when you put your product creator hat on, you'll make a product which has a chance at commercial success. That's the bit the Soviets missed.

If you want to find out more about how to run these types of marketing experiments, I've got your back. My upcoming book Your First Startup Experiment goes into a lot more detail.

Read it.

Apply it.

Profit from it.

altshuller experiment