Experimentation, AI and the Product Model
NOTE: This is a narrative form of a keynote for the Circus Austin Experimentation Conference. While experimentation can be found in various parts of a company, most people think of either product or marketing. This talk is focused on the product organization, the people responsible for building the products the company sells.
Some Notes on Nomenclature
– Experimentation
Depending on the context, there are different definitions of what constitutes an experiment. Since the point of this talk is to attempt to broaden the scope and responsibilities of an experimentation team, I am using the more general sense of the term.
The purpose of any experiment is to learn something. We might be trying to establish causality or prove an outcome, or understand what users find confusing, or see if we understand a compliance constraint, or we simply might want to learn why our ideas are not working as well as we hoped. We might be trying to prove something, or we might simply be trying to better understand why the results are what they are. There are of course very different techniques used for each, and very different levels of data rigor, depending on the objective of the experiment.
More generally, many of our experiments are precisely because we want to avoid building and testing the wrong thing.
– Product Model
This talk focuses on experimentation in the product operating model. Yet most companies use the old project model, where stakeholders create product roadmaps of projects and features, and assign those to product teams to build and launch (output). In contrast, in the product model, product teams are instead given problems to solve, and they are then empowered to come up with a solution that delivers the necessary business results (outcomes).
Experimentation still happens in the project model, but it’s mostly optimization work. But if your goal is innovation and achieving outcomes, then this is where the broader role of experimentation becomes an essential competency.
Overview
The product operating model is all about consistent innovation as the key to delivering outcomes, and experimentation is at the very core of the product model. As Jeff Bezos explains, “If you want to innovate, you simply must experiment.”
While AI is already impacting the mechanics and tasks involved with planning, running and evaluating many types of experiments, the larger role of experimentation, and what we call “building to learn,” has never been more important.
In this talk, we’ll describe what we see as the necessary future of this broader and deeper role of experimentation.
Introduction
I don’t remember precisely when I did my first qualitative product discovery experiment. It would have been in the early 1980’s, and it would have involved creating and testing a prototype with a set of customers.
I do remember when I did my first quantitative product discovery experiment. In 1997 I was running a product organization at Netscape Communications, during the early rapid growth of the Internet. I was dealing with the (still common) problem of an executive that had a very strong opinion about a product’s design, yet I had a product designer with a very different opinion. I mentioned this issue to our company’s design leader, Hugh Dubberly, who had joined us after being a design leader at Apple.
Hugh explained that since the service we were running had a high volume of traffic (we were one of the first companies that had literally millions of daily active users), we could use data to run an A/B test and determine which treatment performed better.
Our CEO was Jim Barksdale, and one of the principles he drilled into us was: “If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”
It was the Internet that enabled product teams to collect the necessary volume of data that made data-informed decisions viable.
It’s hard to describe just how empowering that was; to be able to have the tools to pursue ideas, and overcome the objections of more senior leaders.
Moreover, the product model is all about delivering outcomes, and the way we attribute is with live-data experiments.
The Internet opened up a world of new possibilities, but today, AI has gone further to both change the calculus of product development (where we spend our time), and to change the nature of many of the products we are building (from deterministic to probabilistic).
The time and cost of building has dropped dramatically, which means companies can ship more capabilities than ever before. And for the past year, that is very much what has happened.
Initially companies thought they could simply build quickly, release into the wild, and see what sticks, but the high failure rates come with serious consequences such as frustrated and confused customers, loss of customer trust, and high churn. Even more importantly, few have seen a corresponding improvement to the outcomes.
The result is that there is more awareness than ever on the importance of experimenting in order to discover a solution that will deliver the necessary outcome. The consequences of this may be profound.
The hard part is no longer building, but instead the critical skill is figuring out what to build.
I’m expecting that the next several years will see the rise of experimentation as the core competency of effective product discovery and successful product companies.
For many companies that define experimentation more narrowly, this will require experimentation to play a broader and deeper role: beyond optimization to innovation; beyond proving outcomes, to influencing outcomes. Beyond testing what was built, to helping to drive what gets built.
Experimentation and The Three Types of Product Work
Top product organizations depend on different types of experiments for the three main types of product work:
– Creating New Products
When creating a new product or service, also known as “zero to one” work, we have the broadest set of risks. We not only need to address the normal product discovery risks of value, usability, viability and feasibility, but since this is an all new product, we need to make sure our assessment of viability includes cost structure, monetization strategy, and especially go-to-market strategy. Our goal is to achieve product/market fit.
Many experiments for this type of product work are primarily qualitative, as we don’t yet have large numbers of customers or scale of traffic, yet the insights can lead to real innovation or a significant change in direction.
The emphasis is on rapid learning, and we’re building lots of prototypes all for the purpose of testing before we build.
– Improving Existing Products – Innovation
Once we have an established product, we are now working to sustain a business. This involves work both to create new value (new capabilities), and work to capture existing value (get the most out of our existing capabilities).
We create value with continuous product discovery. We expand the set of problems we are solving for our customers or our company, we work to better solve existing problems, or we bring our solutions to adjacent markets.
This work is characterized by rapid learning through building prototypes and product discovery experiments. Since we have a live product with existing customers, we leverage both quantitative and qualitative experiments.
These first two types of product work are all about determining what to build, and especially the why.
– Improving Existing Products – Optimization
Once we have a live product with actual customers generating real usage data, we are able to leverage our third type of experiment, which is all about helping the current product work to its fullest potential.
Reducing friction, increasing clarity, and optimizing pricing can all help us capture more of the value that we have created.
This work is characterized by primarily quantitative, data-driven experiments. The differences are usually individually small, but when multiplied over thousands or millions of users, the impact can be substantial.
In the experiments for each of the three types of product work, we are trying to maximize our learning.
With quantitative testing, we are working to understand causality and quantify outcomes. With qualitative testing, we are working to understand why our customers are not behaving as we had hoped or expected, and uncover new opportunities to pursue.
AI and Experimentation
AI helps in virtually every step of the product discovery work, for both new products and product improvements, from analyzing and synthesizing user data, to creating live-data prototypes, to preparing for and administering tests and evaluating the testing results, but ultimately these experiments are about decisions and strategy – what to build and why.
For optimization work, AI is rapidly automating many of the tasks within these optimization experiments, including generating variants, running power calculations, estimating sample size, checking instrumentation, monitoring metrics, and detecting issues.
And if we are building or improving an AI (non-deterministic) product, we have an additional layer of complexity. Product discovery includes the eval process, and many of the judgements about the product are subjective. Moreover, an AI product changes the calculus for the use of quantitative tests, as they are generally more expensive and take longer, and the conclusions may not be as durable. We need to use evals to assess quality earlier and focus our A/B tests on establishing the value.
An Expanded Role for Experimentation
Today, there are typically two (or even three) separate groups responsible for running, or at least supporting, these various types of experiments. A user research group, often part of the product design organization; a data analysis/data science group, often part of the product management organization; and a CRO/optimization group which could be part of a product growth team, or a marketing organization.
Yet if the true power of experimentation is to become the company’s core competency, then we need to establish a broader and deeper definition of experimentation, and look at experimentation more holistically.
Unfortunately, to accomplish this, we need to address some organizational politics.
There are two main dimensions to this discussion: the type of experiment, and who runs the experiment.
– What Types of Experiments?
The first dimension refers to the type of experiment. It’s very typical in product companies that one group looks primarily at qualitative experimentation, and a different group looks at quantitative experimentation. Yet there is usually very little synergy when these are two separate groups, and many of today’s challenges need a more holistic approach to learning.
While many companies have a cultural bias for one or the other of qualitative or quantitative experiments, the best companies understand that each is essential, and when it comes to experimentation, they understand that 1 + 1 > 2.
AI can help us increase the scale of qualitative experiments to the point where the line between the two starts to blur. And the eval process is a good example where the rigor of experimentation professionals can contribute to the broader product discovery efforts.
Most people at least intuitively understand the importance of the quantitative experiments, but many don’t appreciate the value of the qualitative experiments.
As Product Discovery Coach Tom Chi argues: “Learning why it worked is way more important than whether it worked. Why it worked is an enduring property. That it worked is an episodic property. The reason why it worked is enduring because it comes from inside the person you’re trying to serve.”
It is becoming increasingly clear that for experimentation to serve the role it needs to, we need to embrace the contributions of both qualitative and quantitative experiments.
– Who Runs Experiments?
The second dimension refers to who actually runs the experiments?
Some groups prefer to organize as a service organization (known as the “agency model”), where product teams make requests for experiments to be run, and the results are presented after the fact as findings.
In other organizations, product teams prefer to run the experiments themselves (known as the “autonomous or democratized model”).
And others prefer a hybrid model where the product teams are responsible for their own experiments, but with coaching and assistance from experts in experimentation (known as the “hybrid or enablement model”).
Each of these three approaches has pros and cons. But after more than two decades in this situation, we have learned some lessons.
There are really two important questions. The first is who has the expertise to run the experiment, and the second is what it takes to cause a product team to take the experiment’s learning to heart? It is easy to focus on the first and ignore the second.
We know that all too often, when teams don’t run an experiment themselves, they do not take the learnings as seriously as they should.
We also know that most teams prefer to own their own experimentation for the same reason they want to own the discovery of the solution. Yet many of these teams do not have the necessary expertise to both run an effective experiment and then effectively interpret the results.
This is why the hybrid model is often the most effective, and what we generally advocate. It lets the product teams own the experimentation, and take responsibility for outcomes, yet with the assistance and coaching of experimentation professionals.
In the hybrid model, the experimentation team has an “enablement charter” where the experimentation team understands that it is their job to enable the product teams to do as much as possible themselves, yet ensure they are working both intelligently and effectively. They often provide actual services (experiment design support, managing test subjects, reporting) as well as ongoing coaching.
The benefit to the product team of the hybrid model is usually quite clear. But there is a major benefit for experimentation professionals as well. Experimentation professionals have often and rightfully complained of not “being in the room where it happens,” where the real decisions are made. Yet in this hybrid model, where they are helping from inside the product team, they are in that room.
Moving To An Effective Product Experimentation Competency
As AI makes the actual building and deployment of product capabilities faster and cheaper, everything depends on what we choose to build.
Experimentation is the core competency for product organizations that are working to discover solutions worth building. It embraces “building to learn” and then testing the prototypes against the product risks.
For organizations that have historically defined most of their experiments as optimization, this means moving from a niche data science CRO team measured by how many A/B tests can be run per quarter, to a product company’s core competency, measured by business and customer impact.
More generally, this means embracing both qualitative and quantitative experimentation1, and experimenting for both value creation and value capture. It means expanding experimentation beyond optimization to innovation, and beyond proving outcomes, to influencing outcomes. It means moving from determining which version won, to determining what we should build and why.
We need to move beyond isolated expertise in user research or data science to experimentation professionals that have deep knowledge of the full range of qualitative and quantitative techniques for the several types of product work, and can assist product teams in not just proving outcomes, but in creating products that drive those outcomes.
- Some people might think that I am describing the concept of “product ops” from a few years ago, as this could be the appropriate vehicle for this experiment enablement team, and I had originally hoped this would be the case, but the product ops movement went in a different direction. I’m thinking this effort has a better chance with a fresh start, perhaps as “experimentation ops.” ↩︎