Episode 19– Flexing your A/B Muscles – Rich Skinner – Head of Data, MAVAN, formerly Google, Zillow, Zynga

Matt Widdoes interviews Rich Skinner, Head of Data at MAVAN, on the importance of data and experimentation for business growth. They discuss the common mistakes businesses make when it comes to data, the key data points to track, and the benefits of running experiments.

Rich emphasizes the need for a data culture and the importance of starting with simple tests before moving on to more complex ones. He also highlights the value of continuous learning and iteration in driving sustainable growth.

 

[0:00:05]

Introduction - Rich Skinner, Head of Data at MAVAN

[0:02:36]

The importance of data for growth and decision-making

[0:04:11]

The difference between those who value data and those who don't

[0:05:43]

The need for data readiness and a data culture

[0:06:53]

The gaps in tracking and measuring data in early stage ventures

[0:07:37]

Lack of measurement around conversion

[0:09:01]

Difficulty in tying together online and offline customer journeys

[0:09:51]

Importance of constant data quality and regular reporting

[0:10:15]

The need to regularly review and analyze data for business understanding

[0:12:04]

Challenges of setting up and leveraging data correctly

[0:14:22]

Starting with the most fundamental metrics that matter to the business

[0:16:54]

Maturation cycle of businesses and evolving metrics over time

[0:18:10]

Making optimizations at every stage of the funnel

[0:19:30]

Importance of experimentation and identifying optimization opportunities

[0:20:14]

Importance causal methodology in business

[0:22:15]

Best practices for early testing in businesses

[0:24:09]

Importance of hero section and emotional appeal in web design

[0:24:59]

Using the jobs to be done framework for landing pages

[0:25:25]

Getting started with A/B testing and statistical significance

[0:26:04]

Challenges and biases in A/B testing

[0:28:58]

Most A/B tests are flat

[0:29:12]

Minor changes and avoiding false positives and negatives

[0:31:49]

Decisions made based on limited data or dirty tests.

[0:32:20]

Making changes in A/B testing has trade-offs.

[0:33:05]

Do no harm thresholds and other metrics impacted by tests.

[0:34:29]

Consider dependencies and silent latent metrics in testing.

[0:35:21]

Accounting for multiple touchpoints and subtests in testing.

[0:36:27]

Addressing seasonality and interactive effects in testing.

[0:38:02]

Ensuring equivalent splits and measuring aggregate impacts.

[0:39:03]

Using holdout groups to measure overall impacts of tests.

[0:40:28]

Success in testing comes from incremental improvements over time.

[0:41:38]

Analogy between A/B testing and weightlifting in the gym.

[0:42:07]

Importance of following best practices in A/B testing and gym workouts

[0:42:59]

The biggest impediment to A/B testing is a culture without a culture of testing

[0:43:58]

Creating a habit loop and culture of testing is crucial

[0:44:29]

Starting small and being consistent in A/B testing yields substantial gains

[0:45:07]

Conclusion - Don’t forget to subscribe to Growth@Scale