Building a Company-Wide Culture of CRO Experimentation #ConvCon

Sean Ellis, the CEO and founder of GrowthHackers, is going to talk about building a company-wide culture of experimentation. It’s a big difference in trying to change culture for a few people or a few hundred people. Dropbox was the first company Sean went into with a goal of creating a culture of growth experimentation. He spent six months as interim vice president of marketing building experimentation into the company.

Seeds of Testing Culture

Here are the steps that Sean took at Dropbox that led to an experimentation culture.

• Built tracking and testing infrastructure
• Shared insights (qualitative and quantitative) with team
• Shared testing wins with team
• Encouraged broader participation

When he left Dropbox, a replacement marketer didn’t join the Dropbox team for another 9 months, but the experimentation culture was so ingrained into the engineering team that it continued on. Today Dropbox has half a billion users.

In previous companies, Sean’s marketing and testing was siloed in safe environments, with a wall in between his experiments and the rest of the company. From these experiments he learned that growth is a company-wide effort. A lot of the powerful levers of growth sits outside of what marketing controls.

The experimenting should happen at all levels including bringing customers back who have gone away. A core growth team that manages that helps a lot. That’s what growth hacking is all about. Moving away from experimenting at the surface level and bringing it to the full funnel.

Testing drives growth. There’s no certainty about what works. Rapid testing is going to take place across all levers:

• Acquisition
• Activation
• Retention
• Referral
• Revenue
• Resurrection

More testing = more learning. You need the feedback loop because the more testing you run, the more you know what works in growing the business.

Of course, change is harder in established company. On the other side of the spectrum from Dropbox is Microsoft. Sean spent some time in the last few weeks at Microsoft and it’s a company going through a growth period. Different teams are at different points on the growth curve. They have a goal across the company to experiment with more frequency, and it’s going to take longer because there are a hundred thousand employees.

Experimentation culture is built on:

• broad participation
• transparent learning
• effective process

Effective process is a loop. Build ideas backlog ➡ prioritize the backlog ➡ launch tests ➡ analyze results and start again. The best companies have a team dedicated to executing this process. At the very least you need a product manager. If the product manager doesn’t have dedicated resources (like a designer or engineer) then they will hit walls and it will be hard to hit testing tempo goals.

When you have a good team in place, then building the idea backlog becomes really important. With a backlog in place, you don’t have to be creative each week of what to test next. Use a system for prioritization; he uses the ICE (impact, confidence and ease) rating system. Choose tests to launch.

Launching tests is next. He likes to make sure that every test has a project manager assigned to make that test go out. They get tools and allocate skills needed to implement.

Analyzing results is the final step (before starting over). Ensure tracking up front for the hypothesis. Run the experiment for a sufficient period of time. Triage analysis by hypothesis first.

Example of a weekly meeting agenda:

• 15 min: KPI review and update focus area
• 10 min: Review last week’s testing sprint
• 15 min: Key lessons learned from analyzed tests
• 15 min: Select tests for this week’s sprint
• 5 min: Check growth of idea backlog

When you’ve gone through this process, you can focus on transparent learning. Every test leads to learning, even the tests where you “fail.” And in every case, you want the whole team to be aware of the test learnings. Proactively share the findings with the idea originator. You may want to do like TripAdvisor and share wins with an opt-in email list. Organize learning for everyone’s easy access. Getting high-fives from your coworkers as you’re going down the hall is the kind of thing that’s going to get people excited about a testing culture.

Here’s how to kick-off the company-wide growth through experimentation initiative, what Sean calls the company kick-off:

• CEO tells everyone to seek opportunities for improvement and growth.
• Share recent wins and insights.
• Provide idea input training.
• Hold pizza and beer brainstorms.

Collect ideas weekly:

Create a standard experiment doc of idea submission. You should even ask them to score their own ICE score to standardize idea input and let them think through how difficult it will be to run a test like this and the expected outcome of the test.

Provide feedback on ideas. Acknowledge all ideas. Praise favorite ideas. Notify if selected. Report back on results.

Keep it fun in the company! You can have a leader board of submitted ideas.

Key Takeaways

• Share learning across the organization.
• Celebrate the wins because that’s where the process gets addictive.
• Empower the full team to generate ideas.
• Ensure core team has process to manage inputs. If you start to collect ideas prematurely and aren’t able to execute, you won’t be able to sustain contributions to the growth process.

Subscribe to the blog to get all the news coming out of Conversion Conference 2016!

Virginia Nussey is the director of content marketing at MobileMonkey. Prior to joining this startup in 2018, Virginia was the operations and content manager at Bruce Clay Inc., having joined the company in 2008 as a writer and blogger.

See Virginia's author page for links to connect on social media.