Duolingo's Evolution: From Third-Party Testing To In-House Innovation With Over 500 Million Users
PLUS: Founding Global Head of R&D at Google& Facebook Engineering exec. building No-Code AI Co-pilot tool
Behind The Scenes
Duolingo, a leading language learning app, has over 500 million users, it has cultivated a culture centered on the principle of "test everything." Launching approximately 30 new experiments weekly, with hundreds running concurrently. Initially, they used a third-party tool for experimentation, but as costs rose and limitations became apparent, they developed their own in-house platform.
Empowering Experimentation:
Duolingo allows team members from any department to propose and run experiments.
They must draft an experiment memo outlining the hypothesis, expected outcome, related work, audience selection, and design specs.
Experiment results are monitored via a custom dashboard that compares various metrics, like lesson starts and conversions, against a control group.
Production and Guardrail Metrics:
Successful experiments are moved into production.
However, "guardrail metrics" related to engagement and retention (as opposed to monetization) require senior-level review before any changes are made.
Duolingo emphasizes the importance of running experiments for extended periods to ensure accuracy.
Pros and Cons of a Testing Culture:
Pros: Repeatability, objectivity in decision-making, autonomy, and a metric-driven approach.
Cons: Requires significant investment, may lead to incremental progress, can create additional QA overhead, and might not effectively drive certain metrics.
Building a Testing Culture:
Duolingo ingrains its culture of testing in new hires, emphasizing operating principles like prioritizing users, taking the long view. Their approach also influences their hiring process, where prospective PMs engage with ongoing experiments.
Getting Started with a Testing Culture:
Ensure a sufficient user base (100,000 users is suggested) for meaningful A/B testing.
Invest in intuitive and efficient tools to facilitate experimentation.
Develop detailed documentation and training for team members to conduct experiments.
Choose and prioritize metrics that genuinely impact the business.
Geeks of the Week
Startup Name: Lepton AI
Geography: US
One-liner: Run AI applications efficiently, at scale, and in minutes with a cloud native platform.
Founder(s) Background: Director of AI Platform at Alibaba, Technical Lead at Facebook AI,
Startup Name: Anytime.AI
Geography: US
One-liner: Enterprise Copilot for Legal Professionals.
Founder(s) Background: Engineering Leadership (LLM and GenAI) at Pinterest, Research Scientist at Facebook AI, VP of Strategy and Transformation at 1010data.
Startup Name: LiveX AI
Geography: US
One-liner: No-code AI Copilot designed to increase sales and improve customer satisfaction, connecting businesses and homeowners seamlessly.
Founder(s) Background: Founding Global Head of R&D at Google, Senior Director of Engineering at Facebook.