Controlled Experimentation in Recommendations, Ranking & Response Prediction

Controlled experimentation, also called A/B testing, is providing the structure and moulding for future innovations in Recommender Systems, Search Ranking, Online Advertising and an array of other response prediction applications. Consumer facing web technology companies, including Amazon, Facebook, Google, Groupon, LinkedIn, Microsoft, Netflix, Yahoo, and countless others often utilize online controlled experiments for testing out new product strategies and approaches. Experimentation is the gold standard for leveraging the overwhelming growth of these sectors by consistently guiding the companies’ data-driven analysis and decisions.

The theory of controlled experiment is simple and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s. However, the deployment and mining of online controlled experiments in practice and at scale can be very complex and challenging. As experimentation gained in popularity, so did the need for properly designing, managing and analyzing experiments, including avoiding pitfalls that can lead to bad and often catastrophic decisions for the aforementioned applications. On the other hand, scientifically instrumented, statistically validated and empirically endorsed approaches lead to a very structured whetting of algorithms, user interfaces and strategies at a brisk pace which is crucial to keep the innovation momentum going and critical for long term success of these applications.

In this workshop, we’ll invite several leading industry experts to talk about their personal experiences on how online experiments are used in ranking for recommender systems, advertising and search applications. Each expert will contribute key lessons, challenges and best practices from their domain and bring to the workshop their years of experience designing and analyzing experiments. This is invaluable given that first-hand accounts discussed at length from so many diverse areas are seldom found in one setting. We will also get a chance to compare experimentation systems used at different companies and identify challenges and trends common to all.

Organizers

• Ya Xu, LinkedIn
• Rajesh Parekh, Groupon
• Juliette Aurisset, Netflix

Workshop Date

Oct 6, 9.00 – 12.30

Location

Syracuse Room

Web site

http://data.linkedin.com/recsys14-workshop-controlled-experimentation