Conducting User Experiments in Recommender Systems

by Bart Knijnenburg (UC Irvine)

There is an increasing consensus in the field of recommender systems that we should move beyond the offline evaluation of algorithms towards a more user-centric evaluation approach. Both researchers and practitioners have found that algorithms account for only a small part of the real-world relevance of a recommender system, and other aspects such as the presentation of recommendations and the user interaction with the system have a very significant impact on the user experience.

User experiments are a scientific method to perform user-centric evaluations. They are essential in uncovering how and why the user experience of recommender systems comes about. However, conducting user experiments is a complex endeavor. How does one measure a subjective concept like “user satisfaction”, and how can they be used to make inferences about user experience?

For the intended audience of recommender systems researchers and practitioners who want to get serious about user-centric evaluation, this tutorial covers all aspects involved in conducting user experiments: developing testable hypotheses, sampling participants from the right population, constructing useful experimental manipulations, robustly measuring behavior and subjective valuations, and analyzing the results using state-of-the-art statistical methods. Although the tutorial will start out at the “beginner” level, even seasoned experimenters are likely to learn something from the more advanced topics covered.

About the Speaker

Bart Knijnenburg is a PhD candidate at UC Irvine, where he does research on user experience and privacy in personalized systems. He is a leading advocate of user-centric evaluation in recommender systems: he established the UCERSTI workshop at RecSys2009 and published the first framework for user-centric evaluation in the recommender systems. Bart has taught university courses on cognition, decision making and research methods, and is an expert reviewer on user-centric evaluation and statistical methods for several journals and conferences.

Bart holds Master degrees in Human-Computer Interaction from both Carnegie Mellon University and Eindhoven University of Technology. His research lives at www.usabart.nl.