Conducting User Experiments in Recommender Systems
by Bart Knijnenburg (UC Irvine)
There is an increasing consensus in the field of recommender systems that we should move beyond the offline evaluation of algorithms towards a more user-centric evaluation approach. Both researchers and practitioners have found that algorithms account for only a small part of the real-world relevance of a recommender system, and other aspects such as the presentation of recommendations and the user interaction with the system have a very significant impact on the user experience.
User experiments are a scientific method to perform user-centric evaluations. They are essential in uncovering how and why the user experience of recommender systems comes about. However, conducting user experiments is a complex endeavor. How does one measure a subjective concept like “user satisfaction”, and how can they be used to make inferences about user experience?
For the intended audience of recommender systems researchers and practitioners who want to get serious about user-centric evaluation, this tutorial covers all aspects involved in conducting user experiments: developing testable hypotheses, sampling participants from the right population, constructing useful experimental manipulations, robustly measuring behavior and subjective valuations, and analyzing the results using state-of-the-art statistical methods. Although the tutorial will start out at the “beginner” level, even seasoned experimenters are likely to learn something from the more advanced topics covered.
About the Speaker
Bart Knijnenburg is a PhD candidate at UC Irvine, where he does research on user experience and privacy in personalized systems. He is a leading advocate of user-centric evaluation in recommender systems: he established the UCERSTI workshop at RecSys2009 and published the first framework for user-centric evaluation in the recommender systems. Bart has taught university courses on cognition, decision making and research methods, and is an expert reviewer on user-centric evaluation and statistical methods for several journals and conferences.
Bart holds Master degrees in Human-Computer Interaction from both Carnegie Mellon University and Eindhoven University of Technology. His research lives at www.usabart.nl.
Personality-based Recommender Systems: An Overview
by Maria Augusta S. N. Nunes (UF Sergipe) and Rong Hu (EPFL)
Over the last 20 decade, recommender systems have obtained great success as an intelligent information system to help deal with the information overload problem, especially in the field of e-commerce. Prior studies on recommender systems mainly consider leveraging user preference information (e.g., user ratings, users’ past behavior), item properties (e.g., price), or user demographic information (e.g., gender). In recent years, other information (e.g., contexts, tags and social information) has also taken into account in the implementation of recommender system. However, few studies have considered addressing the recommendation problem from the angel of users’ psychological characteristics, such as personality. Personality is a critical factor which influences how people make their decisions. People with similar personality characteristics are more likely to have similar interests and preferences. Personality could help us explain why we prefer one option to the other. It is implied that incorporating personality into recommender systems could help understand the reasons that essentially determine user preferences. Currently, some researchers have considered incorporating personality aspects into recommender systems to personalize recommendations and enhance both recommendation quality and user experience. Furthermore, it could be noticed that commercial recommenders have been starting to implement personality-based recommendation engines in their systems. It is an emerging research field. This tutorial will give an overview of personality-based recommender systems and discuss challenges and possible research directions in this topic.
About the Speakers
Maria Augusta S. N. Nunes finished her PhD at LIRMM (Montpellier- France) in 2008. Her thesis was the starting point to the Personality-based Recommender Systems (as described in many papers published in 2011 at ACM, IEEE, UMUAI). Since then her research focus is Affective Computing and how to model and represent the Human Psychological aspects, mainly personality, in computers aiming improve the personalization of information, products and services for humans during their interaction in e-commerce environment, for instance. From 2009 she is an associate professor at DCOMP/Universidade Federal de Sergipe. Her more recent projects include how to extract and store human Personality in order to motivate and personalize services in Recommender System considering mainly the user Psychological aspects. In the last years she wrote many books, book chapters and papers about the use of Affective Computing in order to motivate and personalize information for people. In 2011 shr received 3 awards in projects which considering aspects such as accessibility, Recommendation and Personality Traits.
Rong Hu obtained her PhD from the Human Computer Interaction (HCI) group at Ecole Polytechnique Federale de Lausanne (EPFL) (EPFL) in 2012. Her research interests concentrate on how to incorporate psychological factors (in particular, personality) into recommender systems to improve recommendation quality and user experience. She has published many papers related to personality-based recommender systems at top-tier international conferences, such as IUI, RecSys and UMAP. She is working as a post-doc in the HCI group at EPFL since March 2012.
Building Industrial-scale Real-world Recommender Systems
by Xavier Amatriain (Netflix)
There is an easy way to get up to date on the latest research on Recommender Systems: browse the Recsys conference proceedings for the last few years. But, understanding how this research comes into play in a real-world recommender system is not a trivial task. Take Netflix as an example. Because of our $1M Netflix Prize, people still associate Netflix recommendations with rating prediction. However, ratings predictions is just one of the many components that come into play in our personalization solution. And RMSE is not more than a measure we took as a proxy for user satisfaction with the overall system.
System issues such as scalability or latency have a great impact on perceived quality. And, beyond accuracy, properties of the recommendations such as freshness, novelty, or transparency, will also affect the user’s response to the algorithms. In this tutorial, I will go over some of those practical issues that many times are as important as the theory, if not more, in order to build an industrial-scale real-world recommender system. I will give an overview of the Netflix personalization solution, and the different elements we use in our system. I will also talk about how to involve users in the design process through continuous A/B testing, and how to choose the appropriate model and features. Finally, I will analyze some of the system concerns that we need to bear in mind, and will describe some architectural patterns.
The tutorial is targeting academic researchers that are interested in understanding the practicalities of real-world recommender systems. However, it will also appeal to industry researchers/developers that can compare their approach and learn from a company on the bleeding edge of recommendation technologies.
About the Speaker
Xavier Amatriain is currently managing a team of researchers and engineers creating next generation personalized experiences at Netflix. He is working on the cross-roads of machine learning, software engineering, innovation, and agile methods. Previous to this, he was a researcher focused on Recommender Systems and neighboring areas such as Data Mining, User Modeling, Social Networks, and e-Commerce. He has authored more than 50 papers in books, journals and international conferences, and was General Chair for the 2010 ACM Recommender Systems Conference. Dr. Amatriain has lectured in several universities such as UPF in Barcelona, and UCSB in California.
Best Practices Tutorial and Panel Discussion: The Challenge of Recommender Systems Challenges
by Alan Said (TU Berlin), Domonkos Tikk (Gravity) and Andreas Hotho (U Wurzburg)
Recommender System Challenges such as the Netflix Prize, KDD Cup, etc. have contributed vastly to the development and adoptability of recommender systems. Each year a number of challenges or contests are organized covering different aspects of recommendation. In this tutorial and panel, we present some of the factors involved in successfully organizing a challenge, whether for reasons purely related to research, industrial challenges, or to widen the scope of recommender systems applications.
About the Speakers
Alan Said is a postgraduate researcher at the Competence Center for Information Retrieval and Machine Learning at the Distributed Artificial Intelligence (DAI) Lab of Technische Universität Berlin. He is working in the field of recommender systems, focusing on recommender system evaluation, context-aware and hybrid recommender systems. He has been a co-chair of the Challenges on Context-Aware Movie Recommendation (CAMRa) held in conjunction with ACM RecSys in 2010 and 2011, and the 2012 RecSysChallenge held in conjunction with ACM RecSys 2012.
Domonkos Tikk is the Chief Scientific Officer at Gravity R&D Inc., a recommender solution vendor company. Domonkos obtained his PhD in 2000 in computer science from Budapest University of Technology and Economics. He has been working on machine learning and data and text mining topics in the last decade. His team, Gravity, participated at the Netflix Prize challenge, and was a leader of the The Ensemble team finished tied at the first position of the challenge. The team members founded the company Gravity to exploit the results achieved in Netflix Prize. Domonkos published actively in the field of recommender systems, co-authored about 20 papers in the last years. He also acted as the co-chair of at the recommender system related KDD-Cup 2007, RecsysChallange 2012 and RecSys Doctoral Symposium in 2011.
Andreas Hotho is a professor at the University of Würzburg. He holds a Ph.D. from the University of Karlsruhe, where he worked from 1999 to 2004 at the Institute of Applied Informatics and Formal Description Methods (AIFB) in the areas of text, data, and web mining, semantic web and information retrieval. Andreas Hotho has published over 90 articles in journals and at conferences, co-edited several special issues and books, and co-chaired several workshops, e.g. the Workshop on Recommender Systems and the Social Web in 2011 and 2012 held in conjunction with ACM RecSys and the ECML PKDD Discovery Challenge in 2008 and 2009. He worked as a reviewer for journals and was a member of many international conferences and workshops program committees. His research focuses on the combination of data mining, information retrieval and the semantic web. Further, he is interested in the analysis of social media systems, in particular folksonomies, tagging, and sensor data emerging trough ubiquitous and social activities. As the World Wide Web is one of his main application areas, his research contributes to the field of web science.
