Session: Human Factors
Chair: John Riedl
Date: Tuesday, October 25, 16:15-18:00
- Each to his own: how different users call for different interaction methods in recommender systems
by Bart P. Knijnenburg, Niels J.M. Reijmer, Martijn C. Willemsen
This paper compares five different ways of interacting with an attribute-based recommender system and shows that different types of users prefer different interaction methods. In an online experiment with an energy-saving recommender system the interaction methods are compared in terms of perceived control, understandability, trust in the system, user interface satisfaction, system effectiveness and choice satisfaction. The comparison takes into account several user characteristics, namely domain knowledge, trusting propensity and persistence. The results show that most users (and particularly domain experts) are most satisfied with a hybrid recommender that combines implicit and explicit preference elicitation, but that novices and maximizers seem to benefit more from a non-personalized recommender that just displays the most popular items.
- Rating: how difficult is it?
by E. Isaac Sparling, Shilad Sen
Netflix.com uses star ratings, Digg.com uses up/down votes and Facebook uses a “like” but not a “dislike” button. Despite the popularity and diversity of these rating scales, research offers little guidance for designers choosing between them.
This paper compares four different rating scales: unary (“like it”), binary (thumbs up / thumbs down), five-star, and a 100-point slider. Our analysis draws upon 12,847 movie and product review ratings collected from 348 users through an online survey. We a) measure the time and cognitive load required by each scale, b) study how rating time varies with the rating value assigned by a user, and c) survey users’ satisfaction with each scale.
Overall, users work harder with more granular rating scales, but these effects are moderated by item domain (product reviews or movies). Given a particular scale, users rating times vary significantly for items they like and dislike. Our findings about users’ rating effort and satisfaction suggest guidelines for designers choosing between rating scales.
- A user-centric evaluation framework for recommender systems
by Pearl Pu, Li Chen, Rong Hu
This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users’ point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems’ Quality of user experience), which aimed at measuring the qualities of the recommended items, the system’s usability, usefulness, interface and interaction qualities, users’ satisfaction with the systems, and the influence of these qualities on users’ behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.
RecSys 2011 (Chicago)
Sponsors and Benefactors
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |