Paper Session 1: Human Interaction

Date: Monday, Aug 28, 2017, 10:30-12:30
Location: Main Room
Chair: Peter Brusilovsky

  • LPEducational Question Routing in Online Student Communities by Jakub Macina, Ivan Srba, Joseph Jay Williams and Maria Bielikova

    Students’ performance in Massive Open Online Courses (MOOCs) is enhanced by high quality discussion forums or recently emerging educational Community Question Answering (CQA) systems. Nevertheless, only a small number of students answer questions asked by their peers. This results in instructor overload, and many unanswered questions. To increase students’ participation, we present an approach for recommendation of new questions to students who are likely to provide answers. Existing approaches to such question routing proposed for non-educational CQA systems tend to rely on a few experts, which is not suitable because we want students to be engaged as it positively influences their learning outcomes. In tackling this novel educational question routing problem, our method (1) goes beyond previous question-answering data as it incorporate additional non-QA data from the course (to improve prediction accuracy and to involve a larger part of community) and (2) applies constraints on users’ workload (to prevent user overloading). We use an ensemble classifier for predicting students’ willingness to answer a question, as well as the students’ expertise for answering. We conducted an online evaluation of the proposed method using an A/B test in our CQA system deployed at an edX MOOC. The proposed method outperformed a baseline method (non-educational question routing enhanced with workload restriction) in recommendation accuracy, involving more community members, and average number of contributions.

  • LPThe Magic Barrier Revisited: Accessing Natural Limitations of Recommender Assessment by Kevin Jasberg and Sergej Sizov

    Recommender systems nowadays have many applications and are of great economic benefit. Hence, it is imperative for a success-oriented company to compare different of such systems and select the better one from them. For this purpose, various metrics of predictive accuracy are commonly used, such as the Root Mean Square Error (RMSE), or precision and recall, just to name a few of them. All these metrics more or less measure how well a recommender system can predict human behaviour.

    Unfortunately, human behaviour is always associated with some degree of uncertainty, making the evaluation of recommender systems difficult, since it is not clear whether a deviation is system-induced or just originates from the natural variability of human decision making. At this point, some Authors speculated that we may be reaching some magic barrier where this variability may prevent us from getting much more accurate.

    In this article, we will extend the existing theory of the Magic Barrier into a new probabilistic but yet pragmatic model. In particular, we will use methods from metrology and physics to develop easy-to-handle quantities for computation to describe the Magic Barrier for different accuracy metrics and provide suggestions for common application. This discussion is substantiated by comprehensive experiments with real users and large-scale simulations on a high performance cluster.

  • LPEffective User Interface Designs to Increase Energy-efficient Behavior in a Rasch-based Energy Recommender System by Alain Starke, Martijn Willemsen and Chris Snijders

    People often struggle to find appropriate energy-saving measures to take in the household. Although recommender studies show that tailoring a system’s interaction method to the domain knowledge of the user can increase energy savings, they did not actually tailor the conservation advice itself. We present two large user studies in which we support users to make an energy-efficient behavioral change by presenting tailored energy-saving advice. Both systems use a one-dimensional, ordinal Rasch scale, which orders 79 energy-saving measures on their behavioral difficulty and link this to a user’s energy-saving ability for tailored advice. We established that recommending Rasch-based advice can reduce a user’s effort, increase system support and, in turn, increase choice satisfaction and lead to the adoption of more energy-saving measures. Moreover, follow-up surveys four weeks later point out that tailoring advice on its feasibility can lead to behavioral change.

  • SPEvaluating Decision-Aware Recommender Systems by Rus M. Mesas and Alejandro Bellogin

    The main goal of Recommender Systems is to suggest relevant items to users, although other utility dimensions — such as diversity, novelty, confidence, possibility of providing explanations — are often considered. In this work, in order to increase the amount of relevant items presented to the user, we analyse how the system could measure the confidence on its own recommendations, so it has the capability of taking decisions about whether an item should be recommended or not. A direct consequence of this design is that the number of suggested items decreases, impacting in some of the beyond-accuracy dimensions (especially, coverage). We present an evaluation of different decision-aware techniques that can be applied to any recommender system or to some families of systems, and explore evaluation metrics that allow to combine more than one evaluation dimension.

  • SPUsing Explainability for Constrained Matrix Factorization by Behnoush Abdollahi and Olfa Nasraoui

    Accurate model-based Collaborative Filtering (CF) approaches tend to be black-box machine learning models, such as Matrix Factorization (MF), that lack interpretability and do not provide a straightforward explanation for their outputs. Yet explanations can improve the transparency of a recommender system by justifying recommendations, and this in turn can enhance the user’s trust in the recommendations. Hence, one main challenge in designing a recommender system is mitigating the trade-off between an explainable technique with moderate prediction accuracy and a more accurate technique with no explainable recommendations. In this paper, we focus on MF and further assume the absence of any additional data source, such as item content or user attributes. We propose an explainability constrained MF technique that computes the top-n recommendation list from items that are explainable. Experimental results show that our method is effective in generating accurate and explainable recommendations.

  • SPUser Preferences for Hybrid Explanations by Pigi Kouki, James Schaffer, Jay Pujara, John O’Donovan and Lise Getoor

    Hybrid recommender systems combine several different sources of information to generate recommendations. These systems have been shown to improve accuracy compared to single-source recommendation strategies. However, hybrid recommendation strategies are inherently more complex than those that use a single source of information, and thus the process of explaining recommendations to consumers becomes more challenging. In this paper we show how to use a hybrid recommender system built on a probabilistic programming language to create hybrid recommendations. We then describe an investigation into user preferences for explanations in this system. We present results of an online user survey that evaluates explanations for hybrid algorithms in a variety of text and visual, graph-based formats.

Back to Program

Diamond Supporter
Platinum Supporters
Gold Supporter
Silver Supporter
Special Supporters