Poster Session O3: Late-breaking Results

Session A: 21:0023:00
Session B: 8:0010:00

  • LBRA College Major Recommendation System
    by Samuel Stein (Fordham University), Gary M. Weiss (Fordham University), Yiwen Chen (Fordham University), Daniel Leeds (Fordham University)

    College students are required to select a major but are often provided with only a modest amount of support in making this important decision. A poor decision is detrimental to the student, since it may result in the student later switching to a different major with a delay in graduation—or even result in the student leaving the university. This also impacts the university since time to graduation and retention rate are used to evaluate the quality of a university. There is a general lack of research on recommender systems for college majors, with the most relevant systems focusing on course-level recommendations. This study describes and evaluates a recommender system for selecting an undergraduate major, utilizing nine years of historical student data from a large university. The system bases its recommendations on the courses that the student takes in the first few years of college, and how well they performed in these courses. The system is designed to recommend majors that the student is likely to be interested in and will perform well in. Recommendations are evaluated based on the likelihood that the student’s actual major was in the top five recommended majors, and whether the student performed above average in that major. The recommendation system dramatically outperforms the baseline strategy of randomly selecting a major, and when the recommendation is followed the student is 12% more likely to perform above average in the major.

  • LBRA Joint Dynamic Ranking System with DNN and Vector-based Clustering Bandit
    by Yu Liu (Peking University), Xiaoxiao Xu (Jingdong), Jincheng Wang (, Yong Li (, Changping Peng (, Yongjun Bao (, Weipeng Yan (

    The ad-ranking module is the core of the advertising recommender system. Existing ad-ranking modules are mainly based on the deep neural network click-through rate prediction model. Recently an innovative ad-ranking paradigm called DNN-MAB has been introduced to address DNN-only paradigms’ weakness in perceiving highly dynamic user intent over time. We introduce the DNN-MAB paradigm into our ad-ranking system to alleviate the Matthew effect that harms the user experience. Due to data sparsity, however, the actual performance of DNN-MAB is lower than expected. In this paper, we propose an innovative ad-ranking paradigm called DNN-VMAB to solve these problems. Based on vectorization and clustering, it utilizes latent collaborative information in user behavior data to find a set of ads with higher relativity and diversity. As an integration of the essences of classical collaborative filtering, deep click-through rate prediction model, and contextual multi-armed bandit, it can improve platform revenue and user experience. Both offline and online experiments show the advantage of our new algorithm over DNN-MAB and some other existing algorithms.

  • LBRClosed-Form Models for Collaborative Filtering with Side-Information
    by Olivier Jeunen (University of Antwerp), Jan Van Balen (University of Antwerp), Bart Goethals (University of Antwerp)

    Recent work has shown that, despite their simplicity, item-based models optimised through ridge regression can attain highly competitive results on collaborative filtering tasks. As these models are analytically computable and thus forgo the need for often expensive iterative optimisation procedures, they are an attractive choice for practitioners. We study the applicability of such closed-form models to implicit-feedback collaborative filtering when additional side-information or metadata about items is available. Two complementary extensions to the easer paradigm are proposed, based on collective and additive models. Through an extensive empirical analysis on several large-scale datasets, we show that our methods can effectively exploit side-information whilst retaining a closed-form solution, and improve upon the state-of-the-art without increasing the computational complexity of the original easer approach. Additionally, empirical results demonstrate that the use of side-information leads to more “long tail” items being recommended, benefiting the recommendations’ coverage of the item catalogue.

  • LBRContext-aware Graph Embedding for Session-based News Recommendation
    by Heng-Shiou Sheu (University of Georgia), Sheng Li (University of Georgia)

    Online news recommender systems aim to make personalized recommendations according to user preferences, which require modeling users’ short-term reading interest. However, due to the limited logged user interactions in practice, news recommendation at session-level becomes very challenging. Existing methods on session-based news recommendation mainly focus on extracting features from news articles and sequential user-item interactions, but they usually ignore the semantic-level structural information among news articles and do not explore external knowledge sources. In this paper, we propose a novel Context-Aware Graph Embedding (CAGE) framework for session-based news recommendation, which builds an auxiliary knowledge graph to enrich the semantic meaning of entities involved in articles, and further refines the article embeddings by graph convolutional networks. Experimental results on a real-world news dataset demonstrate the effectiveness of our method compared with the state-of-the-art methods on session-based news recommendation.

  • LBRDo Channels Matter?: Illuminating Interpersonal Influence on Music Recommendations
    by Hyun Jeong Kim (Seoul National University), So Yeon Park (Stanford University), Minju Park (Seoul National University), Kyogu Lee (Seoul National University)

    Researchers and service providers have focused on leveraging social information acquired from interactions between users to improve the accuracy of system recommendations. However, few have explained the characteristics of music recommendations through interpersonal relationships. To investigate how interpersonal relationships affect users’ evaluation of music recommendation, we conducted a survey-based study that compared two types of recommendation channels—interpersonal (i.e., from friends) and non-interpersonal (i.e., from systems). We found that relevance was evaluated higher in music recommended from non-interpersonal channels on average, while diversity, novelty, and serendipity were higher in interpersonal channels. Non-interpersonal channels surpassed interpersonal channels in terms of convenience, frequency, and adoption rate. These results illustrate that interpersonal and non-interpersonal channels have different strengths and that digital streaming platforms, which have mainly provided system recommendations thus far, need to better support interpersonal channels for richer user experience.

  • LBRDRecPy: A Python Framework for Developing Deep Learning-Based Recommenders
    by Fábio Colaço (University of Lisbon), Márcia Barros (University of Lisbon), Francisco M. Couto (University of Lisbon)

    Frameworks that aid the development of Recommender Systems (RSs) are extremely important, since they reduce their development cost by offering reusable tools, as well as implementations of common strategies and popular models. However, it is still hard to find a framework that also provides full abstraction over data set conversion, support for deep learning-based approaches, extensible models and reproducible evaluations. This work introduces a new framework that not only provides several modules to avoid repetitive development work, but also to assist practitioners with these existing challenges. Our evaluation procedure ensures that RSs developed using this new approach are consistent and extensible, by analysing their predictive performance and certain characteristics of their implementation.

  • LBRInferring the Causal Impact of New Track Releases on Music Recommendation Platforms through Counterfactual Predictions
    by Rishabh Mehrotra (Spotify), Prasanta Bhattacharya (National University of Singapore), Mounia Lalmas (Spotify)

    With over 20,000 tracks being released each day, recommendation systems that power music streaming services should not only be responsive to such large volumes of content, but also be adept at understanding the impact of such new releases on, both, users’ listening behavior and popularity of artists. Inferring the causal impact of new track releases is critical to fully characterizing the interplay between artists and listeners, as well as among the artists. In this study, we infer and quantify causality using a diffusion-regression state-space model that constructs counterfactual outcomes using a set of synthetic controls, which predict potential outcomes in absence of the intervention. Based on large scale experiments spanning over 21 million users and 1 billion streams on a real world streaming platform, our findings suggest that releasing a new track has a positive impact on the popularity of other tracks by the same artist. Interestingly, other related and competing artists also benefit from a new track release, which hints at the presence of a positive platform-effect wherein some artists gain significantly from activities of other artists.

  • LBRInvestigating Listeners’ Responses to Divergent Recommendations
    by Rishabh Mehrotra (Spotify), Chirag Shah (University of Washington), Benjamin Carterette (Spotify)

    Recommender systems offer great opportunity not only for users to discover new content, but also for the providers of that content to find new audience, followers, and fans. Users often come to a recommender system with certain expectations about what it will recommend to them, and a recommender system that is optimized for creating opportunities for content creators may provide recommendations that are very different from what a user is expecting. We hypothesize that some users’ expectations have a much wider range of acceptability than others, and users with more ”receptivity” to subversion of their expectations are likely to accept such divergence in the recommended content. Understanding users’ responses to such recommendations is vital to platforms that need to serve multiple stakeholders. In this work we investigate logged behavioral responses of users of an audio streaming platform to recommendations that deviate from their expectation, or “divergent” recommendations. We present three classes of listener response to divergent recommendations that can be identified in interaction logs with the aim of predicting which users can be targeted for future divergent recommendations. We derive a number of user characteristics based on user’s music consumption which we think are predictive of user’s receptivity, train models to predict receptivity of these users, and run a live A/B test to validate our approach by correlating with engagement.

  • LBRInvestigating the Impact of Audio States in Music Streaming Sessions
    by Aaron Ng (University College London), Rishabh Mehrotra (Spotify)

    Music streaming is inherently sequential in nature, with track sequence information playing a key role in user satisfaction with recommended music. In this work, we investigate the role audio characteristics of music content play in understanding music streaming sessions. Focusing on 18 audio attributes (e.g. dancability, acousticness, energy), we formulate audio transitioning in a session as a multiple changepoint detection problem, and extract latent states of different audio attributes within each session. Based on insights from large scale music streaming data from a popular music streaming platform, we investigate questions around the extent to which audio characteristics fluctuate within streaming sessions, the heterogeneity across different audio attributes and their impact on user satisfaction. Furthermore, we demonstrate the promise of such audio-based characterizing of sessions in better sequencing tracks in a session, and highlight the potential gains in user satisfaction on offer. We discuss implications on the design of track sequencing models, and identify important prediction tasks to further research on the topic.

  • LBRLearning Representations of Hierarchical Slates in Collaborative Filtering
    by Ehtsham Elahi (Netflix), Ashok Chandrashekar (Netflix)

    We are interested in building collaborative filtering models for recommendation systems where users interact with slates instead of individual items. These slates can be hierarchical in nature. The central idea of our approach is to learn low dimensional embeddings of these slates. We present a novel way to learn these embeddings by making use of the (unknown) statistics of the underlying distribution generating the hierarchical data. Our representation learning algorithm can be viewed as a simple composition rule that can be applied recursively in a bottom-up fashion to represent arbitrarily complex hierarchical structures in terms of the representations of its constituent components. We demonstrate our ideas on two real world recommendation systems datasets including the one used for the RecSys 2019 challenge. For that dataset, we improve upon the performance achieved by the winning team’s model by incorporating embeddings as features generated by our approach in their solution.

  • LBRTowards Multi-Language Recipe Personalisation and Recommendation
    by Niall Twomey (Cookpad Ltd), Andrey Ponikar (Cookpad Ltd), Mikhail Fain (Cookpad Ltd), Nadine Sarraf (Cookpad Ltd)

    Multi-language recipe personalisation and recommendation is an under-explored field of information retrieval in academic and production systems. The existing gaps in our current understanding are numerous, even on fundamental questions such as whether consistent and high-quality recipe recommendation can be delivered across languages. Motivated by this need, we consider the multi-language recipe recommendation setting and present grounding results that will help to establish the potential and absolute value of future work in this area. Our work draws on several billion events from millions of recipes, with published recipes and users incorporating several languages, including Arabic, English, Indonesian, Russian, and Spanish. We represent recipes using a combination of normalised ingredients, standardised skills and image embeddings obtained without human intervention. In modelling, we take a classical approach based on optimising an embedded bi-linear user-item metric space towards the interactions that most strongly elicit cooking intent. For users without interaction histories, a bespoke content-based cold-start model that predicts context and recipe affinity is introduced. We show that our approach to personalisation is stable and scales well to new languages. A robust cross-validation campaign is employed and consistently rejects baseline models and representations, strongly favouring those we propose. Our results are presented in a language-oriented (as opposed to model-oriented) fashion to emphasise the language-based goals of this work. We believe that this is the first large-scale work that evaluates the value and potential of multi-language recipe recommendation and personalisation.

  • LBRRecommending in changing times
    by Shruti Kunde (TCS), Mayank Mishra (TCS), Amey Pandit (TCS), Rekha Singhal (TCS), Manoj Karunakaran Nambiar (TCS), Gautam Shroff (TCS), Shashank Gupta (VIT)

    “Recommender systems today face major challenges in keeping up with dynamic customer preferences. Disruptions or sudden changes in the environment affect customer preferences drastically and render historical data ineffective for modeling. With businesses relying heavily on Machine Learning(ML) based recommender systems for catering to customer preferences, the accuracy of timely recommendations gains prime significance.
    To address these challenges, we propose a novel concept, LDT (Labeled Data Threshold), a newly defined parameter to determine the sufficiency of available labeled training data. Our proposed scheme, using LDT leads to a significant reduction ( 50X) in the training time for a model, thus enabling recommender systems to adapt quickly to disruptions. We illustrate the efficacy of our proposed scheme, by conducting extensive experimental analysis on six well known, structured data sets from various public domains.”

  • LBRSmart Targeting: A Relevance-driven and Configurable Targeting Framework for Advertising System
    by Yong Li (, Zihao Zhao (, Zhiwei Fang (, Kui Ma (, Yafei Yao (, Changping Peng (, Yongjun Bao (, Weipeng Yan (

    Targeting system is an essential part of computational advertising. It allows advertisers to select and reach their targeted users. Due to various advertising goals and the demand for making budget plans, advertisers have a strong will to configure the final targeting results, or they can become very cautious in spending money on advertising campaigns. Meanwhile, to guarantee the advertising performance, the targeted users should also be relevant to the ads of the advertisers. Recent targeting methods are mainly based on tags produced by the Data Management Platform (DMP) which is easy for the advertisers to configure the targeting results. However, in such methods, the relevance between the targeted users and ads is not technically evaluated and cannot be guaranteed. The biggest challenge is that it is hard for a machine learning model to both model the relevance and take account of the advertiser’s configuration demands. In this paper, we propose a novel relevance-driven and configurable targeting framework called Smart Targeting to solve the problem. Specifically, different from Tag-wise Targeting, we first use a relevance model to retrieve the most relevant users for the ads. To further enable the advertisers to configure the final results, we develop a Delay Intervention Mechanism to leverage the power of DMP. As far as we know, this is the first attempt of combining relevance modeling and advertiser intervention into a unified targeting system. We implement and evaluate our framework on platform with over 300 million users and the results show that it can bring significant improvements to the core indicators such as CTR and eCPM. The long term monitoring also demonstrates that Smart Targeting gradually becomes the most popular targeting tool after its release.

  • LBRThe Connection Between Popularity Bias, Calibration, and Fairness in Recommendation
    by Himan Abdollahpouri (University of Colorado Boulder), Masoud Mansoury (Eindhoven University of Technology), Robin Burke (University of Colorado Boulder), Bamshad Mobasher (DePaul University)

    Recently there has been a growing interest in fairness-aware recommender systems including fairness in providing consistent performance across different users or groups of users. A recommender system could be considered unfair if the recommendations do not fairly represent the tastes of a certain group of users while other groups receive recommendations that are consistent with their preferences. In this paper, we use a metric called miscalibration for measuring how a recommendation algorithm is responsive to users’ true preferences and we consider how various algorithms may result in different degrees of miscalibration for different users. In particular, we conjecture that popularity bias which is a well-known phenomenon in recommendation is one important factor leading to miscalibration in recommendation. Our experimental results using two real-world datasets show that there is a connection between how different user groups are affected by algorithmic popularity bias and their level of interest in popular items. Moreover, we show that the more a group is affected by the algorithmic popularity bias, the more their recommendations are miscalibrated.

  • LBRTuning Word2vec for Large Scale Recommendation Systems
    by Ben P. Chamberlain (Twitter), Emanuele Rossi (Twitter), Dan Shiebler (Twitter), Suvash Sedhain (Twitter), Michael Bronstein (Twitter)

    Word2vec is a powerful machine learning tool that emerged from Natural Language Processing (NLP) and is now applied in multiple domains, including recommender systems, forecasting, and network analysis. As Word2vec is often used off the shelf, we address the question of whether the default hyperparameters are suitable for recommender systems. The answer is emphatically no. In this paper, we first elucidate the importance of hyperparameter optimization and show that unconstrained optimization yields an average 221% improvement in hit rate over the default parameters. However, unconstrained optimization leads to hyperparameter settings that are very expensive and not feasible for large scale recommendation tasks. To this end, we demonstrate 138% average improvement in hit rate with a runtime budget-constrained hyperparameter optimization. Furthermore, to make hyperparameter optimization applicable for large scale recommendation problems where the target dataset is too large to search over, we investigate generalizing hyperparameters settings from samples. We show that applying constrained hyperparameter optimization using only a 10% sample of the data still yields a 91% average improvement in hit rate over the default parameters when applied to the full datasets. Finally, we apply hyperparameters learned using our method of constrained optimization on a sample to the Who To Follow recommendation service at Twitter and are able to increase follow rates by 15%.

Back to Program

Select timezone:

Current time in :

Diamond Supporter
Platinum Supporters
Gold Supporters
Silver Supporter
Special Supporter