Thursday Posters

Date: Thursday October 17
Room: Chamber of Commerce

• TORSA Systematic Study on Reproducibility of Reinforcement Learning in Recommendation Systems
  by Emanuele Cavenaghi, Gabriele Sottocornola, Fabio Stella, and Markus Zanker

  Reproducibility is a main principle in science and fundamental to ensure scientific progress. However, many recent works point out that there are widespread deficiencies for this aspect in the AI field, making the reproducibility of results impractical or even impossible. We therefore studied the state of reproducibility support on the topic of Reinforcement Learning & Recommender Systems to analyse the situation in this context. We collected a total of 60 papers and analysed them by defining a set of variables to inspect the most important aspects that enable reproducibility, such as dataset, pre-processing code, hardware specifications, software dependencies, algorithm implementation, algorithm hyperparameters, and experiment code. Furthermore, we used the ACM Badges definitions assigning them to the selected papers. We discovered that, like in many other AI domains, the Reinforcement Learning & Recommender Systems field is grappling with a reproducibility crisis, as none of the selected papers were reproducible when strictly applying the ACM Badges definitions according to our analysis.

• RESAIE: Auction Information Enhanced Framework for CTR Prediction in Online Advertising
  by Yang Yang (Huawei Noah’s Ark Lab), Bo Chen (Huawei Noah’s Ark Lab), Chenxu Zhu (Huawei Noah’s Ark Lab), Menghui Zhu (Huawei Noah’s Ark Lab), Xinyi Dai (Huawei Noah Ark’s Lab), Huifeng Guo (Huawei Noah Ark’s Lab), Muyu Zhang (Huawei Noah Ark’s Lab), Zhenhua Dong (Huawei Noah Ark’s Lab) and Ruiming Tang (Huawei Noah Ark’s Lab)

  Click-Through Rate (CTR) prediction is a fundamental technique for online advertising recommendation and the complex online competitive auction process also brings many difficulties to CTR optimization. Recent studies have shown that introducing posterior auction information contributes to the performance of CTR prediction. However, existing work doesn’t fully capitalize on the benefits of auction information and overlooks the data bias brought by the auction, leading to biased and suboptimal results. To address these limitations, we propose Auction Information Enhanced Framework (AIE) for CTR prediction in online advertising, which delves into the problem of insufficient utilization of auction signals and first reveals the auction bias. Specifically, AIE introduces two pluggable modules, namely Adaptive Market-price Auxiliary Module (AM2) and Bid Calibration Module (BCM), which work collaboratively to excavate the posterior auction signals better and enhance the performance of CTR prediction. Furthermore, the two proposed modules are lightweight, model-agnostic, and friendly to inference latency. Extensive experiments are conducted on a public dataset and an industrial dataset to demonstrate the effectiveness and compatibility of AIE. Besides, a one-month online A/B test in a large-scale advertising platform shows that AIE improves the base model by 5.76% and 2.44% in terms of eCPM and CTR, respectively.

  Full text in ACM Digital Library

• LBRBalancing Habit Repetition and New Activity Exploration: A Longitudinal Micro-Randomized Trial in Physical Activity Recommendations
  by Ine Coppens (WAVES – imec – Ghent University), Toon De Pessemier (WAVES – imec – Ghent University) and Luc Martens (WAVES – imec – Ghent University)

  As repetition of activities can establish habits and exploration of new ones can provide a healthy variety, we investigate how a recommender system for physical activities can optimally balance these two approaches. We conducted an eight-week user study with 62 physically inactive participants who receive personalized repetition and exploration recommendations in a random order. We distinguish between location, workout, and general activities, and collect participants’ subjective perceptions. Our findings indicate that participants initially preferred exploring general activities, but rated repeating recommendations higher after two weeks. By exploring the optimal transition point from exploration to repetition in personalized recommendations, this study contributes to designing more effective recommender systems for health improvement and healthy habit formation.

  Full text in ACM Digital Library

• INDBridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems
  by Nikhil Khani (Google LLC), Li Wei (Google LLC), Aniruddh Nath (Google LLC), Shawn Andrews (Google LLC), Shuo Yang (Google LLC), Yang Liu (Google LLC), Pendo Abbo (Google LLC), Maciej Kula (Google LLC), Jarrod Kahn (Google LLC), Zhe Zhao (University of California), Lichan Hong (Google LLC) and Ed Chi (Google LLC)

  Knowledge Distillation (KD) is a powerful approach for compressing a large model into a smaller, more efficient model, particularly beneficial for latency-sensitive applications like recommender systems. However, current KD research predominantly focuses on Computer Vision (CV) and NLP tasks, overlooking unique data characteristics and challenges inherent to recommender systems. This paper addresses these overlooked challenges, specifically: (1) mitigating data distribution shifts between teacher and student models, (2) efficiently identifying optimal teacher configurations within time and budgetary constraints, and (3) enabling computationally efficient and rapid sharing of teacher labels to support multiple students. We present a robust KD system developed and rigorously evaluated on multiple large-scale personalized video recommendation systems within Google. Our live experiment results demonstrate significant improvements in student model performance while ensuring consistent and reliable generation of high-quality teacher labels from a continuous data stream of data.

  Full text in ACM Digital Library

• RESCan editorial decisions impair journal recommendations? Analysing the impact of journal characteristics on recommendation systems
  by Elias Entrup (TIB Leibniz Information Centre for Science and Technology), Ralph Ewerth (TIB Leibniz Information Centre for Science and Technology) and Anett Hoppe (TIB Leibniz Information Centre for Science and Technology)

  Recommendation services for journals help scientists choose appropriate publication venues for their research results. They often use a semantic matching process to compare e.g. an abstract against already published articles. As these services can guide a researcher’s decision, their fairness and neutrality are critical qualities. However, the impact of journal characteristics (such as the abstract length) on recommendations is understudied. In this paper, we investigate whether editorial journal characteristics can lead to biased rankings from recommendation services, i.e. if editorial choices can systematically lead to a better ranking of one’s own journal. The performed experiments show that longer abstracts or a higher number of articles per journal can boost the rank of a journal in the recommendations. We apply these insights to an active, open-source journal recommendation system. The adaptation of the algorithm leads to an increased accuracy for smaller journals.

  Full text in ACM Digital Library

• TORSCRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems
  by Yucheng Jin, Li Chen, Wanling Cai and Xianglin Zhao

  An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This article presents our proposed unifying framework, CRS-Que, to evaluate the user experience of CRSs. This new evaluation framework is developed based on ResQue, a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of ResQue (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios: music exploration and mobile phone purchase. The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users’ perspectives.

• RESDNS-Rec: Data-aware Neural Architecture Search for Recommender Systems
  by Sheng Zhang (City University of Hong Kong), Maolin Wang (City University of Hong Kong), Xiangyu Zhao (City University of Hong Kong), Ruocheng Guo (ByteDance Research), Yao Zhao (Ant Group) and Chenyi Zhuang (Ant Group),
  Jinjie Gu (Ant Group), Zijian Zhang (Jilin University) and Hongzhi Yin (The University of Queensland)

  In the era of data proliferation, efficiently sifting through vast information to extract meaningful insights has become increasingly crucial. This paper addresses the computational overhead and resource inefficiency prevalent in existing Sequential Recommender Systems (SRSs). We introduce an innovative approach combining pruning methods with advanced model designs. Furthermore, we delve into resource-constrained Neural Architecture Search (NAS), an emerging technique in recommender systems, to optimize models in terms of FLOPs, latency, and energy consumption while maintaining or enhancing accuracy. Our principal contribution is the development of a Data-aware Neural Architecture Search for Recommender System (DNS-Rec). DNS-Rec is specifically designed to tailor compact network architectures for attention-based SRS models, thereby ensuring accuracy retention. It incorporates data-aware gates to enhance the performance of the recommendation network by learning information from historical user-item interactions. Moreover, DNS-Rec employs a dynamic resource constraint strategy, stabilizing the search process and yielding more suitable architectural solutions. We demonstrate the effectiveness of our approach through rigorous experiments conducted on three benchmark datasets, which highlight the superiority of DNS-Rec in SRSs. Our findings set a new standard for future research in efficient and accurate recommendation systems, marking a significant step forward in this rapidly evolving field.

  Full text in ACM Digital Library

• RESDoes It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations
  by Anton Klenitskiy (Sber AI Lab), Anna Volodkevich (Sber AI Lab), Anton Pembek (Lomonosov Moscow State University (MSU)) and Alexey Vasilev (Sber AI Lab)

  Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user’s history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly.

  We apply several methods based on the random shuffling of the user’s sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.

  Full text in ACM Digital Library

• RESEmbedding Optimization for Training Large-scale Deep Learning Recommendation Systems with EMBark
  by Shijie Liu (NVIDIA Corporation), Nan Zheng (NVIDIA Corporation), Hui Kang (NVIDIA Corporation), Xavier Simmons (NVIDIA Corporation), Junjie Zhang (NVIDIA Corporation), Matthias Langer (NVIDIA Corporation), Wenjing Zhu (NVIDIA Corporation), Minseok Lee (NVIDIA Corporation) and Zehuan Wang (NVIDIA Corporation)

  Training large-scale deep learning recommendation models (DLRMs) with embedding tables stretching across multiple GPUs in a cluster presents a unique challenge, demanding the efficient scaling of embedding operations that require substantial memory and network bandwidth within a hierarchical network of GPUs. To tackle this bottleneck, we introduce EMBark—a comprehensive solution aimed at enhancing embedding performance and overall DLRM training throughput at scale. EMBark empowers users to create and customize sharding strategies, and features a highly-automated sharding planner, to accelerate diverse model architectures on different cluster configurations. EMBark groups embedding tables, considering their preferred communication compression method to reduce communication overheads effectively. It embraces efficient data-parallel category distribution, combined with topology-aware hierarchical communication, and pipelining support to maximize the DLRM training throughput. Across four representative DLRM variants (DLRM-DCNv2, T180, T200, and T510), EMBark achieves an average end-to-end training throughput speedup of 1.5 × and up to 1.77 × over traditional table-row-wise sharding approaches.

  Full text in ACM Digital Library

• RESEmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations
  by Chiyu Zhang (University of British Columbia), Yifei Sun (Meta), Minghao Wu (Monash University), Jun Chen (Meta), Jie Lei (Meta), Muhammad Abdul-Mageed (The University of British Columbia), Rong Jin (Meta), Angli Liu (Meta), Ji Zhu (Meta), Sem Park (Meta), Ning Yao (Meta) and Bo Long (Meta)

  Content-based recommendation systems play a crucial role in delivering personalized content to users in the digital world. In this work, we introduce EmbSum, a novel framework that enables offline pre-computations of users and candidate items while capturing the interactions within the user engagement history. By utilizing the pretrained encoder-decoder model and poly-attention layers, EmbSum derives User Poly-Embedding (UPE) and Content Poly-Embedding (CPE) to calculate relevance scores between users and candidate items. EmbSum actively learns the long user engagement histories by generating user-interest summary with supervision from large language model (LLM). The effectiveness of EmbSum is validated on two datasets from different domains, surpassing state-of-the-art (SoTA) methods with higher accuracy and fewer parameters. Additionally, the model’s ability to generate summaries of user interests serves as a valuable by-product, enhancing its usefulness for personalized content recommendations.

  Full text in ACM Digital Library

• INDEnhancing Performance and Scalability of Large-Scale Recommendation Systems with Jagged Flash Attention
  by Rengan Xu (Meta Platforms), Junjie Yang (Meta Platforms), Yifan Xu (Meta Platforms), Hong Li (Meta Platforms), Xing Liu (Meta Platforms), Devashish Shankar (Meta Platforms), Haoci Zhang (Meta Platforms), Meng Liu (Meta Platforms), Boyang Li (Meta Platforms), Yuxi Hu (Meta Platforms), Mingwei Tang (Meta Platforms), Zehua Zhang (Meta Platforms), Tunhou Zhang (Meta Platforms), Dai Li (Meta Platforms), Sijia Chen (Meta Platforms), Gian-Paolo Musumeci (Meta Platforms), Jiaqi Zhai (Meta Platforms), Bill Zhu (Meta Platforms), Hong Yan (Meta Platforms) and Srihari Reddy (Meta Platforms)

  The integration of hardware accelerators has significantly advanced the capabilities of modern recommendation systems, enabling the exploration of complex ranking paradigms previously deemed impractical. However, the GPU-based computational costs present substantial challenges. In this paper, we demonstrate our development of an efficiency-driven approach to explore these paradigms, moving beyond traditional reliance on native PyTorch modules. We address the specific challenges posed by ranking models’ dependence on categorical features, which vary in length and complicate GPU utilization. We introduce Jagged Feature Interaction Kernels, a novel method designed to extract fine-grained insights from long categorical features through efficient handling of dynamically sized tensors. We further enhance the performance of attention mechanisms by integrating Jagged tensors with Flash Attention. Our novel Jagged Flash Attention achieves up to 9 × speedup and 22 × memory reduction compared to dense attention. Notably, it also outperforms dense flash attention, with up to 3 × speedup and 53% more memory efficiency. In production models, we observe 10% QPS improvement and 18% memory savings, enabling us to scale our recommendation systems with longer features and more complex architectures.

  Full text in ACM Digital Library

• RESEnhancing Sequential Music Recommendation with Negative Feedback-informed Contrastive Learning
  by Pavan Seshadri (Georgia Institute of Technology), Shahrzad Shashaani (TU Wien) and Peter Knees (TU Wien)

  Modern music streaming services are heavily based on recommendation engines to serve content to users. Sequential recommendation—continuously providing new items within a single session in a contextually coherent manner—has been an emerging topic in current literature. User feedback—a positive or negative response to the item presented—is used to drive content recommendations by learning user preferences. We extend this idea to session-based recommendation to provide context-coherent music recommendations by modelling negative user feedback, i.e., skips, in the loss function.

  We propose a sequence-aware contrastive sub-task to structure item embeddings in session-based music recommendation, such that true next-positive items (ignoring skipped items) are structured closer in the session embedding space, while skipped tracks are structured farther away from all items in the session. This directly affects item rankings using a K-nearest-neighbors search for next-item recommendations, while also promoting the rank of the true next item. Experiments incorporating this task into SoTA methods for sequential item recommendation show consistent performance gains in terms of next-item hit rate, item ranking, and skip down-ranking on three music recommendation datasets, strongly benefiting from the increasing presence of user feedback.

  Full text in ACM Digital Library

• LBREnhancing Sequential Music Recommendation with Personalized Popularity Awareness
  by Davide Abbattista (Politecnico di Bari), Vito Walter Anelli (Politecnico di Bari), Tommaso Di Noia (Politecnico di Bari), Craig Macdonald (University of Glasgow) and Aleksandr Vladimirovich Petrov (University of Glasgow)

  In the realm of music recommendation, sequential recommender systems have shown promise in capturing the dynamic nature of music consumption. Nevertheless, traditional Transformer-based models, such as SASRec and BERT4Rec, while effective, encounter challenges due to the unique characteristics of music listening habits. In fact, existing models struggle to create a coherent listening experience due to rapidly evolving preferences. Moreover, music consumption is characterized by a prevalence of repeated listening, i.e. users frequently return to their favourite tracks, an important signal that could be framed as individual or personalized popularity. This paper addresses these challenges by introducing a novel approach that incorporates personalized popularity information into sequential recommendation. By combining user-item popularity scores with model-generated scores, our method effectively balances the exploration of new music with the satisfaction of user preferences. Experimental results demonstrate that a Personalized Most Popular recommender, a method solely based on user-specific popularity, outperforms existing state-of-the-art models. Furthermore, augmenting Transformer-based models with personalized popularity awareness yields superior performance, showing improvements ranging from 25.2% to 69.8%. The code for this paper is available at https://github.com/sisinflab/personalized-popularity-awareness.

  Full text in ACM Digital Library

• LBRExploring Coresets for Efficient Training and Consistent Evaluation of Recommender Systems
  by Zheng Ju (University College Dublin), Honghui Du (University College Dublin), Elias Tragos (University College Dublin), Neil Hurley (University College Dublin) and Aonghus Lawlor (University College Dublin)

  Recommender systems have achieved remarkable success in various web applications, such as e-commerce, online advertising, and social media, harnessing the power of big data. To attain optimal model performance, recommender systems are typically trained on very large datasets, with substantial numbers of users and items. However, large datasets often present challenges in terms of processing time and computational resources. Coreset selection offers a method for obtaining a reduced yet representative subset from vast datasets, thereby enhancing the efficiency of training machine learning algorithms. Nevertheless, little research has been conducted to explore the practical implications of different coreset selection approaches on the performance of recommender systems algorithms. In this paper, we systematically investigate the impact of various coreset selection techniques. We evaluate the performance of the resulting coresets using inductive recommendation models which allow for consistent evaluations to be performed. The experimental results demonstrate that coreset methods are a powerful and useful approach for obtaining reduced datasets which preserve the properties of the large original dataset and have competitive performance compared to the time required to train with the full dataset.

  Full text in ACM Digital Library

• TORSExploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives
  by Christine Bauer, Eva Zangerle and Alan Said

  Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this article, we present an overview of research specifically focused on the evaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, normalized Discounted Cumulative Gain, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.

• RESIt’s (not) all about that CTR: A Multi-Stakeholder Perspective on News Recommender Metrics
  by Hanne Vandenbroucke (imec-SMIT Vrije Universiteit Brussel) and Annelien Smets (imec-SMIT Vrije Universiteit Brussel)

  Recommender systems are increasingly used by news media organizations. Existing literature examines various aspects of news recommender systems (NRS) from a computational, user-centric, or normative perspective. Yet research advocates studying the complexities of real-world applications around NRS. Recently, a multi-stakeholder approach to NRS has been adopted, allowing to understand different stakeholder perspectives on NRS development and evaluation within the news organization. However, little research has been done on the different key performance indicators (KPIs) and metrics considered valuable by different stakeholders. Based on 11 interviews with professionals from two commercial news publishers, this paper demonstrates that stakeholders prioritize distinct KPIs and metrics related to the reach-engagement-conversion-retention funnel. The evaluation of NRS performance is often limited to short-term metrics like CTR, overlooking the multiplicity of stakeholders involved. Our findings reveal how different purposes, KPIs, and metrics are valued from the journalistic, commercial, and tech logic. In doing so, this paper contributes to the multi-stakeholder approach to NRS, advancing our understanding of the real-world complexity of NRS development and evaluation.

  Full text in ACM Digital Library

• RESKnowledge-Enhanced Multi-Behaviour Contrastive Learning for Effective Recommendation
  by Zeyuan Meng (University of Glasgow), Zixuan Yi (University of Glasgow) and Iadh Ounis (University of Glasgow)

  Real-world recommendation scenarios usually need to handle diverse user-item interaction behaviours, including page views, adding items into carts, and purchasing activities. The interactions that precede the actual target behaviour (e.g., purchasing an item) allow to capture the user’s preferences from different angles, and are used as auxiliary information (e.g., page views) to enrich the system’s knowledge about the users’ preferences, thereby helping to enhance recommendation for the target behaviour. Despite efforts in modelling the users’ multi-behaviour interaction information, the existing multi-behaviour recommenders still face two challenges: (1) Data sparsity across multiple user behaviours is a common issue that limits the recommendation performance, particularly for the target behaviour, which typically exhibits fewer interactions compared to other auxiliary behaviours; (2) Noisy auxiliary interactive behaviours where the information in the auxiliary behaviours might be non-relevant for recommendation. In this case, a direct adoption of contrastive learning between the target behaviour and the auxiliary behaviours will amplify the noise in the auxiliary behaviours, thereby negatively impacting the real semantics that can be derived from the target behaviour. To address these two challenges, we propose a new model called Knowledge-Enhanced Multi-behaviour Contrastive Learning for Recommendation (KEMCL). In particular, to address the problem of sparse user multi-behaviour interaction information, we leverage a dual-perspective knowledge encoding componentthat enriches the semantic representations of items, and generate supervision signals through self-supervised learning so as to enhance recommendation. In addition, we develop a cross-behaviour learning component, which includes two contrastive learning (CL) methods, inter CL and intra CL, to alleviate the problem of noisy auxiliary interactions. Extensive experiments on three public recommendation datasets show that our proposed KEMCL model significantly outperforms seven existing state-of-the-art methods. In particular, our KEMCL model outperforms the best baseline, namely KMCLR, by 5.42% on the large Tmall dataset.

  Full text in ACM Digital Library

• RESLearned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfaction
  by Yi Wu (Google), Daryl Chang (Google), Jennifer She (Google), Zhe Zhao (Google), Li Wei (Google) and Lukasz Heldt (Google)

  We present the Learned Ranking Function (LRF), a system that takes short-term user-item behavior predictions as input and outputs a slate of recommendations that directly optimizes for long-term user satisfaction. Most previous work is based on optimizing the hyperparameters of a heuristic function. We propose to model the problem directly as a slate optimization problem with the objective of maximizing long-term user satisfaction. We also develop a novel constraint optimization algorithm that stabilizes objective tradeoffs for multi-objective optimization. We evaluate our approach with live experiments and describe its deployment on YouTube.

  Full text in ACM Digital Library

• RESNeighborhood-Based Collaborative Filtering for Conversational Recommendation
  by Zhouhang Xie (University of California San Diego), Junda Wu (University of California San Diego), Hyunsik Jeon (University of California San Diego), Zhankui He (University of California San Diego), Harald Steck (Netflix Inc.), Rahul Jha (Netflix Inc.), Dawen Liang (Netflix Inc.), Nathan Kallus (Cornell University) and Julian Mcauley (University of California San Diego)

  Conversational recommender systems (CRS) should understand users’ expressed interests, which are frequently semantically rich and knowledge-intensive. Prior works attempt to address this challenge by using external knowledge bases or parametric knowledge in large language models (LLMs). In this paper, we study a complementary solution, exploiting item knowledge in the training data. We hypothesize that many inference-time user requests can be answered by reusing popular crowd-written answers associated with similar training queries. Following this intuition, we define a class of neighborhood-based CRS that makes recommendations by identifying items commonly associated with similar training dialogue contexts. Experiments on Inspired, Redial, and Reddit-Movie benchmarks show our method outperforms state-of-the-art LLMs with 2 billion parameters, and offers on-par performance to 7 billion parameter models while using over 170 times less GPU memory. We also show neighborhood and model-based predictions can be combined to achieve further performance improvements.

  Full text in ACM Digital Library

• RESOh, Behave! Country Representation Dynamics Created by Feedback Loops in Music Recommender Systems
  by Oleg Lesota (Johannes Kepler University Linz and Linz Institute of Technology), Jonas Geiger (Johannes Kepler University Linz and Linz Institute of Technology), Max Walder (Johannes Kepler University Linz and Linz Institute of Technology), Dominik Kowald (Know-Center GmbH and TU Graz) and Markus Schedl (Johannes Kepler University Linz and Linz Institute of Technology)

  Recent work suggests that music recommender systems are prone to disproportionally frequent recommendations of music from countries more prominently represented in the training data, notably the US. However, it remains unclear to what extent feedback loops in music recommendation influence the dynamics of such imbalance. In this work, we investigate the dynamics of representation of local (i.e., country-specific) and US-produced music in user profiles and recommendations. To this end, we conduct a feedback loop simulation study using the LFM-2b dataset. The results suggest that most of the investigated recommendation models decrease the proportion of music from local artists in their recommendations. Furthermore, we find that models preserving average proportions of US and local music do not necessarily provide country-calibrated recommendations. We also look into popularity calibration and, surprisingly, find that the most popularity-calibrated model in our study (ItemKNN) provides the least country-calibrated recommendations. In addition, users from less represented countries (e.g., Finland) are, in the long term, most affected by the under-representation of their local music in recommendations.

  Full text in ACM Digital Library

• RESOne-class recommendation systems with the hinge pairwise distance loss and orthogonal representations
  by Ramin Raziperchikolaei (Rakuten Group, Inc.) and Young-joo Chung (Rakuten Group, Inc.)

  In one-class recommendation systems, the goal is to learn a model from a small set of interacted users and items and then identify the positively-related (i.e., similar) user-item pairs among a large number of pairs with unknown interactions. Most loss functions in the literature rely on dissimilar pairs of users and items, which are selected from the ones with unknown interactions, to obtain better prediction performance. The main issue with this strategy is that it needs a large number of dissimilar pairs, which increases the training time significantly. In this paper, our goal is to only use the similar set to train the models and discard the dissimilar set. We highlight three trivial solutions that the recommendation system models converge to when they are trained only on similar pairs: collapsed and dimensional collapsed solutions. We propose a hinge pairwise loss and an orthogonality term that can be added to the objective functions in the literature to avoid these trivial solutions. We conduct experiments on various tasks on public and real-world datasets, which show that our approach using only similar pairs can be trained several times faster than the state-of-the-art methods while achieving competitive results.

  Full text in ACM Digital Library

• INDPrivacy Preserving Conversion Modeling in Data Clean Room
  by Kungang Li (Pinterest), Xiangyi Chen (Pinterest), Ling Leng (Pinterest), Jiajing Xu (Pinterest), Jiankai Sun (Pinterest) and Behnam Rezaei (Pinterest)

  In the realm of online advertising, accurately predicting the conversion rate (CVR) is crucial for enhancing advertising efficiency and user satisfaction. This paper addresses the challenge of CVR prediction while adhering to user privacy preferences and advertiser requirements. Traditional methods face obstacles such as the reluctance of advertisers to share sensitive conversion data and the limitations of model training in secure environments like data clean rooms. We propose a novel model training framework that enables collaborative model training without sharing sample-level gradients with the advertising platform. Our approach introduces several innovative components: (1) utilizing batch-level aggregated gradients instead of sample-level gradients to minimize privacy risks; (2) applying adapter-based parameter-efficient fine-tuning and gradient compression to reduce communication costs; and (3) employing de-biasing techniques to train the model under label differential privacy, thereby maintaining accuracy despite privacy-enhanced label perturbations. Our experimental results, conducted on industrial datasets, demonstrate that our method achieves competitive ROC-AUC performance while significantly decreasing communication overhead and complying with both advertisers’ privacy requirements and user privacy choices. This framework establishes a new standard for privacy-preserving, high-performance CVR prediction in the digital advertising landscape.

  Full text in ACM Digital Library

• INDRanking Across Different Content Types: The Robust Beauty of Multinomial Blending
  by Jan Malte Lichtenberg (Amazon), Giuseppe Di Benedetto (Amazon) and Matteo Ruffini (Albatross AI)

  An increasing number of media streaming services have expanded their offerings to include entities of multiple content types. For instance, audio streaming services that started by offering music only, now also offer podcasts, merchandise items, and videos. Ranking items across different content types into a single slate poses a significant challenge for traditional learning-to-rank (LTR) algorithms due to differing user engagement patterns for different content types. We explore a simple method for cross-content-type ranking, called multinomial blending (MB), which can be used in conjunction with most existing LTR algorithms. We compare MB to existing baselines not only in terms of ranking quality but also from other industry-relevant perspectives such as interpretability, ease-of-use, and stability in dynamic environments with changing user behavior and ranking model retraining. Finally, we report the results of an A/B test from an Amazon Music ranking use-case.

  Full text in ACM Digital Library

• LBRRecommender Systems Algorithm Selection for Ranking Prediction on Implicit Feedback Datasets
  by Lukas Wegmeth (University of Siegen), Tobias Vente (University of Siegen) and Joeran Beel (University of Siegen)

  The recommender systems algorithm selection problem for ranking prediction on implicit feedback datasets is under-explored. Traditional approaches in recommender systems algorithm selection focus predominantly on rating prediction on explicit feedback datasets, leaving a research gap for ranking prediction on implicit feedback datasets. Algorithm selection is a critical challenge for nearly every practitioner in recommender systems. In this work, we take the first steps toward addressing this research gap.

  We evaluate the NDCG@10 of 24 recommender systems algorithms, each with two hyperparameter configurations, on 72 recommender systems datasets. We train four optimized machine-learning meta-models and one automated machine-learning meta-model with three different settings on the resulting meta-dataset.

  Our results show that the predictions of all tested meta-models exhibit a median Spearman correlation ranging from 0.857 to 0.918 with the ground truth. We show that the median Spearman correlation between meta-model predictions and the ground truth increases by an average of 0.124 when the meta-model is optimized to predict the ranking of algorithms instead of their performance. Furthermore, in terms of predicting the best algorithm for an unknown dataset, we demonstrate that the best optimized traditional meta-model, e.g., XGBoost, achieves a recall of 48.6%, outperforming the best tested automated machine learning meta-model, e.g., AutoGluon, which achieves a recall of 47.2%.

  Full text in ACM Digital Library

• RESRecommending Healthy and Sustainable Meals exploiting Food Retrieval and Large Language Models
  by Alessandro Petruzzelli (University of Bari Aldo Moro), Cataldo Musto (University of Bari Aldo Moro), Michele Ciro Di Carlo (University of Bari Aldo Moro), Giovanni Tempesta (University of Bari Aldo Moro) and Giovanni Semeraro (University of Bari Aldo Moro)

  Given the rising global concerns about healthy nutrition and environmental sustainability, individuals need more and more support in making good choices concerning their daily meals. To this end, in this paper we introduce HeaSE, a framework for Healthy And Sustainable Eating. Given an input recipe, HeaSE identifies healthier and more sustainable meals by exploiting retrieval techniques and large language models. The framework works in two steps. First, it uses food retrieval strategies based on macro-nutrient information to identify candidate alternative meals. This ensures that the substitutions maintain a similar nutritional profile. Next, HeaSE employs large language models to re-rank these potential replacements while considering factors beyond just nutrition, such as the recipe’s environmental impact. In the experimental evaluation, we showed the capabilities of LLMs in identifying more sustainable and healthier alternatives within a set of candidate options. This highlights the potential of these models to guide users towards food choices that are both nutritious and environmentally responsible.

  Full text in ACM Digital Library

• RESRecommending Personalised Targeted Training Adjustments for Marathon Runners
  by Ciara Feely (University College Dublin), Brian Caulfield (University College Dublin), Aonghus Lawlor (University College Dublin) and Barry Smyth (University College Dublin)

  Preparing for the marathon involves many weeks of dedicated training. Achieving the right balance between building strength and endurance and the need for rest and recovery is a must, if a runner is to arrive at the start-line injury-free and ready to achieve their desired finish-time. However, because most recreational runners rely on generic training plans, they can struggle to find this balance, which can impact their motivation, health, and performance. In this paper, we describe a novel case-based reasoning approach to fine-tuning a runner’s training by recommending training adjustments based on the patterns of similar runners at corresponding points in their marathon training. The approach is designed to target training adjustments that are based on similar runners but with varying race goals, to allow runners to adjust their training for slower or faster finish-times, as their training progresses and motivations change. We evaluate the recommendations produced using a large-scale real-world dataset according to several factors: (i) the plausibility of the recommended training adjustment, (ii) the effectiveness of the adjustment when it comes to achieving a particular performance goal, and (iii) the safety of the adjustment in terms of the degree of risk that it will lead to an injury or otherwise disrupt training. Our findings suggest that plausible, effective, and safe recommendations can be generated for runners when evaluated against a range of race goals.

  Full text in ACM Digital Library

• RESRight Tool, Right Job: Recommendation for Repeat and Exploration Consumption in Food Delivery
  by Jiayu Li (Tsinghua University), Aixin Sun (Nanyang Technological University), Weizhi Ma (Tsinghua University), Peijie Sun (Tsinghua University) and Min Zhang (Tsinghua University

  From e-commerce to music and news, recommender systems are tailored to specific scenarios. While researching generic models applicable to various scenarios is crucial, studying recommendations based on the unique characteristics of a specific and vital scenario holds both research and, more importantly, practical value.

  In this paper, we focus on store recommendations in the food delivery scenario, which is an intriguing and significant domain with unique behavior patterns and influential factors. First, we offer an in-depth analysis of real-world food delivery data across platforms and countries, revealing that (i) repeat and exploration orders are both noticeable behaviors and (ii) the influences of historical and collaborative situations on repeat and exploration consumption are distinct. Second, based on the observations, we separately design two simple yet effective recommendation models: RepRec for repeat orders and ExpRec for exploration ones. An ensemble module is further proposed to combine recommendations from two models for a unified recommendation list. Finally, experiments are conducted on three datasets spanning three countries across two food delivery platforms. Results demonstrate the superiority of our proposed recommenders on repeat, exploration, and combined recommendation tasks over various baselines. Such simple yet effective approaches will be beneficial for real applications. This work shows that dedicated analyses and methods for domain-specific characteristics are essential for the recommender system studies.

  Full text in ACM Digital Library

• RESRPAF: A Reinforcement Prediction-Allocation Framework for Cache Allocation in Large-Scale Recommender Systems
  by Shuo Su (Kuaishou Technology), Xiaoshuang Chen (Kuaishou Technology), Yao Wang (Kuaishou Technology), Yulin Wu (Kuaishou Technology), Ziqiang Zhang (Tsinghua University), Kaiqiao Zhan (Kuaishou Technology), Ben Wang (Kuaishou Technology) and Kun Gai

  Modern recommender systems are built upon computation-intensive infrastructure, and it is challenging to perform real-time computation for each request, especially in peak periods, due to the limited computational resources. Recommending by user-wise result caches is widely used when the system cannot afford a real-time recommendation. However, it is challenging to allocate real-time and cached recommendations to maximize the users’ overall engagement. This paper shows two key challenges to cache allocation, i.e., the value-strategy dependency and the streaming allocation. Then, we propose a reinforcement prediction-allocation framework (RPAF) to address these issues. RPAF is a reinforcement-learning-based two-stage framework containing prediction and allocation stages. The prediction stage estimates the values of the cache choices considering the value-strategy dependency, and the allocation stage determines the cache choices for each individual request while satisfying the global budget constraint. We show that the challenge of training RPAF includes globality and the strictness of budget constraints, and a relaxed local allocator (RLA) is proposed to address this issue. Moreover, a PoolRank algorithm is used in the allocation stage to deal with the streaming allocation problem. Experiments show that RPAF significantly improves users’ engagement under computational budget constraints.

  Full text in ACM Digital Library

• INDScale-Invariant Learning-to-Rank
  by Alessio Petrozziello (Expedia Group), Christian Sommeregger (Expedia Group) and Ye-Sheen Lim (Expedia Group)

  At Expedia, learning-to-rank (LTR) models plays a key role on our website in sorting and presenting information more relevant to users, such as search filters, property rooms, amenities, and images. A major challenge in deploying these models is ensuring consistent feature scaling between training and production data, as discrepancies can lead to unreliable rankings when deployed. Normalization techniques like feature standardization and batch normalization could address these issues but are impractical in production due to latency impacts and the difficulty of distributed real-time inference. To address consistent feature scaling issue, we introduce a scale-invariant LTR framework which combines a deep and a wide neural network to mathematically guarantee scale-invariance in the model at both training and prediction time. We evaluate our framework in simulated real-world scenarios with injected feature scale issues by perturbing the test set at prediction time, and show that even with inconsistent train-test scaling, using framework achieves better performance than without.

  Full text in ACM Digital Library

• TORSSelfCF: A Simple Framework for Self-supervised Collaborative Filtering
  by Xin Zhou, Aixin Sun, Yong Liu, Jie Zhang and Chunyan Miao

  Collaborative filtering (CF) is widely used to learn informative latent representations of users and items from observed interactions. Existing CF-based methods commonly adopt negative sampling to discriminate different items. That is, observed user-item pairs are treated as positive instances; unobserved pairs are considered as negative instances and are sampled under a defined distribution for training. Training with negative sampling on large datasets is computationally expensive. Further, negative items should be carefully sampled under the defined distribution, in order to avoid selecting an observed positive item in the training dataset. Unavoidably, some negative items sampled from the training dataset could be positive in the test set. Recently, self-supervised learning (SSL), has emerged as a powerful tool to learn a model without negative samples. In this paper, we propose a self-supervised collaborative filtering framework (SelfCF), that is specially designed for recommender scenario with implicit feedback. The proposed SelfCF framework simplifies Siamese networks and can be easily applied to existing deep-learning based CF models, which we refer to as backbone networks. The main idea of SelfCF is to augment the latent embeddings generated by backbone networks instead of the raw input of user/item ids. We propose and study three embedding perturbation techniques that can be applied to different types of backbone networks including both traditional CF models and graph-based models. The framework enables learning informative representations of users and items without negative samples, and is agnostic to the encapsulated backbones. We conduct experimental comparisons on four datasets, one self-supervised framework, and eight baselines to show that our framework may achieve even better recommendation accuracy than the encapsulated supervised counterpart with a 2×–4× faster training speed. The results also demonstrate that SelfCF can boost up the accuracy of a self-supervised framework BUIR by 17.79% on average and shows competitive performance with baselines.

• INDSliding Window Training – Utilizing Historical Recommender Systems Data for Foundation Models
  by Swanand Joshi (Netflix), Yesu Feng (Netflix), Ko-Jen Hsiao (Netflix), Zhe Zhang (Netflix) and Sudarshan Lamkhede (Netflix)

  Long-lived recommender systems (RecSys) often encounter lengthy user-item interaction histories that span many years. To effectively learn long term user preferences, Large RecSys foundation models (FM) need to encode this information in pretraining. Usually, this is done by either generating a long enough sequence length to take all history sequences as input at the cost of large model input dimension or by dropping some parts of the user history to accommodate model size and latency requirements on the production serving side. In this paper, we introduce a sliding window training technique to incorporate long user history sequences during training time without increasing the model input dimension. We show the quantitative & qualitative improvements this technique brings to the RecSys FM in learning user long term preferences. We additionally show that the average quality of items in the catalog learnt in pretraining also improves.

  Full text in ACM Digital Library

• INDTaming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training
  by Yi-Ping Hsu (Pinterest), Po-Wei Wang (Pinterest), Chantat Eksombatchai (Pinterest) and Jiajing Xu (Pinterest)

  ID-based embeddings are widely used in web-scale online recommendation systems. However, their susceptibility to overfitting, particularly due to the long-tail nature of data distributions, often limits training to a single epoch, a phenomenon known as the “one-epoch problem.” This challenge has driven research efforts to optimize performance within the first epoch by enhancing convergence speed or feature sparsity. In this study, we introduce a novel two-stage training strategy that incorporates a pre-training phase using a minimal model with contrastive loss, enabling broader data coverage for the embedding system. Our offline experiments demonstrate that multi-epoch training during the pre-training phase does not lead to overfitting, and the resulting embeddings improve online generalization when fine-tuned for more complex downstream recommendation tasks. We deployed the proposed system in live traffic at Pinterest, achieving significant site-wide engagement gains.

  Full text in ACM Digital Library

• INDTowards Understanding The Gaps of Offline And Online Evaluation Metrics: Impact of Series vs. Movie Recommendations
  by Bora Edizel (Warner Bros. Discovery), Tim Sweetser (StubHub), Ashok Chandrashekar (Warner Bros. Discovery), Kamilia Ahmadi (Warner Bros. Discovery) and Puja Das (Warner Bros. Discovery)

  Full text in ACM Digital Library

• RESUnified Denoising Training for Recommendation
  by Haoyan Chua (Nanyang Technological University), Yingpeng Du (Nanyang Technological University), Zhu Sun (Singapore University of Technology and Design), Ziyan Wang (Nanyang Technological University), Jie Zhang (Nanyang Technological University) and Yew-Soon Ong (Nanyang Technological University)

  Most existing denoising recommendation methods alleviate noisy implicit feedback (user behaviors) through mainly empirical studies. However, such studies may lack theoretical explainability and fail to model comprehensive noise patterns, which hinders the understanding and capturing of different noise patterns that affect users’ behaviors. Thus, we propose to capture comprehensive noise patterns through theoretical and empirical analysis for more effective denoising, where users’ behaviors are divided into willingness and action phases to disentangle independent noise patterns. Willingness refers to the user’s intent to interact with an item, which may not lead to actual interaction due to different factors such as misclicking. Action denotes the user’s actual interaction with an item. Our analysis unveils that (1) in the willingness phase, high uncertainty in the user’s willingness to interact with the item can lead to high expectation loss which aligns with the findings of existing denoising methods; and (2) in the action phase, higher user-specific inconsistency between willingness and action not only leads to more noise in the user’s overall behaviors but also makes it harder to distinguish between true and noisy behaviors. Inspired by these findings, we propose a Unified Denoising Training (UDT) method for recommendation. To alleviate uncertainty in the willingness phase, we lower the importance of the user-item interaction with high willingness uncertainty recognized by high loss. To ease the inconsistency in the action phase, we lower the importance for users with high user-specific inconsistency as it may lead to noisier behaviors. Then, we increase the importance gap between the clean and noisy behaviors for users with low user-specific inconsistency as their behaviors are more distinguishable. Extensive experiments on three real-world datasets show that our proposed UDT outperforms state-of-the-art denoising recommendation methods.

  Full text in ACM Digital Library

• LBRWhat to compare? Towards understanding user sessions on price comparison platforms
  by Ahmadou Wagne (TU Wien) and Julia Neidhardt (TU Wien)

  E-commerce and online shopping have become integral to the lives of many, with various user behavior types historically identified. Beyond deciding what to buy, determining where to make a purchase has led to the importance of price comparison platforms. However, user behavior on these platforms remains underexplored. Furthermore, web analytics often struggle with tracking users over time and deriving meaningful user types from data. This paper addresses these gaps by defining session types through the analysis and clustering of user logs from a major price comparison platform. The study identifies six distinct session clusters: quick peek, major purchase, constraint-based browsing, knowledge seeking, search and browse and heavy browsing. These findings are intended to inform the design and development of a conversational recommender system (CRS). Often, CRS development occurs without adequate consideration of the existing system into which it will be integrated. The study’s findings, derived from both quantitative analysis and expert interviews, provide valuable contributions, including identified session clusters, their interpretation and indicators on which users might benefit from a CRS on these platforms.

  Full text in ACM Digital Library

Back to program