Session 9: Sequential Recommendation 2

Date: Wednesday October 16, 12:00 PM – 13:10 PM (GMT+2)
Room: Petruzzelli Theater
Session Chair: Justin Basilico

• RES 🕓15Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs
  by Gleb Mezentsev (Skolkovo Institute of Science and Technology), Danil Gusak (Skolkovo Institute of Science and Technology; HSE University), Ivan Oseledets (Artificial Intelligence Research Institute; Skolkovo Institute of Science and Technology) and Evgeny Frolov (Artificial Intelligence Research Institute; Skolkovo Institute of Science and Technology; HSE University)

  Scalability issue plays a crucial role in productionizing modern recommender systems. Even lightweight architectures may suffer from high computational overload due to intermediate calculations, limiting their practicality in real-world applications. Specifically, applying full Cross-Entropy (CE) loss often yields state-of-the-art performance in terms of recommendations quality. Still, it suffers from excessive GPU memory utilization when dealing with large item catalogs. This paper introduces a novel Scalable Cross-Entropy (SCE) loss function in the sequential learning setup. It approximates the CE loss for datasets with large-size catalogs, enhancing both time efficiency and memory usage without compromising recommendations quality. Unlike traditional negative sampling methods, our approach utilizes a selective GPU-efficient computation strategy, focusing on the most informative elements of the catalog, particularly those most likely to be false positives. This is achieved by approximating the softmax distribution over a subset of the model outputs through the maximum inner product search. Experimental results on multiple datasets demonstrate the effectiveness of SCE in reducing peak memory usage by a factor of up to 100 compared to the alternatives, retaining or even exceeding their metrics values. The proposed approach also opens new perspectives for large-scale developments in different domains, such as large language models.

  Full text in ACM Digital Library

• RES 🕓15Transformers Meet ACT-R: Repeat-Aware and Sequential Listening Session Recommendation
  by Viet-Anh Tran (Deezer Research), Guillaume Salha-Galvan (Deezer Research), Bruno Sguerra (Deezer Research) and Romain Hennequin (Deezer Research)

  Music streaming services often leverage sequential recommender systems to predict the best music to showcase to users based on past sequences of listening sessions. Nonetheless, most sequential recommendation methods ignore or insufficiently account for repetitive behaviors. This is a crucial limitation for music recommendation, as repeatedly listening to the same song over time is a common phenomenon that can even change the way users perceive this song. In this paper, we introduce PISA (Psychology-Informed Session embedding using ACT-R), a session-level sequential recommender system that overcomes this limitation. PISA employs a Transformer architecture learning embedding representations of listening sessions and users using attention mechanisms inspired by Anderson’s ACT-R (Adaptive Control of Thought-Rational), a cognitive architecture modeling human information access and memory dynamics. This approach enables us to capture dynamic and repetitive patterns from user behaviors, allowing us to effectively predict the songs they will listen to in subsequent sessions, whether they are repeated or new ones. We demonstrate the empirical relevance of PISA using both publicly available listening data from Last.fm and proprietary data from Deezer, a global music streaming service, confirming the critical importance of repetition modeling for sequential listening session recommendation. Along with this paper, we publicly release our proprietary dataset to foster future research in this field, as well as the source code of PISA to facilitate its future use.

  Full text in ACM Digital Library

• RES 🕓15Repeated Padding for Sequential Recommendation
  by Yizhou Dang (Northeastern University), Yuting Liu (Northeastern University), Enneng Yang (Northeastern University), Guibing Guo (Northeastern University), Linying Jiang (Northeastern University), Xingwei Wang (Northeastern University) and Jianzhe Zhao (Northeastern University)

  Sequential recommendation aims to provide users with personalized suggestions based on their historical interactions. When training sequential models, padding is a widely adopted technique for two main reasons: 1) The vast majority of models can only handle fixed-length sequences; 2) Batch-based training needs to ensure that the sequences in each batch have the same length. The special value 0 is usually used as the padding content, which does not contain the actual information and is ignored in the model calculations. This common-sense padding strategy leads us to a problem that has never been explored in the recommendation field: Can we utilize this idle input space by padding other content to improve model performance and training efficiency further?

  In this paper, we propose a simple yet effective padding method called Repeated Padding (RepPad). Specifically, we use the original interaction sequences as the padding content and fill it to the padding positions during model training. This operation can be performed a finite number of times or repeated until the input sequences’ length reaches the maximum limit. Our RepPad can be considered as a sequence-level data augmentation strategy. Unlike most existing works, our method contains no trainable parameters or hyperparameters and is a plug-and-play data augmentation operation. Extensive experiments on various categories of sequential models and five real-world datasets demonstrate the effectiveness and efficiency of our approach. The average recommendation performance improvement is up to 60.3% on GRU4Rec and 24.3% on SASRec. We also provide in-depth analysis and explanation of what makes RepPad effective from multiple perspectives. Our datasets and codes are available at https://github.com/KingGugu/RepPad.

  Full text in ACM Digital Library

• RES 🕓15Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Models
  by Yu Cui (Zhejiang University), Feng Liu (OPPO Research Institute), Pengbo Wang (University of Electronic Science and Technology of China), Bohao Wang (Zhejiang University), Heng Tang (Zhejiang University), Yi Wan (OPPO Research Institute), Jun Wang (OPPO Research Institute) and Jiawei Chen (Zhejiang University)

  Owing to their powerful semantic reasoning capabilities, Large Language Models (LLMs) have been effectively utilized as recommenders, achieving impressive performance. However, the high inference latency of LLMs significantly restricts their practical deployment. To address this issue, this work investigates knowledge distillation from cumbersome LLM-based recommendation models to lightweight conventional sequential models. It encounters three challenges: 1) the teacher’s knowledge may not always be reliable; 2) the capacity gap between the teacher and student makes it difficult for the student to assimilate the teacher’s knowledge; 3) divergence in semantic space poses a challenge to distill the knowledge from embeddings.

  To tackle these challenges, this work proposes a novel distillation strategy, DLLM2Rec, specifically tailored for knowledge distillation from LLM-based recommendation models to conventional sequential models. DLLM2Rec comprises: 1) Importance-aware ranking distillation, which filters reliable and student-friendly knowledge by weighting instances according to teacher confidence and student-teacher consistency; 2) Collaborative embedding distillation integrates knowledge from teacher embeddings with collaborative signals mined from the data. Extensive experiments demonstrate the effectiveness of the proposed DLLM2Rec, boosting three typical sequential models with an average improvement of 47.97%, even enabling them to surpass LLM-based recommenders in some cases.

  Full text in ACM Digital Library

• IND 🕓10Short-form Video Needs Long-term Interests: An Industrial Solution for Serving Large User Sequence Models
  by Yuening Li (Google), Diego Uribe (Google), Chuan He (Google), Jiaxi Tang (Google DeepMind), Qingyun Liu (Google DeepMind), Junjie Shan (Google), Ben Most (Google), Kaushik Kalyan (Google), Shuchao Bi (Google), Xinyang Yi (Google DeepMind), Lichan Hong (Google DeepMind), Ed Chi (Google DeepMind) and Liang Liu (Google)

  Sequential models are invaluable for powering personalized recommendation systems. In the context of short-form video (SFV) feeds, where user behavior history is typically longer, systems must be able to understand users’ long-term interests. However, deploying large sequence models to extensive web-scale applications faces challenges due to high serving cost. To address this, we propose an industrial framework designed for efficiently serving large user sequence models. Specifically, the proposed infrastructure decouples serving of the user sequence model and the main recommendation model, with the user sequence model being served offline (asynchronously) with periodical refresh. The proposed infrastructure is also model-agnostic; thus, it can be used to support any type of user sequence models (even LLMs) with controllable costs. Empirical results show that large user models deployed with our framework significantly and consistently enhance the quality of the main recommendation model with minimal serving costs increase.

  Full text in ACM Digital Library

Back to program