Session 12: Optimisation and Evaluation 2

Date: Thursday October 17, 09:45 AM – 10:35 AM (GMT+2)
Room: Petruzzelli Theater
Session Chair: Ashmi Banerjee

• RES 🕓5DNS-Rec: Data-aware Neural Architecture Search for Recommender Systems
  by Sheng Zhang (City University of Hong Kong), Maolin Wang (City University of Hong Kong), Xiangyu Zhao (City University of Hong Kong), Ruocheng Guo (ByteDance Research), Yao Zhao (Ant Group) and Chenyi Zhuang (Ant Group),
  Jinjie Gu (Ant Group), Zijian Zhang (Jilin University) and Hongzhi Yin (The University of Queensland)

  In the era of data proliferation, efficiently sifting through vast information to extract meaningful insights has become increasingly crucial. This paper addresses the computational overhead and resource inefficiency prevalent in existing Sequential Recommender Systems (SRSs). We introduce an innovative approach combining pruning methods with advanced model designs. Furthermore, we delve into resource-constrained Neural Architecture Search (NAS), an emerging technique in recommender systems, to optimize models in terms of FLOPs, latency, and energy consumption while maintaining or enhancing accuracy. Our principal contribution is the development of a Data-aware Neural Architecture Search for Recommender System (DNS-Rec). DNS-Rec is specifically designed to tailor compact network architectures for attention-based SRS models, thereby ensuring accuracy retention. It incorporates data-aware gates to enhance the performance of the recommendation network by learning information from historical user-item interactions. Moreover, DNS-Rec employs a dynamic resource constraint strategy, stabilizing the search process and yielding more suitable architectural solutions. We demonstrate the effectiveness of our approach through rigorous experiments conducted on three benchmark datasets, which highlight the superiority of DNS-Rec in SRSs. Our findings set a new standard for future research in efficient and accurate recommendation systems, marking a significant step forward in this rapidly evolving field.

  Full text in ACM Digital Library

• RES 🕓5ConFit: Improving Resume-Job Matching using Data Augmentation and Contrastive Learning
  by Xiao Yu (Columbia University), Jinzhong Zhang (Intellipro Group Inc.) and Zhou Yu (Columbia University)

  A reliable resume-job matching system helps a company find suitable candidates from a pool of resumes, and helps a job seeker find relevant jobs from a list of job posts. However, since job seekers apply only to a few jobs, interaction records in resume-job datasets are sparse. Different from many prior work that use complex modeling techniques, we tackle this sparsity problem using data augmentations and a simple contrastive learning approach. ConFit first formulates resume-job datasets as a sparse bipartite graph, and creates an augmented dataset by paraphrasing specific sections in a resume or a job post. Then, ConFit finetunes pre-trained encoders with contrastive learning to further increase training samples from B pairs per batch to
  O(B2) per batch. We evaluate ConFit on two real-world datasets and find it outperforms prior methods (including BM25 and OpenAI text-ada-002) by up to 19% and 31% absolute in nDCG@10 for ranking jobs and ranking resumes, respectively. We believe ConFit’s simple yet highly performant approach lays a strong foundation for future research in modeling person-job fit

  Full text in ACM Digital Library

• RES 🕓5Unified Denoising Training for Recommendation
  by Haoyan Chua (Nanyang Technological University), Yingpeng Du (Nanyang Technological University), Zhu Sun (Singapore University of Technology and Design), Ziyan Wang (Nanyang Technological University), Jie Zhang (Nanyang Technological University) and Yew-Soon Ong (Nanyang Technological University)

  Most existing denoising recommendation methods alleviate noisy implicit feedback (user behaviors) through mainly empirical studies. However, such studies may lack theoretical explainability and fail to model comprehensive noise patterns, which hinders the understanding and capturing of different noise patterns that affect users’ behaviors. Thus, we propose to capture comprehensive noise patterns through theoretical and empirical analysis for more effective denoising, where users’ behaviors are divided into willingness and action phases to disentangle independent noise patterns. Willingness refers to the user’s intent to interact with an item, which may not lead to actual interaction due to different factors such as misclicking. Action denotes the user’s actual interaction with an item. Our analysis unveils that (1) in the willingness phase, high uncertainty in the user’s willingness to interact with the item can lead to high expectation loss which aligns with the findings of existing denoising methods; and (2) in the action phase, higher user-specific inconsistency between willingness and action not only leads to more noise in the user’s overall behaviors but also makes it harder to distinguish between true and noisy behaviors. Inspired by these findings, we propose a Unified Denoising Training (UDT) method for recommendation. To alleviate uncertainty in the willingness phase, we lower the importance of the user-item interaction with high willingness uncertainty recognized by high loss. To ease the inconsistency in the action phase, we lower the importance for users with high user-specific inconsistency as it may lead to noisier behaviors. Then, we increase the importance gap between the clean and noisy behaviors for users with low user-specific inconsistency as their behaviors are more distinguishable. Extensive experiments on three real-world datasets show that our proposed UDT outperforms state-of-the-art denoising recommendation methods.

  Full text in ACM Digital Library

• RES 🕓5Embedding Optimization for Training Large-scale Deep Learning Recommendation Systems with EMBark
  by Shijie Liu (NVIDIA Corporation), Nan Zheng (NVIDIA Corporation), Hui Kang (NVIDIA Corporation), Xavier Simmons (NVIDIA Corporation), Junjie Zhang (NVIDIA Corporation), Matthias Langer (NVIDIA Corporation), Wenjing Zhu (NVIDIA Corporation), Minseok Lee (NVIDIA Corporation) and Zehuan Wang (NVIDIA Corporation)

  Training large-scale deep learning recommendation models (DLRMs) with embedding tables stretching across multiple GPUs in a cluster presents a unique challenge, demanding the efficient scaling of embedding operations that require substantial memory and network bandwidth within a hierarchical network of GPUs. To tackle this bottleneck, we introduce EMBark—a comprehensive solution aimed at enhancing embedding performance and overall DLRM training throughput at scale. EMBark empowers users to create and customize sharding strategies, and features a highly-automated sharding planner, to accelerate diverse model architectures on different cluster configurations. EMBark groups embedding tables, considering their preferred communication compression method to reduce communication overheads effectively. It embraces efficient data-parallel category distribution, combined with topology-aware hierarchical communication, and pipelining support to maximize the DLRM training throughput. Across four representative DLRM variants (DLRM-DCNv2, T180, T200, and T510), EMBark achieves an average end-to-end training throughput speedup of 1.5 × and up to 1.77 × over traditional table-row-wise sharding approaches.

  Full text in ACM Digital Library

• RES 🕓5AIE: Auction Information Enhanced Framework for CTR Prediction in Online Advertising
  by Yang Yang (Huawei Noah’s Ark Lab), Bo Chen (Huawei Noah’s Ark Lab), Chenxu Zhu (Huawei Noah’s Ark Lab), Menghui Zhu (Huawei Noah’s Ark Lab), Xinyi Dai (Huawei Noah Ark’s Lab), Huifeng Guo (Huawei Noah Ark’s Lab), Muyu Zhang (Huawei Noah Ark’s Lab), Zhenhua Dong (Huawei Noah Ark’s Lab) and Ruiming Tang (Huawei Noah Ark’s Lab)

  Click-Through Rate (CTR) prediction is a fundamental technique for online advertising recommendation and the complex online competitive auction process also brings many difficulties to CTR optimization. Recent studies have shown that introducing posterior auction information contributes to the performance of CTR prediction. However, existing work doesn’t fully capitalize on the benefits of auction information and overlooks the data bias brought by the auction, leading to biased and suboptimal results. To address these limitations, we propose Auction Information Enhanced Framework (AIE) for CTR prediction in online advertising, which delves into the problem of insufficient utilization of auction signals and first reveals the auction bias. Specifically, AIE introduces two pluggable modules, namely Adaptive Market-price Auxiliary Module (AM2) and Bid Calibration Module (BCM), which work collaboratively to excavate the posterior auction signals better and enhance the performance of CTR prediction. Furthermore, the two proposed modules are lightweight, model-agnostic, and friendly to inference latency. Extensive experiments are conducted on a public dataset and an industrial dataset to demonstrate the effectiveness and compatibility of AIE. Besides, a one-month online A/B test in a large-scale advertising platform shows that AIE improves the base model by 5.76% and 2.44% in terms of eCPM and CTR, respectively.

  Full text in ACM Digital Library

• RES 🕓5Right Tool, Right Job: Recommendation for Repeat and Exploration Consumption in Food Delivery
  by Jiayu Li (Tsinghua University), Aixin Sun (Nanyang Technological University), Weizhi Ma (Tsinghua University), Peijie Sun (Tsinghua University) and Min Zhang (Tsinghua University

  From e-commerce to music and news, recommender systems are tailored to specific scenarios. While researching generic models applicable to various scenarios is crucial, studying recommendations based on the unique characteristics of a specific and vital scenario holds both research and, more importantly, practical value.

  In this paper, we focus on store recommendations in the food delivery scenario, which is an intriguing and significant domain with unique behavior patterns and influential factors. First, we offer an in-depth analysis of real-world food delivery data across platforms and countries, revealing that (i) repeat and exploration orders are both noticeable behaviors and (ii) the influences of historical and collaborative situations on repeat and exploration consumption are distinct. Second, based on the observations, we separately design two simple yet effective recommendation models: RepRec for repeat orders and ExpRec for exploration ones. An ensemble module is further proposed to combine recommendations from two models for a unified recommendation list. Finally, experiments are conducted on three datasets spanning three countries across two food delivery platforms. Results demonstrate the superiority of our proposed recommenders on repeat, exploration, and combined recommendation tasks over various baselines. Such simple yet effective approaches will be beneficial for real applications. This work shows that dedicated analyses and methods for domain-specific characteristics are essential for the recommender system studies.

  Full text in ACM Digital Library

• REPR 🕓10Context-based Entity Recommendation for Knowledge Workers: Establishing a Benchmark on Real-life Data
  by Mahta Bakhshizadeh (German Research Center for Artificial Intelligence (DFKI); University of Kaiserslautern-Landau (RPTU)), Heiko Maus (German Research Center for Artificial Intelligence (DFKI)) and Andreas Dengel (German Research Center for Artificial Intelligence (DFKI); University of Kaiserslautern-Landau (RPTU))

  In recent decades, Recommender Systems (RS) have undergone significant advancements, particularly in popular domains like movies, music, and product recommendations. Yet, progress has been notably slower in leveraging these systems for personal information management and knowledge assistance. In addition to challenges that complicate the adoption of RS in this domain (such as privacy concerns, heterogeneous recommendation items, and frequent context switching), a significant barrier to progress in this area has been the absence of a standardized benchmark for researchers to evaluate their approaches. In response to this gap, this paper presents a benchmark built upon a publicly available dataset of Real-Life Knowledge Work in Context (RLKWiC). This benchmark focuses on evaluating context-based entity recommendation, a use case for leveraging RS to support knowledge workers in their daily digital tasks. By providing this benchmark, it is aimed to facilitate and accelerate research efforts in enhancing personal knowledge assistance through RS.

  Full text in ACM Digital Library

• REPR 🕓10Informfully – Research Platform for Reproducible User Studies
  by Lucien Heitz (University of Zurich), Julian Andrea Croci (University of Zurich), Madhav Sachdeva (University of Zurich) and Abraham Bernstein (University of Zurich)

  This paper presents Informfully, a research platform for content distribution and user studies. Informfully allows to push algorithmically curated text, image, audio, and video content to users and automatically generates a detailed log of their consumption history. As such, it serves as an open-source platform for conducting user experiments to investigate the impact of item recommendations on users’ consumption behavior. The platform was designed to accommodate different experiment types through versatility, ease of use, and scalability. It features three core components: 1) a front end for displaying and interacting with recommended items, 2) a back end for researchers to create and maintain user experiments, and 3) a simple JSON-based exchange format for ranked item recommendations to interface with third-party frameworks. We provide a system overview and outline the three core components of the platform. A sample workflow is shown for conducting field studies incorporating multiple user groups, personalizing recommendations, and measuring the effect of algorithms on user engagement. We present evidence for the versatility, ease of use, and scalability of Informfully by showcasing previous studies that used our platform.

  Full text in ACM Digital Library

Back to program