We developed a generic framework which enables Customer Relationship Management (CRM) organizations to deploy an automated ranking system for leads (commonly known as ‘lead scoring’). Leads are records that represent non-customers who might become customers. Lead ranking is a fundamental CRM problem with many flavors. Ranking serves as a prioritization management tool for CRM organizations, with many characteristics similar to those of recommender systems.
We present the system with its most recent developments, emphasizing challenges that go beyond the core of the learning algorithm, and that have played an instrumental role in maturing the system into a trustable feature, robust to different types of organizations and datasets. Particularly, we present features which enable Human in the Loop [1], a dominant concept in both configuration and result consumption. Another type of features demonstrates the addition of domain knowledge into the machine learning based process.
We present the concepts of feature selection, with and without human help, prediction explanations, insights on model inputs, data quality issues, training for UX consistency, and actionability for each individual prediction.

Full text in ACM Digital Library

Offline evaluation is a popular approach to determine the best algorithm in terms of the chosen quality metric. However, if the chosen metric calculates something unexpected, this miscommunication can lead to poor decisions and wrong conclusions. In this paper, we thoroughly investigate quality metrics used for recommender systems evaluation. We look at the practical aspect of implementations found in modern RecSys libraries and at the theoretical aspect of definitions in academic papers. We find that Precision is the only metric universally understood among papers and libraries, while other metrics may have different interpretations. Metrics implemented in different libraries sometimes have the same name but measure different things, which leads to different results given the same input. When defining metrics in an academic paper, authors sometimes omit explicit formulations or give references that do not contain explanations either. In 47% of cases, we cannot easily know how the metric is defined because the definition is not clear or absent. These findings highlight yet another difficulty in recommender system evaluation and call for a more detailed description of evaluation protocols.

Full text in ACM Digital Library

Providing suitable recommendations is of vital importance to improve the user satisfaction of music recommender systems. Here, users often listen to the same track repeatedly and appreciate recommendations of the same song multiple times. Thus, accounting for users’ relistening behavior is critical for music recommender systems. In this paper, we describe a psychology-informed approach to model and predict music relistening behavior that is inspired by studies in music psychology, which relate music preferences to human memory. We adopt a well-established psychological theory of human cognition that models the operations of human memory, i.e., Adaptive Control of Thought—Rational (ACT-R). In contrast to prior work, which uses only the base-level component of ACT-R, we utilize five components of ACT-R, i.e., base-level, spreading, partial matching, valuation, and noise, to investigate the effect of five factors on music relistening behavior: (i) recency and frequency of prior exposure to tracks, (ii) co-occurrence of tracks, (iii) the similarity between tracks, (iv) familiarity with tracks, and (v) randomness in behavior. On a dataset of 1.7 million listening events from Last.fm, we evaluate the performance of our approach by sequentially predicting the next track(s) in user sessions. We find that recency and frequency of prior exposure to tracks is an effective predictor of relistening behavior. Besides, considering the co-occurrence of tracks and familiarity with tracks further improves performance in terms of R-precision. We hope that our work inspires future research on the merits of considering cognitive aspects of memory retrieval to model and predict complex user behavior.

Full text in ACM Digital Library

Temporal set prediction is becoming increasingly important as many companies employ recommender systems in their online businesses, e.g., personalized purchase prediction of shopping baskets. While most previous techniques have focused on leveraging a user’s history, the study of combining it with others’ histories remains untapped potential. This paper proposes Global-Local Item Embedding (GLOIE) that learns to utilize the temporal properties of sets across whole users as well as within a user by coining the names as global and local information to distinguish the two temporal patterns. GLOIE uses Variational Autoencoder (VAE) and dynamic graph-based model to capture global and local information and then applies attention to integrate resulting item embeddings. Additionally, we propose to use Tweedie output for the decoder of VAE as it can easily model zero-inflated and long-tailed distribution, which is more suitable for several real-world data distributions than Gaussian or multinomial counterparts. When evaluated on three public benchmarks, our algorithm consistently outperforms previous state-of-the-art methods in most ranking metrics.

Full text in ACM Digital Library

In the recommender systems literature, it has been shown that, in addition to improving system effectiveness, explaining recommendations may increase user satisfaction, trust, persuasion and loyalty. In general, explanations focus on the filtering algorithms or the users and items involved in the generation of recommendations. However, on certain domains that are rich on user-generated textual content, it would be valuable to provide justifications of recommendations according to arguments that are explicit, underlying or related with the data used by the systems, e.g., the reasons for customers’ opinions in reviews of e-commerce sites, and the requests and claims in citizens’ proposals and debates of e-participation platforms. In this context, there is a need and challenging task to automatically extract and exploit the arguments given for and against evaluated items. We thus advocate to focus not only on user preferences and item features, but also on associated arguments. In other words, we propose to not only consider what is said about items, but also why it is said. Hence, arguments would not only be part of the recommendation explanations, but could also be used by the recommendation algorithms themselves. To this end, in this thesis, we propose to use argument mining techniques and tools that allow retrieving and relating argumentative information from textual content, and investigate recommendation methods that exploit that information before, during and after their filtering processes.

Full text in ACM Digital Library

Retail services in corporate international banks are often constrained by the compliance and infrastructure specificities of the countries in which they operate. In addition, the business model and customers interactions through a banking app differ greatly from other major retail services in sectors like digital streaming or e-commerce platforms. We introduce ING’s Bank retail recommender system, elaborating on how we account for the global vs. local requirements, and how can we benefit from such an approach for our model selection as well as products serving.

Full text in ACM Digital Library

Recommender systems have been successfully applied to diverse areas, but their use in the healthcare domain is still rare. One challenge of applying recommender systems to this domain is related to legal concerns about the consequences of provided recommendations. In this work, we advance an expert-in-the-loop, explanation-first approach to tackle this challenge in a specific healthcare niche: gerontological care. A key aspect of the proposed approach is that both recommendations and explanations reflect the structured questionnaire employed by the practitioner to identify patient needs. Another key aspect is that a clinical dataset of patient assessments and respective assigned interventions is used to estimate effects of alternative interventions during the recommendation process. To evaluate the feasibility of this modelling approach, an explanation style was designed with help of practitioners, and a recommendation model was devised and evaluated against a clinical dataset, which was collected by a partner research group working on gerontological primary care. When compared to other traditional recommendation models, the attained precision was competitive across several evaluation conditions. The results suggest that the proposed approach is feasible and may point new ways of adapting recommender systems to play an assistive role in health care.

Full text in ACM Digital Library