Everyday digital tasks can highly benefit from systems that recommend the right information to use at the right time. However, existing solutions typically support only specific applications and tasks. In this demo, we showcase EntityBot, a system that captures context across application boundaries and recommends information entities related to the current task. The user’s digital activity is continuously monitored by capturing all content on the computer screen using optical character recognition. This includes all applications and services being used and specific to individuals’ computer usages such as instant messaging, emailing, web browsing, and word processing. A linear model is then applied to detect the user’s task context to retrieve entities such as applications, documents, contact information, and several keywords determining the task. The system has been evaluated with real-world tasks, demonstrating that the recommendation had an impact on the tasks and led to high user satisfaction.

Full text in ACM Digital Library

We present Grapevine, a user-controlled recommender that enables undergraduate and graduate students to find a suitable research advisor. This system combines the ideas from the areas of exploratory search, user modeling, and recommender systems by employing state-of-the-art knowledge extraction, grape-based recommendation, and an intelligent user interface. In this paper, we demonstrate the system’s key components and how they work as a whole.

Full text in ACM Digital Library

The prevalence of algorithmic recommendations has raised public concern about undesired societal effects. A central threat is the risk of polarization, which is difficult to conceptualize and to measure, making it difficult to assess the role of Recommender Systems in this phenomenon. These difficulties have yielded two types of analyses: 1) purely topological approaches that study how recommenders isolate or connect types of nodes in a graph, and 2) spatial opinion approaches that study how recommenders change the distribution of users on a given opinion scale. The former analyses prove inadequate in settings where users are not classified into categorical types (e.g., in two-party systems with binary social divides), while the latter rely on synthetic data due to the unobservability of opinions. To overcome both difficulties we present the first analysis of friend recommendations acting on real-world sub-graphs of the Twitter network where users are embedded in multidimensional ideological spaces and in which dimensions are indicators of attitudes towards issues in the public debate. We present a polarization metric adapted to these dual topological and spatial states of social network, and use it to track both the evolution of polarization on Twitter networks where the graph evolves following well-known Recommender Systems, and opinions co-evolve following a DeGroot opinion model. We show that different recommendation principles can sometimes drive or mitigate polarization appearing in real social networks.

Full text in ACM Digital Library

Industrial recommender systems are frequently tasked with approximating probabilities for multiple, often closely related, user actions. For example, predicting if a user will click on an advertisement and if they will then purchase the advertised product. The conceptual similarity between these tasks has promoted the use of multi-task learning: a class of algorithms that aim to bring positive inductive transfer from related tasks. Here, we empirically evaluate multi-task learning approaches with neural networks for an online advertising task. Specifically, we consider approximating the probability of post-click conversion events (installs) (CVR) for mobile app advertising on a large-scale advertising platform, using the related click events (CTR) as an auxiliary task. We use an ablation approach to systematically study recent approaches that incorporate both multitask learning and “entire space modeling” which train the CVR on all logged examples rather than learning a conditional likelihood of conversion given clicked. Based on these results we show that several different approaches result in similar levels of positive transfer from the data-abundant CTR task to the CVR task and offer some insight into how the multi-task design choices address the two primary problems affecting the CVR task: data sparsity and data bias. Our findings add to the growing body of evidence suggesting that standard multi-task learning is a sensible approach to modelling related events in real-world large-scale applications and suggest the specific multitask approach can be guided by ease of implementation in an existing system.

Full text in ACM Digital Library

Recommendation systems shape what people consume and experience online, which makes it critical to assess their effect on society and whether they are affected by any potential source of bias. My research focuses on a specific source of bias — popularity — that is especially relevant in two online contexts: news consumption, and cultural markets. Social media newsfeeds, which are designed to optimize the engagement of users, may inadvertently promote inaccurate and partisan news domains, since these are often popular among like-minded and polarized audiences. More generally, in any type of cultural market, popularity bias may lead to the suppression of niche quality products, which may not attract enough attention. In the context of online newsfeeds, I am investigating whether the political diversity of the audience of a news website can be used as signal of journalistic quality. In an analysis of a comprehensive dataset of news source reliability ratings and web browsing histories, I have shown that the diversity of the audience of a news website is a valuable signal to counter popularity bias and to promote journalistic quality. To further validate these results experimentally, I propose to have direct interactions with social media users through surveys. Here, I provide the details of a field study that I am planning to undertake using an experimental social media platform. More generally, in the context of any online cultural market, political audience diversity may not be applicable but the idea of diversity as a signal for higher quality might still be useful. For example, movies which are liked by an “age diverse” population or books which are read by racially diverse audience may have better quality than other items. However, in this research I have explored audience diversity in the context of social media newsfeeds only. For any online cultural market, I propose a method to quantify the popularity bias through an empirical analysis, using data from several existing markets. Accurately estimating the impact of popularity bias can help us advance our understanding of machine biases and also lead to the development of more robust recommendation systems.

Full text in ACM Digital Library

Recommender systems for fashion have gained in popularity in recent years. An exciting novel application for recommender systems is outfit personalisation. This work discusses the problem of personalised outfit recommendations and presents use cases, challenges and opportunities. This is still a nascent application area in many ways and there are opportunities for innovation in generating outfits, personalising them and displaying them in a coherent way to the user.
Outfits are different from showing complimentary items (e.g. printer paper when you buy ink, socks when you buy shoes). Complimentary items are mostly about upselling the user to buy further items with their main purchase. Outfits can be that, but they are also about presenting the main item in different contexts. Showing the same dress in a work outfit, an evening outfit and a casual outfit showcases the use and value the user would get out of purchasing the dress (the main item). In retail words, outfits help sell the main item and are not necessarily about upselling the other items in the outfit as well. Outfits are often classified as complimentary item retrieval, however there are some important aspects of outfits that are closer to the concepts of contextual recommendations and full page optimisation (what things do you show next to each other).
This extended abstract talks about the different use cases of outfits, gives an overview of the challenges in this domain, and presents scoped problem definitions that are the building blocks that need to be addressed in order to solve personalised outfits.

Full text in ACM Digital Library

Offline evaluation has nowadays become a major step in developing Recommendation Systems in both academia and industry [4, 5]. While academia anchors on offline evaluation due to the lack of proper environments for conducting online tests with real users, the industry uses offline evaluation to filter the most promising solutions for further online testing, aiming at reducing costs and potential damage to customers. Despite the blunt advances observed on this topic recently, consolidating a reliable, replicable, flexible and efficient offline evaluation process capable of satisfactorily predicting online test results remains an open challenge [2]. The community still lacks an integrated and updated view on this topic, useful for practitioners to inspect and refine their current offline evaluation stack.
The main Recommendation Systems venues have plenty of studies with relevant findings, presenting new challenges, pitfalls and divergent guidelines for better offline evaluation procedures [3, 5]. However, inspecting all those studies and keeping an updated perspective about where they agree is impractical, especially for the industry, given the need for fast iterations and deliveries. Thus, it is not rare to observe professionals struggle to obtain solid answers to practical and high-impact questions, such as: What are the main existing pitfalls we should be aware of when setting up an offline evaluation in a given domain? What is the desired evaluation framework for a given recommendation task? How reliable is a given offline evaluation stack, and how far is it from an ideal setting?
In this work, we bring an updated snapshot of offline evaluation standards for Recommendation Systems. For this, we reviewed dozens of studies published in the main Recommendation Systems venues in the last five years, dealing with recurring questions related to offline evaluation design and compiling the main findings in the literature. Then, we contrasted this curated body of knowledge against practical issues we face internally at SEEK, aiming to identify the most valuable guidelines. As a result of this process, we propose an integrated evaluation framework for offline stacks, a reliability score to monitor signs of progress on our stack over time, and a list of best practices to bear in mind when starting a new evaluation. Hence, we have organised the work into three parts:
Part I – Integrated Evaluation Framework. We present an offline evaluation framework that compiles the primary directives, pitfalls, and knowledge raised in the last five years by representative studies in the Recommendation Systems literature. This framework aims to compile the main steps, flaws and decisions to be aware of when designing offline tests. Also, it aims to present the leading solutions suggested in the literature for each known issue. The proposed framework can be seen as an extension of Cañamares’ work [1], in which we expand the factors, steps and decisions related to the design of offline experiments for recommenders. Figure 1 depicts the main steps of the framework along with some of the main pitfalls recurrently related to each step. It is noteworthy that this framework should not be deemed as a rigid and thorough set of steps and rules that all professionals must consider in every scenario. It is rather an organized collection of concerns raised in different situations, in which the strength and potential impact of each of them should be carefully inspected through the lens of each evaluation scenario.

Part II – Reliability Score. We also propose a Reliability Score to quantify how close a given offline evaluation setting is from the idealised framework instantiated to a given domain and task. This score is derived from a question-driven process that estimates the current state, effort, and impact that each known issue has for each team or company. These questions represent a non-closed set of concerns related to distinct steps of the evaluation process that should be addressed by a reliable evaluation framework. The final score ranges from 0 to 1 and the higher its value, the more reliable a given offline evaluation setting is, considering the specific needs and perspectives of a team or company. Further, this score allows teams of professionals to monitor progress in their offline evaluation settings over time. The proposed score empowers companies to compare the maturity of different teams w.r.t. offline assessments using a unified view. In order to illustrate the practical utility of the Reliability Score, we also present a few internal use cases that demonstrate how the proposed score helped us at SEEK to identify the main flaws in our offline settings and outline strategies for refining our current evaluation stack.
Part III – Best Practices & Limitations. Finally, we compiled a list of best practices derived from academic works, experience reports from other companies, and our own experience at SEEK. We expect the proposed list to serve as a starting point for practitioners to qualitatively review their decisions when designing offline assessments, as well as that these professionals would contribute to refining and growing it over time.

Full text in ACM Digital Library

Recently, recommendation systems have been proven to be susceptible to malicious perturbations of the model weights. To overcome this vulnerability, Adversarial Regularization emerged as one of the most effective solutions. Interestingly, the technique not only robustifies the model, but also significantly increases its accuracy. To date, unfortunately, the effect of Adversarial Regularization beyond-accuracy evaluation dimensions is unknown. This paper sheds light on these aspects and investigates how Adversarial Regularization impacts the amplification of popularity bias, and the deterioration of novelty and coverage of the recommendation list. The results highlight that, with imbalanced data distribution, Adversarial Regularization amplifies the popularity bias. Moreover, the empirical validation on five datasets confirms that it degrades the diversity and novelty of the generated recommendation. Code and data are available at https://github.com/sisinflab/The-Idiosyncratic-Effects-of-Adversarial-Training.

Full text in ACM Digital Library