- LPFrom the Lab to Production: A Case Study of Session-Based Recommendations in the Home-Improvement Domain
by Pigi Kouki (RelationalAI), Ilias Fountalis (RelationalAI), Nikolas Vasiloglou (RelationalAI), Xiquan Cui (The Home Depot), Edo Liberty (HyperCube), Khalifeh Al Jadda (The Home Depot)
E-commerce applications rely heavily on session-based recommendation algorithms to improve the shopping experience of their customers. Recent progress in session-based recommendation algorithms shows great promise. However, translating that promise to real-world outcomes is a challenging task for several reasons, but mostly due to the large number and varying characteristics of the available models. In this paper, we discuss the approach and lessons learned from the process of identifying and deploying a successful session-based recommendation algorithm for a leading e-commerce application in the home-improvement domain. To this end, we initially evaluate fourteen session-based recommendation algorithms in an offline setting using eight different popular evaluation metrics on three datasets. The results indicate that offline evaluation does not provide enough insight to make an informed decision since there is no clear winning method on all metrics. Additionally, we observe that standard offline evaluation metrics fall short for this application. Specifically, they reward an algorithm only when it predicts the exact same item that the user clicked next or eventually purchased. In a practical scenario, however, there are near-identical products which, although they are assigned different identifiers, they should be considered as equally-good recommendations. To overcome these limitations, we perform an additional round of evaluation, where human experts provide both objective and subjective feedback for the recommendations of five algorithms that performed the best in the offline evaluation. We find that the experts’ opinion is oftentimes different from the offline evaluation results. Analysis of the feedback confirms that the performance of all models is significantly higher when we evaluate near-identical product recommendations as relevant. Finally, we run an A/B test with one of the models that performed the best in the human evaluation phase. The treatment model increased conversion rate by 15.6% and revenue per visit by 18.5% when compared with a leading third-party solution.
Full text in ACM Digital Library
|
- LPRecommending the Video to Watch Next: An Offline and Online Evaluation at YOUTV.de
by Panagiotis Symeonidis (Free University of Bozen-Bolzano), Andrea Janes (Free University of Bozen-Bolzano), Dmitry Chaltsev (Free University of Bozen-Bolzano), Philip Giuliani (Keep in Mind), Daniel Morandini (Keep in Mind), Andreas Unterhuber (Keep in Mind), Ludovik Coba (Free University of Bozen-Bolzano), Markus Zanker (Free University of Bozen-Bolzano)
The task “recommend a video to watch next?” has been in the focus of recommender systems’ research for a long time. However, adequately exploiting the clues hidden in the sequences of actions of user sessions in order to reveal users’ short-term intentions moved only recently into the focus of research. Based on a real-world application scenario, in this paper, we propose a Markov Chain-based transition probability matrix to efficiently reveal the short-term preferences of individuals. We experimentally evaluated our proposed method by comparing it against state-of-the-art algorithms in an offline as well as a live evaluation setting. In both cases our method not only demonstrated its superiority over its competitors, but exposed a clearly stronger engagement of users on the platform. In the online setting, our method improved the click-through rate by up to 93.61%. This paper therefore contributes real-world evidence for improving the recommendation effectiveness, by considering sequence-awareness, since capturing the short-term preferences of users is crucial in the light of items with a short life span such as tv programs (news, tv shows, etc.).
Full text in ACM Digital Library
|
|
|
- LPIn-Store Augmented Reality-Enabled Product Comparison and Recommendation
by Jesús Omar Álvarez Márquez (University of Duisburg-Essen), Jürgen Ziegler (University of Duisburg-Essen)
We present an approach combining the AR-based presentation of product attributes in a physical retail store with recommendations for items only available online. The system supports users’ decision-making process by offering functions for comparing product features between items, both physical and online, and by providing recommendations based on selecting in-store products. The physical products may thus serve as anchors for forming the user’s preferences, also offering a richer and more engaging experience when exploring the products hands-on. Both objective product attributes as well as the visual appearance of a physical product are employed for generating recommendations from the online space. In this way, the advantages of online and in-store shopping can be combined, creating novel multi-channel opportunities for businesses. An empirical evaluation showed that the comparison and recommendation functions were appreciated by users, and hinted some possible benefits of a hybrid physical-online shopping support system. Despite the limitations of the study, there is sufficient evidence to consider this a viable approach worth to be further explored.
Full text in ACM Digital Library
|
- INBalancing Relevance and Discovery to Inspire Customers in the IKEA App
by Balázs Tóth (IKEA Group), Sandhya Sachidanandan (IKEA Group), Emil S. Jørgensen (IKEA Group)
“IKEA stores are designed to engage and inspire customers as they make their way from the entrance to check-out. To enable a great experience for every individual, our home-furnishing experts strive to provide the best possible input and advice, catering to individual needs and helping customers realise their vision for life at home.
As we move rapidly towards a multi touch point shopping vision with digital at heart, data and machine learning are crucial to 1) enable a consistent journey across, and 2) extend the domain expertise found in stores to our purely digital touch points, like the new IKEA App. A specific example of a digital product where we aim to inspire is our newly launched Inspirational Feed. In the Feed, customers browse their way through ”shoppable” images, in which our products are displayed in purposefully atmospheric settings for a wide range of room types. In this talk, we present our newest algorithmic approach to personalise the Feed based on contextual bandits [2, 3, 4], and highlight some of the technical challenges regarding implementation at scale.
The past decade has led to an explosion in research on recommender systems, often with personalised content as a motivating factor [1]. To identify a suitable class of algorithmic strategies for the Feed, note that:
the Feed needs to be relevant and inspire; many of our existing panels on IKEA.com or in the App can be resolved using classical recommendation strategies, as they focus primarily on relevance. The Feed demands a higher degree of discovery with an element of surprise.
the Feed is a new product; hence, data scarcity is a central challenge that we need to actively address. In particular, we want to avoid ending up with sub-optimal recommendation policies that could originate from the selection bias of supervised models fitted to historical behaviour data.
To capture the described trade-off between relevance and discovery, as well as matching data for how customers on average shop with IKEA, contextual bandits with batch learning from logged bandit feedback inspired by [4] have shown especially promising results among numerous other exploitation-exploration strategies. This approach induces a fair amount of exploration and minimizes the risk of showing irrelevant images to the customer based on the principle of counterfactual risk minimisation. In our contextual bandit setup, a bandit session is defined as a user session of the Feed taking place in the App (the home of the Feed). An action is displaying an image, while a reward is generated whenever the user clicks on it. The bandit context consists of previous user behaviour collected from the App, combined with product metadata. Our model takes advantage of embeddings of the input features, similar to [1].
The implementation of the strategy requires us to handle several interesting edge cases, e.g. we are looking into various techniques for increasing exploration as new images are added to our dynamic content catalogue.”
Full text in ACM Digital Library
|
- INOn the Heterogeneous Information Needs in the Job Domain: A Unified Platform for Student Career
by Markus Reiter-Haas (Talto GmbH), David Wittenbrink (Talto GmbH), Emanuel Lacic (Know-Center GmbH)
“Finding the right job is a difficult task for anyone as it usually depends on many factors like salary, job description, or geographical location. Students with almost no prior experience, especially, have a hard time on the job market, which is very competitive in nature. Additionally, students often suffer a lack of orientation, as they do not know what kind of job is suitable for their education. At Talto1, we realized this and have built a platform to help Austrian university students with finding their career paths as well as providing them with content that is relevant to their career possibilities. This is mainly achieved by guiding the students toward different types of entities that are related to their career, i.e., job postings, company profiles, and career-related articles.
In this talk, we share our experiences with solving the recommendation problem for university students. One trait of the student-focused job domain is that behaviour of the students differs depending on their study progression. At the beginning of their studies, they need study-specific career information and part-time jobs to earn additional money. Whereas, when they are nearing graduation, they require information about their potential future employers and entry-level full-time jobs. Moreover, we can observe seasonal patterns in user activity in addition to the need of handling both logged-in and anonymous session users at the same time.
To cope with the requirements of the job domain, we built hybrid models based on a microservice architecture that utilizes popular algorithms from the literature such as Collaborative Filtering, Content-based Filtering as well as various neural embedding approaches (e.g., Doc2Vec, Autoencoders, etc.). We further adapted our architecture to calculate relevant recommendations in real-time (i.e., after a recommendation is requested) as individual user sessions in Talto are usually short-lived and context-dependent. Here we found that the online performance of the utilized approach also depends on the location context [1]. Hence, the current location of a user on the mobile or web application impacts the expected recommendations.
One optimization criterion on the Talto career platform is to provide relevant cross-entity recommendations as well as explain why those were shown. Recently, we started to tackle this by learning embeddings of entities that lie in the same embedding space [2]. Specifically, we pre-train word embeddings and link different entities by shared concepts, which we use for training the network embeddings. This embeds both the concepts and the entities into a common vector space, where the common vector space is a result of considering the textual content, as well as the network information (i.e., links to concepts). This way, different entity types (e.g., job postings, company profiles, and articles) are directly comparable and are suited for a real-time recommendation setting. Interestingly enough, with such an approach we also end up with individual words sharing the same embedding space. This, in turn, can be leveraged to enhance the textual search functionality of a platform, which is most commonly based just on a TF-IDF model.
Furthermore, we found that such embeddings allow us to tackle the problem of explainability in an algorithm-agnostic way. Since the Talto platform utilizes various recommendation algorithms as well as continuously conducts AB tests, an algorithm-agnostic explainability model would be best suited to provide the students with meaningful explanations. As such, we will also go into the details on how we can adapt our explanation model to not rely on the utilized recommendation algorithm.”
Full text in ACM Digital Library
|
