Recent times have seen data analytics software applications become an integral part of the decision-making process of analysts. The users of these software applications generate a vast amount of unstructured log data. These logs contain clues to the user’s goals, which traditional recommender systems may find difficult to model implicitly from the log data. With this assumption, we would like to assist the analytics process of a user through command recommendations. We categorize the commands into software and data categories based on their purpose to fulfill the task at hand. On the premise that the sequence of commands leading up to a data command is a good predictor of the latter, we design, develop, and validate various sequence modeling techniques. In this paper, we propose a framework to provide goal-driven data command recommendations to the user by leveraging unstructured logs. We use the log data of a web-based analytics software to train our neural network models and quantify their performance, in comparison to relevant and competitive baselines. We propose a custom loss function to tailor the recommended data commands according to the goal information provided exogenously. We also propose an evaluation metric that captures the degree of goal orientation of the recommendations. We demonstrate the promise of our approach by evaluating the models with the proposed metric and showcasing the robustness of our models in the case of adversarial examples, where the user activity is misaligned with selected goal, through offline evaluation.

Full text in ACM Digital Library

“In the Netflix user interface (UI), when a row or UI element is named “Because you Watched…”, “More Like This”, or “Because you added to your list”, the overarching goal is to recommend a movie or TV show that a member might like based on the fact that they took a meaningful action on a source item. We have employed similar recommendations in many UI elements: on the homepage as a row of recommendations, after you click into a title, or as a piece of information about why a member should watch a title.
From an algorithmic perspective, there are many ways to define a “successful” similar recommendation. We sought to broaden the definition of success. To this end, the Consumer Insights team recently completed a suite of research projects to explore the intricacies of member perceptions of similar recommendations. The Netflix Consumer Insights team employs qualitative (e.g., in-depth interviews) and quantitative (e.g., surveys) research methods, interfacing directly with Netflix members to uncover pain points that can inspire new product innovation. The research concluded that, while the typical member believes movies are broadly similar when they share a common genre or theme, similarity is more complex, nuanced, and personal than we might have imagined. The vernacular we use in the UI implies that there should be at least some kind of relationship between the source item and the recommendations that follow. Many of our similar recommendations felt “out of place”, mostly because the relationship between the source item and the recommendation was unclear or absent. When similar recommendations tell a completely misleading, incorrect, or confusing story, member trust can be broken.
We will structure the presentation around three new insights that our research found to have an influence on the perception of similarity in the context of Netflix as well as the research methods used to uncover those insights. First, the reason a member loves a given movie will vary. For example, do you want to watch other baseball movies like Field of Dreams, or would you prefer other romances like Field of Dreams? Second, members are more or less flexible about how similar a recommendation actually needs to be depending on the properties of and their interactions with the canvas containing the recommendation. For example, a Because You Watched row on the homepage implies vaguer similarity while a More Like This gallery behind a click into the source item implies stricter similarity. Finally, even when we held the UI element constant, we found that similar recommendations are only valuable in some contexts. After finishing a movie, a member might prefer a similar recommendation one day and a change of pace the next. Research methods discussed will include single-arrangement Inverse Multi-Dimensional Scaling [1], survey experimentation, and ways to apply qualitative research to improve algorithmic recommendations.”

Full text in ACM Digital Library

“Online learning has become the primary source of education in 2020 with the impact of COVID-19 forcing millions of students to stay at home. In order to personalize the learning experience we have built a recommendation system that takes advantage of the (1) Rich content developed at Chegg (2) An excellent knowledge graph that organizes content in a hierarchical fashion (3) Interaction of students across multiple products to enhance user signal in individual products.
Chegg is a centralized learning platform. Students visit Chegg to get help with homework using Chegg Study, learn from flashcards for their tests, practice examinations, learn relevant concepts, work with a tutor online to get one on one learning experience. This represents a large amount of content available to students. In order to organize the content we have developed a Knowledge Graph with nodes representing a hierarchical scheme of concepts taught at different educational institutions and the content at Chegg. In order to create edges between concept nodes and content we build text classifiers to tag content with concept nodes.
Often students will interact with one or two products such as Chegg Study or Text Book rentals and browse other products such as flashcards, exam practice, etc. In order to suggest relevant content to them, we deduce the concepts they have been studying in products where they are more active and suggest content in products where they are less active.
In this presentation I will talk about the general framework for developing personalized recommendations at Chegg and do a deep dive into (1) Text classifiers required for content tagging and (2) Building cross product recommendations. In text classification, I will go into details about working with noisy training data and model improvements using multi-task learning [1]. For cross product recommendations I will talk about combining user signals from multiple products [2] to deduce general pattern of student interest and use that information to retrieve relevant content — for example flashcards or practice exams for users.”

Full text in ACM Digital Library