Discovering Things We Truly Love

Update: I’ve now moved my blog to my new company’s website. Please join me there to continue the conversation!

Each day, we face countless decisions about how to live our lives: Where should I eat? Which movie should I watch? Which sunglasses should I buy? Which hotel should I stay in? Which podcast should I listen to? The list is almost endless. Sometimes, we seek something familiar (such as watching our favourite movie for the hundredth time). But often, we’re looking for something new. In these cases, we face a discovery problem: How do I discover something that brings me satisfaction? Or, even better: How do I discover something I truly love?

Before the internet era, most of our discovery problems involved only a handful of options, so simple approaches such as “research all the options” or “ask peers for recommendations” usually provided reasonably good outcomes, reasonably quickly. Fast-forward to today, and our discovery problems typically involve thousands or even millions of options. In theory, this greater level of choice is wonderful. In practice, however, it’s paralyzing. Research from experimental psychology indicates that when we’re faced with so many options, we feel anxiety while choosing and we experience lower satisfaction from any eventual choice. In short, we face what psychologist Barry Schwartz calls “The Paradox of Choice”: We have access to more options than ever before, but this greater level of choice actually causes us to experience worse outcomes.

To help address this problem, many large online platforms (including content platforms, such as Netflix, and e-commerce platforms, such as Amazon) implement recommender systems, which use AI techniques to generate personalized recommendations. Although these systems play a major role in how many of us approach our discovery problems, they also entail important drawbacks. As a result, they typically lead us to discover things we find vaguely OK, not things we truly love.

In this post, I’ll describe these drawbacks in detail. I’ll also highlight how the dominant paradigms of the internet make these challenges very difficult for existing recommender systems to overcome. I’ll argue that to make significant progress, we’ll require a new approach, based on new foundations. Motivated by this analysis, I’ll conclude by throwing my hat into the ring. I’m building a new company to tackle this age-old problem: To help people discover things they truly love.

The Problem

“The ‘success’ of modernity turns out to be bittersweet, and everywhere we look it appears that a significant contributing factor is the overabundance of choice. Having too many choices produces psychological distress, especially when combined with regret, concern about status, adaptation, social comparison, and perhaps most important, the desire to have the best of everything — to maximize.” — Barry Schwartz, “The Paradox of Choice”

Let’s start by considering some concrete examples of discovery problems:

• Paul would like to buy a book to read on his upcoming long-haul flight. He mainly enjoys fiction, and he prefers recent books to older classics. His favourite book is “Life of Pi” by Yann Martel. He would like a book that will keep him entertained for the full duration of his flight, which is about 7 hours.
• Saira and Naomi would like to watch a movie together this evening. They subscribe to Netflix, NowTV and Amazon Prime Video. Saira enjoys psychological thrillers, and her favourite movie is “The Silence of the Lambs”. Naomi likes European cinema, including movies with subtitles, but not those that are dubbed. Her favourite movie is “Amélie”. Naomi would like to watch something that’s no longer than 90 minutes.
• Foodie is a London-based start-up with 4 employees. Each month, the employees get together for an after-work social event in a restaurant. Each person has their own preferences regarding cuisine, and also their own dietary requirements. They all live in different parts of London, and they’d like to choose a restaurant that’s roughly equal-distance between all of them. The maximum budget is £150.

Although the details are quite different in each case, these examples — and many more — can all be represented by the same job story:

When faced with a discovery problem, people want to choose the option that best aligns with their personal preferences, so they can experience the greatest possible satisfaction.

Discovery problems are as old as civilization. In recent years, however, the ever-growing role of the internet has suddenly made these problems much more difficult. The reason is simple: In the internet era, we are faced with an overwhelming number of options to choose from. In the mood for some music? Spotify has more than 70 million tracks on offer. Or do you prefer reading? Amazon lists more than 20 million paperbacks. Or maybe you’d like to book your next vacation? There are more than 7 million properties listed on AirBNB (not to mention the 325,000 on Hotels.com).

It’s worth taking a moment to reflect on these numbers. This level of choice is truly staggering. Research from the field of psychology illustrates that we struggle to solve discovery problems that involve even just a handful of possible options, so how on earth are we supposed to navigate this level of choice in our lives? When faced with these situations, we feel considerable anxiety, which causes us to experience lower satisfaction from any eventual choice — or simply to choose nothing at all.

Sounds like we’re going to need some help.

Existing Approaches to Discovery Problems

“There is lots of compelling evidence that existing tech-based approaches to this problem bring big benefits for companies like Netflix, Spotify, and so on. But it’s hiding in plain sight that those same approaches aren’t doing a very good job for consumers. They are doing an OK job at helping us find stuff that’s OK — but not more. For example, I rarely discover anything truly outstanding via a recommender system. To find the really great stuff, I have to use old-school techniques like asking my friends or reading reviews on websites like Pitchfork.” Ian Hogarth, Entrepreneur & Investor

Today, there are three main approaches to solving discovery problems: Conducting research, asking peers for recommendations and using a recommender system.

Conducting Research

The first approach to solving discovery problems is to conduct research about the possible options, then to use a decision-rule or heuristic to make a choice, based on this research. If the number of options is small, it’s feasible to conduct research for each one, then to choose the option that looks the best. This strategy is called maximizing. If the number of options is large, it’s often necessary to use a heuristic that Nobel Prize-winning economist and psychologist Herbert A Simon called satisficing: Rather than researching all possible options, we instead research the options one-by-one, and we choose the first option that looks “good enough”.

For anyone who’s spent hours scrolling through countless possible options on a menu screen, the challenges of implementing this strategy will be painfully familiar. Conducting research is a labour-intensive process, and evaluating even a single option requires time and mental energy. If we use multiple research sources, we also need to make subjective judgments about the quality of information that we receive from each one. When painted in this light, conducting research doesn’t really solve a discovery problem, but rather remixes it: Instead of asking “which option should I choose?”, the problem instead becomes “which sources should I trust?”. Solving this problem may be harder than solving the discovery problem itself!

Asking Peers for Recommendations

The second approach to solving discovery problems is to ask our peers for recommendations. Put simply, when faced with a discovery problem in a specific domain (e.g., “which book should I read to learn more about object-oriented programming?”), we can make a list of our peers with relevant domain knowledge, then ask those people for advice on which option we should choose. Although typically easier and faster than conducting research, this approach also entails considerable challenges:

• First, we might not have access to relevant peers to ask. If we don’t know anyone who is a professional software developer, then who should we ask for recommendations about object-oriented programming?
• Second, even if we do have some peers to ask, it’s not clear whether those people are skilled at solving the specific problem we face. For example, they may not have sufficient knowledge of the set of all possible options, or they may not have a clear sense of our preferences (which may be very different from their preferences).
• Third, if we ask several different peers for recommendations, it’s likely that each person will provide different recommendations. How do we reconcile these different opinions into a single choice?

In essence, asking peers for recommendations amounts to deferring our choice to other people. If we have access to relevant peers, all of whom have a deep understanding of our personal preferences, and all of whom agree with each other, then this approach can yield great results. But how often does that happen in practice? It still feels like there’s a lot of room for improvement. Maybe technology has the answer?

Using a Recommender System

The third approach to solving discovery problems is to use a recommender system. A recommender system is a digital product that uses AI to generate personalized recommendations. Over the past 20 years, recommender systems have come to play a central role on many large online platforms. Some recommender systems run prominently in the foreground of the user experience (UX), and operate as core features or even brands (such as Spotify’s much-loved Discover Weekly). Other recommender systems run quietly in the background, and manifest by changing small elements of the UX, such as reordering the items on a page.

The core mechanics of individual recommender systems also vary considerably across different platforms. Some recommender systems seek to understand what we’re looking for, then attempt to assess how well the possible options meet our preferences. These approaches are called knowledge-based or content-based recommender systems. For example, if I’m looking for an action movie that’s similar to Crouching Tiger, Hidden Dragon, I might use a knowledge-based recommender system to help me find one. Other recommender systems seek to uncover patterns in the choices made by many different users. These approaches are called collaborative-filtering recommender systems. For example, a collaborative filtering system might seek to identify a set of other users whose tastes are similar to mine, then recommend me things that are popular among those other people.

Knowledge-based and content-based recommender systems are essentially digital incarnations of the traditional “conducting research” approach. Similarly, collaborative-filtering recommender systems are essentially digital incarnations of the traditional “asking peers for recommendations” approach (albeit with “peers” that we don’t necessarily know). Thanks to the power of the internet, recommender systems can conduct huge amounts of research in the blink of an eye. They can also identify relevant peers whose tastes closely match ours, and use mathematical tools to reconcile differences of opinion. Therefore, recommender systems can overcome many of the drawbacks associated with the traditional approaches to solving discovery problems.

In theory, recommender systems sound like the perfect solution to our discovery problems. But something doesn’t quite add up: If that’s the case, why are we still having this conversation? In practice, most people find that recommender systems do an OK job of finding things that are OK, but that they rarely help us discover things we truly love. What’s holding them back? I argue that the key challenge boils down to five interrelated problems.

• First, most recommender systems are not actually designed to solve the discovery problems that we face as consumers. To see why this is the case, let’s assume that Nikhil is shopping on Amazon for a halloween costume. Amazon implements many different recommender systems to provide Nikhil with recommendations. From Nikhil’s perspective, the objective is clear: Choose a great option that satisfies my preferences. Let’s call this Nikhil’s discovery objective. Importantly, this is not the same objective that’s being pursued by the recommender systems. Instead, those systems pursue (and, crucially, have been trained on) objectives such as “maximize the probability that the customer makes a purchase” or “maximize the lifetime value of the customer”. Let’s call these supply-side objectives. Supply-side objectives are not entirely unrelated to Nikhil’s discovery objective, because supply-side platforms know that Nikhil won’t buy things he hates. But they aren’t perfectly aligned either. As a simple example: If there are multiple different products that could do the job equally well, then Nikhil is incentivized to buy the cheapest product, whereas suppliers are incentivized to sell him the most expensive product.
• Second, most existing recommender systems do not allow us to state our preferences explicitly. Instead, they seek to run “automagically” or “in the background”, so as not to disrupt the core UX on the supply-side platforms that they inhabit. Such platforms understandably optimize their UX to best serve their supply-side objectives — which, again, don’t necessarily align with our discovery objectives. For example, if Nina is looking for movie recommendations, she might open Amazon Prime Video and browse the recommendations on the home-page. These recommendations are optimized to keep Nina engaged on the platform, but they don’t allow her much freedom to state her preferences explicitly. As a result, she ends up watching something she finds vaguely OK, but not something she truly loves.
• Third, most existing recommender systems do not have access to any data outside of their own supply-side platform. For example, if Lee watches lots of true-crime documentaries on Netflix, then it stands to reason that he might also enjoy listening to true-crime podcasts on Spotify and buying true-crime books on Amazon. But because each of those platforms processes data in a silo, Lee’s data from Netflix has no impact on the recommendations he receives from Spotify or Amazon. [Important note: I strongly believe that it’s vital for everyone to have control over where and how their data is used (much, much more on this in my subsequent posts!). I’m certainly not suggesting a data-sharing free-for-all. I’m simply highlighting that walled-garden ecosystems produce suboptimal outcomes for discovery problems, and I’m highlighting that there could be other ways to approach this, while still preserving people’s privacy.]
• Fourth, most existing recommender systems make their recommendations based on relatively low-quality training data. Yet again, this issue stems from the problem that most existing recommender systems are owned and operated by supply-side platforms, who seek to make their core UX as frictionless as possible. It’s perfectly reasonable that they would build their recommender systems in this way, because the core function of those recommender systems is to support their supply-side objectives. However, this approach doesn’t yield the best possible outcomes to a discovery problem. For example, most supply-side platforms rely only on implicit data (i.e., data that measures whether we consumed a given option), not explicit data (i.e., data that measures whether we liked a given option). Of course, there are challenges associated with collecting explicit data, but those challenges are not insurmountable — especially for a company that is not optimizing for supply-side objectives.
• Fifth, most existing recommender systems fail to offer us recsplanations (i.e., human-understandable explanations for why they’ve recommended specific things). This makes it difficult for us to estimate how much satisfaction we might experience from a given option. Research in the field of recommender systems shows that recsplanations increase user satisfaction and user trust, yet few existing recommender systems provide them. At heart, this isn’t simply a matter of choosing not to show us their recsplanations; rather, it’s a problem that most recommendation algorithms aren’t amenable to providing them at all. For example, when implementing collaborative filtering, the recsplanation is often: “This option is popular among other users who have similar tastes to you”. That’s vaguely nice to know, but it doesn’t help us work out whether or not we’re likely to enjoy something.

Of course, it’s possible to come up with exceptions to all the above observations. Some existing recommender systems already optimize for discovery objectives, not supply-side objectives. Some allow us to state our preferences explicitly. Some use data from outside their own domain. Some use explicit data. And some provide recsplanations. This suggests that no single one of these points is a panacea for this issue. But this is a messy problem, and this post isn’t intended to be a check-box list for how to solve it. Instead, it’s intended to inspire an observation: To make step-change progress on the discovery problem, we need a recommender system that’s built from the ground up, with the specific objective of helping people discover things they truly love. Every aspect of the recommender system needs to support that core aim, from the UX, to the design, to the underlying AI, to the data that powers it, to the way that the product integrates into people’s lives. All these things need to work coherently and in concert to help people discover things they truly love.

The Opportunity

Let’s recap: We all encounter many discovery problems every day. These problems have been around for generations, but in the internet era, the number of possible options has grown to a bewildering level. This means that we can’t rely on traditional research-based approaches, because we’d need to invest a crazy amount of effort to explore all the options. As an alternative, we could try asking trusted peers, but in doing so, we’re really just delegating our decision to other people. In theory, digital products called recommender systems can overcome these challenges, but in practice most modern recommender systems suffer from drawbacks that prevent them from doing a better job. Most of these drawbacks stem from a structural issue: They are owned by supply-side platforms, so they are optimized to support supply-side objectives and they are integrated into a user experience that is not specifically designed for solving discovery problems. As a consequence, existing recommender systems often return options that are “good enough”, but they rarely help us discover things we truly love.

What if there was another way?

As you’ve probably guessed by now: I believe there can be. And I’ve decided to take a shot at building it. Together with my cofounder, Johnny Hunter, I recently founded a new company to tackle this challenge. Together, we’re creating a new digital product that helps people solve discovery problems. In an attempt to overcome the challenges faced by existing recommender systems, we’re building something that’s specifically designed to help people discover things they truly love. Every aspect of the product — and every decision along our path to building it — will align with that core outcome.

Make no mistake: This is a tough climb. To crack it, we’ll need to combine cutting-edge AI, exceptional product design and world-class engineering to create something entirely new. All of these things are difficult. But if we succeed, we’ll help millions (or even billions) of people solve multiple discovery problems each day. That’s an incredible opportunity.

I began this article with some questions: Where should I eat? Which movie should I watch? Which sunglasses should I buy? Which hotel should I stay in? Which podcast should I listen to? In many ways, our entire lives are defined by questions like these. And, when portrayed in this way, it becomes clear that these many small questions add up to a rather larger one: How should I live a good life?

Johnny and I care deeply about discovery problems because making great decisions can bring people joy — and, ultimately, improve people’s lives. This leads us to our company mission:

To make life better for everyone by helping them discover things they truly love.

In future posts, I’ll be sharing more of our thinking, and providing regular updates on our company’s progress. Up next, I’ll be describing how we’re working hard to ensure that we’re building a force for good, by holding ourselves accountable to three core values: privacy, transparency and algorithmic responsibility. These values shape everything we do as a company, from how we design prototypes, to how we think about data, to how we interact. They’re the guiding lights that keep us moving in the right direction. More soon!