Delivering a great search experience means consistently providing relevant results to your users. At the same time, your users expect search to be flexible enough so they do not have to put a lot of effort to get the relevant results.
In this article we are going to:
Rethink relevance and redefine its traditional meaning
Devise a strategy for improving relevance using the new definition
Background
In a previous article, I discussed the formal definition of relevance which explained that relevance is driven by 2 metrics:
Recall: The percentage of relevant documents returned over total relevant documents
Precision: The percentage of documents returned that are relevant
As we explored real scenarios trying to improve recall and precision, the relationship between the two became obvious. Attempting to increase recall tends to reduce precision and vice versa. So how do we predictably improve relevance if it is not possible to increase recall and precision at the same time?
Rethinking Relevance
While it is important to understand the definition of relevance you should not lose sight of the end goal. A great search experience is one where the user consistently finds what they are looking for without a lot of effort. Is increasing relevance by means of recall and precision the correct approach to improve the search experience?
Search Experience and Relevance
Relevance alone is not the only factor that affects the search experience. Consider the following the search results:
60 results are returned
10 results are very relevant to the user and represent what the user is looking for
The remaining results (50) are not that relevant to the user
The 10 relevant results show first
The above search results have low relevance because of the bad precision (10 relevant results / 60 results total). However, the above search experience meets the end goal. The user was able to find what they were looking for without looking past the first 10 results. So although relevance was low, the similarity ranking of the results created a good search experience. Similarly, if the same set was ranked in a way where the relevant matches were in the bottom 30 results, the user would have never have found what they were looking for.
Redefining Relevance
The above example demonstrates that the formal definition of relevance alone does not help us improve search results and that we also should focus on improving similarity. It is more important that the top results the user would see to be of high relevance as opposed to high relevance across the entire set.
Recommended Strategy
Now that relevance has been redefined, a predictable approach can be used to improve search results. The approach is to "cast a wide net" in terms of recall and then push the most relevant results towards the top to increase precision for the top results.
The steps for this strategy are:
Increase recall which may increase or decrease overall precision
Tune similarity scoring so that the most relevant matches show in the top results
Repeat step 1 if more entries should have been returned in the results
Future posts will explore different ways to improve recall and introduce the concept of degrees of similarity which allows for better precision tuning.