A Strategy for Improving Relevance

Delivering a great search experience means consistently providing relevant results to your users. At the same time, your users expect search to be flexible enough so they do not have to put a lot of effort to get the relevant results.


In this article we are going to:


  1. Rethink relevance and redefine its traditional meaning

  2. Devise a strategy for improving relevance using the new definition

Background


In a previous article, I discussed the formal definition of relevance which explained that relevance is driven by 2 metrics:


  • Recall: The percentage of relevant documents returned over total relevant documents

  • Precision: The percentage of documents returned that are relevant

As we explored real scenarios trying to improve recall and precision, the relationship between the two became obvious. Attempting to increase recall tends to reduce precision and vice versa. So how do we predictably improve relevance if it is not possible to increase recall and precision at the same time?


Rethinking Relevance


While it is important to understand the definition of relevance you should not lose sight of the end goal. A great search experience is one where the user consistently finds what they are looking for without a lot of effort. Is increasing relevance by means of recall and precision the correct approach to improve the search experience?


Search Experience and Relevance


Relevance alone is not the only factor that affects the search experience. Consider the following the search results:


  1. 60 results are returned

  2. 10 results are very relevant to the user and represent what the user is looking for

  3. The remaining results (50) are not that relevant to the user

  4. The 10 relevant results show first

The above search results have low relevance because of the bad precision (10 relevant results / 60 results total). However, the above search experience meets the end goal. The user was able to find what they were looking for without looking past the first 10 results. So although relevance was low, the similarity ranking of the results created a good search experience. Similarly, if the same set was ranked in a way where the relevant matches were in the bottom 30 results, the user would have never have found what they were looking for.


Redefining Relevance


The above example demonstrates that the formal definition of relevance alone does not help us improve search results and that we also should focus on improving similarity. It is more important that the top results the user would see to be of high relevance as opposed to high relevance across the entire set.


Recommended Strategy


Now that relevance has been redefined, a predictable approach can be used to improve search results. The approach is to "cast a wide net" in terms of recall and then push the most relevant results towards the top to increase precision for the top results.


The steps for this strategy are:


  1. Increase recall which may increase or decrease overall precision

  2. Tune similarity scoring so that the most relevant matches show in the top results

  3. Repeat step 1 if more entries should have been returned in the results


Future posts will explore different ways to improve recall and introduce the concept of degrees of similarity which allows for better precision tuning.



Did you like this article? Subscribe to our blog by adding your email address to the form below. You can also email me at andreas@inventaconsulting.net or schedule a call to find out how Inventa Consulting can help you with Elasticsearch.

SUBSCRIBE

Stay up to date with articles about guides and best practices for Elasticsearch

Thanks for subscribing!