Recently, a routine check on a search engine log report suddenly started showing zero matches for a very common query. This seemed impossible, so I checked the search index, but there were plenty of items that matched the query term, and I could search for and find the term without any problems. Nor did the search engine or configuration settings seem to have any bugs that could cause this.
After some head-scratching, I realized that the query term had zero matches only when it was filtered, and that was not expressed in my search reports.
Example: search + some filters = no matches
These kinds of filters are search limits imposed either by a user action or a configuration setting. Examples include filtering by country code, language, file type, company, product, genre, and so on. In my case, it was a geographic zone set in the user account, so they didn’t even know the filter existed.
Analytics reports must reflect filter scope
In these situations, the zero match reports cannot be aggregated, because the scope is incorrect. Each filtered section’s zero-match reports must be examined separately for accurate analysis. Then it’s possible to evaluate whether the filters are working with the users expectations of content availability, and improve vocabulary and content coverage, or consider widening the search automatically.
Even user-friendly features such as autocomplete and spelling suggestions can cause a lot of problems with filtered search scope, as they can lead users to believe they will get results and then leave them with zero matches. It may be possible to filter these automated options to only display those that contain matches within the scope. Best bets — manually managed search recommendations — should not have the scope problem, as they link to specific locations not dependent on search.
Facet issues
In contrast to static filters, dynamic metadata facets should not trigger zero match results, as by definition they do not allow users to combine conditions that contain no matches. Facet interfaces either do not display these combinations or display them as inactive with 0 items. However, a user may drill down by facets and then do a search: if the search engine incorporates the existing facets in the query, it may find zero matches within that scope, although there are items available that match within the index. Facets are not a panacea.
Analytics reports must reflect filter scope
Given all this, I recommend adding an overall zero-match filter/facet metrics report to the search analytics — it could be as simple as a list of zones with their query frequencies, or as complex as a matrix with multiple facet/filter zero match queries. The idea is to identify the most important issues first.
In the following example, an ice cream site with tens of thousands of queries per day has two kinds of filters: dessert type and flavor.
| Frequency of Zero-Match Queries By Filter |
|
[no type filter] |
ice cream filter |
fr. yogurt filter |
gelato filter |
| [no flavor filter] |
5,234 |
580 |
1411 |
697 |
| chocolate filter |
855 |
221 |
5 |
20 |
| vanilla filter |
621 |
46 |
91 |
44 |
| strawberry filter |
288 |
11 |
87 |
23 |
| pistachio filter |
473 |
173 |
26 |
12 |
This table makes it clear that the priorities for fixing the zero-matches result by adding content or synonyms should be:
- where there is no filter
- where either flavor or type filter is selected
- the most common few multi-filter situations, such as ice cream/chocolate, ice cream/pistachio, and frozen yogurt/strawberry.
Very small numbers of no-matches results with filters are probably not important, unless they apply to very expensive items such as jewelry, or vital ones such as legal precedents.
Filters and analytics for query frequency and clickthrough
Scope issues affect several analytic reports addition to zero-match cases. Query frequency reports are less influenced by filters, because users often search without regard for current filter constraints, so it causes fewer problems with aggregating query terms. However, for the most commonly-used filters, it’s worth separating and monitoring the specific query frequencies. Ecommerce sites frequently allow product managers to check reports within a specific product line or category.
Clickthrough rankings can vary widely based on filter or facet mode: if the best matches for a query are not within the scope, users may either click on inappropriate matches or abandon the search altogether. With enough traffic, they could affect the data context and reduce the accuracy of the rankings.
Forewarned is forearmed
This scope problem does make analytics more difficult, but at least knowing about the issue avoids wasting time with mysterious and impossible Zero Match situations.
Search Analytics Resources
Search Tools Consulting Blog 