Solr Migration, Consolidation For Online Search/Advertising Co.

Highlights

Lucid Imagination helped an online search and advertising company move to Solr from a proprietary commercial search platform:

• Migrated search to Solr by replacing a commercial solution and returning search application support to in-house resources
• Consolidated search footprint from over 130 servers to about 2 dozen
• Provided guidance on integrating Web 2.0 data into search results, such as user comments, reviews, and location

Customer Overview

An online and mobile search and advertising company owns several different online properties connecting businesses with prospective customers. Search results include business listings, different levels of information such as URLs, addresses, hours of operations, advertising, and so on. Searches are performed across 20 million documents, with significant “time of day” spikes in the normally high volume of searches.

Business Issue

The company’s search application faced several issues. Primarily, they needed help in deciding how to model and present their search data in a way that would balance their advertisers’ need to achieve proper placement in search results against their users’ need for relevant results ranking. Search results also needed to factor in location to support geographic search, and include relevant end-user comments. In addition, the company wanted to gain Solr expertise as part of an effort to switch away from a proprietary commercial search solution.

Solution Description

The company engaged the following services from Lucid Imagination:

• Search Health Check
• Training. These courses were customized for developers new to Solr, and search data administrators.
• ExpertLink: Support agreement plus consulting. Review of planned development and in-depth collaboration on specific issues.

For this project, the company utilized the full portfolio of Lucid Imagination’s services: development, support, in-depth consulting, and training. Lucid experts kicked off the engagement with about two weeks on site, training new administrators and developers, and working with the company’s staff, reviewing the company’s architectural plans, and talking through specific issues. Subsequent engagement has been exclusively remote consulting, by phone and online.

A key challenge for the company was to include user-generated data—comments, ratings, and feedback on search results—for a number of its properties, and address the fact that such user-generated inputs are not always relevant or concise. Lucid helped the company decide when and how to incorporate these results by building a scoring model appropriate to both the advertisers’ and consumers’ needs, based on relevancy and location. This required searching in ways beyond keywords, such as exact-match, or “invisible queries” (fake-and-invisible-queries) to achieve the expected results.

Relevancy was a key focus of the effort. Searches were rated by text-string relevancy, geographic nearness, category, ad placement, and user comments. Searches for a business name generally received one to three worthwhile results, with the rest found to be not useful. Usually two to five more results made some sense but were again not the desired result, with the rest essentially meaningless. The customer wanted to increase the relevant results and also minimize the meaningless results. Lucid developed a set of “fallback” rules that made the search far more effective. These included making a second different request if the first one did not work, adjusting the relevancy of different fields based on the user’s location, using the categories to drill down, and judging when to include user comments.

Migration

During the course of the transition from the commercial solution, Lucid helped build a blended-index capability that gradually introduced Solr-based search into the search service. In replacing the proprietary commercial enterprise search solution, they were able to downsize from over 130 servers to only about 25 upgraded servers, a much smaller footprint.

Outcomes

The outcome of the engagement was well received. The company’s online properties now offer rich search results, factoring in location, users reviews, and ratings, alongside paid advertising. By taking advantage of Solr’s open source model, the company is able to develop and maintain its search capabilities in-house, without relying on proprietary expertise. Furthermore, some of the innovations developed during the engagement were submitted as changes and improvements to Apache Solr, in full accordance with the customer’s wishes. Subsequently evaluated within the community guidelines and processes, these improvements were ultimately committed into the Solr source tree. Because these improvements are now part of the standard public Solr distribution, the company benefits by reducing the cost of supporting custom code