Publishing Case Study: McClatchy Interactive Employs Open Source Search

Highlights

• Dramatically increased index speed improves timeliness and relevance of user searches
• Reduced the number of required servers by 85% over the previous proprietary search platform
• Eliminated license fees associated with search
• 8 months from management approval to seamless integration across dozens of media sites

Staying competitive is the name of the newspaper game

The third-largest newspaper publisher in the U.S., McClatchy Company owns 30 daily newspapers in 29 markets across the country. This includes the Sacramento Bee, its first paper and one of the oldest in the West, the Miami Herald, The News and Observer, and others.

The unforgiving economic climate has hit the industry hard, accompanied as it is with fundamental shifts in reader and advertiser behaviors. As a market leader, McClatchy is vitally focused on their online business model. McClatchy Interactive has more experience than most—they launched one of the first electronic news sites on the Web, The Nando Times, in the Spring of 1994. Today, McClatchy’s mission is to establish its local papers as the leading source of online news within their respective markets, with compelling local information online, along with comprehensive news, advertising, e-commerce and other services. Helping online readers find the relevant information they want is a cornerstone of their online initiative.

Open source technologies contribute directly to McClatchy’s business model by minimizing the both capital and operational costs without compromise in customer experience. McClatchy Interactive chose the leading open source search solution, Solr, to search classified ads across all online properties. Solr has already helped McClatchy Interactive properties reduce costs dramatically, while improving on the same high quality of service to its online readers.

“Solr has done wonders for us. It is easy to understand and deploy, and reduced our costs drastically,” according to Doug Steigerwald, software developer at McClatchy Interactive.

Making the leap from proprietary system to open source search

The company’s legacy search system was built on a proprietary, pre-packaged enterprise search software solution that was increasingly expensive to maintain and modify while addressing new business requirements. In 2007, the company decided to bring in-house its Web site search functionality, and build a search service that could handle classified ads across all McClatchy properties.

Primary responsibility for development fell to Doug, although he had some part-time assistance from another member of the software development group.

There were two requirements. The first was that the new search system had to have the same features and capabilities as the existing search product. Second, the solution had to be seamless, transparent to support personnel and the McClatchy newspapers. “It had to be absolutely no work for anybody other than development,” said Doug. “And it had perform the same or better as what we already had.”

Why Solr was chosen

Solr is the leading open source search solution, and features high performance and comprehensive capabilities. These include faceting—the ability to drill down on characteristics of the desired result. For example, when looking for a used car, a reader may want to limit search results by price range or model type, and then drill down to find only those cars that are nearby.

Any search solution would need to enforce McClatchy’s search policy requirements. Stories—which can be searchable or non-searchable—appear in many places across McClatchy Interactive properties. A searchable story must remain so, even if it appears in a non-searchable section. The converse is also true, controlling which results are excluded from a search.

Before a final decision was made, extensive comparisons of Solr and the existing search package were undertaken, in which Solr delivered better, more relevant results. The developers also checked to make sure that Solr’s search processes stored data in the same way as the legacy system, so no problems would arise with other related applications after the new solution was rolled out to production. McClatchy’s Solr implementation was able to replace the legacy software without changing any significant back end functionality. Doug says, “After researching it, we felt Solr could do everything we’d done with our existing proprietary system.”

Real, fast results

The project began in September 2007 and was in production by April 2008. With experience in open source development, the developers began evaluating and building a search system using Solr. Basic development, including the training classes for the two developers, was completed in January 2008. Once the pilot implementation was up and running, efforts moved to improving speed and accuracy. Two months of QA ensured everything would work as intended.

In order to meet their targets for both performance and relevance, McClatchy engaged Lucid Imagination’s consulting services to help them get the most out of their implementation as soon as possible. The team put together a presentation about what McClatchy was doing and asked for an expert opinion on how the system could do better in terms of performance.

Lucid worked with McClatchy to carry out extensive analysis of logs and production for all sites, to profile the volume, diversity, and frequency of data being indexed. For McClatchy, Lucid’s recommendations focused largely on configuration and schema—for example, what field options could be closed or disregarded—to increase indexing and query response time.

Doug says, “Lucid Consulting was definitely worth it. They helped things move along more quickly by pointing out ways to better tune Solr, based on our usage. We could have spent a couple of weeks benchmarking and refining our implementation, but Lucid Consulting was able to optimize our specific search environment in a day or two.”

McClatchy Interactive today

The Solr implementation achieved impressive cost savings and performance. Because there are no licensing costs associated with Solr, McClatchy enjoyed significant savings by eliminating the licensing fees associated with the previous packaged software application they had purchased. In addition, one of the most impressive benefits of this Solr implementation was the dramatic reduction in hardware requirements. The old search required 24 production servers, while McClatchy’s Solr search service runs on just three of the existing servers. The remaining 21 servers have been freed up for other uses. “We really reduced the number of servers we needed,” states Doug. “We may add one or two more in the future to handle extra traffic and roll out some new services.”

All 30 McClatchy dailies are running on the new system, and the combined number of classified and news searches reach 700,000 per day—primarily from classified queries. Every ad category and classification is stored within a Solr index. Solr also hooks into the company’s content management system, which serves all its newspapers, to index other searchable data such as stories, movie show times, and so on. In addition, the entire front page of online properties is powered by Solr, including news search, movie search, and more.

Partly because it was so easy to implement, Doug recommends Solr to other developers who are just beginning to explore open source. “Solr is free and has a huge, active user base, which is very helpful,” says Doug. “It was a different experience than the proprietary software we were using. We found it was easier to get support by going open source. The Solr community provides support through the developer e-mail list, where you can find out how others have already solved problems you might have.”

McClatchy Interactive is evaluating Solr to take advantage of other capabilities. For example, Doug is evaluating Solr’s More Like This functionality, to help readers find related stories and increase site stickiness. And as Doug points out, there’s no extra charge for this functionality.

Hardware

• 1 master server, with 8GB RAM
• 2 query servers, each with 16GB RAM

Software

• Red Hat Enterprise Linux 5.3
• Jetty
• Ruby on Rails
• Perl
• Java