Open Source Myths: Open Source Search Needs More Assistance

Does open source Lucene/Solr require more outside assistance than comparable commercial products?

No. (I’m tempted to stop there, but that’s not quite fair.) That was, in essence, a question posed by Daniel Tunkelang on his blog in his commentary on the launch of Lucid Imagination. It’s a fair one to ask and an issue that has been raised before. First, some distinctions. Lucene is a Java search library with best practice indexing and query capabilities that have been continuously improved over the past 10 years. It’s cleanly implemented and not very hard to embed in an application, as thousands of Lucene-based application developers have discovered. But it’s only a library, so there is a certain amount of infrastructure that you have to build around it if you want to create an application, especially if you want something that’s flexible and easily maintained. Enter Solr, released by Apache in January, 2007. Solr is a scalable search platform. It puts Lucene over http and adds infrastructure such as schema-based management of fields (including how they’re defined and processed), an admin interface, cache management, tools for replication and data loading, logging and statistics and more. It also adds faceting, which is not an infrastructure improvement but an additional user-visible feature for doing the kind of categorizing of results that has become very popular at places like Amazon. With Solr you no longer need to write a Java program. One of the popular talks at ApacheCon is ‘Solr out-of-the-box’ – all the things you can do with Solr without writing code. People can try this out by downloading the Solr tutorial included with the Lucid Solr download or by downloading it from Apache. Does Solr have all the packaging and features found in the best commercial search engines? Not today. But neither does every user need all (or even most) of them. What’s more important for most developers is how easy it is to turn a search platform – commercial or open source – into an application, and how easy or hard it is to maintain it. Lucene/Solr is well written software with a modular architecture, and it’s easy to plug in or unplug various available modules … or, if needed, to write new ones. Disk overhead, relevancy ranking, throughput and query speed are all on par with, and often superior to, the best commercial engines. That translates to less needed customization and ongoing attention to a Lucene/Solr application. There are also the obvious advantages to open source: complete access to the code to figure out what’s going or make a change that you want, when you want it; an active community that responds to questions or problems; and no license costs.

Do open source projects often require consultants?

In my experience, no more than commercial applications, some of which pretty much require the vendor’s professional services group to make a complex product usable, or to make needed, even trivial modifications, over and above the license cost of the product. Consultants may ’stand out’ more in some open source situations… but that’s because they’re often independent consultants, rather than the paid services a commercial company provides with its product.

Might you be inclined to hire consultants for open source applications?

I think yes…but that’s because you have the freedom to hire them to get the application to do just what you want, and getting search to be really good is still an art. Much of the work we’ve done at Lucid has been to extend it for customers who wanted an extension when they needed it, or to assist them with planning their search application.

So…is there any down-side to open source Lucene/Solr?

Yes…but it’s not really about complexity or customization. What you don’t have with Lucene/Solr is a comprehensive support contract – Apache doesn’t provide them. It’s solid enough (and has source available) that this hasn’t stopped organizations of all sizes from moving to Lucene/Solr, but community and self-support are not for everyone. What you also don’t have with Lucene/Solr is a single assured place to turn to when you do need expert assistance, or want training. Those are gaps Lucid Imagination hopes to fill by providing high quality responsive support and services from a staff with over 70 years of cumulative experience building search applications.