Lucene and Solr: the Java of data

Among its recent technology news (beyond Windows 7 and the predictably delightful Apple riposte), two items of note, with previews of Sharepoint 2010 and the news around syndication of Twitter through Bing and Google. There’s a single underlying theme, as Alex Williams of readwriteweb/enterprise puts it:

Wikis, micro-blogs and collaboration technologies get a lot of attention for their use in the enterprise but one need remains constant.
Search.
Who[ever] wins the search battle will come home with a lot of prizes and big wins in the enterprise.

Williams goes on to talk about search capabilities differentiating across CMS vendors, but the observation applies to the enterprise and the web worlds in equal measure. The better/faster you can match people with content, the more your content is worth.

In the enterprise, Social/Web 2.0 is not only proving as powerful at busting work boundaries as the Blackberry has been. Social technology has lit a fire under the world of content in a big way – not only as more content comes faster, but as content becomes continuously more diverse. Tackling diversity and volume together means search application developers need search application development with ever greater depth and flexibility. One criticism of Lucene and Solr I hear, especially from search incumbents, is that “it’s just a toolkit”. That misses the point completely:  search is not merely a service, or a solution – it’s an application development platform.

In its early days, Java emerged as a way to write applications and overcome the heterogeneity of the underlying network and transaction  space, eventually maturing into the distributed and server-side incarnations where it is arguably delivering the most productivity (as with its Microsoft doppelganger, Dot-net).

That transition is now well under way in search. Just as Java flowered through J2EE to JEE and its SOA successors, Lucene is flowering through Solr. By providing a deep, flexible stable platform for application development, silos of resources can be addressed in consistent, powerful fashion and unlock ever more value.

Packaged search applications offer many options for navigating existing data relationships, but shortcomings in search and query APIs make it difficult to stray from those boundaries — and constrain solutions that need to introduce logic into the search above and beyond what’s already there. One new thing you can do with Lucene and Solr more readily, is to use the new numeric search to encode proprietary business rules around ranges within data. In general, since the data is unstructured and always changing, the transparency of these APIs is an important premium unavailable in packaged solutions.

Packaged applications – whether legacy search solutions or in legacy back office ERP –  have their place, but where organizations need competitive differentiation, they write their own applications. And, with the full force and innovation of open source, search is moving from a structured, constrained problem that hews closely to existing business processes, to one that itself becomes an adaptive driver of business processes.