Lucene goes from Enterprise Search to search platform

Somewhere in the soap opera that might be titled ‘As Hewlett Packard Turns’, the question of the impact of the Autonomy acquisition on the market once known as ‘Enterprise Search’ hasn’t gotten a great deal of explicit treatment.

When we bump into people from commercial enterprise search software providers (a lot of them seem to be looking for jobs, sadly) the words we hear about Lucene/Solr are ‘it’s just a toolkit’, as though that explains why people don’t buy it. Umm … it’s open source, people don’t buy the software, I always find myself thinking.

But measuring the economic impact of Lucene/Solr on what was the Enterprise Search market is pretty tough going, if you want to run the numbers. That didn’t stop Leslie Owens of Forrester, who put together a nice general purpose use case about open source with some very sharp observations, starting with “What Search Buyers Need Is Not What Search Vendors Sell.” I hope the folks at Forrester won’t mind if I quote the report. (As an aside, you can Google the report, and you’ll be shocked to learn that the vendor who got the most points announced in a press release that they are distributing the report on their website in order to capture your name and phone number. I won’t be surprised if you think that winning the bake-off and funding for the report were not unrelated, as the economics of the analyst business are a little easier to understand.)

Search tools aim to solve a well-known and critical problem — information overload. Yet few CIOs see enterprise search as a fundamental service to deploy to their workforce — similar to email. Unless this shift occurs, vendors can’t position their wares around knowledge worker efficiency and expect to make a sale. As a result many search vendors are guilty of over-intellectualizing what search can do — pushing “conceptual and optimized decision dashboards” for example, when workers simply want enterprise search.

Operating systems, collaboration tools, email clients, and enterprise content management (ECM) systems have decent retrieval capabilities. Is there even a need for enterprise search software? Yes. Enterprise search comes into play when no primary repository exists. It is a key component of an Information Workplace (IW) strategy that organizations build to unify their diverse information environments. Enterprise search lets workers find information in heterogeneous sources and can also serve as a development platform to build custom search-driven applications.

The density of observations here is fabulous, certainly good for more than one blog post. I’ll just pick up on the last phrase — ‘custom driven search applications’. If you’re reading this blog post, no doubt you get it, and you know what role open source plays. But don’t look for open source to be covered in the Forrester report. Josette Rigsby of CMSWire picked this up, too:

“Popular open-source enterprise search engine Lucene/Solr was excluded due to “lack of track record and/or in-house deployment expertise” according to the report. However, Lucene is the underlying platform for IBM’s solution, which was included on the list. Given the size of the Lucene/Solr open source community, the complexity of applications supported by the software and the current number of deployments, it should have been included.”

Couldn’t have said it better myself. In fact, it’s in the nature of customized search-based applications that they’re all different, very much a product of the business and technology context in which they are built and deployed — and harder to count as a result.

Customized search-based applications are without question gaining traction, and the penetration of open-source technologies into the ‘enterprise’ (that lovely ’80s word for ‘this app makes money, don’t screw it up’) has definitely reached critical mass. Customized search ain’t happening just because there’s information overload; it’s because people are tackling information overload with Lucene/Solr.

Is Lucene/Solr Enterprise Search technology ‘Enterprise’ search? Of course it is, it’s become one of those technologies you can’t screw up because it’s making money. Is ‘Twitter’ enterprise? They moved to Lucene because they couldn’t screw up search, either. No, search in the enterprise is no more limited to ‘enterprise search’ than mobile phones are limited to Blackberry (oh, HP found that out, too).

FAST went to Microsoft, Exalead went to Dassault, Autonomy’s selling their autonomy to HP, Endeca’s going to Business Intelligence. To us, the signs are clear: search is a platform. It’s a platform for the money-making enterprise, whether it’s a modern tech-only business or a classic IT datacenter organization. And if the platform has to have the power, the flexibility, and the cost-effectiveness to support a money-making enterprise, Lucene/Solr is an obvious choice. Most of our customers have reached exactly this conclusion. In fact, it’s probably a key factor in why HP wanted to buy Autonomy to begin with.

One other thing: use Lucene/Solr to build your search application, and it cuts the risk the search technology you’re using will vanish into some vendor falling to an acquisition.