Open Source: the industry standard

I recently had the pleasure of speaking to Gary McFadden, who writes for Wikibon, an interesting market analyst premised on the dual notions of open source meritocracy with openly modifiable “expert” opinions. I’d recommend his article on the future of enterprise search:

By 2012, unified search platforms that combine both enterprise and Internet search capabilities will be widely adopted by knowledge workers seeking flexibility to tune information access solutions for diverse workloads across a variety of content sources and data types.

His survey covers the range of players in our actively growing market (disclosure: I interviewed for the article). And his recommendation sounds to me like a broad endorsement of open source search. As I’ve said before, open source does diversity best.

I was struck by one of the questions he asked me, which is “What industry standards does Lucene support for data integration?”. Aside from the immediate answer, I was immediately reminded of the old joke: “The great thing about standards is that there are so many of them”. And certainly, any marketers worth their salt are quick to declare their wares an “industry standard”. Be that as it may, open source search with Lucene/Solr is the closest thing you can get to an industry standard in search.

The subject has been debated before: some years ago, Jonathan Schwartz (anyone remember him? He gave “pravda” a bad name) imputed some superiority to open standards and open protocols over open source. In referring back to that article as I was thinking about my conversation with Gary, I came across a thoughtful refutation by Peter St. Andre, which I think has worn well with the passage of more than six years since he wrote it.

So, what else is new? I certainly don’t hear as much about industry standard battles as we used to, and that’s because open source in the real world has turned standards from a passive into an active artifact. Not only do contributors have to meet the test of openness and robustness through the meritocracy of participation such as one gets with Apache; more importantly, the process produces a single, coherent implementation of the standard.  As many discarded standards have shown, the proof of the pudding is in the implementation. Going or gone are the days where a single, deep-pocketed vendor could out-muscle all other ideas (just ask Larry Ellison). And on this front, Lucene/Solr performs well — and not merely as a meritocracy that cultivates and releases disruptive new feature implementations (why pay for faceting when Solr licenses it to you for free?).

That’s not to say there’s no market for enterprise virtues around open standards and open source: getting the best implementation takes time, and sometimes, if you’ve bet your business on search, time is critical. The community can and will help, in time. On the other hand, if you need answers now, and time is money, we’re here to help. But whether you call us or not, if you are counting on search to drive your business, implementing on Lucene/Solr gives you more control over your search destiny than any other alternative. The proof of the pudding is in the implementation.