Search Intelligence

Why do you need open source search? Here’s a good reason:

[It’s] translation of disparate data and information to actionable intelligence via focused real time contextual analysis.

Credit for this crisp description goes to Mats Bjore, who’s built a cool site called Silobreaker.com (come see him speak at Lucene Revolution in Boston, October 7-8) Anyone who’s ever worked in a company larger than your nuclear family knows why breaking silos is a good thing; the site is a demo for enterprise intelligence gathering. But that’s not the only place this approach has cut its teeth — variously known as open source intelligence, public intelligence, and naked intelligence, these guys are pushing the envelope in turning content from diffuse raw material to something you can use to make decisions. There’s an interview with Bjore in the Beyond Search blog:

[We] anchor all we do in a strong belief in that information overload is not evil but a reassuring consequence of freedom and innovation, but that it is the ability to refine this overload and extract benefits from it that truly create the “killer app” that everybody needs.  … The focus on “how many hits at what speed” feels very much like first generation features and haven’t really helped with information overload.  Context, analysis, and query customizations will be the challenges for next generation algorithms and services.

Speaking of the world of government and public intelligence, check out the following post from David Smiley, who is also speaking at Lucene Revolution

Today at work (at MITRE outside DC) there was (is) a day of technical presentations about topics related to information dissemination and discovery (broad squishy words there, but mostly covered “search”) at which I spoke about the value of faceting, and gave a quick Solr pitch. …  I pointed out how open-source Solr has “democratized” (i.e. made free) search and faceting when it used to require paying lots of money.  …   Speaking of performance, on a large scale search project where we’re using Solr …  the search results were so fast (~150ms) … that the UI engineers building the interface on top of the XML output thought Solr was broken because it was so fast. The quote was “It’s so fast, it’s broken”. In other words, they were used to 2-3 second response times and so if the results came back as fast as what Solr has been doing, then surely there’s a bug. There’s no bug.

David’s the author the book Solr 1.4 Enterprise Search Server, which if you haven’t seen, you’ll likely find pretty handy.