Why Lucene? Why Lucid?

I’ve spent a third of my life, my entire professional career, doing what I love: developing software.

More specifically, even though it was never a particular ambition of mine, nearly every major project I’ve worked on in my career was developing “Search” software.  Why search?  Well the short answer is “Because my bosses told me to”, but the longer answer is that getting Search “right” is hard work.  In any software system you have to worry about correctness, maintainability, performance, efficiency, scale, etc. With Search software, all of those things are still important, but there are additional fuzzy metrics that in the most basic form get labeled “precision vs. recall”.  When you write database software, the input you deal with is SQL instructions and your output are sets of rows.  There is no subjectivity involved in deciding which rows match which SQL statements — the results of a SQL operation are completely objective.  In search applications however, almost everyone has their own subjective view on what the “best” matches are for a given search.  Finding ways to implement Search applications that will satisfy as many subjective opinions as possible is incredibly challenging and interesting.

For the past 5 years, all of my “search” related projects have involved Apache Lucene (starting with the initial development of Solr on top of Lucene back in 2005).  Why Lucene?  Because even though there are lots of good search systems out there that can serve as the backbone of custom search applications, Lucene happens to have an amazing collection of properties that put it in a unique position:  high performance, open source, easily extensible & tunable, healthy community support, active development of new features, friendly license, etc. Working with Lucene & Solr these past few years has been a dream compared to some of the proprietary systems I used before.  Not only can you build applications that are stable, and reliable, and performant; you can also see exactly why it behaves the way it does.  You can trace the behavior to make sense of the “subjective” results, and tune that behavior to your heart’s content, in a way that’s just not possible with any other black box search system.

So now I’ve started a new chapter of my life: working as a software engineer at Lucid Imagination.  Why Lucid?  Because the folks here at Lucid don’t just “get search”, they also understand the importance of openness in search, and are dedicated to working with the Lucene community to make Lucene & Solr even better then they are today.

So here I am.  Let’s see what we can do in the next 5 years.