Fun with Solr Functions

For a long time now, Solr has had a good chunk of functions available for use to boost relevance based on the content of a field, but I’ve always been on the user side of them and never on the writing side.  At least, that is, until recently.  This week I have been putting the finishing touches on an article on using Lucene and Solr for spatial search.  As part of the article, I had a chance to write up a bunch of new functions for Solr, including several for calculating distance between two points.  These functions are:

1. hsin and ghhsin (geohash based haversine) – Calculate the Haversine distance (Great Circle) between two points on a sphere
2. L^p norm – i.e. the 1-norm (Manhattan distance), 2-norm (Euclidean), etc.
3. radian/degree converter
4. Geohash converter – convert latitude/longitude pair to a geohash value.

On top of that, Yonik added in support for pretty much all of java.util.Math, including cosine, sine, tangent, etc. so now Solr really has some very significant mathematical capabilities, all of which can powered by the values selected by doing a search.  Not too mention, they can be used for filtering too, using the FunctionRangeQParser (frange for short).

As an example, here’s a few Solr requests I generated for my article:

http://localhost:8983/solr/select/?q={!func}recip(hsin(0.78, -1.6, lat_rad, lon_rad, 3963.205), 1, 1, 0)

http://localhost:8983/solr/select/?q=*:*&fq={!frange l=0 u=10}dist(2, 32, -79, lat, lon)

In the first example, I’m scoring all my docs based on the great circle distance from a specific point on the globe, while in the second, I’m filtering out docs based on distance.

I’ve also seen functions used in all sorts of different ways.  One customer was even using them to simulate alternate scoring models by using the ExternalFileValueSource.  Additionally, one of the nice things they can also do is give you much higher precision document/field boosts.

Up until now, I had never realized how dead-simple they are to write.  Basically, take in a ValueSource, convert it to a DocValues instance which gives you the value of the field for each document and then do your calculation.  Finally, register it in the solrconfig.xml.  For more on this, see the SolrPlugins wiki. page.

Finally, keep an eye on Solr 1.5, as I have an inkling to add what I’m tentatively calling aggregating functions, which I think can be used to power more business intelligence applications.  These aggregating functions would be a significant extension of Solr’s StatsComponent and allow for much more expressive calculations over result sets.