At Lucene Revolution this morning, the first technical breakout session I’ve attended is by Erik Arnold, who’s presenting on his team’s work on search.usa.gov. Some pretty striking things about this website, both from a search implementation perspective and in terms of how open source technology delivers value to the federal government — which those of us in the 50 states are helping to pay for.
First, search.usa.gov is interesting because it represents an integration of two data streams (federated search? No pun intended) query results are drawn from Bing, and then indexed and searched in real time along with multiple government collections that have been added to the ever-broadening dataset for search.usa.gov. If you search “hurricane” for example, the search.usa.gov service (with handy search-as-you-type) integrates things like real time weather reports from NOAA. The use case is not only interesting in the fold-it-in-on-the-fly, but in the use of Apache Solr to orient search results to the user base: searching government data for “yosemite” can and should return different results than a media company’s search or even an ad-driven commercial consumer search engine (National Park Service should outrank hotels, for example).
The second, and more striking change this represents is the inception of an incursion — a revolution? — in the government moving to open source. In the past, procurement cyles for Government IT would pick a proprietary enterprise software vendor, the contract would last 2-3 years, but of course the technology would age in the years between the requirements spec and evolving real world needs, growth, etc. The old answer? Do a new bid, throw away all the code from old bid, and start from scratch. With search.usa.gov (and whitehouse.gov, recovery.gov, etc.), the government stack is migrating to an open source foundation is more ‘evergreen’. Implementations stay transparent, and vendors compete on forward execution, not past installed binaries.
The evergreen quality of open source is tied to another kind of green: the green of dollars and cents that are needed to pay for the cost of change. Scaling, and handling change, have at their root economic proposition. Given the appetite for growth across so many sectors of society and the economy, lowering the cost of growth seems like a straightforward opportunity.