Looking Back at Search in 2017

2017 was a big year in search technology. As we chronicled last month in our rundown of trends for 2018, search technology has moved far beyond just keywords, faceting, and scale. But let’s take a look back  at the trends that have continued through the past year.

Continued Industry Consolidation

We’ve continued to see consolidation with the exit of Google Search Appliance from the market. Now organizations are re-evaluating technologies, like Endeca, that have been acquired by vendors and products like FAST and have been embedded in other products. Ecommerce companies that have traditionally thought of search as a primary part of what they do, have already migrated to newer systems. In 2017, IT companies stuck with maintaining technology not intended for today’s scale, moved away from legacy technologies in earnest.

Meanwhile other vendors have been downsizing staff but continuing to support the locked-in long tail installation base. You can figure out which ones by looking at current vs past employees on LinkedIn. In 2017, customers started to get wise. No one wants to be the last one on a sinking ship.

In this same time period, I’m proud to say Lucidworks continued to grow in terms of code written, revenue, employees, and even acquisitions.

Technology and Data Consolidation

Not long ago, larger companies tended to have more than one IT department and each of those individual departments had their own search solution. So there would be a search application for sales which was deployed by the sales IT group and then another search app for the HR department deployed by their IT group and then probably yet another search solution for the product teams built by their IT group. With IT consolidation, an ever-increasing mountain of data, and new integrated business practices, there is a greater need than ever to consolidate search technology. There are still single source solutions (especially in sales) but last year, IT departments continued to push to centralize on one search technology.

Meanwhile there are more data sources than ever. There are still traditional sources like Oracle RDBMS, Sharepoint, and file shares. However, there are newer data sources to contend with including NoSQL databases, Slack, and SaaS solutions. With the push towards digital business, and turning information into answers, it is critical to build a common search core to pull data from multiple sources. In 2017, we saw continued movement in this direction.

Scale Out

Virtualization replaced bare metal for most companies years ago. The trend was the joining of the private and public cloud. This move continued against a business backdrop of continued globalization and a technology backdrop of continued mobilization. In 2017, modern companies often conducted business all over the world from palm-sized devices, tablets and laptops.

Meanwhile there are new forms of data emerging. Customers now generate telemetry from their mobile devices. Buildings can now generate everything from presence data to environmental and security information. Factories and brick-and-mortar storefronts now generate data forming the so-called Internet of Things. With machine learning and search technology, companies are now starting to make better use of this sort of data. These trends were nascent in 2017, but still observable.

In a virtualized, cloud-based global world where data is generated from everything everywhere all of the time, companies need search technology that can handle the load whenever, wherever, and however it comes. Old client-server technology was no longer enough to handle these demands. In 2017, horizontal scale was no longer a luxury, but a necessity.

Personalization and Targeting

2017 saw simple search start to abate. While AI and machine learning technologies are relatively new to the search market, some of the more mature tools saw widespread deployment. Many organizations deployed search technology that could capture clicks, queries, and purchases. Modern search technology use this information to provide better, more personalized results.

Collaborative filtering (boosting the top clicked item for a given query) is the most common optimization followed by similarity (MoreLikeThis) but we also saw companies start to deploy Machine Learning powered recommendations especially in digital commerce. These recommendations use information about what a user or similar users have done to suggest choices.

Mainly Custom Apps, but The Rise of Twigkit

In 2017 most companies were still writing their own custom search apps. Unlike previous years, these apps are very AJAX/JavasScript-y/dynamic. Frameworks like Angular ruled the search application market. At the same time, savvy organizations realized that writing yet another search box with typeahead was a waste of time and they started using pre-built components. One of the best toolboxes of pre-tested pre-built components was Twigkit.

Twigkit had been around since 2009 and was a widely respected force in the search industry with relationships with all of the major vendors and customers all over the world. Lucidworks had been recommending it to our customers and even using it in some deployments so we decided to acquire the company and accelerate the technology. The future of Twigkit was announced as at our annual conference last September with the technology becoming part of Lucidworks App Studio.

Happy New Year

Goodbye to 2017 but hello to 2018. It was a great year for search, but not as good as what is coming. If you want to see what’s on the way in 2018, here’s my take on what to watch for in the coming year.

If you find yourself behind the curve, Lucidworks Fusion and Lucidworks App Studio are a great way to acquire the technologies you need to catch up. You might also sign up for Fusion Cloud.