Search Is Sexy! (Again?)

You’re not taking crazy pills: Reports of the death of enterprise search are exaggerated. Search is once again in the headlines. Search is back, baby!

But you can be forgiven if you didn’t know it because it’s draped in buzzwords.

Search this, search that, search here, search there, search me, search you…. so much search. It felt like for most of the aughts, all anyone could blabber on about was enterprise search. “Connecting employees to data so they can do their jobs better…” blah blah blah OMG we get it.

Then Google came along and raised everyone’s expectations on what search could be and should be. Gradually those expectations moved into the workplace and enterprise search took another step in sophistication and speed.

Nonetheless, it was such an improvement in what we had that we all settled happily into using Google at home and accepting Google-like results at work.

The Need for Search Was Everywhere

It’s not surprising. The enterprise search industry is pretty long in the tooth, and we were all kinda over all of it by 2010. But search was once fresh, exciting, and new.

Computer science pioneer Karen Sparck-Jones laid the groundwork for the modern search engine in 1972 with her work in natural language processing. She introduced  term frequency–inverse document frequency (tf-idf), which weights terms on how often they appear in a document and in a corpus. Tf-idf remains the underpinnings of the search engine today, and it was revolutionary.

Sparck-Jones’ work, combined with database advancements in the 1970s saw the birth of enterprise search. Query languages followed in the 1980’s, the Internet exploded in the 1990s, and big data happened in the aughts. The need for effective search kept growing.

Tf-idf has its limits though. It relied heavily on word counts to compute document similarity, and that can be slow for large vocabularies. Plus, these methods leave out adding in semantic calculations – the meaning of the words in a document and how they relate to each other. And these days your corpus of data might not be all text. You could have images, audio, video, and other multimedia formats you’re trying to index and understand.

Regardless, vendors and platforms sprang up and attempted (or failed) to meet the challenges companies of all sizes were having getting their employees connected to the data and documents that they needed to do their jobs.

In the last 15 years, we saw client-server model technologies from companies such as Verity, FAST and Endeca advance what’s possible for enterprise search. These search deployments were great for their time, indexing and accessing simple file servers, a web server or three, and data (assuming it was well-structured).

As the technologies matured, things were starting to fall apart at the strategic level. Vendors were charging per document, so licensing costs grew and grew until many organizations threw up their hands and went to open source, driving costs down. Hello Apache Lucene, Solr, and Elasticsearch.

We saw consolidation as Autonomy bought Verity, and HP bought Autonomy. Oracle bought Endeca, Google EOL’ed Search Appliance, and Microsoft bought FAST.

And whether these solutions were knowledge management or intranet search or knowledgebase search or customer support search or expert finders – we still called it all search. You had a box and a query and the search engine came back with a set of results. It was search. And usually it worked pretty well. Sometimes it didn’t. But we were happy with it.

But sometimes analysts get bored and marketers get tired. And that’s when the obfuscation began.

Insight Engines?

Who wanted old, ordinary, plain-Jane search when we could have oh-so-many other things?

Analyst firm Gartner changed the name of the vertical of their famed Magic Quadrant from enterprise search to insight engines. I know I read that and thought, “What fresh hell is this?” and thought, “Analysts gonna analyst.”

Turns out, there was a point to its re-framing. Let’s look at how Gartner described insight engines in the past few years. In their 2017 report:

“Insight engines provide more-natural access to information for knowledge workers and other constituents in ways that enterprise search has not.”

The emphasis is on the access to the information, making it more intuitive and easier. Then in 2018, Gartner revised its definition with:

“Insight engines augment search technology with artificial intelligence to deliver insights — in context and using various modalities — derived from the full range of enterprise content and data.”

This is where Gartner tucks in the use of AI and tilts the focus to the intention of the user – that they want insights from the questions that they ask their data. They want insights that help them do their jobs and take the next best action.

Cognitive Search?

Meanwhile, down in the Forrester PDF mines, analysts were trying to figure out their take – and what they were going to call it. They settled on cognitive search and knowledge discovery solutions. Rolls right off the tongue. Here’s the definition  in the last Forrester Wave for this market segment:

“A new generation of enterprise search solutions that employ AI technologies such as natural language processing and machine learning to ingest, understand, organize, and query digital content from multiple data sources.”

Forrester brought AI into the definition just as Gartner did, but it’s specifically calling out natural language processing and where it is going to be a part of the user’s experience.

Oh God, Here Come the Marketers

And aside from the analysts, at each vendor’s HQ the same ruminating was going on, “Well, what in the world are we supposed to call what we do now?”

This whole enterprise search thing. We don’t want it to be limiting but it needs to evoke the continuing evolution of search across the organization. It can’t be just “Now with AI!”

Or can it?

Here at Lucidworks we’re calling it AI-Powered Search. It’s a perfect Reese’s cup of the word search – with all the historical meaning that we need so that people have some sense of what we do – along with AI-powered to say it’s the new-new thing. It’s like the old thing, but new and better and more powerful and helpful. Uniquely personal. The search and data applications you use at work should be just as responsive and intuitive as Google or Amazon at home.

But is AI even descriptive enough?

We’ve seen the phrase artificial intelligence start to lose its luster, and we see more industry messaging and marketing around machine learning with nods to deep learning starting on the horizon. As customers get more educated, they are getting better attuned to what they need to shop for in their next search platform. AI and machine learning isn’t just a nice-to-have, it’s a critical component of building search systems that index millions of documents and serve thousands of users with dozens of applications.

Search By Any Other Name Is Still Sweet

Stop someone on the street and say, “Excuse me, what’s the word you use for when you’re looking for something among a bunch of other things?” No one is  going to say information foraging, knowledge management, cognitive search, search and discovery, or talk about using an insight engine. They’re going to call it search. Searchity Search Search. Searchy McSearchface. You get the idea.

Search remains the universal interface. Doesn’t require much training. Doesn’t require having to know Boolean operators or SQL structure. If you want to know something or do something you just ask it, and the system takes your query and uses natural language processing to figure out the specifics of what you’re wanting to know – and then gets you the right answer so that you can decide what to do next.

This is being embedded in the internal apps you use at work and the external apps and devices you use once you get home. Most of the apps on the phone in your pocket right now are search-based. Songs. Podcasts. Google Docs. Instagram thots.

If you’re trying to find something in a huge pile of other things, it’s probably a search app. Even if you don’t call it that.

So is search back? Well, it never left but it looks like we’re calling it search again. For now. Maybe in a few months or years we’ll have a new name for it. But it’s still search, same as it ever was.

Andy Wibbels is a published author and has been featured in The Wall Street Journal, USA Today, Entrepreneur, Wired, Business Week, and Forbes. He’s worked at several startups including Typepad, Get Satisfaction, InMobi, Keas, and Mindjet and is currently Director of Marketing at Lucidworks. andywibbels.com