Top 5 Considerations When Migrating From Attivio to Lucidworks

Tim Naylor is a Lucidworks guest blogger. He is a software engineer at Accenture.

Recently I’ve been working with my team to help an organization migrate from Attivio to Lucidworks Fusion for their various internal and public-facing search applications. Similar to many enterprise projects, one of the migration requirements is the ability to handle a massive volume of data (millions of records) stored in multiple repositories, with data types ranging from unstructured content like pdfs and Word documents to structured content like tables and database entries.

As both Attivio and Lucidworks were contenders in Gartner’s Magic Quadrant for Insight Engines (before Attivio was acquired by ServiceNow at the end of 2019), I’ve had conversations with our Innovation Lead and teammates around these search engines’ capabilities, limitations, and potential migration challenges. We then mapped out various considerations for organizations looking to migrate from Attivio to Lucidworks. In this blog, I’ll discuss the top 5 considerations.

1. Content Sources

For organizations with content residing in disparate sources, the first thing to consider is whether Lucidworks Fusion has all the required connectors that are being used on Attivio. Whether the connector is built-in or customized, we need to cover any special custom features so that it can work seamlessly with the new system. In addition, content crawlers should be set up to honor the same include/exclude patterns in the existing system.

It’s worth noting that, while Attivio has a joining database connector that can automatically merge rows with the same ID, Lucidworks Fusion doesn’t. As a result, some data normalization will need to happen during database migration.

One of the most critical aspects of connectors is how security is handled. We need to ensure that the same security levels and access permissions are maintained when migrating from Attivio to Fusion. For organizations with complex security and customization requirements, Accenture provides a range of connectors that complement these search engines’ built-in connectors.

2. Infrastructure

The underlying technology is another key aspect to consider in any search engine migration. A fault tolerant Attivio cluster runs on Hadoop as a platform, and is supported in a single data center. Lucidworks Fusion 5 is based on Kubernetes, which provides a cloud-native architecture. Your organization’s existing infrastructure – cloud-based, on-premise, or hybrid – will be an important factor determining the amount of effort to migrate.

3. Workflows

Among the requirements we discussed was “How can we migrate Attivio’s sophisticated index and query pipeline structures to Lucidworks Fusion and can we reuse the same workflow components?”

Well, while Lucidworks has available tools for schema conversion, many of the custom workflows built in Attivio will need to be re-created in Lucidworks. But the good news is that both Attivio and Lucidworks Fusion use JavaScript for scripted custom modules and Java for more complex modules. This allows the re-use of the most complex algorithms, eliminating the need to rewrite the workflows in a whole new language.

Our team has learned from various migration projects that it’s critical to re-index all of the content when migrating and rebuilding workflows as well as to audit the new index. To do this, you may consider a side-by-side comparison between the old and new indices. For example, once the new index is ready, you can assess which documents got ingested and show up, the order in which they show up, etc. Going one step further, consider doing engine scoring for the “golden queries,” such as your most-used queries or queries with manual boosting, to validate the new index’s performance.

4. Analytics

As with all search engines, analytics is key to monitoring and enhancing search performance overtime. So, what should we consider when migrating system analytics and monitoring from Attivio and Lucidworks Fusion? Fortunately, most of the standard analytics features in Attivio are also present in Lucidworks.

One of the most innovative analytics features in Lucidworks is its ability to use click tracking and query signals to update search relevancy based on real-time usage. Consider exporting search usage data from the existing system and leverage those signals at go-live.

5. Ontology Tool

If your organization, like many of our enterprise clients, has large datasets from various sources, ontology is an important migration consideration. One of Lucidworks Fusion’s highlighted features is its join capability which provides tools for handling block joins and parent-child joins during migration.

In Summary

So, these are the five key considerations before starting an Attivio-to-Lucidworks migration project. It’s worth noting that there are machine learning capabilities available on Lucidworks, including Learning-to-Rank, that promise a more intelligent search experience. I’m also excited to explore Lucidworks’ Chatbots and Natural Language Processing capabilities and leverage them to enhance the user experience.

Are you embarking on a search engine migration project? Connect with us to discuss your requirements and learn how we can help.

Special thanks to these blog contributors:

• Paul Nelson, Innovation Lead, Search & Content Analytics, Accenture Applied Intelligence
• Andrew Gullett, Data Engineering Manager, Search & Content Analytics, Accenture Applied Intelligence