Making Fusion Work for You: A Custom JSON ObjectMapper in a JavaScript Stage

„It is not down in any map; true places never are.“ – Herman Melville Out-of-the-box, Fusion contains a wealth of…

„It is not down in any map; true places never are.“

– Herman Melville

Out-of-the-box, Fusion contains a wealth of mapping capabilities. In some cases, however, we may need to create a custom mapping stage. For the purposes of this blog, we’ll create a custom JSON ObjectMapper in a JavaScript stage. We’ll be using the Local File System datasource, however it would be possible to use it in a Web Crawler datasource with no changes.

Step One: Create the Datasource

In the Fusion UI, open or create a collection, and add a Local Filesystem datasource. Scroll down to ‚Start Links‘ and add a full path to your target start directory, E.g. /path/to/my/start/dir. This should be a valid path to one or more JSON files. Save your datasource.

Note: This pipeline assumes that all files crawled will be JSON files.

Step Two: Create the pipeline

Because we’re doing something non-standard with the files being crawled, we’ll create a proprietary pipeline to handle the processing.

Open your „Index Pipelines“ view for the new datasource, and click .

Give your new pipeline a name, E.g. custom_object_mapping_pipeline.

You’ll use three (3) stages for this pipeline: 1) Apache Tika Parser; 2) JavaScript; 3) Solr Indexer, in this order.

First, add the Apache Tika Parser stage. In the settings for this stage, make sure the „Return original XML and HTML instead of Tika XML output“ is checked. Save this stage.

Next, we’ll add the JavaScript stage. This is where all the processing (mapping) will be handled. Add a JavaScript stage from your list of stage selections, give it a name, and add the following code to the „Script Body“ field:

function (doc) {
    if (doc !== null && doc.getId() !== null) {

        // class import declaration
        var ObjectMapper = com.fasterxml.jackson.databind.ObjectMapper;
        var ArrayList = java.util.ArrayList;
        var Map = java.util.Map;
        var StringUtils = org.apache.commons.lang3.StringUtils;
        var String = java.lang.String;
        var e = java.lang.Exception;


        try {

            // local variable declaration
            var mapper = new ObjectMapper();
            var content = doc.getFirstFieldValue("body");
            if (content !== null) {
                var mapData = mapper.readValue(content, Map.class);
                if (mapData != null) {
                    logger.info("Read data OK");
                    var result = mapData.get("result");
                    var obj = java.lang.Object;
                    var key = java.lang.String;
                    var list = java.util.ArrayList;
                    for each(var key in result.keySet()) {
                        obj = result.get(key);

                        if (obj instanceof String) {
                            logger.info("Key: " + key + " object: " + obj.getClass().getSimpleName() + " value: " + obj);
                            doc.addField(key, obj);
                        } else if (obj instanceof ArrayList) {
                            list = obj;
                            logger.info("Key: " + key + " object: " + obj.getClass().getSimpleName() + " value: " + StringUtils.join(list, ","));
                            doc.addField(key, StringUtils.join(list, ","));
                        }
                    }

                }

            } else {
                logger.info("Content was NULL! ");
            }

        } catch (e) {
            logger.error(e);
        }
    } else {
        logger.warn("PipelineDocument was NULL");
    }

    return doc;
}

Breaking it down:

So what’s going on in the above code? In the scope of this blog, I’ll discuss the significant processing pieces. If you are interested in further reading on how to use the Nashorn JavaScript engine, you can find further reading in Oracle’s documentation here. At the heart of it, we’re taking the content from the ‚body‘ field (created during the Tika parsing stage) and spinning that up into a Map object using the Jackson ObjectMapper. This instantiation is accomplished with these lines of code:

   var mapper = new ObjectMapper();
            var content = doc.getFirstFieldValue("body");
            if (content !== null) {
                var mapData = mapper.readValue(content, Map.class);
                if (mapData != null) {
                ...

From here you would have to think about the structure of the json you’d be parsing. Note: In the context of this blog, I’m pulling the ‚result‘ top level object from the Map and iterating over the values therein. However, depending on the structure of the JSON you’re parsing, your algorithm may vary.

Während Sie über die Werte iterieren, sollten Sie prüfen, welche Art von Objekt zurückgegeben wird. Wenn Sie generische Objektklassen verwenden, gibt Jackson folgendes zurück:

In diesem Beispiel überprüfe ich nur auf ‚String‘ und ‚ArrayList‘. Auch hier gilt: Je nachdem, welche Art von JSON Sie parsen, kann Ihr Algorithmus variieren.

Sobald wir den Klassentyp des Objekts ermittelt haben, wandeln wir ihn in eine Zeichenkette um (es sei denn, es handelt sich bereits um eine Zeichenkette) und fügen ihn dann wie folgt zu unserem PipelineDocument hinzu:

 if (obj instanceof String) {
     doc.addField(key, obj);
     ...

Fügen Sie schließlich eine ‚Solr Indexer‘-Stufe hinzu, damit Ihr PipelineDocument gespeichert wird. Hinweis: Sie sollten sicherstellen, dass alle Felder, die Sie in der JSON-Datei indizieren, in Ihrer Sammlung vorhanden sind und im erwarteten Format initialisiert sind (z.B. ’strings‘).

Viel Spaß beim Indizieren!