Pig and HBase with Lucidworks Big Data

One of the goals of Lucidworks Big Data (LWBD) is to facilitate interoperation of components within the Hadoop ecosystem. One pathway we use commonly in LWBD is reading and writing data in HBase through Pig. At the most basic level, this is made possible through the HBaseStorage load/store backend provided by Pig. We have found there is a bit of an impedance mismatch between the way HBaseStorage returns records and the way Pig manipulates data. Specifically, HBaseStorage returns associative arrays (maps) corresponding to columns, but Pig is very record oriented and lacks support for map manipulation.

To solve this problem, we have written a simple Pig UDF (User Defined Function) called AddToMap which will add or update a map with one or more key-value pairs. Let’s see it in action.

First, let’s define two text files:

text1.txt

text2.txt

We will read in the first text file and use the built-in TOMAP function to construct a map which we will store in a column family “cf1” in HBase.

If we scan “test-table” in HBase, we see:

Now we read these records from HBase, read the second text file, join the two, and update our existing records in HBase

Here we load two column families into maps, although only “cf1” has any data (mainly just to demonstrate the syntax for multiple column families). The “loadKey” flag indicates that you want to include the row key as a field in Pig (it is the 1st field and has type bytearray).

The interesting part of this can be seen when we create relation “F”. Here we are using the AddToMap UDF to modify our “cf1” map which will get mapped back to individual columns in HBase. AddToMap makes this kind of read/modify/write action possible.

If we scan “test-table” again, we now see the new fields:

The same technique can be applied to reading and writing documents in LWBD’s document service. The major difference is that we use complex row keys in HBase for efficient lookups. To enable encoding/decoding our custom row keys, we have provided two UDFs: ToDocumentRowKey and FromDocumentRowKey. These UDFs transform the non-human-readable row keys into a tuple of (collection, id) for convenient manipulation in Pig.

HBase gives you fast random access to your data and Pig makes it very easy to process heterogeneous data sets. This bridge between the two lays the foundation of a very powerful big data processing pipeline.