The Shard Splitting feature was first introduced in Solr 4.3 to SolrCloud users. Solr 4.3.1 (released yesterday) fixes many bugs identified in shard splitting and makes it production ready.
Shard splitting allows seamless on the fly splitting of an existing shard into two without any downtime. You can keep searching and indexing documents while a split operation is in progress so that users of the cluster are not affected.
A point worth noting is, even though a split can be performed while indexing and querying is in progress, it is recommended that you split shards during off-peak hours so that the parent shard leader can handle the extra load.
Sharding in SolrCloud
There are two ways to distribute your documents in a SolrCloud cluster. The first method is to decide, on your own, outside of Solr, the shard on which a particular document must live on. This is also known as “Custom Sharding” in SolrCloud terminology. The other method is auto-sharding which requires you to decide the number of shards when you setup your collection. Solr assigns a hash range to each shard and routes documents to the shard whose range contains the hash value of the unique key. Searchhub has an excellent post on Solr Cloud and Document Routing if you want to read further.
The Shard Splitting feature is applicable only for deployments using the second method (as of Solr 4.3.1). So, to use this feature, you must be using SolrCloud and your collection must have been created with the numShards parameter.
Quick start – How to split a shard
Splitting a shard is as easy as making an HTTP request. There is a new SolrCloud collection API called “splitshard”. Here’s how to invoke it against your existing SolrCloud cluster:
This HTTP API will split the specified shard into two “sub-shards” — each containing exactly half the hash range of the original (parent) shard.
Please note that this does not mean that the number of documents are divided equally among the sub-shards. This is because the split happens on the mean of the integer range of the parent shard and not on the number of documents. Hence, this may lead to an uneven split in case of a non-uniform distribution of hash values.
Once the split is complete, the parent shard will be made ‘inactive’ and subsequent requests will be forwarded automatically to the sub-shards.
Why to split a shard?
Auto-sharding in SolrCloud requires you to fix the number of shards (numShards) in your collection while creating it. Predicting the right number of shards for your use-case is not easy. Underestimating that number can lead to large shards and make both indexing and search rather slow. Overestimating, on the other hand, can lead to shards that are just so small that they become an overhead.
Before shard splitting came along, the general advice was to err on the side of more shards rather than less because there was no way to increase the number of shards in a collection once it has been created. If you absolutely had to change the sharding setup, you had to recreate the collection itself and re-index documents. Now you can start with your best guess and increase the number of shards by splitting them as they grow.
Let’s try this new feature out by on a simple SolrCloud cluster with 2 shards.
Step 1 – Follow the SolrCloud wiki to setup the demo cluster here.
Step 2 – Optionally, you can add more replicas.
Follow the instructions [Simple two shard cluster with shard replicas] to add a replica to each shard created in step 1.
Step 3 – Add data into the newly created collection
# cd exampledocs
# java –jar post.jar
Copy paste this into the browser or use curl to confirm the documents are in the index.
Here’s the response:
<response><br /> <lst name="responseHeader"><br /> <int name="status">0</int><br /> <int name="QTime">6</int><br /> <lst name="params"><br /> <str name="q">*:*</str><br /> <str name="rows">0</str><br /> </lst><br /> </lst><br /> <result name="response" numFound="32" start="0"></result><br /></response>
This should reflect 32 documents which were posted.
Step 4 – Using the collections API to split shard1.
Once the call finishes, open the Solr Admin UI to see the new shards..
Step 5 – After calling the ShardSplit API, you’ll need to commit to see the documents on the new sub-shards.
What happens behind the scenes
The operation splits the original index into 2 and creates new shards in Construction State. The new shards start maintaining a transaction log from the updates received from the parent shard. They do not ever receive any queries in this state. The queries continue to be routed to the parent shard.
An important thing to note here is that when the parent shard’s hash range is split, the partitioning logic takes care to round the hash range’s end value. This ensures that the documents having the same shardKeys still end up together i.e. composite id based co-location is respected.
After we have a leader for both the new sub-shards, we start applying buffered updates on these shards.
New replicas are created for each sub shard to match to match the replication factor of the parent shard at the time of split.
Once the replicas are created, we do an atomic switch of shard states. The parent shard is then marked inactive and the new sub-shards take over as the active ones.
Though this has been more talked about as the Shard Split collections API, there is a lower level core admin API call too.
An important thing to note here is that SolrCloud users should completely avoid this CoreAdmin call for all their nodes.
This API lets users to make a lower level call to split a core into ‘n’ parts. Here’s how to perform a core admin split:
Note: The “path” parameter is multi-valued.
The above call splits the core, as specified by the core_name or shard_name and places the smaller sub-cores at the paths specified using the “path” parameter.
The Road Ahead
Shard Splitting in SolrCloud is a rather new feature which has a lot of scope for improvement. To begin with, being able to specify the number of sub-shards to be created, as opposed to a fixed value of 2 would come in handy.
Also, as mentioned earlier, Shard Splitting only works for the case of auto-sharding or Hash based routing and having it work with “Custom Sharding” would be a great thing to have.
Allowing sub-shards to be created intelligently i.e. by splitting equal chunks from all shards in a collection for more equitable distribution of data could come in handy.
The newly created sub-shards have leaders on the same node as the parent. Providing a way to create sub-shards on new nodes directly or a mechanism to migrate sub-shard leaders to new nodes would be highly desirable too.
As of now, the parent shard is marked inactive after the shard split, in other words, no document updates or queries are sent to these shards. However, the data and the shards physically remain. There should be a mechanism to unload all of that and clean the shards up. The good thing is, the work for this is already in progress [SOLR-4693] and should be out with the next release of Solr.
The really amazing Solr Admin UI also needs to be changed to start respecting the state of a Shard, which was introduced only with 4.3. As of now, though the Shard becomes inactive it still shows up in green on the Solr UI. There’s already a JIRA on this [SOLR-4777].
The split HTTP API is a synchronous call with a large timeout (currently 5 minutes) but the operation can take longer on large indexes. Be aware that a read timeout on the HTTP call does not necessarily mean that the operation failed. The call is supposed to complete or automatically retry incase of a failure. We’d suggest not to retry it manually without looking at the logs and confirming. For the same reason, we’d also want better status reporting on the call.
Even though there is plenty of scope for improvement, the Shard Splitting feature should come in as a useful one for a lot of users with growing data.
Thanks to Shalin Shekhar Mangar for helping me out with this post.