Reimagining the Solr Reference Guide

The Apache Solr Reference Guide is considered the official documentation for the Apache Solr project. Officially, it’s released as 700+ page PDF, but an online version has been available for a long time in the Apache Confluence instance, aka “cwiki”.

Soon the Lucene/Solr community will reintroduce the Solr Reference Guide in a new location, published with a completely new model.

Why Change

There are a lot of issues with trying to manage something the size of the Solr Reference Guide with Confluence, and they primarily center around control and access.

Confluence is designed to be a collaboration tool, but the Lucene/Solr community decided that for us to produce authoritative documentation, we should limit editing to committers only. Non-committers, who are the primary audience and users of the documentation, are shut out of this process, except to make comments on pages. Yet there are some very knowledgeable and talented users of Solr who aren’t committers. Wouldn’t it be nice if it was easier for them to participate more?

A second issue with the current approach is that we’re only able to keep one version online at a time.  This means the online version is always for the next release of Solr. At the same time, we believe online is the main way the Guide is consumed. Wouldn’t it be nice if we could have an online version tied to every release?

There’s also a disconnect between code in one place and documentation in another. This means documenting new features is divorced from development of features. That places extra burdens on the developer and frequently means someone else is trying to write docs for code they didn’t write. Wouldn’t it be nice if documentation changes could be made at the same time code changes are committed?

Finally on the short list of issues (there’s definitely a longer list!), we’re basically stuck with the look and feel of Confluence, unless we want to try some drastic and invasive changes that would break with every system upgrade. Here’s what the old Guide looks like today:

Wouldn’t it be nice if we could easily redesign that as needs change?

What’s Happening

Back at the end of August 2015, I started a side project to see if I could migrate the existing Solr Reference Guide to a new model. It sat on my back-burner for a long time, but last year I took it up in earnest, enlisting the help of my colleagues for some of the thornier technical issues. The proposed change has been accepted by the Lucene/Solr community, and we’re well on our way to making it happen. In fact, for Solr 6.6, the Solr Reference Guide will be published in this new model.

So, what are we changing? First, we will launch the Solr Reference Guide at a new online location (https://lucene.apache.org/solr/guide). The Guide will be versioned for each release of Solr, so you’ll always be able to access the Guide for Solr 6.6, or 7.0, or 7.1, etc., at a stable URL (like the Javadocs today).

The PDF will remain the “official” version of the Guide, and will be accessible from the online version for download just as it is today. However, when a new release of Lucene/Solr occurs, we’ll publish the Guide for that release online.

The new Guide is integrated with the Apache Comments system, so while existing comments in Confluence will not be carried over, you’ll still be able to comment on pages, to share feedback or tips with other Solr users.

If you notice a problem, and you have the Lucene/Solr source, you will be able to submit a patch for it to be reviewed and committed to the docs for future releases. If you think there should be new docs on a topic, you can write them!

We’ll also have full control of all aspects of the look and feel, and the new Guide will look much different, with a more dynamic design that will work better on smaller screens. Here’s what the new Guide will look like:

Behind the Scenes

Now some technical details of how we’ve made this happen. The current pages you see in Confluence will be converted to AsciiDoc format. This is a text markup format, very similar to Markdown but with additional extensions for flexibility. Specifically, we’re using tools from Asciidoctor in our build processes, so we’re focused on that project’s particular flavor of AsciiDoc to ensure compatibility.

For the online version, we’re using Jekyll to make the HTML pages. Jekyll is a static site generator, which takes our raw AsciiDoc-format files and applies templates to them to make a full site. This separation of content from presentation provides a lot of power, flexibility, and control over the documentation experience.

The raw AsciiDoc-format files will live in the Lucene/Solr source code, in a new directory “solr-ref-guide”. For now it has it’s own build.xml file to build the PDF and HTML versions (at the same time, or separately).

I’ll write some more posts in the future to introduce writing in AsciiDoc and submitting patches for documentation.

Timing

We have all of the big things we wanted to have in place before cutting over to this process: build scripts, Jenkins jobs, decisions about process, etc. The work currently exists in a branch from the main Lucene/Solr branch, and we’re tracking all the work in sub-tasks to SOLR-10290. You can take a look at the branch in the Git repo if you are more curious.

This weekend (starting 5 May), Confluence will be locked to all edits while we export the content and migrate it. About 90% of the migration is scripted, but there are a few things that just need a human being to resolve. My hope is this effort only takes a few days, and then we will start updating the content for Solr 6.6, which is planned for release in the next few weeks.

Stay tuned, there will be more announcements coming soon!

Some brief acknowledgements: While I kicked off the effort, we wouldn’t be this far along without the help of several people in the Lucene/Solr community as well as my colleagues and managers at Lucidworks. I’d like to especially thank the Lucidworks Fusion documentation team for blazing the trail and sharing their experience and some scripts to get us started.