Considerations for a production Setup

Java Settings

Make sure you are using a server version of Java in production. The output of:

$ java -version

should include Java HotSpot(TM) Server VM or Java HotSpot(TM) 64-Bit Server VM. You can force the Java VM into server mode by calling it with the -server command. Do not try to run Solr with versions of OpenJDK or other non-official Java versions. They tend to not work well or at all.

Depending on the size of your Solr index, you need to configure the Java VM to have enough memory. Good starting values are -Xms128M -Xmx256M, as a rule of thumb keep Xmx double the size of Xms.

You can configure these settings via the java_opts value in the collective.recipe.solrinstance recipe section like:

java_opts =
  -server
  -Xms128M
  -Xmx256M

Monitoring

Java has a general monitoring framework called JMX. You can use this to get a huge number of details about the Java process in general and Solr in particular. Some hints are at http://wiki.apache.org/solr/SolrJmx. The default collective.recipe.solrinstance config uses <jmx />, so we can use command line arguments to configure it. Our example buildout/solr.cfg includes all the relevant values in its java_opts variable.

To view all the available metrics, start Solr and then the jconsole command included in the Java SDK and connect to the local process named start.jar. Solr specific information is available from the MBeans tab under the solr section. For example you’ll find avgTimePerRequest within search/org.apache.solr.handler.component.SearchHandler under Attributes.

If you want to integrate with munin, you can install the JMX plugin at: http://exchange.munin-monitoring.org/plugins/jmx/details

Follow its install instructions and tweak the included examples to query the information you want to track. To track the average time per search request, add a file called solr_avg_query_time.conf into /usr/share/munin/plugins with the following contents:

graph_title Average Query Time
graph_vlabel ms
graph_category Solr

solr_average_query_time.label time per request
solr_average_query_time.jmxObjectName solr/:type=search,id=org.apache.solr.handler.component.SearchHandler
solr_average_query_time.jmxAttributeName avgTimePerRequest

Then add a symlink to add the plugin:

$ ln -s /usr/share/munin/plugins/jmx_ /etc/munin/plugins/jmx_solr_avg_query_time

Point the jmx plugin to the Solr process, by opening /etc/munin/plugin-conf.d/munin-node.conf and adding something like:

[jmx_*]
env.jmxurl service:jmx:rmi:///jndi/rmi://127.0.0.1:8984/jmxrmi

The host and port need to match those passed via java_opts to Solr. To check if the plugins are working do:

$ export jmxurl="service:jmx:rmi:///jndi/rmi://127.0.0.1:8984/jmxrmi"
$ cd /etc/munin/plugins

And call the plugin you configured directly, like for example:

$ ./solr_avg_query_time
solr_average_query_time.value NaN

We include a number of useful configurations inside the package, in the collective/solr/munin_config directory. You can copy all of them into the /usr/share/munin/plugins directory and create the symlinks for all of them.

Replication

At this point Solr doesn’t yet allow for a full fault tolerance setup. You can read more about the Solr Cloud effort which aims to provide this.

But we can setup a simple master/slave replication using Solr’s built-in Solr Replication support, which is a first step in the right direction.

In order to use this, you can setup a Solr master server and give it some extra config:

[solr-instance]
additional-solrconfig =
  <requestHandler name="/replication" class="solr.ReplicationHandler" >
    <lst name="master">
      <str name="replicateAfter">commit</str>
      <str name="replicateAfter">startup</str>
      <str name="replicateAfter">optimize</str>
    </lst>
  </requestHandler>

Then you can point one or multiple slave servers to the master. Assuming the master runs on solr-master.domain.com at port 8983, we could write:

[solr-instance]
additional-solrconfig =
  <requestHandler name="/replication" class="solr.ReplicationHandler" >
    <lst name="slave">
      <str name="masterUrl">http://solr-master.domain.com:8983/solr/replication</str>
      <str name="pollInterval">00:00:30</str>
    </lst>
  </requestHandler>

A poll interval of 30 seconds should be fast enough without creating too much overhead.

At this point collective.solr does not yet have support for connecting to multiple servers and using the slaves as a fallback for querying. As there’s no master-master setup yet, fault tolerance for index changes cannot be provided.

SolrCloud

You can read more about the Solr Cloud effort which aims to provide this.