Performance and monitoring
Once you have your OMERO server running and secured, a second critical step will be tuning various configuration parameters in order to get optimal performance. Assorted timeouts can be found under Performance but the more critical properties are outlined below.
The configuration properties starting with
omero.db control how OMERO manages JDBC
connections to your database. For a production system,
omero.db.poolsize is the most important property to
modify. By default, a limited number of simultaneous connections
(e.g. 10) are allowed. You should plan for allowing a few connections
per concurrent user.
$ omero config set omero.db.poolsize 100
OMERO should automatically configure itself to take advantage of the physical memory installed on a system whilst leaving room for other services. You may wish to override the defaults on a production server, for instance if your machine is solely dedicated to running OMERO you can increase the amount of memory that OMERO will use. You may also need to modify your settings if you are seeing out-of-memory errors when dealing with certain types of images.
A number of configuration properties starting with omero.jvmcfg control the calculation of how much memory to allocate to various OMERO services on startup, most importantly:
Configuration properties can either be applied to all three service
types at the same time by omitting the service type (e.g.
omero.jvmcfg.strategy) or to each individually by including it
For example, the default, PercentStrategy, is equivalent to making the call:
$ omero config set omero.jvmcfg.strategy percent
This could be changed to use the ManualStrategy for all servers:
$ omero config set omero.jvmcfg.strategy manual
A couple of strategies are available for calculating the effective JVM settings from the provided configuration properties.
Default. Reads the percent configuration property which can also be set globally or on a service-type basis. This percentage (0-100) of the system memory is used for the process, subject to minimum and maximum limits which can be changed.
omero.jvmcfg.max_system_memoryare all used for defining the system memory seen. The default percentages are: blitz and pixeldata 15%, indexer 10%. If
omero.jvmcfg.heap_sizeare explicitly set, they will be used directly as with the ManualStrategy.
Simply provides the values given as the JVM settings. If no value is set for a particular configuration property, then the default is used: heap_size=512m and perm_gen=128m These values are equivalent to the defaults in OMERO 5.0.2 and earlier.
$ omero config set omero.jvmcfg.percent.blitz 50
would raise the blitz heap size to 50% of the system memory seen.
$ omero config set omero.jvmcfg.system_memory 24000
would set the system memory seen to 24GB regardless of the actual amount of memory present in the system. The PercentageStrategy would use this as the basis for setting the Java heap sizes for all services.
$ omero config set omero.jvmcfg.max_system_memory 64000
would raise the maximum system memory seen by an OMERO installation to 64000MB of system memory. Assuming there was at least 64000MB of memory installed blitz would default to using 9600MB.
$ omero config set omero.jvmcfg.strategy.indexer manual
$ omero config set omero.jvmcfg.heap_size.indexer 2000
would set the indexer heap size to 2000MB without modifying the settings for the other services.
View the memory settings that will apply to a newly started server.
$ omero admin jvmcfg
After modifying any memory settings, be sure to restart your server.
$ omero admin restart
Forum thread on PixelData JVM memory settings
- Grid configuration
Section of the advanced server configuration documentation describing
In addition to watching the OMERO log files, the JVM itself provides a number of tools that you can use to determine the health of your server. JVisualVM, for example, can be used to visualize the memory use of each JVM:
You will need to have the PID for the service you want to monitor, which will usually be the main Blitz process. You can find the PID either via omero admin diagnostics or alternatively via the jps command found in the JDK.
Another tool, JConsole, also provides access to the memory statistics for your JVM, but also lists the JMX management beans which provide extensive information about the running process. Information includes the number of queries that have been run, the number of open file handles, the system properties that were set on startup, and much more. Further, the ome.system.metrics package makes use of JMX to expose further properties.
With further configuration, JMX properties can also be accessed remotely which can be very useful for monitoring your server with Checkmk, Nagios, Zenoss, or similar. However, care must be taken to protect the exposed ports.
The commands above require the Java JDK as opposed to the JRE.
Building on top of Coda Hale’s Metrics library, OMERO provides the ome.system.metrics package which measures a number of internal events and makes them available both via JMX as described under Monitoring but also prints them to the log files.
By default, these values are printed to each of the JVM-based log
etc) once per hour. This value can be configured via
omero.metrics.slf4j_minutes. A typical value might look like:
11:28:18,923 INFO [ ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.fulltext.FullTextIndexer.batch ...
Values include basic statistics (count, min, max, mean, etc.) as well as 75th, 90th, 95th, etc percentiles. Further, the rate over the last minute, the last 5 minutes, and the last 15 minutes is provided (m1, m5, m15). For example:
Useful metrics include:
The number and rate of errors that have been logged. (All services)
The ratio of used to available file descriptors. (All services)
The number of items in the queue. (Indexer-only)
Time taken to generate min/max values per plane. (PixelData-only)
Time taken to generate tiled-pyramids for a big image. (PixelData-only)