Remote JVM profiling via SSH tunnels

This document provides a java class and instructions on performing JVM monitoring via JMX, where the monitoring client connects to the JVM via SSH tunnels.

Rationale

Using JMX on a firewalled server can be difficult due to the unpredictable second port that the client must connect on. Advice is sometimes given that ports 1024-65535 should be opened up to the machine that will be running the JMX client (such as jconsole). But this may not be an acceptable solution:

Policy may prevent opening so many ports, even to a single trusted system.
The traffic may not be encrypted. Or if it must be encrypted, it will be necessary to configure the connection to use SSL.

Another solution is to install a customized RMI agent that uses a specific second listening port. This permits the JMX client to access the JVM via SSH tunnels. No additional ports through the firewall need be opened, and the traffic will be encrypted.

Installation

The code for this agent is provided as an attachment to this page. It may be added to the Sakai source tree, and installed with Maven.

Installation consists of the following steps:

Extract the code into the Sakai source tree.
Install the custom RMI agent class.
Modify Tomcat's startup.sh script to set options that enable the JMI.

That suffices to get the JMI up and running. See "Usage" below, for information on how to set up SSH tunnels and use the client.

Custom RMI agent class

Download, then extract and install:

cd $SAKAI_SOURCE
tar xzvf ~/tunneling_jmx.tar.gz
cd jmx
mvn -Dmaven.tomcat.home=$CATALINA_HOME install sakai:deploy

Tomcat's startup script

The file $CATALINA_HOME/bin/startup.sh must be modified to activate the JMI.

At this point you must identify two free ports (greater than 1024) on the server, which will be used for the JMI. The second port must also be free on the client, since its tunnel must use the same port on both ends.

Call them JMX_PORT and RMI_PORT. The default for the first port is 1099. There is no default for the second port, but adding 1 to the first port is reasonable.

The following options must be set in CATALINA_OPTS in startup.sh:

Option	Value	Description
com.sun.management.jmxremote.port	$JMX_PORT	The primary listen port for JMX.
com.sun.management.jmxremote.authenticate	false	Don't use any authentication method. Anyone who connects can use the JMI.
com.sun.management.jmxremote.ssl	false	Don't use TLS.
edu.ucmerced.jmx.tunnelingrmiagent.port	$RMI_PORT	The secondary listen port (for RMI). IMPORTANT: This port must be the same as the port opened up on the client's side of the tunnel.
java.rmi.server.hostname	127.0.0.1	Indicates local host.
javaagent	$CATALINA_HOME/shared/lib/tunneling-rmi-agent-M2.jar	Loads our custom agent class.

Example:

Here is an example of lines added to startup.sh. There are four lines. They can go anywhere above the final "exec".

JMX_PORT=1099
RMI_PORT=1100
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Dedu.ucmerced.jmx.tunnelingrmiagent.port=$RMI_PORT -Djava.rmi.server.hostname=127.0.0.1
-javaagent:$CATALINA_HOME/shared/lib/tunneling-rmi-agent-M2.jar"
export CATALINA_OPTS

Observe catalina.out as you restart Tomcat. The first line should state:

Starting tunneling RMI agent on port <port>

Tomcat will then be ready for JMI connections.

Usage

This section describes opening SSH tunnels and starting jconsole. The provided "jmx" script can perform these steps automatically.

Select a client machine with SSH access to the server.

You must set up two tunnels: one to each of the listen ports.

Select a free port on the client system. Name it CLIENT_PORT_1. The second port, named CLIENT_PORT_2, must be the same as RMI_PORT.

Then construct the tunnel commands as follows:

ssh -f -N $TOMCAT_HOST -L $CLIENT_PORT_1:127.0.0.1:$JMX_PORT
ssh -f -N $TOMCAT_HOST -L $CLIENT_PORT_2:127.0.0.1:$RMI_PORT

Furthermore, you will need a JMX service URL to give to jconsole. Its form is:

service:jmx:rmi://localhost:$CLIENT_PORT_1/jndi/rmi://localhost:$CLIENT_PORT_2/jmxrmi

This URL can be passed on the command line, or provided in the "Advanced" tab of the connection dialog.

Examples

In these examples, I will use port 9000 on the client. The server is sakai.ucmerced.edu.

Creating tunnels:

ssh -f -N sakai.ucmerced.edu -L 9000:127.0.0.1:1099
ssh -f -N sakai.ucmerced.edu -L 1100:127.0.0.1:1100

Running jconsole:

jconsole service:jmx:rmi://localhost:9000/jndi/rmi://localhost:9001/jmxrmi

Again, note that for the RMI connection, the port on the client must be the same as the one on the server. In this case, they are both 1100.

jmx script

The provided jmx script can be used to automatically open SSH tunnels and start jconsole for monitoring specific JVMs. See README.jmx in the attached archive.