Distributed APM Agents
 


Introduction
JXInsight's agent architecture has been designed to support practically all deployment topologies for client applications and servers. JXInsight's management console and terminal tools connect directly into agent processes for real-time snapshot retrieval but they can also perform offline snapshot aggregation , inspection and analysis.

JXInsight's trace and profile extensions can work in environments consisting of different middleware and communication technologies including JMS, CORBA, RMI, HTTP and other proprietary protocols. An unique feature of the solution's architecture is the ability to trace and profile both client and server JVM processes without having each individual process connected to a central management server process. Importantly JXInsight correlation of distributed traces from client to server(s) can be performed offline. The benefits of the JInspired JXInsight's solution include:


Security
: There is no need to open up ports between the monitored processes and the management server so as to profile each JVM runtime.

Communication: Customers can use existing communication protocols (HTTP, FTP, JMS) to transfer performance snapshots from agents to shared directories accessible to JXInsight management consoles and terminals. Access authorization can be delegated to underlying platform file system security.

Fault Tolerance
: With each agent having a high degree of autonomy there is no single point of failure. Multiple managment consoles and terminals can be run to routinely monitor, analyse, and store profiling snapshots. Snapshots recording for particular partitions can be replicated across consoles and terminals.

Partitioning: Unlimited flexibility in partitioning the management domain across operations staff. Management consoles and terminals can be configured to monitor different client and application server processes. The JXInsight management console can aggregate snapshots into clusters containing arbitrary sets of application client and application server processes.

Distributed Tracing: Distributed tracing can be performed offline and does not require a central management server to perform real-time trace identification generation, correlation, aggregation and association. JXInsight's approach is so flexible that is possible to correlate traces at various levels of granulatity: cluster, host, process, thread group, thread, request parameters, request timestamp, trace context stacks, component or object instance, etc. JXInsight unlike other management systems does not require an unique trace identifier to be generated and maintained in memory across multiple processe and to be resolved by a central system. Tracing across JVM processes can even be performed by simply passing system properties from one process to another via the command line [Ant/JUnit Example].


To show how this can be applied in practice I am going to run the CORBA sample application which is located in the /samples/corba directory in the JXInsight installation root. To (re)build the application classes you will need Apache Ant installed and available on the system path. The application is very simple with a single CORBA interface providing a function that returns the addition of two numeric parameters. Here is the CORBA IDL definition.

module add {
   interface Adder {
       long add(in long a, in long b);
   };
};

To build the application navigate to the samples/corba directory in a shell and execute the build target in the build.xml file
> ant build

Note: By default the CORBA client and server processes use the ORB that ships with the Sun JDK but it is possible to configure another ORB such as Borland VisiBroker for Java.


Naming Service
Before starting the CORBA server and client you will need to start a naming service used for registering and locating CORBA server objects. The naming service can be started by executing the naming.bat script located in the /samples/corba. The ant build.xml defines a property, orb.port, that is used by both CORBA client and server processes to locate the naming service. The script executes ant runnaming.




CORBA Server
Following the successful execution of the naming service the CORBA server can be started by executing the server.bat script. The script file executes ant runserver after appending the native agent library to the system library path. The runserver target executes a Java process with the following JVM arguments.

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE
Public trace extension classes have been migrated from com.jinspired.jdbinsight.trace.* to com.jinspired.jxinsight.trace.*. For backward compatibility the classes in com.jinspired.jdbinsight.trace.* still remain though in most cases they merely extend from the migrated versions in com.jinspired.jxinsight.trace.*. The JDBInsight Driver and DataSource classes remain unchanged.

For the CORBA Distributed Tracing Extension please make the following configuration change.
RENAME com.jinspired.jdbinsight.trace.ext.corba.ORBInitializer
TO com.jinspired.jxinsight.trace.ext.corba.ORBInitializer




The JXInsight agent is loaded by the -Xrunjdbinsight argument. The JXInsight CORBA distributed trace extension is installed by registering a ORBInitializerClass.


Run Client
After the CORBA server has been started and the object registered in the naming service the CORBA client can be started by executing the client.bat script. The script file executes ant runclient after appending the native agent library to the system library path. The runserver target executes a Java process with the following JVM arguments.



The JXInsight agent is loaded by the -Xrunjdbinsight argument. The JXInsight CORBA distributed trace extension is installed by registering a ORBInitializerClass.

For each Adder.add(...) request dispatched the client will print out a line to the shell with the randomly defined parameters values and result. The client will continue indefinitely.

Looking at the source code for both client and server you will see that no references are made to JXInsight's libraries. The instrumentation of distributed tracing is completly transparent to both client and server. This transparency is achieved via CORBA's PortableInterceptors.


Shutdown
To show the offline analysis capabilities of the JXInsight solution lets shutdown the client and then the server JVM's. To mimic a JVM shutdown and highlight an useful developer capability of the console I mounted both processes (server=localhost:1515, client=localhost:1516) within the console and then executed the Shutdown menu command from the Selected drop down menu. The Shutdown action invokes the java.lang.System.exit() call in the remote JVM. The CORBA client was shutdown and then the CORBA server was shutdown. The following screen shot shows that the processes are no longer available to the management console.




Automatic Export of Snapshot on Shutdown
By default all JXInsight servers (JVMPI agent and profiler or tracer) will export snapshots (profile, metric, and timeline) on shutdown of the JVM container. The location of the exported snapshots is configured via the system property

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE
All system properties have been changed from jdbinsight.* to jxinsight.*. For backward compatibility existing jdbinsight.* properties will be recognized though it is strongly recommended that all customers upgrade their existing configurations.

-Djdbinsight.server.shutdown.exportsnapshots.dir=${user.dir}

Additional system properties related to snapshot exportation.
jdbinsight.server.shutdown.exportsnapshots.enabled=true
jdbinsight.server.shutdown.exportsnapshots.profile.include=true
jdbinsight.server.shutdown.exportsnapshots.metric.include=true
jdbinsight.server.shutdown.exportsnapshots.timeline.include=true
jdbinsight.server.shutdown.exportsnapshots.fileprefix=${host}.${process}.${timestamp}

After shutdown of both the CORBA client and server processes there should 4 snapshot files located in the samples/corba directory. A profile and metric snapshot for each JVM. As the JXInsight installation root directory is mounted in the management console we can immediately navigate to the directory in the console browser tree and load each profile snapshot file activating the Traces perspective.

The Traces Table view in the Traces perspective for the CORBA client profile snapshot is shown below. Both the Count and Clock Total statistics are present but there are no values for Count (R) and Clock Total (R) which relate to costs incurred in external processes. The clock total for the client side processing is 598 ms.



The Traces Table view in the Traces perspective for the CORBA server profile snapshot is shown below. The trace stack for the invocation on the server implementation is prefixed with the the client trace stack that was active following the interception of invocation on the stub (proxy). Client traces are delimited by pairs of ${client.trace}=< and ${client.trace}=> trace identifiers. The host trace iiophost has values for Clock (R) and Clock Total (R) because the console was able to match this identifier with the first trace identifier in the client trace. The clock total for the server side processing is 47 ms.




Snapshot Merging - Inspired
By looking at each individual snapshot in the console we can determine that the client side processing took 598 ms of which 47 ms related to the server side processing. But switching between snapshots and noting statistical values is not very efficient if not impractical for any reasonably sized J2EE/CORBA application. The solution is very simple but extremely powerful - merge both the client and server snapshots into a new snapshot.

Select the Merge Snapshots... menu item from the File menu. Navigate to the samples/corba directory selecting both client and server profile snapshots and clicking Finish.



After loading and merging the selected snapshots select a directory and enter a file name in the Export dialog for the merged snapshot. Click the Finish button to save the snapshot and then load the newly created snapshot in the console.



The Traces Table view in the Traces perspective for the merged profile snapshot is shown below. The client trace sub tree /iiophost/1516 now shows values for Count (R) and Clock Total (R). This is calculated by scanning the profile model for client trace subsequences matching a tree path (root to node). In the example the tree paths /iiophost, /iiophost/1516, /iiophost/1516/IIOP/, /iiophost/1516/IIOP/add where found prefixing a client trace appearing under the tree path /iiophost/1515/.



This elegant solution is extremely powerful in providing distributed tracing offline analysis with configurable levels of granularity. The granularity is determined by configuration of the trace root system property and the trace identifier system properties for the JXInsight trace extensions involved in the execution of a client request spanning multiple JVM's.

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE
All system properties have been changed from jdbinsight.* to jxinsight.*. For backward compatibility existing jdbinsight.* properties will be recognized though it is strongly recommended that all customers upgrade their existing configurations.

Fine Grain Distributed Tracing: By appending the trace timestamp to the trace identifier for the CORBA request it is possible to trace a particula CORBA request across multiple JVM's with aggregated memory allocation, cpu time, and clock time statistics.
-Djdbinsight.server.tracer.ext.corba.clientrequest.trace.identifiers=${corba.protocol}.${corba.operation},${trace.timestamp}

Coarse Grain Distributed Tracing: By changing the default trace root system property in multiple client JVMs to include a string literal and removing host and process information from the trace root it is possibly to view distributed trace statistics representing lets say the name of an client application or user group.

-Djdbinsight.server.tracer.root=servicedesk
-Djdbinsight.server.tracer.root=supportgroup

For deeper insight into the configuration of JXInsight trace extensions please read Configuring Servlet Traces.


And Finally - A Console Tip
A feature of the Traces perspective is the ability to select a trace identifier in the main Traces Table view and see the matched client traces in the Remote Traces Table view. Double clicking on a row in the Remote Traces Table view will result in the main Traces Table view expanding its tree nodes and making the matched client trace sequence the selected row.




William Louth, JXInsight Product Architect