Introduction
I have just comeback from a visit with a customer who had a rather urgent requirement to trace IIOP calls from a Swing application to Enterprise JavaBeans deployed in a Sun Application Server and then from the Sun EJB's calls to Enterprise JavaBeans deployed in a JBoss container followed by remote procedure calls to a proprietary application server running a helpdesk and service management system. JXInsight had already shipped distributed trace capabilities for the IIOP leg via the CORBA PortableInterceptors trace extension. The calls from the EJB's within the JBoss container to the proprietary application server could be traced via a special transport protocol trace extension we previously developed for the vendor. The only leg in this distributed execution not being traced where the calls from the EJB's deployed in the Sun server to the EJB's deployed in JBoss server.

Fortunately JBoss does have client-side and server-side invocation interceptors that largely resemble CORBA PortableInterceptors including the ability to insert contextual information into the invocation payload (service context in CORBA). After creating an class implementing org.jboss.ejb.plugins.Interceptor, extending org.jboss.proxy.Interceptor , routing preinvoke and postinvoke events to the JXInsight Tracer, inserting client trace stacks into the payload for remote invocations, as well as adding useful configuration options (trace identifiers) we were able to provide a complete distributed trace across the 4 JVM's with aggregated statistics for clock time, cpu time, object allocations, GC, thread monitor blocking and waiting. The JXInsight console was able to show the resulting database transactions from a single mouse click on a button in the Swing application which was 4 JVM's removed from the database tables and columns that the transaction worked was performed on.

Impressed? Interested? Yes, then begin downloading the latest JXInsight installers from the downloads section while reading the following instructions detailing the configuration steps required for this new distributed trace extension. I have chosen to use the the Sun J2EE™ Samples Application - Dukes Bank. Instructions and additional code and configuration for building, deploying and running the Dukes Bank application with JBoss application server can be downloaded from the JBoss web site.

Step 1
Download, build and deploy the Dukes Bank Application on the JBoss Application Server

Step 2
Perform the JXInsight integration steps described in JBoss Integration Guide

Step 3
Add the JXInsight client and container interceptors to the all relevant containers defined in standardjboss.xml file. The interceptor class, com.jinspired.jdbinsight.trace.ext.jboss.client.EJBTraceInterceptor, should be added to the interceptor stacks for all relevant invoker-proxy-bindings. The interceptor class, com.jinspired.jdbinsight.trace.ext.jboss.container.EJBTraceInterceptor, should be added to the interceptor stacks for all relevant container configurations. The following screen shots show XML snippets of both client and container interceptor configurations.

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE Public trace extension classes have been migrated from com.jinspired.jdbinsight.trace.* to com.jinspired.jxinsight.trace.*. For backward compatibility the classes in com.jinspired.jdbinsight.trace.* still remain though in most cases they merely extend from the migrated versions in com.jinspired.jxinsight.trace.*. The JDBInsight Driver and DataSource classes remain unchanged. For JBoss EJB Interceptor please make the following configuration change. RENAME com.jinspired.jdbinsight.trace.ext.jboss.client.EJBTraceInterceptor TO com.jinspired.jxinsight.trace.ext.jboss.client.EJBTraceInterceptor RENAME com.jinspired.jdbinsight.trace.ext.jboss.container.EJBTraceInterceptor TO com.jinspired.jxinsight.trace.ext.jboss.container.EJBTraceInterceptor

Step 4

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE The old library jdbidriver.jar has been renamed and partitioned into multiple jars to support the class loading of particular trace extension into child class loaders which is typical within J2EE application server and web container classloader hierarchies. When deploying a jxinsight-ext-*.jar library the jxinsight-core.jar must be deployed instead of jxinsight-all.jar. The jxinsight-core.jar must be visible to all trace extensions via the same classloader or a direct/indirect parent classloader. For backward compatibility the jxinsight-all.jar (previously named jdbidriver.jar) is built with all JXInsight and JDBInsight classes and resources.

To record distributed tracing from client application to server with active client trace stacks inserted into the invocation payload the client JVM required the jxinsight/lib/jdbidriver.jar to be added to the class path as well as the jdbinsight JVMPI native agent library to be made available on the library path and loaded on client startup via the JVM argument -Xrunjdbinsight. The build.xml shipped with the Dukes Bank application contains a target used for running the client. The target was changed to the following:

IMPORTANT JXINSIGHT 4.0 CHANGE NOTE All system properties have been changed from jdbinsight.* to jxinsight.*. For backward compatibility existing jdbinsight.* properties will be recognized though it is strongly recommended that all customers upgrade their existing configurations.

To easily identify the client JVM process in the JXInsight console the system property, jdbinsight.server.tracer.root, included a name-value pair which labeled the process as 'swingclient'. A similiar property was added to the JBoss run script to label its process as 'jboss'.

Step 5
Run the JBoss run script (with the integration changes). Run the Ant target 'run-client'. Select 'Customer Admin' -> 'View Customer Information' -> '200' > 'OK'.

Step 6
Startup the JXInsight console. Create a new server cluster called 'cluster' adding servers localhost:1515 and localhost:1516. Click on the cluster and invoke the selection menu 'Snapshot'. Switch from the 'Transactions' perspective to the 'Traces' via the drop down menu in the top right hand corner of the console. The traces table displayed contains columns for local and remote statistics for clock time, cpu time, object allocations, gc, thread monitor blocking and waiting.



Step 7 (Optional)
Turn on the timeline analysis mode for each server within the cluster by selecting each one and then executing 'Selected' -> 'Monitoring' -> 'Timeline' (check) -> 'Finished' . Execute an operation in the Dukes client application. Select the cluster in the JXInsight console and switch to the timeline analysis mode. Select a client trace execution block in the timeline mode. The following screen shot shows a client trace selected with a slightly shaded area representing the time window with the associated server trace as well as the transaction (small bracket above server trace execution block).




I hope that this short introduction to distributed tracing for JBoss has proved valuable in understanding the required configuration steps as well as the inherent extensibility of the JXInsight trace and profile solution. Additional information on the various system properties associated with the JBoss trace extension can be found here.

William Louth, JXInsight's Product Architect