ClassCastException on Hibernate 4.3.x and Glassfish 4.x

I am attempting to utilize Hibernate 4.3.8 in a service that I am creating that will be running on Glassfish 4.1. When I attempt to

read an object from the db, such as the example below:

Product product = entityManager.find(Product.class, 980001);

The following exception is thrown

java.lang.ClassCastException: com.lynden.allin.service.Product cannot be cast to com.lynden.allin.service.Product

At first glance this may seem a bit strange since the 2 classes appear identical, and they are, but the issue is that there are 2 instances of the class being loaded by different classloaders. When the entityManager attempts to cast the class, it grabs a version that the service itself doesn’t know about since the service the reference that the service has was created from a different class loader.

After some searching it appears that this is a know issue with Hibernate 4.3.6 and newer:

The solution for the time being is to downgrade hibernate to 4.3.5 in order to avoid this issue in Glassfish.

twitter: @RobTerpilowski

Streamlining a Deployment Pipeline with a Custom Jenkins Plug-In

In a previous post (located here) I discussed setting up a deployment pipeline with Jenkins which would retrieve a specified version of one of our web applications from our Nexus repository and deploy it to a specified Glassfish server.  One of the challenges with the Jenkins job was that most of the fields on its deployment page were text fields, allowing users to free form text which was prone to errors.  The original deployment page appears below and allows the user to select the server and application via dropdown box, but leaves all other properties open to free form text.

image1.png (945×640)

The other drawback to the screen above was that each application needed to be listed in a text file which Jenkins would read  in order to populate the Application dropdown box.  This meant that there was a manual step involved with deploying an application that had not been deployed before.

With this in mind I decided to modify the Jenkins Dynamic Parameter Plugin to allow us to auto populate dropdown boxes with valid values based on our environment here at Lynden. I started with this plug-in since it already had functionality to change the contents of dropdown boxes based upon user selections. ie The selection of one dropdown could be used to determine the contents of another dropdown box.  The requirements for this modified plug-in were that it should:

  • Get a list of applications that are in our Nexus release repository
  • Get a list of versions of the selected application from Nexus.
  • Get a list of the application versions that are currently deployed to the Glassfish instance the user has specified for deployment.
  • Enable the user to select a web context to use when deploying the app, or select “Other” if they want to enter a context that does not appear in the list.

The Server dropdown box for this new Deployment plug-in allows the user to select which instance of Glassfish they would like to deploy to.  Each line in the dropdown contains the server name, Glassfish instance name, HTTP port, and admin port of the instance.


Once the user has selected a server, they can then select an application from the Application dropdown box.  For this field, the Jenkins plug-in goes out to our Nexus repository and retrieves the names of all of our applications, populating the list with these values.


Once the application has been selected, the user can select which version of the application to deploy.  This list of versions is also obtained from Nexus by the plug-in, which displays up to 25 of the most recent versions.


Once the user has selected the version to deploy, they can optionally specify a version to undeploy.  The plug-in populates this field by looking at the Glassfish instance that the user selected and querying it to determine if any versions of the specified application are currently deployed to it.  In the example below, HelloWorldWeb 1.1.5 and 1.1.4 are both currently deployed to the Glassfish instance running on “webstg”.  This field is optional, and if a value is selected, the Deploy task will first undeploy the specified version before deploying the new version.


Finally, the user can select a context to use when deploying the application, which is the portion of the URL after the hostname that the app server uses to identify requests for the application.  The two most common context names are included in the drop down, which are <app-name> and <app-name>-<version-number>.  So for example, if HelloWorldWeb-1.1.3 were selected below, the application would have a URL of


If a different application context is needed, the user can select “Other” from the dropdown box (in the screenshot above), and then enter an app context name in the “Other” text box below the Context drop down box.


At this point the user can select the “Build” button which will invoke an Ant script detailed in the previous post, downloading the binary from Nexus, undeploying the previously deployed version from Glassfish (if specified), deploying the new version of the application to Glassfish, and then saving information about the deployment to a database which can be viewed from a separate web application.


Since nearly all the fields are dynamically populated, there isn’t too much work for configuring the job in Jenkins.  Below is the Configuration screen for this deployment job.  There are 2 parameters which must be defined for this job.  The first is the environment this job will deploy to, ie. Test, Staging or Prod. The second is a list of valid Glassfish instances which can be deployed to.  The location of the maven repository and the location of the Glassfish command line interface are defined as system properties in Jenkins.


By using a Jenkins plugin to prepopulate the fields which are required for deploying an application we will streamline the deployment process and make it less prone to error.

twitter: @RobTerp

Creating a Deployment Pipeline with Jenkins, Nexus, Ant and Glassfish

In a previous post I discussed how we created a build pipeline using Jenkins to create application binaries and move them into our Nexus repository. (Blog post here)  In this post I will show how we are using Jenkins to pull a versioned binary out of Nexus and deploy to one of our remote test, staging or production Glassfish servers.  By remote I mean that the Glassfish instance does not live on the same box as the Jenkins CI instance, but both machines are on the same network.

In a previous post I also discussed how to set up Glassfish v3 to allow deployments pushed from remote servers (Blog post here), so if you haven’t explicitly configured your Glassfish to allow this feature, you will need to do so before you get started.

On our Jenkins CI box we have an Ant script, which will be executed by a Jenkins job manually kicked off by a user. The script defines tasks to perform the following operations:

  • Ensure all needed parameters were entered by the user. (app name, version number, admin username/password, etc).
  • Copy the specified version of the application from Nexus to a local temp directory for deployment to a remote Glassfish instance.
  • Undeploy a previous version of the app from the target Glassfish instance.  (Optional).
  • Deploy the app from the temp directory to the target Glassfish instance.
  • Record the deployment info in a deployment tracking database table.  Historical deployment info can then be viewed from a web app.

Ant Script

Below are some of the more interesting code snippets from our Ant script that will be doing the heavy lifting for our deployment pipeline.  The first code snippet below defines the Ant tasks needed for deploying and undeploying applications from Glassfish.  These Ant tasks are bundled with Glassfish, but not installed by default.  If you haven’t installed them, you will need to do so from your Glassfish update center admin page.

<taskdef name="sun-appserv-deploy" classname="org.glassfish.ant.tasks.DeployTask">
      <pathelement location="/nas/apps/cisunwk/glassfish311/glassfish/lib/ant/ant-tasks.jar"/>

<taskdef name="sun-appserv-undeploy" classname="org.glassfish.ant.tasks.UndeployTask">
      <pathelement location="/nas/apps/cisunwk/glassfish311/glassfish/lib/ant/ant-tasks.jar"/>

Once we have the tasks defined, we create a new target for pulling the binary from Nexus and copying it to a temporary location from where it will be deployed to Glassfish.

<target name="">
   <echo message="copying from nexus"/>    
   <get src="http://cisunwk:8081/nexus/content/repositories/Lynden-Java-Release/com/lynden/${}/${version.number}/${}-${version.number}.${package.type}" dest="/tmp/${}-${version.number}.war"/>

Next is a target to undeploy a previous version of the application from Glassfish.  This step is optional and only executed if the user specifies a version to undeploy from Jenkins.

<target name="undeploy.from.glassfish" if="env.Undeploy_Version">
   <echo message="Undeploying app: ${}-${undeploy.version}"/>
   <echo file="/tmp/gf.txt" message="AS_ADMIN_PASSWORD=${env.Admin_Password}"/>
   <sun-appserv-undeploy name="${}-${undeploy.version}" host="${}" port="${admin.port}" user="${env.Admin_Username}" passwordfile="/tmp/gf.txt" installDir="/nas/apps/cisunwk/glassfish311"/>
   <delete file="/tmp/gf.txt"/>

Next, we then define a target to do the deployment of the application to Glassfish.

<target name="" if="">
    <sun-appserv-deploy file="/tmp/${}-${version.number}.war" name="${}-${version.number}" force="true" host="${}" port="${admin.port}" user="${env.Admin_Username}" passwordfile="/tmp/gf.txt" installDir="/nas/apps/cisunwk/glassfish311" contextroot="${App_Context}"/>

And then finally, we define a target, which will invoke a server and pass information to it, such as app name, version, who deployed the app, that it can be recorded in our deployment database.

<target name="tag.uv.deploy.file">
      <format property="time" pattern="yyyyMMdd-HHmmss"/>

   <!--Ampersand character for the URL -->
   <property name="A" value="&amp;"/>
   <get src="${}${A}date=${time}${A}environment=${deploy.env}${A}serverName=${}${A}serverPort=${server.port}${A}adminPort=${admin.port}${A}serverType=${server.type}${A}version=${version.number}${A}who=${deploy.user}${A}contextName=${App_Context}" dest="/dev/null"/>

   <echo message="tagging deployment info to UV"/>

Jenkins Configuration

Now that we have an Ant script to perform the actions that we need to do a deployment, we set up a Jenkins job to deploy to servers in each one of our environments (test/staging/prod).

In order to kick off a deployment to one of our servers, the appropriate environment is selected from the screenshot above, and the “Build Now” link is clicked which presents the user with the screen below. In this case we are deploying to a test Glassfish domain named “bjorn” on

The user can select from the drop down list the server they wish to deploy to and the application they wish to deploy.  The version number is a required entry in a text field.  If the script can’t find the specified version in Nexus, the build will fail.  There are also optional parameters for specifying an existing version to undeploy as well as an application context in the event the app name shouldn’t be used as the default context.

In the screenshot below we are deploying version 5.0.0 of the Crossdock App to the “bjorn” domain running on


Once the job completes, if we log into the bjorn Glassfish admin page on, we see that Crossdock-5.0.0 has been deployed to the server.


The screenshot below is an example of undeploying version 5.0.0 of Crossdock, and deploying version 5.0.1 of Crossdock.  Also, in this example, we are telling the script that we want the web context in Glassfish to be /Crossdock, rather than the default /Crossdock-5.0.1


The screenshot of the Glassfish admin page below shows that Crossdock-5.0.0 has been unistalled, and that Crossdock-5.0.1 is now installed with a Context Root of /Crossdock.


Deployment History

Finally, as I mentioned previously, the Ant script is also saving the deployment information to a historical deployment table in our database.  We have written a simple web application which will display this historical data.  The screenshot below shows all of the applications that have been deployed to our test environment. (We have similar pages for Staging and Production as well).  Included in this information is the application name, version number, date, and who deployed it, among some other miscellaneous info.


We can then drill into the history of a specific application by clicking the “Crossdock” link on the screen above and get a detailed history about the deployments for that application including version numbers or dates.  We maintain more than 60 different web applications serving various purposes here at Lynden, so this has been a great tool us to see exactly what versions of our applications are currently deployed where, as well as see the history of the deployment of a specific application in the event we need to roll back to a previous version.


As we have learned firsthand, Jenkins is a very useful and versatile tool that is easy to extend for purposes beyond automated builds and continuous integration.

twitter: @RobTerp

Migrating an Automated Deployment Script from Glassfish v2 to Glassfish v3

Just recently we attempted to update an Ant script which we are using to do automated deployments to Glassfish v2 servers, to deploy to our new Glassfish v3 servers instead.  This Ant script is invoked from our Jenkins automated deploy pipeline process (another blog post about this later) and copies a .war file from our Nexus repository and installs it to a Glassfish instance running on a remote server.  I ran into a number of issues that prevented the Ant script from being used “as-is”.

The first thing that this assumes is that there is a Glassfish v3 server running on the build machine from which this script is executed.  The build script needs access to the “asadmin” tool in the glassfish/bin directory in order to deploy a .war file to a remote glassfish server.

The first issue was that the Ant deploy/undeploy tasks for Glassfish are in a different location in v3.  Actually, not only is the .jar file with the Ant tasks in a different location in the newer version of Glassfish, its not installed by default!  In order to get the Ant task you’ll need to go to the Update Tool on the admin web page of your Glassfish v3 instance and install the “glassfish-ant-tasks” component.  Once you’ve done that then you can modify your Ant script to use the new Ant tasks (which are also located in a different Java package also).  The code snippets from an Ant script below compare the differences between the Glassfish v2 and v3 ant task usage.

<!—Glassfish v2 ant tasks –>

<taskdef name=”sun-appserv-deploy” classname=””> <classpath> <pathelement location=”/nas/apps/cisunwk/glassfish/lib/sun-appserv-ant.jar”/> </classpath> </taskdef>

<!—Glassfish v3 ant tasks –>

<taskdef name=”sun-appserv-deploy” classname=”org.glassfish.ant.tasks.DeployTask”> <classpath> <pathelement location=”/nas/apps/cisunwk/glassfishv3/glassfish/lib/ant/ant-tasks.jar”/> </classpath> </taskdef>

The next change that needs to be made to the build script is the call to the sun-appserv-deploy task itself.  The “”installDir” property has been changed to “asinstalldir” for Glassfish v3.  A comparison of the v2 vs. v3 code snippets are below.

<sun-appserv-deploy file="/tmp/${}-${version.number}.war"
name="${}-${version.number}" force="true" 
host="${}" port="${admin.port}" user="${env.Admin_Username}" 
passwordfile="/tmp/gf.txt" installDir="/nas/apps/cisunwk/glassfish"/>

<sun-appserv-deploy file=”/tmp/${}.war”

name=”${}” force=”true” host=”${}” port=”${admin.port}” user=”${env.Admin_Username}”

passwordfile=”/tmp/gf.txt” asinstalldir=”/nas/apps/cisunwk/glassfish311″/>

The final task in getting this to work is to enable remote commands on each of the target glassfish instances to which the apps will be deployed.

./asadmin enable-remote-admin –port <glassfish-admin-port>

where <glassfish-admin-port> is the admin port that domain (4848 by default).

The last “gotcha” to keep in mind here is that the glassfish that is defined in the <sun-appserv-deploy> task with the “asinstalldir” property MUST be the same version as the remote target Glassfish instances where the web app will be deployed to.  At least this was the case when we attempt to specify a Glassfish v3.0 instance in the installDir property, deploying to a Glassfish v3.1 remote instance.  When we updated the installDir to point to a v3.1 instance, the deployment went fine. (v3.1 to v3.1.2 may be OK)

Hopefully this will help others that have encountered similar issues while attempting to do deployments to remote Glassfish instances either from Jenkins, Ant or other automated processes.

Twitter: @RobTerp

Hacking ActiveMQ to Retrieve Connection Statistics

This hack is so awful that I am almost embarrassed to blog about it, but this was the only way to obtain the information we needed from ActiveMQ to display in an MBean in JConsole.

We have written a Java proxy server to allow Window CE devices running a C# application to connect to our JMS message broker.  The proxy server runs within a Glassfish instance to make managing its lifecycle a bit easier.  For system diagnostics and debugging purposes we created an MBean to be accessed via JMX so we could get real-time statistics from the proxy through JConsole or VisualVM, such as # of clients connected, message counts, as well as total connections open from the proxy to the broker.

The number of connections that the proxy had opened to the broker seemed like it *should* be a fairly straightforward task.  Enable statistics on the ActiveMQConnectionFactory instance by calling setStatsEnabled(true) and then retrieve the connection statistics from the factory via a call to the getStats() method, and then get the total number of open connections. However, no matter what we did, the getStats() call always returned null, and for good reason, upon inspecting the ActiveMQConnectionFactory source code we found:

218    public StatsImpl getStats() {
219        // TODO
220        return null;
221    }

In the ActiveMQConnectionFactory class there resides a field called factoryStats of type JMSStatsImpl.  The field is protected and there are no get() methods to retrieve this field, so we had to resort to drastic measures if we really wanted to know how many open connections the proxy had open to the broker.  The code to get the connection information from the factory appears below.

    public int getBrokerConnectionCount() {

        int count = -1;
        try {
            Field factoryStatsField = ActiveMQConnectionFactory.class.getDeclaredField("factoryStats");
            JMSStatsImpl factoryStats = (JMSStatsImpl) factoryStatsField.get(connectionFactory);
            count = factoryStats.getConnections().length;
        } catch (Exception ex) {
            logger.error("Attempted to get Broker connection count, but failed due to: " + ex.getMessage());
        return count;

Even though the field is protected, reflection can be used to set the field to “accessible” and then the value of the field can be read directly from the factory object.  This is not guaranteed to work in future releases of ActiveMQ, but since this is only for informational purposes in JConsole this was good enough for what we needed.

Twitter: @RobTerp

Debugging Web Service Issues When Migrating from Glassfish Version 2 to 3.

For the most part our migration of web applications (including web services) from Glassfish version 2 to version 3 has been relatively painless. We did however encounter a strange issue with one of our server apps that has both a web service and an RMI server component. The application has been running without problems on Glassfish 2, but when we attempted to deploy to Glassfish 3 we encountered the error message shown in the screenshot below:

“Error occurred during deployment: Exception while preparing the app: Servlet CarrierBillWS implements 2 web service endpoints but must only implement 1”


The initial thought was to go ahead and migrate the application from Java EE 5 to Java EE 6, and hope that corrected the issue. This however did not change anything and we were left with the same error. Googling didn’t produce any useful results so I decided to take a look through some of the glassfish code where the exception was being thrown from.

         WebServicesDescriptor webServices = bundle.getWebServices();
          Collection endpoints =

         if( endpoints.size() > 1 ) {
             String msg = "Servlet " + getWebComponentLink() +
                     " implements " + endpoints.size() + " web service endpoints " +
                     " but must only implement 1";
             throw new IllegalStateException(msg);

It appears that there could possibly a name collision with classes in the application, which wasn’t being checked for in Glassfish 2. The web service class itself is pretty simple:

public class CarrierBillWS {

    protected CarrierBillWSActor actor;
    protected Logger logger;

    public CarrierBillWS() {
        logger = Logger.getLogger(getClass());
        actor = new CarrierBillWSActor(PropertyManager.getInstance().getProperty(PropKey.UV_SESSION));

    @WebMethod(operationName = "reopenBill")
    public void reopenBill( String dispatchId, String locationCode ) { "Web service Request received to reopen carrier bill: " + dispatchId + " from location: " + locationCode );
        actor.reopenBill(dispatchId, locationCode);


For fun, I commented out the @WebService annotation and deployed the application without any issues, with the obvious result being that there was no CarrierBillWS web service listed in the Glassfish admin console, but I was at least able to deploy. I then did a grep for “CarrierBillWS” and to my surprise found another class with the same name in one of the web service app’s dependencies. This dependency is a .jar file that is shared across multiple applications here at Lynden, including the web service. So, why was there a class with the same name in this library? It was because the library also contained code for a client of this CarrierBill web service which other applications depended on for communicating with the service. The CarrierBillWS application itself depends on this .jar file for other functionality that is shared across multiple apps. When the jar was built, it pulled down the WSDL from the web service and auto generated the client side classes, which included a CarrierBillWS client side class. This .jar was then bundled in the web service’s .war file, and Glassfish was finding both classes and then complaining.

The solution to the problem was pretty simple; override some of the default values of the @WebService annotation, and then rebuild the dependency using a new WSDL.

@WebService(serviceName="CarrierBillWebService", name="CarrierBillWebService")
public class CarrierBillWS {

The new “serviceName” and “name” properties on the WebService annotation will cause the client project to generate code with a class name of CarrierBillWebService rather than CarrierBillWS. When the .jar is built, the Web Service and its client related classes in the dependent jar now have different names and Glassfish v3 loads the application without any issues.

Since there wasn’t much info out there on this particular issue, I’m hoping that this may help someone in the future who encounters a similar problem.

Twitter: @RobTerp