Powered by Jetty

One of our customers had been running into some trouble with one of  their Fisheye instances after we performed a recent upgrade. They were getting java.lang.OutOfMemoryError errors with no specific cause, such as permgen.  Initially we thought that due the size of the instance, and possibly some new features, it might just require a bigger foot print but after some investigation we tracked down that it was actually a bug in the Jetty container that comes packaged with Fisheye (see http://jira.atlassian.com/browse/FE-2477 for details).

The customer wanted us to go ahead and try upgrading the bundled container.  While this is something we hadn’t ever attempted (and probably is not a recommended practice except where warranted), we serve as the IT department for our customers Internet infrastructure and are always up for a challenge.  We started looking into what was required to make this happen.

Well it turns out that this is actually rather easy.  So easy in fact that all you have to do is replace 3 jars from the installation.

Here is the basic rundown:

Go get the new version from Codehaus at http://dist.codehaus.org/jetty/jetty-6.1.23/.  Grab these 3 jars from the package:

• lib/jetty-6.1.23.jar
• lib/jetty-util-6.1.23.jar
• lib/ext/jetty-ajp-6.1.23.jar

In the fisheye container directory under lib you will find these three jars:

• jetty-6.1.16.jar
• jetty-ajp-6.1.16.jar
• jetty-util-6.1.16.jar

Shutdown the fisheye instance and move them someplace safe. Copy the three new jars you grabbed from the jetty package and put them in place of the three jars you just moved to the safe place. Bring the instance back up and your good.

So if you find yourself getting heap errors from Fisheye that look a little like this:

java.lang.OutOfMemoryError
at java.util.zip.Deflater.init (Native Method)
at java.util.zip.Deflater.(Unknown Source)
at java.util.zip.Deflater.(Unknown Source)

Then it is likely you’ve been bit by this bug and you might want to consider an upgrade to your jetty container.

Back in December when things slowed down a bit we were asked to try to use the free time to catch up on documentation.  One of the specific areas of documentation that is usually neglected is that of what is on the servers.  My first thought when I saw the request was “why isn’t this happening semi-automatically”.  In order for us to document a server we run commands and then format the output to be understandable and pretty.   There is little or no reason why these commands could not be put into a script and ran automatically.

So I started out writing a bunch of little scriptlets to do the different parts of the documentation procedure.  When Mark saw what I was doing he suggested that we take this a step further by having one big script that does all the documentation and outputs it in wiki-markup that could go straight to confluence.  So we collaborated on building a big documentation script.  Then through work on another project to build a tool that would allow engineers to copy files to a customers server through the central midpoint server I realized we had the potential to push and pull this documentation with little effort.

Meanwhile Mark was working on a program that would allow an engineer to post a document to confluence from the command line.  With this program and the copy program the workflow was complete.  By building a wrapper script using some of the logic in the copy program I could push the documentation script up to a customers server then pull the output back, remove the litter files on the server then push the output document up to confluence all in one easy command.

Once things were working as intended it struck me that our initial attempt at the documentation script was not nearly detailed enough.  So I went back and rewrote the hardware section to use the Dell utilities or fall back on standard Unix utilities if the Dell software isn’t present.  Then I started to rework the J2EE application section.  So far the J2EE section has grown over ten fold in size and the documenting of Tomcat containers is almost finished.  There is much work to be done yet on getting better information for the other containers and applications within them.  After that will be a rework on the apache vhost documentation.

A sample screenshot of magic documentation