Back in December when things slowed down a bit we were asked to try to use the free time to catch up on documentation.  One of the specific areas of documentation that is usually neglected is that of what is on the servers.  My first thought when I saw the request was “why isn’t this happening semi-automatically”.  In order for us to document a server we run commands and then format the output to be understandable and pretty.   There is little or no reason why these commands could not be put into a script and ran automatically.

So I started out writing a bunch of little scriptlets to do the different parts of the documentation procedure.  When Mark saw what I was doing he suggested that we take this a step further by having one big script that does all the documentation and outputs it in wiki-markup that could go straight to confluence.  So we collaborated on building a big documentation script.  Then through work on another project to build a tool that would allow engineers to copy files to a customers server through the central midpoint server I realized we had the potential to push and pull this documentation with little effort.

Meanwhile Mark was working on a program that would allow an engineer to post a document to confluence from the command line.  With this program and the copy program the workflow was complete.  By building a wrapper script using some of the logic in the copy program I could push the documentation script up to a customers server then pull the output back, remove the litter files on the server then push the output document up to confluence all in one easy command.

Once things were working as intended it struck me that our initial attempt at the documentation script was not nearly detailed enough.  So I went back and rewrote the hardware section to use the Dell utilities or fall back on standard Unix utilities if the Dell software isn’t present.  Then I started to rework the J2EE application section.  So far the J2EE section has grown over ten fold in size and the documenting of Tomcat containers is almost finished.  There is much work to be done yet on getting better information for the other containers and applications within them.  After that will be a rework on the apache vhost documentation.

A sample screenshot of magic documentation

Contegix has been hosting the initial meetings of the Gateway Groovy Users, a group dedicated to Groovy and Grails.  The first two meetings were on the small side ( 6 people ) but they were productive.  Plans are underway for our first “big” meeting.

Contegix is the natural place to host a group for Groovy and Grails, since both of these open source projects are part of codehaus.org, which is hosted at Contegix.  Attendees to our meetings have the unique privilege of being in the hallowed halls that house the main source code repository of the tools they love!

If you are already using Groovy or Grails (or Griffon) or are interested in these technologies, please join us at our next meeting on Wednesday, March 11, 2009 at 6:30pm.  We will be having a presentation on developing desktop GUI applications with Groovy.  Then, for those who want to stay, we will be starting on a group project.  It promises to be a great learning experience and a lot of fun.

Hope to see you there!

Prior to the existence of Mod Proxy AJP being built into Apache HTTP we used Mod JK for any Tomcat based instance we hosted here. All of our Resin containers would use Mod Caucho, and all the Tomcat containers used Mod JK, and all was well in the world. We never had any significant complaints with Mod JK (Caucho’s another story for another day), other than it being a bit of a pain to setup, and it needing to be built from source. This created a situation where maintaining Mod JK was problematic, because it wouldn’t automatically be updated in our routine RHEL updates forcing us to manually rebuild for critical security updates. Then you added in the fact that the configuration files totalled in at three, with a mod_jk.conf, a worker.properties file, and the actually JkMount itself in the virtual host configuration, and you had yourself a maintenance headache. Not too mention the slightest of typos and a simple Apache reload turns into Apache going belly up like a poisoned whale.

Needless to say I suppose, when Proxy AJP came into our lives we as engineers were smitten. We were all quite familiar with ProxyPass, and simply swapping “http://” with “ajp://” to proxy to Tomcat containers seemed like the best solution we had ever seen. We were practically partying over the idea that we could finally ditch JK like a bad habit, and ride off into the sunset with our new ProxyPassing beast of a friend. The only downside is it required Apache 2.2, so our RHEL 5 boxes were all set, but the RHEL 4 boxes left us in the hands of old faithful, Mod JK. The ultimate question continued to linger though: Which would hold it’s own under high stress more effectively? In terms of average browsing, I don’t think you’d ever notice a difference, even after the tests I’ve done. The difference, to the human eye at least, is negligible. Nevertheless, I constructed a few test to put both through their paces.

Test Environment:

• Apache 2.2
• Tomcat 5.5.20
• Java 1.6 64 bit
• PostgreSQL 8.3.5 (tweaked for performance)
• Confluence 2.10.1

Test Tools:

• Pylot
• BrowserMob

To get the ball rolling I started with Pylot. This was a quick and free way to start some preliminary testing. My initial tests fired up a constant flow of 200 (virtual) users at the instance for 10 minutes to see what would happen. To my initial surprise, Proxy AJP was a pretty clear winner to start things off. It was able to serve 1-2 more requests per second, while serving an average response time of roughly 1 second better than JK. Surely that couldn’t be right, it couldn’t possibly be that easy right? Right, not that easy. With some adjustments to Mod JK, like giving it a connection pool, I was able to get the results from my tests with Pylot to average out to be roughly the same. Under constant pounding from 200 agents, both were able to average an average response time of approximately 8.3 seconds, while serving roughly 23 requests per second. I was pretty pleased with my test environment at this point, and ready to kick it up a notch.

We got in touch with Patrick from BrowserMob, which is a site that offers real load testing with real browsers. I was hoping that with his service, I’d be able to find distinguishing differences between the two AJP connectors for Apache. After crafting myself a script using Selenium, and testing it ad nausem locally, I was prepared for my first test against Proxy AJP. I ran a 15 minutes ramp up to 100 users, and then put a constant load of 100 users across 45 minutes on the new kid in town. Surprisingly, the throughput looked pretty incredible:

If you click the image you can get a better view of the results, and see that we topped out around 140MB per minute being served. You’ll also see that the ebb and flow of data being pushed out was incredibly spiky. There are numerous jumps up and down the graph as more and less intense pages are hit throughout the tests. At first this seemed a bit scary, because I somewhat expected a smoother line across the board. I left all configurations the same though, and proceeded to test Mod JK, where I received the following throughput graph:

In a side by side comparison (or for better results lay the images over the top), Mod JK suffers from similar, but less severe spikes. It never quite reaches the highs that Proxy was able to reach, and never dips to the lows the Proxy had succumbed to. However, there’s something else at play here that caused this to happen. Mod JK caused Apache to reach MaxClients, and hold fairly steady there. There were only 100 users, and there were 256 MaxClients available in Apache, so numerically speaking, Mod JK should have been able to handle this without cracking under the pressure.

However, since it was at max connections, it didn’t have the downward spikes that Proxy did, because it was still trying to serve past requests. At the end of the day, we saw roughly 630 timeouts from JK in the tests, with the tests timing out at 30 seconds. Yes, I could have cranked up the timeouts to allow every step to completely finish, but this is real world testing. When was the last time you allowed a page more than 30 seconds to load? Proxy AJP on the other hand only sufferred from 450 such timeouts, showing me that it was able to pump out requests, and break off connections faster than Mod JK. The way I currently see it, JK failed to serve 200 people. That’s 200 people that think my site’s down, or heaven forbid, a Google bot that is prepping to denote me as a downed site. That’s 200 requests I’m not able to serve advertisements on, and thus not able to make money off of. Might not seem like much, but it is a big deal to some degree.

Now it would be interesting to crank up the MaxClients, and see what happens in that scenario. However, I needed to keep parity between the tests, so they were both performed within the exact same scenarios. We can always do more tweaking, and make more adjustments though. However, at the end of the day, Proxy has a lot going for it. It’s a built in module in 2.2, configuration is insanely easy, balancing is a cinch, it works with the balancer manager of mod_proxy_balancer, it receives automatic security fixes via RHEL updates, and so far, it out performs JK. We plan on doing more tests with altered configuration in the future, so look forward to future rounds. Perhaps JK can make up some ground in Round 2? Perhaps you’ll also get some insight on methods to tweak your Confluence installation to squeeze a bit more performance out of it as well. Many updates to come…