At Contegix, the NOC engineers spend a lot of time working with Atlassian’s products. We are in a constant cycle of installing, maintaining, and upgrading Confuence, JIRA, etc. for customers. We install their plugins, help work out the kinks, and make sure the applications stay running as close to 100% of the time as possible. Oddly, up until lately we’ve mainly only used one application of Atlassian’s internally - Confluence. Of course, we use Crowd here and there as well, but it’s transparent and I never need to worry that it even exists in our infrastructure. JIRA is used for projects with our development consultants and special projects. Everything we document ends up in Confluence, and that allows us to be more productive as we have this incredible encyclopedia of knowledge constantly at our disposal. The need for JIRA by the engineers didn’t every really seem excessively relevant in the past, nor did Bamboo, Fisheye and Crucible. We’re a hosting company with system administrators, not a software development company. In our minds, it didn’t make as much sense. At least, it wasn’t overtly clear that we needed JIRA.

We have Subversion running to handle the code for most of our internal projects, emails to and fro between engineers worked as bug reports for the internal scripts we use, and email was used to announce new versions of various scripts. We’d also use Confluence in a backwards way to help manage some of our internal projects as well, which wasn’t the best solution. It worked, yes, but it wasn’t optimal. We didn’t know any better though. We’re administrators, not users! How were we supposed to know that JIRA was so slick?

Well, the advent of JIRA Studio has taught us a solid lesson, the NOC engineers needed JIRA a looooong time

ago. We often ran into problems in the past where one person would write a script or an application, but it wouldn’t gain widespread use amongst our engineers. The simple fact was that either not everyone knew about it or the script would need updates; however, it wasn’t being properly maintained. This would lead to a wide variety of editions of the script floating around - the absolute death of the script in the first place. Then, we’d revert back to everyone doing everything by hand again. It was an endless cycle that would start everytime someone wrote a mediocre or decent script. We wouldn’t give the script the proper care and love it deserved, it’d gain moderate use for a bit, and later find itself in the script graveyard. The other alternative reality was the individual engineers wrote their own scripts, but never really found the means to openly share their scripts. All of that soon changed, as our illustrious leaders bestowed upon us a great and magical gift… JIRA Studio (add your own fantasy based sound effects here, I prefer trumpets personally).

All of a sudden, we went from having no way to manage our many scripts that have been tossed around the office like dirty laundry, to having so far five projects managed in JIRA Studio. We haven’t had the instance up for more than a couple weeks, and I’m sure the number of projects we maintain internally with JIRA Studio will only increase. It’s amazing how much easier it has made life for us already though. We have subversion repositories for our individual projects, issue tracking, code reviews, great tools to analyze our repositories with, plus a solid documentation backbone. Each project is sectioned off to what feels like its own little world, yet it’s still a part of the big picture that is our developmental operations here at Contegix. We have scripts for customers, programs that make our internal life less chaotic, along with our very website just in case anyone feels the need to make improvements to it.

I think the best part of the whole experience is that we’re all finally starting to share code, discuss new ides for automation, and expanding our thinking quite a bit. Gone are the days of not enough time, not enough resources, or too much effort. We don’t have to worry about those issues now. Now I can whip up a script that has a solid base to help out our company, create a JIRA Studio project, and as a team we can nurture the script to a fruitful life.

It’s odd how just having a launching pad for our internal development projects has opened our eyes quite a bit. Before we were quite content doing a lot of our work by hand, because our development process wasn’t exactly the best. The big problem with automating what we do is we’re dealing with your (our customers) production systems. If we’re going to develop automation tasks for your systems, they absolutely must be 100%. We won’t mess around with your systems by testing half baked scripts on what for many of our customers is their livelihood. That was the past though, because now we have solid testing sandboxes setup for our automation tests, along with JIRA Studio to help us manage the process of developing our applications. We’re already starting to see the benefits as bug reports roll in, fixes roll out, and new projects are being started. I’d say this is the beginning of a new era for Contegix, as we’re now more capable of streamlining of our efforts thanks to JIRA Studio

I guess my overall point is that if you think JIRA Studio isn’t right for your company, because you’re not a development team, you may want to reconsider. I don’t believe we ever though we needed JIRA, and we don’t need it, but I sure don’t want to go back to life without it!

Beginning in September, we began to quietly upgrade our backup system. The previous system worked well and never lost customer data. It was the perfect system when we purchased it in December 2005. It suffered a few problems, including slow startup restore times and a sad dependence on filesystem inodes for the indexing system. This led to scaling issues that would have drastically impacted us by mid-2008.

After a lengthy investigation into option, we decided to use NetVault from BakBone Software. The decision was made due to fulfilling specific criteria - native Linux client and server software, support for MySQL, PostgreSQL, and Oracle hot backups, Disk-to-Disk-to-Tape (D2D2T) capabilities, and the ability to implement quickly, very quickly. The last one was key to minimizing our costs for the implementation as we knew we would need to maintain the old system for a minimum of 60 days after the last server was migrated. This also led to RedHat and Bakbone producing a case study.

Feel free to read it here.

In an ongoing effort to improve service to customers, Contegix will be making some upgrade infrastructures to our core DNS services. Effective October 20, 2007, Contegix will be making changes to cache value, called TTL (http://en.wikipedia.org/wiki/Time_to_live#Time_to_live_of_DNS_records), of all DNS records to one hour (3600 seconds). In the past, most TTL values were set at 24 hours except during migration or changes where this value was lowered in order to minimize downtime. This was the Contegix default.


Hopefully, these FAQ below will help answer any questions:

Q: Why is Contegix making this change?

A: This change is being done in order to help facilitate quicker changes to customer DNS records. Customers previously had to wait up to 24 hours to see the change replicate among all DNS servers throughout the Internet. With the change, this time has been shortened to 1 hour.

Q: According to the Wikipedia link, lower TTLs cause heavier loads on the name servers. Is the infrastructure capable of handling this?

A: Absolutely. The core Contegix DNS infrastructure is capable of handling this load and much higher. Current name servers are sitting approximately 99% idle with an average response time of 5-8 milliseconds. This level is being held even with TTLs at 600 seconds or lower for over 4,000 DNS records. In addition, the change for reverse DNS (IP to hostname lookup) has been using the new value for over 6 months with no noticeable load difference.

Q: Will Contegix still allow custom TTL values - lower and higher?

A: Absolutely. Contegix will continue to offer customers the option of having lower and higher TTL values. Contegix remains one of the few providers who allow TTL with values below one hour. Please file a support request if you wish to have a different value set for an entry after the changeover.

Q: What action do customers need to take?A: Unless the customer has specific TTL requirements, no action is needed upon the part of the customer.

Q: What if I do not host my DNS with Contegix?

A: No changes are required on your side.

Q: I like what I hear about Contegix’s redundant DNS infrastructure. Can I move my domain(s) there?

A: Contegix provides DNS hosting as part of our services. You are welcome to move your DNS hosting to us. Contact support at your convenience and the engineers will guide you through the process. Please do not change your DNS until you have contacted support and planned a migration strategy.

As always, thank you for your continued business. We look forward to helping you build the best infrastructure available. As promised, this is one additional announcement on our continued improvements to drive more customer value.

Our single most important commitment to our customers is to provide the absolute best support possible. Part of our support commitment is continuously pushing the response from reactive to pro-active. We want to be able to address an infrastructure problem before our customers realize it is a problem. When it is an application issue, we want to be able to provide you every piece of data available to help you resolve the application level issue. A key component to this goal is our monitoring system. With this goal in mind, I am please to announce that we are making a significant improvement to this core infrastructure.

We have selected Hyperic (http://hyperic.com) as the foundation of our new monitoring infrastructure. Specifically, we are deploying the Hyperic HQ Enterprise Edition (http://hyperic.com/products/hq_for_ent.html). Hyperic HQ drastically increases our monitoring capabilities by focusing on the health of the server and the applications in your infrastructure. Purpose-built for web infrastructure, and architected to consider all layers of infrastructure including hardware, middleware, virtualization and applications, Hyperic HQ delivers system monitoring, trending, and analysis.

So what does this mean for you - our customer? Let’s look past the server health and examine the application infrastructures:

• For our customers utilizing Apache, we will now be monitoring the health of Apache, including its availability and number of available Workers/Servers.
• For our customers utilizing Java, we will now be monitoring the health of the JVM via JMX. These statistics will be collected to determine the health of your application.
• For our customers utilizing Rails, we will now be monitoring the health of individual Mongrel instances.
• For our customers utilizing Zimbra, we will now be monitoring the individual components, including availability, speed of IMAP connections, and LDAP connections.

All of this means faster support, faster resolution, and great visibility. And that’s just the beginning…

We have decided to raise the bar on the managed hosting industry and deliver even more value through our trademark Beyond Managed Hosting. For our managed customers, the monitoring will continue to be included in your services. This drives more value from the management services we provide with more visibility into your infrastructure. Beginning this weekend, our support engineers will begin making the necessary changes to customer firewalls to accommodate the Hyperic agent.

The rollout of the agent will begin in approximately two week with completion by March 15, 2008. After the rollout is complete, direct customer access to the system will begin rollout with completion by end of March 2008.

If you are a colocation customer, feel free to contact your account executive or sales@contegix.com to find out how you can take advantage of this monitoring solution at a reasonable per-device cost.