A while ago I read an excellent blog post from Timothy Fitz called Continuous Deployment at IMVU: Doing the impossible fifty times a day, that got me started on trying to improve the way we deploy our websites at UKFast but just as important – being able to roll back to a previous version at the drop of a hat.

We had a few requirements such as, hook into current systems, deploy code with near zero downtime, run some post deploy commands, stop deploying on error and keep an archive of old deployments for simple roll back amongst other things.

As you’ve probably gathered by the title, we run Subversion as our version system, it works well as a standalone but that’s never enough for us at UKFast. The system that was developed so far has achieved everything that we need, new or existing sites are deployed usually within 30 seconds but that depends on the size of them, the freshly deployed code is activated online without any loss of service to visitors to the site which is probably one of the best features about this system. It also means that roll backs are done transparently and due to another feature, archives, depending on the (user configurable) number of archived versions we keep, we can roll back to a previous version almost instantaneously as the code is already deployed and ready to go – if we need to go back further past the archived version then a checkout is done on the old version.

Now once a site is deployed, that typically wouldn’t be the end of it. If someone had tried to perform the update manually then it most likely wouldn’t be case of just FTPing up the new website, for one, we usually have some heavy customisation on folder structure, permissions or just because we want something different. So the system needs to cope for this as well, it will allow users to issue post deploy commands from a predetermined list of allowed options. This has so far covered us from all the usual requests our web developers usually ask for and without have to give them a ssh shell!

In the event that any errors are detected at any point the deployment is halted and the user performing the update is notified, no need to roll code back in anyway as the new version hasn’t even gone live yet.

This system has meant we can stop using ssh / ftp accounts for website management, determine fine granular access to developers and mainly, allow us to update our websites at any stage of the day, knowing that it won’t impact site visitors and if it does, then we have a roll back process that takes us 10 seconds to perform and the site will be restored.

Unfortunately the way in a lot of our systems work means I can’t discuss it on an open blog, competitors might be watching. But if anyone wants to pick my brains then by all means, get in touch.

So without further ado, below you will find the monitoring software I have installed and tested. These are fully functional in all aspects, the only thing limited is the ability to change passwords so that people can’t lock one another out and also, smtp is blocked on this server meaning that any alerts generated won’t get sent out.

Nagios:
http://demo.nocmonitoring.co.uk/nagios/
username: admin
password: admin
Notes: You cannot add / edit / remove devices from this demo as Nagios relies on editing of configuration files directly which isn’t supported natively.

OpenNMS:
http://demo.nocmonitoring.co.uk:8980/opennms/acegilogin.jsp
username: admin
password: admin

Zenoss:
http://demo.nocmonitoring.co.uk:8080/
username: admin
password: admin

Zabbix:
http://demo.nocmonitoring.co.uk/zabbix/
username: admin
password: admin

I’ve got to say, after going through and testing all of these, at the basic level of just adding a device to be monitored for things such as http, it’s an extremely difficult process on all of them (Nagios was the easiest but again, this is probably due to my time served with this software). From the perspective of simplicity, I can’t understand how such a task of adding a device whether by hostname or IP can be such a complicated task.

Anyway, feel free to make use of the online demo’s and if you would like me to add anymore NMS systems to this list then please feel free to get in touch and I will see what I can do.

Enjoy

**Update

Jane has kindly pointed out the login for Zabbix didn’t work – changed it to the actual correct details so now it should.

No, that’s the simple answer. The longer answer is more complicated and depends on what your System Administrators can support from an application point of view. A lot of the NMS systems available are your standard Perl / C implementation on a flat file or MySQL/PostgreSQL backend so should be supported on most systems. Others however require Java or Python support, whilst this is simple enough to install and run on most Linux distributions, what happens when things go wrong? How many companies have IT staff who have taken the time to learn Java/Python for the next big Web 2.0 site?

I’ve used Nagios for a while now (some 5 years) and have always gone back to the start and looked over the alternatives at various times to see what they offer in terms of features and integration. Integration, that’s the key for me when it comes to choosing an NMS system, if it’s not able to offer a level of integration with the systems we already have (and we’re willing to do some work to make that happen), then it’s a non-starter no matter how laden with features it might be. Over the past couple of years, a wider choice has become available which makes choosing to switch, that much harder! Having the luxury of time to test these newer systems isn’t something that’s available to everyone including me so my ‘experience’ of other systems is limited compared to my time with Nagios, however, I know what I want and need so it doesn’t always take long before commissioning an app to the dusty code graveyard.

Let’s get into some of the more popular NMS systems available at present and what my impression of them has been…..

OpenNMS

OpenNMS is actually a really nice application to use for Network Monitoring, it’s discovery feature works really well and being configurable form XML files makes it extremely easy to setup and maintain. Installation is relatively straight forward if you are using the pre-packaged versions available or are a dab hand with Tomcat. The pre-configured range that I created for Network discovery worked fine and it detected all devices which responded to ICMP and monitoring of individual services/interfaces was simple if not time consuming if you have a large selection of devices to monitor.

The bad points for me are the fact that auto-discovery is the primary way of adding new devices to be monitored. You can add new devices in via the command line on the NMS server, this isn’t too much of an issue depending on how many new devices are added to your infrastructure on a day by day basis. If it’s a sizable amount then this isn’t going to be an option for long and with no way to add single devices in via a web interface your only options left is by some form of integration by way of a script. The next problem is managing the individual services/interfaces that are available for a particular device, this again appears to be a manual process with no easy way to integrate into your current NOC.

PostgreSQL is the supported DB of choice for this project, we haven’t at present migrated over to PostgreSQL which means that the maintenance of this solution would be higher than a MySQL based back end. That’s something to bear in mind in any solution you may migrate to or implement. PostgreSQL is gaining in popularity and features so this at some point will become a mute issue.

Zenoss

Wow! Zenoss appears to have improved quite a bit since the last version that I tested and looks to be highly recommended now. It’s features include a comprehensive API to allow integration with existing systems, this would enable the setup of monitoring new devices quite easy. Installation on platforms with supported binaries appears to be straight forward along with the configuration and setup of your first ‘Devices’ that need to be monitored. Auto-discovery is still an option and is more intelligent than OpenNMS, it provides a handy feature of ‘walking’ your network via routers to find all devices located on your network, this is quite a powerful feature on it’s own.

It supports the ability to expand your single monitoring server to a High Availability solution, whilst this isn’t quite out of the box, it really isn’t a complex setup for a Linux Sysadmin (Setup Guide). This enables you to grow your monitoring environment as your infrastructure grows or provide a level of redundancy to ensure that you know what is going on 24×7.

The changes and improvements that have been made since my last evaluation of Zenoss means that it’s about time that I tested it again – if it became a viable alternative then a lot of work would have to go into the migration from Nagios to Zenoss but it looks like it could be worthwhile.

Zabbix

I’m not a big a fan of the Documentation for Zabbix, everything is dumped into a single PDF which makes it difficult to filter out what is part of configuration and what is part of administration. For instance, to refresh my memory whether you could add a single host into the setup via the web administration, I checked the documentation. Now this was  a brief check, it was 00:15 but could I find anything other than auto-discovery? Nope, not a single thing, the system of course does allow this, you just have to struggle in the docs to find it. Not a great start but not a show stopper if an API was available – it doesn’t appear to be, I can see comments about this on the forum but so far nothing seems to have a materialised so far.

Repeat notifications has now been implemented since I last tested Zabbix, this is something that was extremely lacking in previous versions and is a must for any NOC, especially if your using email/sms/pager alerting. These methods are inherently unreliable when it comes to critical service so sending more than one alert is always a handy feature which meant that before, Zabbix would have sent a single alert when a device went down – and that was it, if you didn’t get that alert for whatever reason then you would be unaware of any issues until someone logged into the administartion system.

Distributed monitoring is included and seems to be extremely simple to setup, this is one of the better features of Zabbix and something worth considering if this is a requirement for your environment. In general Zabbix seems to have improved quite a lot since the last testing I did, the restrictive admin interface means it would be something that I wouldn’t really consider in a live environment.

Finally, a comprehensive list comparing the available NMS applications available is currently hosted on Wikipedia, go and check it out for a list of more NMS applications.

http://en.wikipedia.org/wiki/Comparison_of_network_monitoring_systems

8. Use of Your Content

a. Adobe does not claim ownership of Your Content. However, with
respect to Your Content that you submit or make available for inclusion
on publicly accessible areas of the Services, you grant Adobe a
worldwide, royalty-free, nonexclusive, perpetual, irrevocable, and
fully sublicensable license to use, distribute, derive revenue or other
remuneration from, reproduce, modify, adapt, publish, translate,
publicly perform and publicly display such Content (in whole or in
part) and to incorporate such Content into other Materials or works in
any format or medium now known or later developed.

For now, I’d recommend avoiding using the service until they resolve this which they are doing according to sources.

Microsoft seem to be getting more right nowadays, some of the new features being released indicate that things are definitely changing for the better, this could easily change though and it’s not the first time that Microsoft has made promises both before and during a launch only to be shot down due to bugs/faults in the release. One thing that they have ventured into but from the talks don’t seem to have got quite right is the new Windows Server 2008 Core install, this new install version will give you a non-gui CLI only Windows installation – I can only assume to be able to compete with *nux. However, they have failed to be able to provide ASP extensions meaning that all you can do is server static content or run PHP. Now this might be good for those who do only code static pages (are their anyone who still do this?) or run PHP, however the lack of ASP extensions means that Windows developers are really going to stick with the standard GUI interface to be able to run the apps they develop. This seems to defeat the object of having a CLI version as if I want to run PHP then I’ll just stick a Linux box online to do that job!

They do however have a lot going on with their new HyperV technology which looks particular interesting with the ability to manage other vendors Virtualisation software such as Xen and VMware. The interface for managing installs of this seems extremely simple and well thought out which could come as a shock to those used to running Virtuozzo (or parallels server virtualization as it is now know as)  due to the, well lets put it simply, completely crap interface they provide!

Something else that they seem to be incapable of doing is converting fault / log id’s into text, they demo’d to us how you can now enable full auditing when running a DC, as the demo showed, when you disable or enable an account in the DC, it now logs that this has been done and according to the person showing us, code 62345 (made that code up but it’s like that) means the account has been enabled, ffs, why not just tell us that rather than having us look through a bunch of codes to find out it’s meaning?

All in all it was a good day, very informative and well presented. The free drinks and chocolate bars seemed to go down very well which is not surprising with a building full of 1000+ geeks! They also laid on a spread for lunch in the expo area which is the first time I’ve seen that been done although the food wasn’t exactly the best

They didn’t just have the one sizable office that you might imagine, they had 3 all next to one another in a U shape, each building was a pretty decent size and comparable to a full size football pitch with 2-3 floors. We were there for a seminar day on the Cisco WAE and the Cisco ACE but expecting more of a sales pitch than anything else. From the moment you walk in and sign the guest book, your shown where you need to go by a receptionist as they had around 2 or 3 at the desk! The seminar was really technical and turned out to be one of the better ones I’ve been on as for once, it didn’t turn into a sales pitch

Lunch was brought round to us with a mixture of sandwiches, pies and free drinks from a fridge stocked with a variety of different soft drinks and bottled water. During lunch I bumped into our what I suppose you would call account manager for Cisco who spent 10 minutes seeing how business was going before HAVING to show me one of their teleconference rooms which he described before we got in as 24 like He wasn’t wrong and looking back I wish I had asked to take a photo just to show you how this hasn’t been done by half. 3 40″+ plasma screens surrounded a bank of cameras for viewing who you would be on a conference call with, speakers placed around the desks would have enabled around 6-8 people to participate in a conference call from this office alone. It was truly amazing and we chatted briefly about possibly ripping out our new phone system to go Cisco……

After the seminar I promptly headed to reception to book a taxi to take us back to Heathrow T1 (which I’m currently still sat in ), reception kindly informed me that a free taxi would be around in 10 minutes to take us back and pointed me in the direction we needed to wait – the only bad thing I had to say about the experience was that it would have been nicer to wait inside as it was freezing out

Me being chuffed at being at the Cisco office.

Either this wasn’t the case and I was being a bit paranoid, or we really are that good – for those of you who are customers of ours then I will let you decide that but I definitely would like to think the way the company is run, the support that we offer and the way we are always striving to improve and take on criticism as a growing business has a lot to do with it. Obviously I am going to be biased towards where I work but that also means I have first hand experience of the work and dedication that the staff put in, especially the support team who do go above and beyond the call of duty to help clients get to where is needed.

Our MD, Lawrence Jones accepted the award and thanked the team behind making things happen but I do have to say – couldn’t wish for a better person to work for. The amount I have learned as time has gone on with regards to performing the role I have within the company is amazing, having come from pretty much a full technical background with a bit of management experience has meant it’s been a strange but interesting learning curve which I’m sure is going to continue……..

Needed another time to test it, a bike ride would work and luckily we had one planned for the Saturday just gone! Another trip over to Watergrove Reservoir sorted that out. I remembered before setting off, to get the sat nav locked onto our location and fired up NST. The software itself is so simple to use and provides a wealth of information about the journey you’ve been on. Obviously using the in-built GPS means that the battery life gets sapped pretty quickly, couple with the fact the missus had been using it the day before as well meant it didn’t have long to live.

Luckily it held out for three quarters of the day before powering itself off. I’d not used nst when the battery has died before but it handled it well, it reported that it had crashed upon the next reload but the full details of what we had done was still there just 1 hour short I’ve uploaded the days activities and you should be able to view it here.

NST has pre-defined setup for Walking, Cycling, Running, Skiing and 2 other custom activities. It supports things like Live sharing where you can constantly update your progress online, Auto pause when your speed drops below certain levels speed averaging. If you have GPS enabled Nokia phone then I highly suggest downloading and installing this – well unless your a bit of a fatty

Still no sign of a new hardware version with 3 or 3.5G available which is still a bit of a problem/pain for some. However with the updates in software, most relevant to me is the MS Exchange ActiveSync for mobile push email and the Cisco VPN client.

Having just signed a new deal to port our work mobile phones over to O2, we have been given a free one. The other phones we are going for means that we are finally making the move from an Open Source email solution to a MS Exchange based system with blackberry BES system for the few users who will be getting blackberry handsets. So this means the odd one out – the iPhone, will eventually be able to mingle with the fold once again

Don’t really use Windows much now apart form at work but could this be a good step in the right direction?

Think i’ll give it a go on my asus to try it out…….