My one pain point was state flapping. A service can go unresponsive for a fraction of a second under heavy load and come back right again. All to frequently my inbox would get swamped with problem…ok…problem…ok….problem…ok messages. By the time I’d log in everything had settled down. It’s not a perfect situation, but doesn’t warrant getting out of bed to repair either. Nagios has a flapping detection function but Zabbix doesn’t.

What I’ve found is that the AVG function can be used to average the status values (“1″ or “0″) over a given period and only trigger an alert if it drops below the alert value. For example

{server01:net.tcp.port[, 80].avg(30)}<1

will only trigger a second alert if the web server stays responsive for at least 30 seconds between outages. Also

{server01:net.tcp.port[, 80].avg(30)}=0

Will only trigger an alert if the service stays down for more than 30 seconds at a time. Further you can combine the two in

{server01:net.tcp.port[, 80].avg(60)}<0.5

to only trigger an alert if the service stays down for more than 30 seconds and only trigger an OK if it stays back up for 30 seconds.

Because my current work style is very fluid, where I hop between my office and home PCs, a laptop, a netbook and an iPhone, it made sense to try and keep my data where I could access it regardless of device. I basically have 3 categories of data: email, my web browser environment (bookmarks, stored passwords, history) and unstructured data (Word, PowerPoint, Excel, JPEGs etc. files). I use a few cloud based services to supplement these, but more on them another time.

Documents: I’m fairly neat when it comes to organising data on my laptop. Everything is in the My Documents folder. I used to use Beyond Compare to manually sync my desktop and laptop but for the last while I’ve been using Microsoft Live Mesh Beta to keep everything in sync automatically. A change or addition on any system ripples around to all the rest in real time. This means I can start work on a document in the office and seamlessly pick up where I left off when I get home without having to do a thing. On the new PC I just installed the Mesh client and left it to do it’s thing over night. It pulled my data from the online copy and placed it just where I’d expect it.

Email: POP is for noobs; IMAP is king. Seeing as Google are generous enough to give away 7GB+ of email storage I’d be a fool not to use it. A few years ago (pretty much as soon as it was available) I created an @garypigott.net Google Apps account and shoved all my email up there. Right now there are 118,527 emails, occupying 2298MB of space, and vitally, it’s completely searchable in exactly the same way you’d search the web using Google. I access exactly the same data using Gmail, Outlook, Outlook Express, Windows Live Mail, Thunderbird, and the iPhone mail client.

Web Browser Environment: My browser of choice is Firefox,  which is handy a) because it’s multi-platform and b) because of Weave. Weave is an add-on that moves your bookmarks, history and passwords into the cloud. You can install the add-on on each of your computers, regardless of OS and everything is kept in sync. Weave is open source, so surely some enterprising and generous developer out there will churn out a Weave add-on for IE, Opera and Safari…

title Ubuntu
rootnoverify (hd0,2)
chainloader +1

This has the effect of kicking off the bootloader on /dev/sda3 (i.e. the Ubuntu one) when selected, booting Ubuntu!

The factory installed Linpus distro is OK-ish, but it doesn’t get upgraded as new versions of the component software gets released, so still includes Firefox 2, rather than 3. Also, it tries to simplify things too much.

I tried installing XP Home, which isn’t as easy as you’d think because there’s no optical drive. I followed this guide to installing XP using a USB key, and after an age (well 2 1/2 hours actually) I had a working XP install. The problem is that this isn’t the speediest of hardware (the SSD in particular is woefully slow) so there’s a huge amount of tricks and hacks to get it running at an acceptable speed. In the end I decided to revert to Linux.

There are a few distros tweaked to run well on these systems. About the best is Ubuntu Netbook Remix. It’s an official branch of Ubuntu, so gets continuous support/maintenance. I ditched Evolution for Thunderbird, but other than that it’s pretty much perfect out of the box.

Aaaagggghhh

Why is it that VMware hide their VMware server downloads behind a password?

Like many I run headless Linux servers without a GUI installed. I’d like to be able to download VMware Server (700MB+) using wget via an SSH session, but I can’t because you need to log in first, and it’s not a vanilla HTTP login that I can work through wget either! I’m expected to log into the site on my desktop, download the installer and then SFTP it up to the Linux box. That would be OK if the server was on the same LAN, but it’s in a data centre and I’m on the end of an ADSL line with a 672Kbps upload. Surely controlling access to the serial numbers is good enough as the installer is useless without it. Still, at least I could use w3m in the end.

I decided to use him as a guinea pig for Restore, and open source client-server disk backup platform. He has a reasonably powerful server in the office (I bought it for him) and a web hosting account with 10GB of space (also thanks to me). I wanted to set it up so it would backup the desktops to disk on the server, which I’d then rsync off-site. I’ve used VMware Server a lot, so the plan was to take Restore’s “VMware Appliance”, drop it on the server as a totally self contained virtual server, spend a 1/2 hour configuring it and I could go back to the day job.

It didn’t work out that way. The VM was created using VMware Workstation 6, whose VMs are incompatible with VMware Server 1.X. All they needed to do was click “VM” > “Upgrade or Change Hardware” before publishing the VM and everything would be fine. Now my 30 minute favour for a friend will balloon into a 3 hour job because I need to install it the old fashioned way. Grrrr!

SiliconRepublic.com: Irish Stock Exchange firms not tracking web visitors

1. While the application is fully 64 bit, the installer isn’t quite, and the error it throws isn’t the most illuminating.  You need to install 32 bit support by running “apt-get install ia32-utils”
2. You can’t connect to your freshly minted VMware server with the VMware Server Console. You get the error “There was a problem connecting: Cannot connect to host X.X.X.X: No connection could be made because the target machine actively refused it”. It’s because VMware-mui hooks into inetd, which doesn’t get installed in a bare-bones Debian Etch 64 bit system. “apt-get install inetutils-inetd” is the quick fix.

The server originally came practically free with some disk arrays we bought a while back so I wasn’t expecting the best, but was still slightly peeved when I saw it only had a CD-ROM drive, so my copious quantities of DVD+/-R media were useless. Off I went to my local mirror and downloaded the “ia 64 netinst” CD image and I burned it to my last CD-R. I bunged it in the drive and hit the switch and it booted straight into Windows….. I tried a bootable Windows CD and it booted off that fine. Hmmm. So I Googled and found this. apparently if you want to use an E64MT CPU you need the AMD 64 bit install CD, rather than the ia64 one, which only works on Itanium CPUs. I’m off to the shop now to buy more CDs.

Re: Debian Installer on DELL PowerEdge 850/860

 VMware recently announced the first beta of their free low-end VMware Server 2. It is a beta, and as such the sensible part of my brain tells me to stay away and let someone else take the pain. Unfortunately the gadget & gimmick loving part of my brain made me install it on my mail/web/everything else server that had a perfectly functioning v1.0.4 setup. Bad idea!

Version 1 has this very nice web UI for monitoring, starting/stopping, and tweaking stuff, and a separate Windows/Linux application for creating new VMs and accessing the console of each one. Version 2 has dispensed with the application and instead relies on a beefed up web interface, with a browser plugin to display the consoles. Unfortunately being beta, the web UI is ugly and unfinished, and there’s so much un-optimised debug code the VMs run at less than half the speed they did on the old version. I run Debian Linux web, email and SyncML servers, a Windows XP desktop, and the odd test Linux VM, all on a single 2.4GHz Celeron Dell server that I got for free. Obviously the beta had to go…..

Rolling back the server and toolset installs was a bit long winded, but it was straightforward enough. What wasn’t straightforward was un-scrambling the network config. Each VM has at least two interfaces, usually bound to the host system’s wireless interface (I keep the server in my garage, where I can’t hear it, so there’s no Ethernet out there) and a “host only” LAN for NFS, backups etc. The reinstall remapped everything arseways.

VMware Server 2.0 Beta – VMware

We’ve started to grow, both in head count and sophistication, so now we’ve got a lot more documents to manage and a lot more people wanting to manage them. Our documents are usually authored by one person, but in many cases require review and comment by sales, marketing and technical people prior to release. What would be great is something that would manage the workflow, with notifications and audit tracking, and integrate with our website to make the final result available to clients, all in a controlled manner. We need a Document Management System. Our company likes to use open source software wherever possible, so my first port of call was Google, which gave me the following:

• xinco
• contineo
• KnowledgeTree
• Epiware
• Alfresco

Most of the above are Java applications, which I really don’t like, and I’ll cover in a later blog post. KnowledgeTree is written in PHP and ticks all the boxes. Alfresco is a Java app, but it’s functionality is too good to dismiss straight away. Let the testing begin!

Because of this we had loads of different options if you needed email. You could use an “enterprise” platform like Microsoft Exchange, Lotus Notes and the like, or you could roll your own with any number of SMTP, POP or IMAP servers running on Linux (my preference right now is Postfix & Dovecot running on Debian).

Now days you can’t easily get away with the DIY approach. E-mail is has to a large extent replaced voice & fax as the prime business communication method. We need to support huge inboxes, instant delivery of messages, access on the road, and near perfect uptime and backups. Most of these things can be achieved with any modern email setup, however some require a little help from outside.

Remote access is a major requirement these days. An executive feels naked on the golf course unless he’s got a Blackberry, a Treo, a Symbian or a Windows Mobile smartphone on his hip (mine is a Nokia e61). You’ve got three options if you want to get email onto these things:

1. Scheduled checking of a POP/IMAP server: It works, but there’ll be a delay because the message doesn’t get pushed to the device as soon as it comes in.
2. Blackbery Enterprise Server: Native support for Blackberry units, and add-in support for other platforms. It only supports Exchange, Notes, or GroupWise servers though
3. Microsoft Exchange Direct Push: Native support for Windows Mobile units and add-in support for other platforms. Only supports MS Exchange servers obviously.

The other major issue is how to back them up. You not only have to worry about DR backups, but also compliance. A nightly backup isn’t good enough any more. You either need to run a continuous backup or run a parallel archiving solution.

At issue here is that the remote access and backup requirements require 3rd party support. 3rd parties target their efforts where they’ll get the most business, so There’s no support outside the big three (Exchange, Notes & GroupWise). Nobody’s deploying new Notes or GroupWise installations anymore, so as much as I hate to say it, Exchange is the only option if you need to provide effective remote access and require effective backup.

I came across OpenPsa recently and it seemed perfect for what we do. It’s a CRM tool with heavy project management and document repository tools that could be perfect. It runs on a LAMP stack, so I don’t need to dedicate a server (or VM) exclusively to it, like I would do if it was a Java app.

The one thing I always look for when choosing an Open Source product is the commitment of the developers. I’ve been burned in the past by committing to platforms only to find that the lead developers have no interest in improving the product or fixing the bugs. The most recent posting on the news page was a link to a blog entry of one of the lead developers on a Saturday stating that the 2.0 release was coming out the following Monday….. That was last May. He has updated his blog since, but no mentions of OpenPsa in any of them. There is no updates since about developers falling out, selling out or dropping dead. It’s the software equivalent of the Marie Celeste. Too bad. I’d have loved to play around with it.

VMWare Command Line Tools from VM Back is an unofficial open source command line version of VMWare Tools that enables you to shrink your disk images and sync your clocks on any Linux, DOS, BSD or Solaris VM.

Install a bare-bones Etch system by skipping the role and package selection screens during the installation.

Edit the apt sources.list to include the “contrib” and “non-free” packages by adding the following lines to /etc/apt/sources.list:

deb http://ftp.debian.org/debian/ etch main contrib non-free
deb-src http://ftp.debian.org/debian/ etch main
deb http://security.debian.org/ etch/updates main

Run “apt-get update” to get the package list up to date.

Run “apt-get install sun-java5-jdk” to install Sun Java

The JAVA_HOME environment variable needs to be set for any Java app to run so adding the following to /etc/profile will do the trick

JAVA_HOME=”/usr/lib/jvm/java-1.5.0-sun”
export JAVA_HOME

Run “apt-get install tomcat5.5 tomcat5.5-admin tomcat5.5-webapps” to install Tomcat

Run “apt-get install wget unzip” to install wget so we can download jbilling from the server, and unzip so we can decompress the archive.

“cd /opt” and run “wget http://heanet.dl.sourceforge.net/sourceforge/jbilling/jbilling-1_0_6.zip” to download it and “unzip jbilling-1_0_6.zip” to decompress it.

Edit /opt/jbilling/server/default/conf/jbilling.properties to suit.

Run “chmod 700 /opt/jbilling/bin/*.sh” so we can start and stop the server easily.

Run “/opt/jbilling/bin/run.sh” to start the service and then open http://<IP_Address>/billing/signupEntity.do?action=setup in a web browser and follow the Getting Started Guide here.

This is easy as pie to set up in any desktop email client, but I check my email on the move with my Nokia e61, on my laptop when I get a chance to sit down, Outlook, Thunderbird, Outlook Express or Evolution when at one of my work desktops, and webmail everywhere else! Therefore I need the emails to be sorted on the server, before they get downloaded to an email client.

Sieve seems to be the tool of choice, but you have to compile it from source on Debian, which defeats the point. People use Debian because of it’s package management. I log into each of my servers each week and run “apt-get update;apt-get upgrade” and I know everything gets brought up to date. Having to manually track a specific application’s release cycle and bugs just makes things messy.

The search continues….

Asigra supports Fedora Core, Red Hat ES and SUSE ES. I don’t like Fedora for a number of reasons, and this is for home use so I’m not paying for RH or SUSE, so I’ll use CentOS, the free clone of RHES. I’ve allocated a 4GB disk file for the VM. The OS itself should fit under 1GB, but the Postgresql database can hit 2GB, so I need the space. I’ve allocated 256MB RAM to the VM, but backups will run a lot faster with more.

At the OS “Installation Type” screen, you select “custom” to keep the disk footprint as small as possible. Disable the firewall and SELinux. On the Package Group Selection screen, tick the box for minimal install and complete the installation.

The application requires Postgres, so firstly you install it with:
yum install postgresql-server postgresql-jdbc

You need to run it once to build the database and create the config files
/etc/init.d/postgresql start
does the trick.
To make postgresql launch at boot, I use “chkconfig –add postgresql“, “chkconfig –level 345 postgresql on” and to confirm “chkconfig –list postgresql“. Postgres needs some minor tweeks for Asigra. you must add the line “tcpip_socket = true” to postgresql.conf and “host all all 127.0.0.0 255.255.255.255 trust” to pg_hba.conf. You then need to restart postgres to load the new config by running “/etc/init.d/postgresql restart“.

The client has some other dependencies that get resolved by installing “yum install compat-libstdc*” The client installer must be run with the “-console” switch unless you’ve installed X. Agree to the EULA and accept the default install path of “/usr/local/DS-Client”. We’re only going to install the DS-Client on this box, with the Java DS-User UI to be installed elsewhere. We’ll select a normal installation. The self contained install is an evaluation mode where everything is stored locally. We enter “localhost” for the DB host, and “postgres” as the username. We can put in anything for the password. We also want the service to start at boot time. It is normal for the client to give a warning about a missing database during an initial install. At this stage you get prompted for username, client ID and server location. These would be provided to you by your service provider.

The next thing is the encryption level. Asigra supports everything from DES (very weak, avoid) right up to military grade AES256. AES is slow, and the higher the key length, the slower the progress. I usually go for AES128. It’s significantly faster than 256, and it will take millions of years to break, rather than trillions, so it’s still good enough. Asigra uses two encryption keys; a key specific to the PC and an account key shared amongst all backup clients on the WAN. This enables common files to be encrypted and stored once, with this common key, rather than multiple copies being stored with individual keys (inefficient), or all sites sharing a single key (insecure).

Installing it on Debian is a 2 liner if you read this page. “apt-get install procmail” and adding “mailbox_command = /usr/bin/procmail -a “$EXTENSION” DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir” to the postfix config file and it’s done! Now all I have to do is dump my old procmail configs back in and I’m in business.

Linux shouldn’t be that bad, because the kernel structure is rigidly defined and every application is written and packaged in a way to just work. In fact most major distros release schedules are marketing or QA driven, rather than by a feature addition. Red Hat release a new major version of their Enterprise OS every 18 months, regardless of what new features are added in the mean time by the community. They do this because it means their customers can plan ahead for the major upgrade upheaval.

Because every package comes from a different source, you need to use the built in package management tools to do updates in an efficient manner. If I want to support PHP 5 on my server and I’m running Debian Sarge, I’m out of luck unless I want to remove the packaged PHP 4.4.3 and install version 5.2.0 from source. This now means that I need to keep an eye on Debian’s bug list and PHP’s one, and follow an extra update process.

Debian are the best and the worst of the major distros. Because they’re one of the few non-commercial operators, they can wait to release a version until it’s ready, so when they finally do release Etch, it’ll be rock solid. The down side is that you can be waiting years between versions. You can get around that by using their second release structure. They have a “stable“, “testing” and “unstable” release structure also. “Stable” is identical to their current full release version. If they release a new full version, in theory your server will switch over to this new version overnight, “testing” is their next release. It’s under constant modification until they freeze development to get everything lined up for release. “unstable” is bleeding edge stuff.

Because so many of us need current versions of packages available to us, we have to either install and maintain them from multiple sources or run the “testing” version of Debian. Unfortunately, Debian don’t recommend using anything but “stable” in the real world, and don’t put the “testing” updates through the same QA as the “stable” version. This morning, I ran an update, and saslauthd got broke. Sasl is the bit that provides the glue between postfix, my mail server, and authentication. It allows me to force a password check before allowing my server to relay email from another computer. This way I can have a globally accessible outbound email server so I don’t have to go reconfiguring my email client when I’m out and about. I’m not happy

Debian don’t “sell” their software at retailers. I download the ISOs from debian.org when I need them, so why should I download a disk image that was fixed 12, 18 or even 24 months ago. Why can’t Debian switch to the stable/testing/unstable structure as their primary release framework? Stable can still be a point in time fork of the testing development track, and deliver the the static OS experience that some need. Additional testing resources for the testing version could deliver a slowly evolving, always up to date, modern distro, with the ease of maintenance that Debian is famous for.

I’m back with Postfix/Courier-IMAP, but I’ve still got the original problem of synchronising calendars and contacts.

I setup courier-IMAP-ssl, postfix, procmail, and clamAV on my Debian Sarge server, used fetchmail to pull emails from Gmail, work and a few other places, and things were good. I had the same consolidated set of virus-free mail folders everywhere, it was IMAP so I had the choice of almost every email client on every platform when it came to reading email. For web caf????s and hotels, I installed RoundCube|Mail as a web front end (very pretty interface and it gets the job done).

That was fine for a while, but I’ve begun to rely more and more on the tasks, calendar, contacts, notes functionality of Outlook on my laptop to manage work stuff, and at this stage it’s key to my daily work. It’s always backed up, but I need a way of accessing it in the same way I access my email. I need the same up to date copy of the data wherever I go, with whatever device I’m using. At one stage I was even thinking of using Exchange! Buying in a server, Windows 2003 and a copy of Exchange was overkill, so I parked it for a while. I played with some Apache/webDAV based stuff that in theory allowed multiple Outlook instances to share data, but it still didn’t fix the problems of accessing the same data on PDAs, other OSs and no OS (web browser only). There are other web based groupware products (like eGroupWare, PHProjekt, Open-Xchange, Zimbra, SugarCRM etc.) but they either don’t connect to Outlook, connect poorly/incompletely, or have a payware connector (I’m cheap!).

A couple of weeks ago I saw a mention of Scalix on a mailing list I subscribe to. It’s a commercial product, but there’s a free, virtually feature-complete version available for sites of <= 25 users. It includes an Outlook connector that provides an almost perfect replication of Exchange’s functionality. It runs on Red Hat and SUSE, so there was a ???????0 cost to me! I don’t fancy running it on Fedora, don’t like SUSE, and don’t want to pay for Red Hat Enterprise Server (they make enough out of me in the office!) so I went for CentOS, which is a free unbranded version of RHES. Scalix needed a bit of persuasion to install. It actually refuses to install on any other distros, even if all the dependancies are there, so I edited the /etc/redhat-release to fool Scalix into thinking I was installing on RHES 4.