How to add a timeout to a Zabbix alert trigger
Monday, May 31st, 2010I’ve replaced Cacti and Nagios with Zabbix to monitor the company infrastructure. Nagios is nice but there are some things you can only monitor using agents as Linux’s SNMP support is just too fragile. It’s also nice having monitoring and charting built into one easy to configure web application as opposed to drowning in config files.
My one pain point was state flapping. A service can go unresponsive for a fraction of a second under heavy load and come back right again. All to frequently my inbox would get swamped with problem…ok…problem…ok….problem…ok messages. By the time I’d log in everything had settled down. It’s not a perfect situation, but doesn’t warrant getting out of bed to repair either. Nagios has a flapping detection function but Zabbix doesn’t.
What I’ve found is that the AVG function can be used to average the status values (“1″ or “0″) over a given period and only trigger an alert if it drops below the alert value. For example
{server01:net.tcp.port[, 80].avg(30)}<1
will only trigger a second alert if the web server stays responsive for at least 30 seconds between outages. Also
{server01:net.tcp.port[, 80].avg(30)}=0
Will only trigger an alert if the service stays down for more than 30 seconds at a time. Further you can combine the two in
{server01:net.tcp.port[, 80].avg(60)}<0.5
to only trigger an alert if the service stays down for more than 30 seconds and only trigger an OK if it stays back up for 30 seconds.

I bought a
Because my current work style is very fluid, where I hop between my office and home PCs, a laptop, a
Documents: I’m fairly neat when it comes to organising data on my laptop. Everything is in the My Documents folder. I used to use
Email: POP is for noobs; IMAP is king. Seeing as Google are generous enough to give away 7GB+ of email storage I’d be a fool not to use it. A few years ago (pretty much as soon as it was available) I created an @garypigott.net Google Apps account and shoved all my email up there. Right now there are 118,527 emails, occupying 2298MB of space, and vitally, it’s completely searchable in exactly the same way you’d search the web using Google. I access exactly the same data using
Web Browser Environment: My browser of choice is 