PuppetConf 2012

I will discuss the evolution of our organisation's automated host monitoring. Specifically, I will discuss the performance issues and custom solutions we implemented whilst progressing from a static and inconsistent monitoring solution to centralising our logic and requirements in Puppet through the use of exported resources. Unhappy with the performance of a more traditional solution using exported resources with Nagios across a large number of nodes, we opted for a home grown and somewhat crazy solution. We now rely on Puppet to automatically generate each host's customised monitoring configuration on the host itself; we do not use Puppet's stored config database as a transport mechanism. We implemented a hybrid solution that collects and validates pre-generated monitoring configuration out of band, whilst maintaining full process automation. Our solution compresses and transports configuration using NRPE, but the general idea could easily be applied to other monitoring solutions. The consequence is that we avoid the performance impact of exporting a large number of monitoring resources per host; rather we simply export one resource: is this host monitored? In a large, highly customised environment, this greatly improved our run times and scalability. Our solution wouldn't be appropriate for everyone, but is an interesting story for those using Puppet and trying to automate Nagios or other monitoring configuration for sanity. Our requirements are likely to align in many areas with the community at large; especially for other providers managing larger clusters of machines with Puppet.

Rated: Everyone
Viewed 556 times
Tags: There are no tags for this video.