I recently stumbled upon Ansible, which has an interesting approach to managing systems. It's not what I'd choose for a large scale organisation with lots of sysadmins and developers needing to maintain standards, but for the lone admin or small skilled group with good version control habits, it might be the ideal choice.
The primary features of Ansible are:
- Custom inventory scripts (with a little more work, also dynamic inventory)
- Only python and json/simplejson are required on the remote servers (if you want to use the built-in library modules)
- Existing library modules are written to be idempotent
- Playbooks letting you combine multiple library modules, and pass variables between them.
In every large organisation where I've worked, we've always used a system management database of some sort (usually home rolled) to keep track of the systems and usually their (top-level) configuration. In my current position, I have access to several several thousand systems and occasionally have to run commands on a subset of them. Until now I've been using an inventory-search script that I wrote to extract system lists, and GNU parallel to execute commands in parallel, but I've found it limiting at times (and a mess of hostlists, and failure scenarios are problematic)
In Ansible, I found the external inventory better suited to small-scale system management, but it didn't take me long to work around the limitation with my own wrapper script.
As an example, I can now run "ans app=foo dc=lon01 env=prod -a uptime" and it will run the uptime command on all "prod"uction systems configured to run the "foo" app in the "lon01" datacenter. I could also run "ans '*.foo.lon01.*' -a uptime" to match based on the FQDN. ALL the matching is now done by my inventory script, and ansible just uses the results en-masse. My script is a lot more efficient than Ansible's grouping functions.
At home, I store information about my local systems/VMs and cloud-based VMs in a stateful cluster manager, and I have an inventory script for that too. I'm also more likely to use playbooks at home, as Ansible is lighter weight than running Puppet or Chef agents, but has much of the same functionality.