Great Circle Waypoints

LOPSA board member Trey Harris has posted an excellent message outlining his thoughts on effective organization and scheduling for groups of sysadmins in a high-interrupt, high-profile, high-availability environment (Amazon, Google, etc.).

Trey's message was part of a very interesting discussion taking place on the LOPSA "discuss" mailing list regarding "interruptions coverage" for sysadmins. The basic question under discussion is, given that much of system administration work is by nature interrupt-driven, how can an organization best shield some of its sysadmins' time from these interrupts, so that the sysadmins can get long-term work done (and maintain their own sanity!)? To read the whole thread, search for "Interruptions coverage" in the list's archive.

I think that this discussion (and Trey's contribution in particular) is an excellent example of the sort of thoughtful discussions from experienced professionals which you can expect from LOPSA, which is why I'm encouraging everyone involved with system administration to join and support this important new organization.

USENIX has made available an audio recording (in MP3 format) of the Incident Command for IT: What We Can Learn from the Fire Department invited talk (Adobe Acrobat PDF format) that I did at the 2005 LISA conference a couple of weeks ago. You'll want to skip past approximately the first 3 minutes (2 minutes, 56 seconds, to be exact) of the recording, which are silence and administrivia announcements from before the start of the presentation; it would have been nice if USENIX had edited that out, but they didn't.

I'm back home today after two weeks on the road, and it's good to be home. The first week, I was in San Diego for the annual USENIX LISA conference (where LOPSA was a major topic of discussion), then I was home just long enough to do laundry and repack before heading out to New York City for the Interop conference and exhibition (where I was helping Splunk showcase their product in the InteropNet NOC).

LISA and LOPSA

At LISA, I participated in the Social Technologies and Advanced Technologies Workshops (small, day-long discussions among senior practitioners who are especially interested in a particular topic), gave an invited talk on Incident Command for IT: What We Can Learn from the Fire Department (Adobe Acrobat PDF version), and chaired a Birds of a Feather session on network automation.

Much of the "hallway track" discussion at LISA, of course, was about the implosion of SAGE and subsequent formation of LOPSA, the League of Professional System Administrators. I think that LOPSA has a chance to become a world-class organization, but the first year or so are going to be critical, and it isn't going to happen if everybody takes a "wait and see" attitude. What's going to matter most are money and membership numbers. To try and make sure LOPSA reaches critical mass, I've joined as a Platinum Individual Sponsor, and I encourage others to join/donate/volunteer as they are able.

Interop and Splunk

At Interop, I was working in the NOC for the InteropNet show network, helping Splunk showcase their product. Splunk's product is a really powerful troubleshooting tool for interactively working your way through huge volumes of any sort of text-based system logs (syslog, SNMP traps, whatever). It is designed as a tool for use by smart people who understand what the log messages mean, if only they could wade through the flood; the software doesn't try to understand the messages itself, just makes it easier for a system/network/security administrator to navigate through the flood of messages and find the needle in the haystack. There's a very powerful free version of their software (the paid version adds various features, but the free version is fully functional by itself) available from the Splunk web site; check it out!