Automated Tragedy

A couple years back, an HP engineer at CommBank in Australia managed to format some 9,000 workstations using Microsoft’s System Center Configuration Manager. The problem is…they weren’t trying to. Turns out that an operating system deployment task sequence was inadvertently advertised to nearly every computer in the environment, meaning those machines, whether they needed an OS reinstall or not, ended up getting one. Let’s hope that backups were good. At the time, I shared this horror story with a client where we were preparing to do a global Windows 7 deployment using Configuration Manager, and as we tried to comprehend the sinking feeling that the at-fault administrator must have felt when he realized what he’d done, we laughed and thought, “this will go down as an IT legend…no one will make that mistake again.”

We were wrong. Looks like on or about May 14th, Emory University managed to accidentally accomplish the same thing. I’ve seen a couple sources on the internet surmising that this was the result of a bug or vulnerability in Configuration Manager, but before even reading Emory’s post mortem, I would have bet good money that this was, again, an accidental advertisement of a task sequence to machines that shouldn’t have been targeted. And it looks like that’s exactly what happened:

“A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks.”

From a very general standpoint, I think this serves as a very important warning. Automation can be hugely advantageous, but it can also do things that are really, really bad. Sure, I can reimage machines in just a few minutes with Configuration Manager, and that’s great when prepping 100 workstations for a new branch office, but I can also reimage machines in just a few minutes with Configuration Manager, and that can be career ending if I accidentally do it to a datacenter full of servers. Please don’t misunderstand: I’m not trying to scare anyone away from automating repetitive tasks. Automation is, in general, great when it’s used to save time and increase consistency of results. But because tools like Configuration Manager and System Center Orchestrator and their stored credentials and agent permissions allow an administrator to reach out to and perform operations on machines there they wouldn’t otherwise have permissions, they need to be properly implemented, scoped, and secured.

Of course, a dose of “measure twice, cut once” doesn’t hurt when you’re about to click “OK” on a broadly scoped OS deployment in Configuration Manager, or when you’re about to kick off a Runbook in Orchestrator that might loop through all the user accounts in an Active Directory OU. Run your task sequences and automation runbooks against test collections or small groups of machines/objects to make absolutely sure they are doing exactly what you want before deploying them to a broader production audience. I still get nervous before clicking “OK” on a Configuration Manager deployment that’s targeting “All Systems” or “All Workstations” because I know there’s no big stop button I can hit if I got something wrong.  What I am saying is, don’t be afraid to have someone double check your settings before clicking “OK” on a big deployment or before promoting an automation task in to production.

Now, being a big Configuration Manager guy, I have to point out a way to avoid this particular Configuration Manager issue. In both of these cases, it sounds like not only were all machines targeted with the task sequence when they shouldn’t  have been, but also that the machines were allowed to run the OS deployment from within the currently installed operating system. Although there are a million ways to configure targeting behavior and user interaction in Configuration Manager, a scenario I’ve almost never understood was advertising OS deployments to currently active workstations so that they are presented to the users. That basically means that as a user I’m going to get a pop-up prompting me to run the new OS deployment that’s been made available, which will then probably format my workstation and reinstall Windows. I’d need to be very confident in the use of network file shares, folder redirection, and/or backup routines in my OS deployment for that to be something I want a user to self-initiate. To make matters worse, it sounds like for both CommBank and Emory, the OS deployment was mandatory, meaning machines would run it even without user interaction. Thankfully, there are ways to prevent task sequences from ever prompting users from within Windows, or from running automatically on existing workstations. Here are some tips and tricks to consider.

  • If you’re using Configuration Manager 2007 or Configuration Manager 2012 RTM:
    • You have to “trick” things a little bit, by telling your task sequence that it is only eligible to run on a specific OS. For example, I like to use Windows 2000 SP4, since I’m hoping no one is still running that OS in the wild. Once set to only run on that specific OS, any existing Windows XP or Windows 7 machines will receive the OS deployment task sequence, but evaluate that OS version condition, and reject the advertisement. It’s a bit of a hack, but believe it or not, a Configuration Manager developer told me that this was the same way he did it when I asked for a best practice.
    • When you now boot workstations to PXE, USB, or CD-ROM boot media, they will still be able to run the advertisement in question, since the Windows PE boot environment for Configuration Manager does not check the currently installed OS against the condition. Under these circumstances,  I have advertised task sequences to every computer in the environment, but never run in to a scenario like that at CommBank or Emory.
  • If you’re using Configuration Manager 2012 SP1 or later:
    • Thankfully, Microsoft has now made this an easy GUI setting. When creating your task sequence, simply select whether you want your task sequence to be available to Configuration Manager clients (meaning active Windows operating systems with the client installed), boot media and PXE, or both. Under these circumstances, you can easily restrict task sequences so they never prompt users or start from within Windows.
    • Important note: these settings are only obeyed by Configuration Manager 2012 SP1 clients or later – any machines in your environment with the 2012 RTM client will not know how to follow the setting, and end up executing the same nightmare scenario that happened to CommBank and Emory. A great reason to make sure your clients are upgraded, if you haven’t already!

osd-sp1-01

Your email address will not be published. Required fields are marked *

Phone: 312-602-4000
Email: marketing@westmonroepartners.com
222 W. Adams
Chicago, IL 60606
Show Buttons
Share On Facebook
Share On Twitter
Share on LinkedIn
Hide Buttons