A Day in the Life- When WLAN Pieces Don’t Live Up to Their Responsibilities

I stared into the darkness, and softly spoke
“What the shit is this? Why didn’t it reboot?”
The early morning mocked me
The clouds and the birds and the rising sun
Even my first cup of coffee
All sang and screamed and laughed
“Your stupid WLC didn’t reboot! It didn’t reboot!”
And so I laughed, like an idiot, as not to cry.
Beef Wellington, from The Controller Chronicles

Sigh… Sometimes things don’t do what they’re supposed to do. Like in the case of a simple Cisco 8540 controller upgrade. It matters not that I’ve done this procedure about a hundred times through the years. THIS TIME, the controller had it’s own idea about how this code upgrade would go down.

And Time, a maniac scattering dust,
And Life, a Fury slinging flame.
Tennyson, from In Memoriam

no reboot

The maintenance window was claimed. Change control was done. Code was downloaded to the 8540. And… the required reboot was scheduled May 30, at 0400.

Yet… 05:16 rolled around on May 30, and the reboot was still configured for 0400.

Have you seen the bruise on that man’s head?
   -Professor, on Gilligan’s Island: Waiting for Watubi episode

The bruise is mine. From beating my head against the wall. But whatever… forcibly reboot the 8540 (slightly outside of the maintenance window- but don’t tell anyone). Now all is good- except for dozens of APs that lose their config in the process.

APs default

EXCEPT not ALL of the APs that went to defaults REALLY went to defaults. Only about 20% did. The rest come back proper, with full config, if you remove and restore their power. It makes no difference that they are correctly showing in CDP, drawing good inline power, etc. You’ll reboot if you want them back. That other 20%? They really are defaulted. Build their configs from scratch, and shut up about it.

I need you, I need you, I need you right now
Yeah, I need you right now
So don’t let me, don’t let me, don’t let me down
I think I’m losing my mind now
It’s in my head, darling I hope
That you’ll be here, when I need you the most
So don’t let me, don’t let me, don’t let me down
D-Don’t let me down
Don’t let me down
Chainsmokers, Don’t Let Me Down

Sorry, Chainsmokers. Letting people down is kind of a way of life in/for these parts.

6 thoughts on “A Day in the Life- When WLAN Pieces Don’t Live Up to Their Responsibilities

      1. Alan Wang

        We just upgraded code on one of our main campus 8540 pairs from 8.2 MR4 to 8.3 MR2 and we experienced the same thing you had happen in this blog post where APs came back up and didn’t have their config intact. I noticed that after a poe power cycle on the switch that the APs seemed to have failed to download the code during the initial upgrade procedure and once cycled, picked up the new code. On a controller of ~1700 APs exactly 1% of them are in this limbo state.

      2. wirednot Post author

        Hi Alan- this is where I get really irritated. To have this go on for multiple customers, and then to hear from the vendor that “you are probably spending too much in OpEx!” This IS OpEx, bitches!

  1. Pingback: Sequel: A Week in the Life- Cleaning Up Afterwards- When WLAN Pieces Don’t Live Up to Their Responsibilities | wirednot

Tell me what YOU think.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s