Category Archives: Cisco Networks

Figuring Out What Bothers Me About Wi-Fi and “Analytics”

I’ve been to the well, my friends. And I have drank the water. 

I was most fortunate in being a participant in the by-invitation Mobility Field Day 3 event, this past week. Few events get you this close to so many primary WLAN industry companies and their technical big-guns, on such an intimate level and on their own turf. For months leading up to MFD3, something  has been bothering me about the discreet topic of “analytics” as collectively presented by the industry- but I haven’t been able to nail down my unease until this past week.

And with the help of an email I received on the trip back east after Mobility Field Day was over.

Email Subject Line: fixing the wifi sucks problem

That was the subject in the email, sent by an employee of one of the companies that presented on their analytics solution at MFD3 (Nyansa, Cisco, Aruba Networks, Fortinet, and Mist Systems all presented on their own analytics platforms). The sender of this email knew enough about me to do a little ego stroking, but not enough to know that only a matter of hours earlier I was interacting with his company’s top folks, or that I’ve already had an extensive eval with the product he’s pitching at my own site. No matter… a polite “no thanks” and I was on my way. But his email did ring a bell in my brain, and for that I owe this person a thank you.

The subject line in that email set several dominoes of realization falling for me. For example-  at least some in the WLAN industry are working hard to plant seeds in our minds that “your WLAN sucks. You NEED us.” Once that hook is set, their work in pushing the fruits of their labor gets easier. The problem is, all of our networks don’t suck. Why? These are just some of the reasons:

  • Many of our wireless networks are well-designed by trained professionals
  • Those trained professionals often have a lot of experience, and wide-ranging portfolios of successful examples of their work
  • Many of our WLAN environments are well-instrumented with vendor-provided NMS systems, monitoring systems like Solar Winds and AKIPS, and log everything under the sun to syslog power-houses like Splunk
  • We often have strong operational policies that help keep wireless operations humming right
  • We use a wealth of metrics to monitor client satisfaction (and dis-satisfaction)

To put it another way: we’re not all just bumbling along like chuckleheads waiting for some Analytics Wizard in a Can to come along and scrape the dumbness off of our asses.

In all fairness, that’s not a global message that ALL vendors are conveying.  But it does make you do a double-take when you consider that a whole bunch of data science has gone into popping up a window that identifies a client that likely needs a driver update, when those of us who have been around awhile know how to identify a client that needs a driver update by alternate means.  Sure, “analytics” does a lot more, but it all comes as a trade-off (I’ll get into that in a minute) and can still leave you short on your biggest issues.

Like in my world, where the SINGLE BIGGEST problem since 2006, hands-down and frequently catastrophic, has been the buggy nature of my WLAN vendor’s code. Yet this vendor’s new analytics do nothing to identify when one of it’s own bugs has come to call. That intelligence would be a lot more useful than some of the other stuff “analytics” wants to show.

Trade-Offs Aplenty

I’m probably too deep into this article to say “I’m really not trying to be negative…” but I’ll hazard that offering anyways. Sitting in the conference rooms of Silicon Valley and hearing from many of the industry’s finest Analytics product’s management teams is impressive and its obvious that each believes passionately in their solutions. I’m not panning concepts like AI, machine learning, data mining, etc as being un-useful as I’d be an idiot to do so. But there is a lot of nuance to the whole paradigm to consider:

  • Money spent on analytics solutions is money diverted from elsewhere in the budget
  • Another information-rich dashboard to pour through takes time away from other taskings
  • Much of the information presented won’t be actionable, and you likely could have found it in tools you already have (depending on what tools you have)
  • Unlike RADIUS/NAC, DHCP/DNS, and other critical services, you don’t NEED Analytics. If you are so bad off that you do, you may want to audit who is doing your network and how

Despite being a bit on the pissy side here, I actually believe that any of the Analytics systems I saw this week could bring value to environments where they are used, in an “accessory” role.  My main concerns:

  • Price and recurrent revenue models for something that is essentially an accessory
  • How well these platforms scale in large, complicated environments
  • False alarms, excessive notifications for non-actionable events and factors
  • Being marketed at helpdesk environments where Tier 1 support staff have zero clue how to digest the alerts and everything becomes yet another frivolous trouble ticket
  •  That a vendor may re-tool their overall WLAN product line and architecture so that Analytics is no longer an accessory but a mandatory part of operations- at a fat price
  • Dollars spent on big analytics solutions might be better allocated to network design skills,  beefy syslog environments, or to writing RFPs to replace your current WLAN pain points once and for all
  • If 3rd party analytics have a place in an industry where each WLAN vendor is developing their own

If all of that could be reconciled to my liking, much of my skepticism would boil off. I will say after this last week at MFD3, both Aruba and Fortinet did a good job of conveying that analytics plays a support role, and that it’s not the spotlight technology in a network environment.

Have a look for yourself at Arista,  Aruba, Cisco, Fortinet, Mist and Nyansa telling their analytics stories, linked to from the MFD3 website.

Thanks for reading.

Another Example of How Important Wire is to Wireless

A house built on a shaky foundation cannot endure. And a WLAN built on a shaky wiring foundation likewise cannot endure, I tellya. My friends, is your foundation shaky? Is it? CHECK YOUR FOUNDATION NOW. (I happen to sell foundation-strengthening herbal supplements on the side, if you need that sort of thing…)

I’ve long been a proponent of recognizing installed UTP as a vital component in the networking ecosystem. Too many people take Layer 1 for granted, and forgivable sins of of our 10 Mbps and Fast Ethernet pasts won’t fly in a Gig world. Toolmakers like Fluke Networks sell cable certification testers that take the guesswork out of whether a given cable run can be relied on to perform as expected. Don’t use one of these testers at time of cable installation, and you are only assuming you have a good station cable.

I just had an interesting situation come up that I helped a very skilled field tech with. He was working in several different small buildings, each serviced by a Cisco Catalyst Switch and a handful of 3802 802.11ac access points. The switches and cable had been in place for years, and the APs for many months, all with no issues whatsoever.

Then, we changed out the old 3560X switches for shiny new 3650s (curse you Cisco for your bizarre fascination with part numbers so close together), and suddenly some APs weren’t working any more. Between us, we checked all switch settings, POST reports, CDP tables, logs, etc- everything you can dream up on the switch. We put the APs that weren’t working back on the old switches, and they came right up. Hmmm… thoughts turned to PoE/code bugs, but then I went a-Googlin’ before consulting TAC.

I found this document that put me on the path to righteousness. Though we weren’t having “PoE Imax Errors”, a couple of nuggets jumped out at me about our new switches.

PoE Imax

Holy guacamole- We got us a situation! But wait… THERE’S MORE!

PoE Imax2

Shazam! Which, of course, translates in Esperanto to “maybe your cable is actually kind of iffy, and all the CDP stuff that happens at the milliwatt level before PoE gets delivered worked OK with your old switch but not with the new one that has the enhanced PoE controller”.

If you don’t know that the newer switch does PoE differently, you might wrongly assume that your cabling is “good” because the APs worked on it when those APs used the old switches connected to that wiring. By now, you can probably guess where I’m headed…

Our tech tested the cabling on the new-switch-problem APs and in each case found that they needed help to work with the new switch. He re-terminated and tested each, with the APs then coming up with no issues. I have no doubt that this cable was certified 10-12 years ago, but in that time a lot can happen to either end of those cables depending on the environments where they are used.

Live and learn!

 

 

Cisco’s Latest AP is Mind-Blowing (and a quick history lesson)

Aironet 4800 Access PointFeast your eyes on that little Chiclet-looking thing… No image can do justice to Cisco’s latest powerhouse AP. That innocuous looking image represents a full 5.6 pounds (2.5 kg) of all kinds of Cisco’s latest technology in the company’s new 4800-series access point. You got 4×4 802.11ac Wave 2 radio wizardry,  a built-in hyperlocation antenna array, and BLE beacon capability. And… regardless of whether you buy into Cisco’s DNA Center story, the new 4800 has a lot of DNA-oriented functionality. It’s big in size, functionality, and at least for a while- price.

You don’t need me regurgitating the entire data sheet- that can be viewed here. You’ll also want to hear the full story of the 4800 and DNA Center when you get a chance, because it’s nothing less than fascinating. (My own take: DNA-C might be revolutionary- but I’d rather see new controllers with a new WLC operating system rather than bolting DNA-C’s future-looking promise onto yesterday’s fairly buggy wireless parts and pieces. That’s just me speaking from experience- take it or leave it).

I’ve seen the 4800 with the outside cover removed, and even that is profoundly thought-provoking when your eyes take in how much is really going on with the various antennas- get a look at that if you can (I’m not comfortable sharing the images I’ve seen, not sure where NDA starts and stops on that).

So a huge access point story is afoot, and I applaud Cisco on that bad-lookin’ mammajamma. But I also got sparkley-eyed by something else fairly nerdy while looking through 4800 materials and links to other links.

Here’s a screen grab of the 4800 power specs:

4800 power

Nothing real exciting there, right? New APs generally need the latest PoE+, and we’re a few years into that story. But I somehow stumbled across this document, that shows this picture:

and it took me way back to my own early days of wireless. My WLAN career started with a 4-AP deployment of those 350s, which ran the VxWorks for an operating system and had only 802.11b radios… (cue the flashback music here).

Also included in that doc is this brief history of PoE:

PoE Hist

As I read that over, my mind goes back to all of the Cisco APs that have come and gone in my own environment- 350, 1130, 1200, 2600, 3500, 3600, 3700, and our latest in production, the 3800. In this list, there have been multiple models from the different series of AP leading to the thousands of APs that are now deployed in my world.

On the operating system side, VxWorks became IOS, and in turn AireOS. Now we have AP-COS on the latest Wave 2 APs (don’t Google “AP-COS”, most of what comes back is bug-related, sadly).

It’s interesting to reflect back, on operating systems, PoE, radio technologies, and feature sets. As Wi-Fi has gotten more pervasive, it has also gotten more complicated on every level. Seldom is the latest access point THE story any more, now it’s about all of the features that come with the whole ecosystem that the vendor wants that access point to operate in- if we as customers buy into the bigger story.  I’m not passing judgement on anything with that statement, or intentionally waxing nostalgic (well, maybe a little bit).

It’s pretty neat how one image or a certain document can suddenly flash your your entire wireless history before your eyes.

Good stuff.

A WLAN Doer Contemplates the Cisco/Apple Partnership

I’ve been in the wireless game with Cisco products since long before thin was in. These days, I support many thousands of access points and tens of thousands of Wi-Fi clients on those APs. At least half of those client devices are Apple products, and in some spaces in my environment, as many as 85% of all clients are Apple. Obviously, I hope for the best of outcomes from the new Cisco and Apple partnership, as my customers would benefit from those positive outcomes. There’s no meanness intended in what follows, just reflection on days past and what I hope comes of these two market leaders becoming more collaborative.

Code Counts as Much as Hardware

Cisco and Apple both put out beautiful hardware with premium price tags. Many purists who worship either or both companies have a hard time believing that anything defective could come in hardware that is so robustly built, pretty, and expensive. If my iDevice isn’t working, your network MUST be to blame. And if my WLAN is acting up, it must have been designed wrong because Cisco code isn’t cheap… and it comes from the market leader, by golly. Both Cisco and Apple are at the top of their games as measured by volume of devices in many large and small WLAN environments. And both frequently, too often, put out mediocre (or horrible) code that leaves people like me holding a bag full of smelly network pain.

In Cisco’s case, their WLAN controller code is just short of being chronically buggy, and a culture of “get it out the door and let our customers QA it!” seems to rule the product line. (Greg Ferro sums it up nicely in the opening paragraph of this article.) It’s not uncommon to spend days on the phone with TAC only to find out that randomly rebooting controllers or some oddball client behavior is actually a known bug.

For Apple, you never know what you’re going to get related to Wi-Fi behavior with OS and iOS upgrades and patches. Release notes are scant, and it seems that the Wi-Fi area of Apple devices is always being tinkered with back on the mothership. From a history of sticky-client behavior to curve-balls in how you are “allowed” to configure profiles to decidedly non-enterprise quality gimmicks like Bonjour, it has been an interesting ride administering business networks that have lots of Apple wireless clients on them. (This is not just me ranting, the Apple support forums are chock full of frustrations with Wi-Fi client behavior through the years.)

Features? What About Standards (and stability)?

Cisco networks also have to support a lot of non-Apple client devices. Making Apple’s consumer-centric AirPlay/Bonjour feature sets work in large business enterprises can be a nightmare. And though Cisco (and other vendors that do similar) mean well with mechanisms like band-steering and load balancing across APs, these enhancements cause their share of problems in the Wild West of widely varying client types found on big WLAN networks. It would be nice to see more focus on standards-based interoperability and feature sets rather than vendor-proprietary juju.

Looking Forward

I used to marvel a bit at Apple’s mastery of talking out of both sides of their corporate mouth when it came to their place under the network sun. Sometimes they were unequivocally not an Enterprise company, and sometimes they were. It seemed to depend on the audience, and how well their unyielding way of doing things fit into the general networking landscape where they were trying to gain specific market share. Now, with the Cisco alliance in play, Apple is emphatically stating that they are an Enterprise player. Hopefully, the company gives strong consideration to what that means to all of the users who love Apple gear but get frustrated because too much of the “Living Room, Single Class C Subnet Network” mentality is in play.

From the Cisco side, ideally my Wi-Fi vendor won’t skew their already frequently-frustrating code too far in the Apple direction at the expense of the rest of the client devices that have no use for Apple-specific features. Also ideally, Cisco would also find a way to end the code bug madness before it starts tweaking WLCs to do magic things for iDevices, lest bugs beget bugs.

This could be absolutely wonderful for environments like mine, or it could just be more of the same- but disappointingly amplified. I’m crossing fingers that both companies get it right…

In Cisco Prime Infrastructure, Let the Picture Tell a Radio Story

Say what you will about PI (and I say plenty, sometimes under my breath), there is a lot packed into this Network Management System. One of my favorite features for remotely characterizing how individual APs see each other may not be so obvious, so a quick show and tell is provided here.

The goal? Let me:

  • see a floorplan with APs on it
  • ask any one of those APs how it sees it’s neighbors from the radio perspective as they are all mounted
  • show me one band at a time
  • show me signal strengths of APs on the same floorplan, and NOT on the same floorplan

And as a bonus- show me a slew of radio specific data for any AP’s individual radios.

Before the run-through, a quick note on when I use this functionality… In areas where RRM is allowed to do it’s thing, this is a handy way to let the radios themselves tell you how they see their places in the RF world in relation to each other. Where RRM is over-ridden for whatever reason, this becomes one more sanity-check data point on whether you got the intra-AP power settings right. What this does NOT do- does not show you the system from client device perspective, and so is only part of the bigger picture. But is a handy one at that…

Got it? Then here we go.

  1. Monitor\Site Maps\specific building\specific floor

    Wash 1

  2. Select an AP, drill into one of it’s radios

    Wash 2

  3. Notice the radio details in table form. Good stuff, sure… now click into “View RX Neighbors”

    Wash 3

  4. The example AP is on 5th floor. It sees one other AP on the same floor, and two more from the floor below, at discreet strengths. You get a sense of propagation with this simple view through walls and floors if you know the power levels in play. Again, is one more place to gain a useful data point or two in complicated environments that gets beyond just heatmap juju.

    Wash 4

  5. Then there’s the click into all of the radio data available for a given interface

    Wash 5

  6. And behold- all of the RF stuff you might need for troubleshooting, contemplating RF changes, etc. (Yes, this info can also be accessed from Monitor AP path).

    Wash 6 Wash 8

Nothing earth-shaking here, but I’ve met enough fellow PI users that are either new to the platform or who just didn’t know about these features that sharing seemed prudent.

Thanks for reading!

In Appreciation of White Box Guest Access

“Guest Access” means different things to different people, and organizations. Certainly if you’re a traveler using hotel or conference Wi-Fi, you have a general set of expectations and desires. If you’re a company or a school, the guest wireless service you provide is likely shaped by organizational policy. And for many of us, the guest environment also tends to act a s a catch-all for client devices that don’t fit on our secure WLANs- a place for “free passes” and MAC exceptions. But the devil is in the details, and I have found finding the right guest access feature set can be difficult.

What you WANT may not be what you can HAVE

Having designed a number of guest environments for large and small networks, I’m always astounded to engage a WLAN vendor on the topic and to find how far their guest offering is from what I’m looking for (more on that in a bit). Worse, seldom do I hear “what are your requirements?” as it tends to be more like “this is what we think everyone should want and accept”.

Simplicity? Fat chance… 

Guest access can also have a lot of moving parts, depending on how it’s implemented. Overall functionality tends to be broken up and scattered across access points, controllers, RADIUS servers, credential stores, web servers, and sometimes switches. It all has to click, or you have problems. And for me, despite the typical complexity of guest services, I still find myself frustrated at features that are not included.

What worked for my environments

Years ago, for my big honkin’ 3,000 AP environment (and our small branches alike), we arrived at a desired feature set that went more or less like this:

  • Our guest SSID would equal a single dedicated guest VLAN
  • 24-hour individual self-sponsoring is a must
  • Alternatively, ANYONE authorized to use our wired or secure wireless network could sponsor a guest
  • For self-sponsoring, a ten-digit mobile number capable of accepting a text must be provided and within seconds a password would be sent
  • For large events, a shared account could be generated
  • All accounts were time limited with role-granularity
  • The system would have easily configurable firewall rules and (generous) rate limiting capabilities
  • On the admin side, we could add MAC exceptions and login-bypass
  • The system would provide NAT to preserve public IP addresses
  • Reporting would be easy, as would user quarantine (rarely used)
  • ALL OF THIS WOULD HAPPEN UNDER ONE HOOD-VIA A SINGLE INTERFACE
  • A programmer would not be needed to stitch it all together
  • Ideally, it would have vendor support (for a number of reasons, open source not desirable)

Going back those several years, our WLAN vendor (Cisco) didn’t come close to being able provide what we wanted. In their defense, nor did any other market leaders at the time. We heard that Colubris Networks had a gateway that might fit the bill, but they had just been bought by HP and try as we might, we couldn’t locate anyone that could talk with us about what we were looking for.

Then we found Bluesocket (now Adtran) and their BSC Controllers. When I first contacted Bluesocket, we came to the mutual realization that they could do about 75% of what I wanted. They weren’t really initially open to developing the self-sponsored texting and “anyone authorized can sponsor a guest” features. So… we thanked each other for our time, and I kept searching. Then a week or so later Bluesocket called back, and said they were game for a bit of development, and saw the value in what would become a feature set that they were able to market to others. They were able to do everything I was looking for in a single, kick-ass box in a matter of hours.

What Bluesocket was able to deliver after actually listening to our requirements has been in play for us for lots of years. We’ve served thousands and thousands of guests with it, along with using it as a mechanism for supporting wonky devices like Google Glass (turn head, spit) that weren’t built with enterprise security support, and so can’t be on the WLANs we’d rather they used.

It’s been absolutely great, and I know of at least three other schools that pursued the same guest access model after experiencing ours.

Looking forward

Our old Bluesocket boxes are getting, well… old. They are appliances, and Adtran seemingly has no desire to virtualize what we need into an OVA or the like. In fact, on newer Adtran wireless products, what we appreciate about the BSC has been moved to Adtran APs that we’ll never buy, so the research for a suitable replacement starts again.

The thing is, we absolutely love what we get out of our aging guest solution, and in a perfect world, I’ll find a similar third-party, one-box bolt-on for our big Cisco WLAN. (I will give Cisco another chance to catch me up on how their native guest access services have improved, but I also know that my requirements are firm). I have also inquired to Adtran one last time about the possibility of somehow preserving this wonderful magic, but the silence thus far is pretty telling.

Which brings me to Meraki. The features I need for my guest environment are pretty much included in the WLAN side of the Meraki product line, and we use it with great success in our Meraki-enabled branch sites. But… to bolt the Meraki capability up to my Cisco WLAN in a way that would replace Bluesocket, I’d need the guest features made available in the Meraki MX security appliances and not just in the AP feature set. I’m hoping to get Meraki’s ear on this anyway, because guest access needs also do tend to pop up on the wired side occasionally, too. Right now, wired guest needs are a gap in the MX.

If Meraki can accommodate, a big MX would snap in nicely where my Bluesocket sits now for guest access. If not, I’ll have to consider things like pfSense, Packetfence and other one-offs that I’d rather not get into after being happy with a commercial solution. Or, I’ll have to rethink our requirements, which would really suck, as they really are what we consider requirements, not just nice-to-haves.

There will obviously be more to follow to this evolution.  I am curious if anyone else is facing a similar situation, and how you might be approaching it.

(Please- I’d love your comments, just don’t blast me with pointless “you should switch to vendor X for your WLAN!” type feedback.)