Category Archives: Networking

A Little Quiet Please- The Jabra Evolve 75

Jabra 75 EvolveFew things are more annoying when you’re troubleshooting in a noisy environment while trying to work with technical support or a coworker over the phone and you can’t hear them (or they can’t hear you). There’s usually enough stress in play to begin with, and having to repeat yourself and say “what? I just can’t hear you…” over the exhaust fans of network equipment in packed racks only makes it worse. Enter the Jabra Evolve 75.

Quiet Down!

Marketed as “the best wireless headset for concentration in the open office”, the Evolve 75 from Jabra is impressive for a number of reasons.

I don’t work in an open office environment so I can’t validate Jabra’s claim in that regard, but I can tell you that the Evolve 75 impresses in the noisy data center and telecommunications closets frequented by network technicians and administrators. I recently took an evaluation unit for a spin, and the timing was perfect for a real-world test under extremely loud background noise conditions. Not only could I hear the support engineer on the other end perfectly, but he also said my audio was clear despite the Evolve 75 mic being  inches away from blowing server fans. It’s hard to ask for more than that, but there is more to talk about.

Classy

Jabra has always made fairly elegant Bluetooth headsets for cellphones, in my experience. This was my first outing with a professional grade Jabra product, and it blew the doors off of other headsets I’ve tried to use in similarly noisy situations. The Evolve 75 is fairly light, but substantive. It feels good on my big head, and the mic raises and lowers smoothly, and in both positions is unobtrusive. The overall look and feel whether it’s on your head or in it’s beefy charging cradle is one of quality.

Functional

I’m finding that the Evolve 75 has really, really good battery life for my usage patterns- but I also do not wear it as long as a call-center person or the like might. The ANC (Active Noise Cancellation) functionality works like a dream, and these may be the absolute quietest phones I’ve ever used. I manipulated the volume buttons and mute functions on WebEx, Skype, and simple phone call without having to fumble around, and it doesn’t take long to feel at a disadvantage when you take a call without the benefit of this headset.

I also made use of the unit’s ability to pair with two Bluetooth devices at once, which proves handy more often than you might think. When paired simultaneously to my computer and my smartphone, the new-found mobility and seamless transitioning between the two was just a joy to experience. I’m not one to gush, but I kinda fell in love the Evolve 75 and don’t mind saying so.

Nice Bonuses

I’m not sure that I qualify as an audiophile after all the beating my ears took working around loud fighter jets in my Air Force days, but I do enjoy music and the occasional podcast. Jabra makes both activities more enjoyable with the sound quality, volume, and purity that the Evolve 75 delivers. It also comes with a really nice travel case.

The only downside I see with the Evolve 75 is that it’s one of those really, really nice tools  that you miss greatly when you don’t have it.

 

NETSCOUT’s Next-Gen LinkRunner Tester Is Ready For The Changing Network Landscape

Just when you thought that maybe all of the cool testing innovation was reserved for Wi-Fi and the likes of the AirCheck G2, NETSCOUT brings out an equally impressive wired networking tester. The new LinkRunner G2  (shown on left below) sports the same color scheme and physical profile as the AirCheck G2:

IMG_20171127_140442975

But a closer look topside reveals some tell-tale features:

IMG_20171127_142046889_BURST000_COVER_TOP

What isn’t obvious from these images is that NETSCOUT made the LinkRunner G2 with a whole new user-customizable philosophy in mind that applies to both hardware and software. See the fiber SFP on the left and the USB WLAN adapter on the right side in the second image? You don’t HAVE TO get those from NETSCOUT if you find a better price on similar interfaces elsewhere. Unlike some vendors, NETSCOUT opted to be very accommodating of 3rd party adapters. And the LinkRunner G2 is actually a hardened Android computing platform that you can tweak in a number of ways- but let’s come back to that in a bit.

Recognizing the LAN-Scape For What It’s Becoming

In talking with LinkRunner’s product management, I was able to hear the inside scoop on where the company sees the product fitting into the connected world. It’s no secret that the number and type of network-connected devices “out there” is skyrocketing, but investment in support staff and their capabilities isn’t for many organizations. That being said, the LinkRunner G2 is viewed by NETSCOUT as the “smart network tester for the connected world”. Now, I’m as buzzphrase-adverse as anyone, but the deeper you dig into the LRG2, the more you realize that NETSCOUT is not over-hyping the new tester’s capabilities. With strong physical layer support capabilities, LRG2 is handy before the network even goes live. On active networks, Ethenet and core services are tested and characterized nicely. Then there are the true differentiators- and Power over Ethernet (PoE) functionality is a prime example.

Robust PoE Measurement and Charging

NETSCOUT points out that today’s LAN environment is no longer just a client-access domain, it’s also a power-distribution system for a growing number of devices. Beyond VoIP phones, CCTV cameras and wireless access points, we now have lighting systems, locks, and Bluetooth sensors among the many devices using PoE for operating power.

Netscout describes the LRG2 as the only current tester fully able to support all versions of PoE including Universal PoE (UPOE) that sources 60 watts at the switchport and up to 51 watts at the field jack. And this LinkSprinter also charges off of PoE!

Screenshot_20171127-184138

Link-Live

Depending what other NETSCOUT (or Fluke Networks) tools you use, you may already be a Link-Live user.

 

Link-Live makes the storing, sharing, and reporting of test results and absolute cakewalk. When you have a number of staff with varying skillsets, NETSCOUT’s Link-Live-capable tools can bring a uniformity of testing that reduces errors and faulty troubleshooting, and makes the results available for reference and escalation. It’s a free service, and LinkSprinter G2’s tests are as well formatted as the likes of the AirCheck G2’s. I’m a fan.

The Multi-Function Tester That You Can Customize

This tester is still a LinkSprinter, so you’d expect to see views like this on the crystal-clear touchscreen as you test copper and fiber links:

 

But there is sooooo much more. Remember, I said that this is a full-blown Android device. It also happens to have many “phone-like” features including a built-in flashlight, camera (attach pictures to Link-Live reports or email them from the LRG2), web browser, screen shot capabilities, Micro SD slot for file storage and portability, and even access to other apps that can be installed on the tester.

Screenshot_20171127-184637

This is so handy, and almost too good to be true. The caveat to the starting app paradigm? You don’t have access to the full Play Store. This is a hardened Android device after all, and you do not log in to the Play Store with your own account. But you do have access to a wide range of vetted network/documentation/productivity apps via NETSCOUT’s partnership with Google. If you find an app that you’d like, NETSCOUT provides an easy path to request it.

Within five minutes after discovering this capability, I had my test unit nicely loaded with some of the same Wi-Fi and network apps I use on my own Android phone and tablets, and the ability to run them all off of a robust network tester feels incredibly empowering.

Screenshot_20171127-184155

There is just a lot here to appreciate in the LinkRunner G2. It’s clear that NETSCOUT was shooting for versatility and expandibility with this network tester, and they hit both targets nicely.

Learn more at product web site.

 

Future-Proofing Networks with Fabric-Attached Wi-Fi: Q&A with Extreme Networks’ Director of Wireless Product Management & Strategy

It’s easy to become desensitized to the onslaught of marketing that surrounds networking concepts like “fabric” and “unified networks” when every vendor has their own version of them. Naturally, each marketing department promises that their solution is the best, but reality shines through when you start to look past the buzzwords for substance. I was recently  introduced to (and impressed by) Extreme Networks’ own fabric accomplishments, and wrote about my impressions here. Soon after, I had the chance to talk with Extreme’s director of wireless product management and strategy, Mike Leibovitz, about where WLAN specifically fits into the company’s fabric approach.

Leibovitz is one of those people that I’m always glad to catch up with. I’ve spent time with him at different Tech Field Day events and  IT conferences, and have had opportunities to socialize with him. Beyond just being an all-around nice guy, Leibovitz has a passion for his job and believes strongly in Extreme’s products, methods and his company’s future. Our most recent conversation evolved into an informal Q& A about the Extreme Automated Campus solution and Wi-Fi. Here are the highlights from that discussion (I’m in italics).

Mike, Extreme has been busy integrating the likes of ExtremeWireless WiNG from Zebra/Motorola and Avaya’s fabric portfolio (from recent acquisitions) with Extreme’s own wireless product lines. How’s all that going?

It’s been a great run, for us and our customers. We’re fully supporting all product lines, and it’s only getting better for the end users, regardless of which hardware they use. Looking forward, the best of all our product lines will be fused into new feature options that customers of either ExtremeWireless WiNG or ExtremeWireless can take advantage of without forklift upgrades.

We’ll get to fabric and Wi-Fi in a bit, but first- is there anything on the horizon that is particularly driving Extreme’s WLAN-specific evolution, and do you have any examples of where ExtremeWireless WiNG might bring something new to Extreme’s story that customers can appreciate?

Aside from our fabric architecture taking deeper root, we see the coming of 802.11ax as significant, and that does figure into our current product evolution. As the radio side of the equation gets higher in performance, we’ll continue to leverage things like Motorola’s unique excellence in access point design for challenging and high-ceiling environments, for instance. Also, we have the successful integration of the Azara Cloud into ExtremeCloud as an example of how we make what’s good even better.

It seems that Extreme goes to great lengths to make sure that new customers gained through acquisitions are treated just as well as long-time Extreme customers. Is that a fair characterization?

Absolutely, and that’s something we work hard at. You’ve experienced and written first-hand about being a customer on the losing end of an acquisition, when the purchasing company doesn’t get it right when it comes to integrating support for its new customers. Despite being well-established, Extreme has more of a start-up mentality in that all of our customers matter. We take none of them for granted. No one should have to guess at what’s going to happen when they need support just because their vendor was acquired.

Amen to that, Mike. Now onto fabric, Extreme Automated Campus, and wireless specifically. I know that you are pumped up about this area. What’s the first thing that potential customers should know about Extreme when it comes to fabric and WLAN?

I’d say first that people should realize that our fabric offering is mature, proven, and is shipping now. That includes how our Wireless solution connects to the fabric. Other market leaders have their fabric stories ahead of their deliverables to a certain degree, but Extreme doesn’t use customers as guinea pigs while we figure out how to keep promises.

Give me a sense of how that integration of Wi-Fi to the fabric works. Do you have any  examples?

Sure. Let’s start with ExtremeControl, which competes with ISE and Clearpass for functions like onboarding, authorization, and role-based policies. ExtremeControl has always excelled at extremely granular policy constructs used to program per-session behavior of the access point, the data plane, and the likes of QoS and analytics. That’s what we’ve been doing for years. Now add in the Avaya fabric contribution. Instead of just bridging traffic to a controller or to an AP you can now bridge wireless sessions to different fabric segments, uniquely for each connected device. That’s a new level of micro-segmentation that basically means you can traffic engineer wireless user traffic literally anywhere in the enterprise campus with the policies you set for RBAC, Layer7 control, QoS, and analytics carried all the way through.

So… we’re used to thinking of wireless access points or AP/controller pairings as bridges that have 802.11 on the radio side, and 802.3 Ethernet on the wired side. Am I reasonable in suggesting that now we can replace Ethernet with fabric on the wired side when we think about access at the WLAN edge?

That’s a good way of picturing it for functional discussion.

Can you give a specific scenario where fabric-attached Wi-Fi yields obvious, easy-to-highlight benefits that solve real-world problems?

We’re already leveraging fabric-connected WLAN in healthcare environments. As a wireless networker, you know the technical importance of reducing the number of SSIDs in a given wireless environment. Think about having one single SSID for everything, with a slew of different security and policy constructs going on behind it with no dependence on VLANs. From doctors’ unique security requirements to guest access to IoT devices and their various limitations – all are configured via ExtremeControl and micro-segmentation on the fabric. We can bridge traffic anywhere it needs to be for any user or use case. It’s really impressive, and no other vendor is even close to this level of functionality yet.

 Does the new magic come at the cost of CPU or memory utilization anywhere?

 That’s a great question, but actually the opposite is true. You can even add new policies on the fly, non-disruptively, directly on our access points. The flow technology that came way back from our Enterasys purchase works wonders in keeping resource utilization low.

This is great information, Mike. It’s awesome to learn of real-world, low-hype network fabric technology that is proven, shipping, and mature. What else do you want people to know as we close?

It sounds silly to say that “fabric is the future” because for Extreme Networks, fabric is now. At the same time, our fabric today does future-proof customer environments by providing unparalleled flexibility in security, segmentation, simplicity, control, and analytics that will only evolve for the better. Extreme will be ready to add 802.11ax into our fabric-connected Wi-Fi strategy when it comes, and we’re a natural fit for IoT in its many incarnations. Our roadmap is exciting, and I encourage our customers and analysts like you to watch us as we evolve.

FTC-required disclosure: I was compensated to comment on the Extreme Networks Automated Campus referenced in this blog, by PR company Racepoint Global. I have no direct business relationship with Extreme Networks, and in no way claim to be an Extreme Networks customer or representative of Extreme Networks. At the same time, I have known Mike Leibovitz for years.

Extreme Networks Has Good Footing to Lead Network Fabric Evolution from Hype to Reality

If you manage a  network today, you are likely getting peppered by the drumbeat of  ideas for new ways of doing networking. Concepts like SDN, automation, AI, machine learning and fabric are becoming the next-generation lexicon of connectivity. Sure, us long-timers have heard it all before in different incarnations- but this is a pot that is really beginning to simmer while the industry tries to collectively move the way enterprise networks are done forward.

Meanwhile, those of us in the trenches have production environments to run. It’s not particularly comfortable to contemplate moving our own cheese in response to abstract promises of better ways and sunnier days, but Extreme Networks,Inc. may just be the company to break down the wall of hype and deliver the industry to the actual realization of the promise of network fabric architectures.

Before I get into why I think Extreme is the most likely company to show that the new network magic can actually be delivered in a way that leads to wide-scale adoption, let me share one of the best whitepapers I’ve read yet on what vendors are actually trying to do with the latest fabric initiatives. All the expected promises of simplification and reduced OpEx are in the Extreme Automated Campus document, but so is an excellent summation on some of the not-so-obvious advantages and evolutions that come with a properly implemented automated network. Among them:

  • The use of 802.1aq Shortest Path Bridging (SPB) as essentially a single-protocol replacement for traditional building blocks like MPLS, BGP, multicast PIM, OSPF, VLANs, and others. That’s huge, and reduces complexity by several orders of magnitude in large environments.
  • The notion that hop-by-hop network provisioning is a thing of the past. The network core is essentially unseen to most network admins, and all changes are done on the edge (live and without outages/maintenance windows).
  • User and device policies are the basis for automated network changes, and constant analytics provide feedback used to tune performance and anticipate issues.
  • By employing hyper-segmentation, a security breach in one part of the network is contained like never before, as the rest of the network is invisible to the bad guys because the old protocols leveraged for nefarious purposes are no longer present.
  • The use of APIs mean that third-party network components can interoperate with Extreme’s Automated Campus.

Extreme 3

There’s a lot more to the whitepaper, and I encourage anyone who’s been underwhelmed by other explanations of what network fabrics/automation are supposed to deliver read it as an excellent primer.

As I digested insights from Extreme’s whitepaper, I also found myself reminded that obsolescence can be insidious with the legacy methods we do networking with now. Dated designs can underperform today and fail tomorrow while we miss subtle signs of trouble because of disparate logs and dashboards. This isn’t news to anyone running large business networks, and is why automated analytics has a fairly strong appeal. This brings me back to Extreme and what puts them at the head of the pack within the networking space.

Extreme pioneered and set the bar high for network analytics with its ExtremeAnalytics platform. The value proposition has been proven in many cases, via a range of customer relationships. Where other networking companies are relying on third -parties or are just getting around to developing analytics solutions, Extreme has been optimizing networks based on machine-learning analytics for years.

Extreme 1

Then there is Extreme’s purchase of Avaya earlier this year. By my estimation, Avaya was the absolute creator of SDN-enabled network fabric environments. I visited the company’s Silicon Valley facilities in 2014 during Tech Field Day, and got a first-hand look at the impressive technology that  has become part of Extreme’s fabric offerings. Extreme now has real-world fabric customers and a mature offering among newcomers to the game.

Extreme 2

The fabric/SDN thing is here to stay as evidenced by the market leaders all talking it up as “what comes next” in unified networking. But how to get there – and whether you want to stay with your incumbent networking vendor for the leap – is a more complicated discussion. Some of the new initiatives feel cobbled-together, i.e. placing  frameworks of APIs into legacy hardware that may not have the best track-records for reliability. I’m of the opinion that some vendors are trying to figure out how to proceed with network-wide fabric methods,  while painting beta-grade efforts up with glitz and catchy slogans (though lacking depth and a track-record). This just isn’t the case for Extreme.

Extreme has done a great job in integrating their acquired Avaya fabric assets with their established portfolio and consolidating it all (along with their excellent technical support) into the Extreme Automated Campus. It’s new, on paper, but made up of mature industry-leading building-blocks. This is why I see Extreme as the one to beat in this space.

Learn more about the Automated Campus solution here.

Register for Extreme’s upcoming Automated Campus webinar here.

 

FTC-required disclosure: I was compensated to comment on the Extreme Networks Automated Campus referenced in this blog, by PR company Racepoint Global. I have no direct business relationship with Extreme Networks, and in no way claim to be an Extreme Networks customer or representative of Extreme Networks. The opinions expressed here are my own, and absolutely true at the time of publication.

Transparency as a Service- Yes, Please

Whether it’s in our personal relationships or technical careers, honesty and transparency go a long way. None of us are perfect, and even our best efforts can be undermined by an errant cut and paste, a cocked connector, or any number of soft or hard goof-ups. You know the routine- fix it quick, own up to it, and have a talk with yourself (even if the boss gives you a free pass) about what you’ll do different next time to not repeat the error.

Transparency and honesty show character and confidence- you’re big enough to admit your snafus, because they hopefully don’t happen often. But take it in the opposite direction and your credibility goes in the toilet. Show that lack of transparency or repeat the offense too often, and there may be no salvaging your good name. None of this is news, right? And what does this have to do with networking?

Consider this message that many Meraki customers recently received:
Meraki Lost

In this case, I lost a handful of floor plans that had APs placed on them from just a couple of my many sites, other people lost more. Meraki came out with guns a’ blazin’, and basically said “we screwed up”. I like the approach, and I also value that the cloud dashboard provides a natural conduit for the vendor to push information in front of the customer. 

Then there’s this sort of thing, from Aruba, Cisco, and other vendors:
Aruba Sec

That one came in my email, and the proactive notice is appreciated as it saves me from having to go out and dig around. But… vendors can do more. Even in the absence of the ability to push notifications as with a cloud dashboard, they can leverage email culled from support contracts to warn of catastrophic bugs ahead of customers hitting them.

I’m not inviting vendors to spam us with every bug that any customer hits, as that does nobody any good and wouldn’t be practical. But I can remember a day when my environment was ground zero for discovering a fairly catastrophic bug that had profound implications for the stated capabilities of a given hardware platform. As best as I can tell, pretty much no other customers were made privy to the information, and I saw at least half a dozen cases over the next couple of years where the same limitation was hit (the product data sheet should have been updated to reflect the discovery- it was that bad and blatant). Customers talk and share information. This situation felt real, real sleazy from where I sat, and seemed a natural candidate for sharing with anyone who had that component on paid support. Instead, vendor credibility was bludgeoned.

I like these from Cisco, released quarterly (some Cisco-sensitive content removed from attached):
cisco code

This is something all vendors should be doing. At the same time- there is so much bad code out there, customers deserve better communications on what really shouldn’t be used. It’s just confusing as hell when the “recommended” code is several versions behind others that are out in the wild available for download. I propose crystal clear warning labels on the download page, and calling the non-recommended code versions “beta”, as they often feel as such.

In closing, whether “honesty is the best policy” is applicable, or “sunlight is the best disinfectant” seems more appropriate, you get the point. Enterprise systems just cost too much and budget-minded IT teams are being tasked with doing ever more with less resources. We need that transparency thing from vendors, now more than ever. It keeps us from making mistakes that can be prevented if we only knew what the vendor already knows, and keeps the vendor’s credibility in good standing- and that is one thing you can’t put a price on.

 

The Red Hot Cable Peppers

It was only a dream. But it was a c-r-a-z-y dream. There were no chemicals ingested prior to the slumber that contained the dream, either- so just get those thoughts out of your head up front. But it seemed so real. Me and the Red Hot Chili Peppers… doing some cabling work. I kid you not. Here’s what I recall of it, and the lessons that these awesome rockers took away from our imaginary time together.

I was up front at a Peppers concert, and they were just getting into By The Way.

Standing in line
To see the show tonight
And there’s a light on
Heavy glow
By the way I tried to say
I’d be there, waiting for
Dani the girl
Is singing songs to me
Beneath the marquee, overload

And BOOM! All the stage equipment went dead as soon as Anthony got to “overload”.

For some reason, I was suddenly backstage with the band (in my own favorite incarnation- Anthony, Chad, Flea, and John Frusciante). Chad looked disgusted, and before he wandered off he said something like “I’m getting too old for this. Why the hell do we run our own data cables any more for these shows?” That was the last I saw of him. Flea (who is not English, but he had a Cockney accent for some reason in this scenario) shouted “I told you wankers to actually LABEL things and test your work!” Then he too disappeared.

Anthony said nothing, but he looked seriously pissed. John asked me “can you help us? We gotta get this stuff going again, man…” Now, I have no idea why a bunch of data cables would have anything to do with the lights and sound on stage crapping out for the Peppers,  and I’m here to tell you that it’s irrelevant. These guys needed my help.

For some reason we had to climb on top of an RV to where like fifty or so UTP cables were hanging, and a bunch had sloppily crimped-on ends and were coupled together with RJ45 couplers. Flea was right, nothing was labeled. Anthony continued to say nothing, and John did all the talking. In magic dream speed, he showed me a few patch panels, their patch cables, and lots of odd little things that needed straightening out. We had to rerun a bunch of cables, and even put a new rack in the RV.  John got a roadie to film the whole thing, so he could play it back to Chad and Flea later, which I thought was really good thinking.

Anthony worked with us, still never saying a word and looking angry, sometimes at me. It was freaking me out, because I was trying to help him.

So how it finished out… We basically got all their wiring issues fixed. John was excellent, and he told me Anthony was just intense, and not really pissed which I was okay with. Anthony actually gave me a hug, and a carton of Dunkin Donut Holes. The band got back on stage to finish the concert, and I got to hang out offstage and monitor their “LEDs”, having no idea what I was looking at. The crowd didn’t seem to notice that the band was gone for however long it took to fix all the wiring. At the end, Anthony said “Goodnight, Poland! We love you!” and I was now mildly worried how I’d be getting home from Poland.

Before we went to what certainly would have a been a kick-ass afterparty, Anthony called us all into a room and wanted to white-board what they would do different on their next cabling job. Here’s what that amounted to:

Alas, I did miss the dream party because I woke up, but felt that I got to be pals with John Frusciante which was pretty cool. And I KNOW that if the dream Chili Peppers keep running their own dream data cables for other people’s dream concerts, they have me to thank for doing it right from now on.

(This is a true story- I bored my wife with it at breakfast.)

 

Don’t Forget Visual Inspection When Network Troubleshooting

My small engines shop teacher said it in high school. Countless Air Force electronics instructors said the words when I went through Electronic Warfare school. I myself even harped on it when I became an Air Force instructor, and again years after when I taught basic electronics classes at a local vo-tech center.

Always first do a visual inspection when you’re troubleshooting. Always.

It’s easy to say, and just as easy to blow right past. Like I did yesterday when troubleshooting a wireless bridge link, which cost an extra hour of troubleshooting time.

In this scenario, a farm campus is tied together by three Ubiquiti bridges. It’s an environment that I took over and cleaned up a few years ago. I had my hands full eliminating all the oddball consumer routers that were in way too many places and moving the entire environment to a manageable topology that both I and the owner could understand. I inherited two M5 Nano Station bridge links, that were actually pretty well done- or so it seemed. Later, I would add a 900 MHz bridge link to get past a large stand of tall pines for a new connection, but this tale of my own shortcomings focuses on one of the M5 links.

The trouble call was for the single PC in the Robot Barn- a facility used for automatic feeding of dairy cow calves. The PC has two network connections; one goes to the modem that uplinks the robot feeders on proprietary low-voltage protocols, and the other connects to one of the M5s and ultimately back to the Meraki MX that head-ends the network. Basically, nothing was working.

A quick stop at the barn, and I found that the PC was in the kind of shape that comes when someone doesn’t know what they are doing, but are trying to fix it anyway. Both adapters had all kinds of oddball, nonsensical settings. I quickly got the dairy application side up so the important robot data was at least being buffered, and it could upload to offsite servers when I got the network link figured out.It was pretty clear that the PC was not talking back into the network, nor would my own laptop. But… from the remote end I could get to the far-side bridge admin interface, and see that it showed link down. On the way out of the building, I took a quick look and saw this:
M5.JPG

Then, I drove to the other end of the farm to where the root bridge is. As I walked in to the building to check to make sure the root had link-light and such, I got distracted by one of the owners. He told me he had re-arranged some of the power cords and the monitor for the CCTV system, which are co-located with the network equipment the same time the problem started. Ah-hah! I’m highly skeptical of coincidences, and bit right into the probability that THIS MUST BE THE PROBLEM. I sat down, got into the root bridge UI, and started thinking desperate thoughts. Like… even though I can get into the UI on both bridges, maybe one died on the radio side. Or maybe one of the cheap power supplies wasn’t getting it done (despite both bridges eagerly presenting their UIs to me).

For the next hour, I let myself go down goofy rabbit holes. I replaced both bridge power injectors. I dorked with settings on each bridge. I falsely concluded that one bridge or the other was at least corrupted, if not bad. My next step was to take them both down and see if I could reset them and start over getting them to talk. I walked outside with one of the owners to show her where I needed to get access to take down the root bridge- and then felt profoundly stupid.

The root bridge was not where it was supposed to be. It was laying down on the metal roof, looking sadder than a country song on a Sunday morning. Remember, I inherited this bridge, along with the others. The “mast mount” was an anemic two sheet metal screws into the thin metal peek of the roof, and it’s amazing it held up as long as it did. Up I scurried, and cobbed it back into place with wire as it was getting dark with proper mounting to follow. And- the link came back up.

LESSONS:

  • When I took responsibility of this network over, I should have looked closer at the shoddy way this bridge was mounted and dealt with it then.
  • Whoever hosed up the computer shouldn’t have. The owners will work with the staff to ensure that doesn’t happen again.
  • I SHOULD HAVE gotten out of my vehicle and walked immediately to where I could see the root bridge installed, after having verified all at the non-root site was seemingly fine.
  • I SHOULD NOT HAVE gotten starry eyed jumping to the conclusion that the problem came from things being touched near the network equipment.

Having skipped the important visual inspection step at the root end pushed me into a trap of bad judgement that we all land in occasionally, and when I realized that had happened my mind was immediately flooded with voices from the past (including my own) saying yet again “Always do a visual inspection first!”.

Whether you’re looking for a wireless bridge laying on a roof, a burnt-out resistor on a circuit board, a corroded Ethernet jack, or a damaged fiber cable, a quick once-over with the eyes is sound practice before you start digging in on configurations.

Had I followed my own guidance, I would have had my client back in service a lot quicker.

(And yes… I did make sure all of the other bridges were mounted right before I left!)