Category Archives: Networking

Some Advice for Ubiquiti Forum Posters

Having just migrated Wirednot HQ to Ubiquiti (LAN, WLAN, CCTV, and a P-P link), I find myself in the Ubiquiti forums more than I have been in the past. The community discussions are the main means of support on this ever-evolving (and expanding) product set, and you’ll find both fellow Ubiquiti customers and company employees engaged in discussions. It’s an interesting framework, and like any discussion forum where lots of people participate, you get good advice, odd advice, and exposure to a lot of different personalities (we’ll come back to this point).

I have noticed a few trends that I want to call out for the benefit of both those in the Ubiquiti forums, but also for anyone dealing with general networking issues that may benefit from some basic network troubleshooting advice.

  • The Physical Layer matters- bigtime. I notice a lot of “my network performance isn’t what I expect- something has to be wrong with my Ubiquiti gear” kind of laments in the forum. And the discussions that follow often NEVER get into the physical layer. Chances are, many of the same folks that use UniFi gear are also buying offshore-sourced (which is polite-talk for cheap) cable products, or terminating it themselves without having a certification tester to prove wiring and performance are up to snuff. So be it, remember that the physical layer is where troubleshooting should start. Do SOMETHING to verify your cable is not the problem, if nothing else than swapping out to another cable to see if the problem follows. And when you engage in the forums looking for help, tell us how you’ve verified the cabling is OK as part of your troubleshooting so far.
  • Network switches have stories to tell, but you have to listen. Just like we can’t assume that cabling is good when trouble hits, we also can’t assume that network connections between devices are behaving as they should. Check for speed and duplex status for the ports in the path of your trouble– like so:
    ubntspeedduplex
    and check for RX and TX errors (receive and transmit) that could indicate bad cable, bad jack, bad NIC, or misbehaving SFP module:
    ubnterrors
    If you find errors, I suggest you clear the counters and then watch to see if the errors continue to increment. If they do, you have at least part of your problem figured out.
  • What Access Point is your client device connecting to? I see plenty of “my Wi-Fi is slow” postings, and many of these are in environments where multiple access points are in use. Not only do you need to know what the output power of the access points are and what channels are in use so interference is minimized, you also need to know what access point your “slow” client is connecting to, and how good that connection is. Client devices do not always connect to the nearest or strongest AP, or to the radio (5 GHz or 2.4 GHz) you might assume they should.
    ubntclient
    There are other views that will tell you more, but “slowness” may be normal, based on the connection properties in play. Often the “fix” is to update the client device drivers or firmware.
  • Speedtest to the Internet isn’t the end-all. Understand what is actually being tested. To state the obvious, your Internet speedtest results can’t exceed your ISP connection capacity. If you have a 25 Mbps down/5 Mbps up connection, you won’t see any more than those numbers on Internet speedtests. And… if other devices are using the Internet while you are speedtesting, your results will be less because you are sharing “the pipe”.
    When you run the Internet speedtest, you are exercising one discreet path- the connectivity between your specific device and the server out on the Internet. If it feels slow and you are on WIRELESS, you need to verify that your wireless connection is healthy as described above.
    If speedtest feels slow and you are on WIRED, check the specific port behavior for your connected device, also as described above. If speedtesting from multiple devices feels slow, try to move as close to the edge router as you can and retest. If it suddenly perks up, you may have to “divide and conquer” to find what part of the network is slow versus what is behaving normally.
    Also know that some Internet speedtest sites can be fairly erratic, based on a number of factors. Try a couple of different ones, and never come to conclusions based on a single test.
  • Consider learning iPerf, possibly getting an internal testing device. Just like Internet speedtests can be fairly ambiguous, there are tools that can be pretty damn accurate in characterizing exactly how a network is behaving between Point A and Point B. Consider iPerf as an excellent freebie, or something like WLAN Pi that can be built for well under $100 (WLAN Pi also gives a slew more functionality than just throughput testing). However you get there, it’s empowering to be able to test between different points on the local network as you try to isolate perceived problems. This is where you make sure that switch to switch connections are actually delivering Gigabit, for example.

There are more basics to talk about (like being on the right FW versions), but these are a good start. I encourage using these tips every time BEFORE you reach out for help, as they will lead to better resolution faster, and you will also become more self-sufficient in solving your own problems (or in helping others to overcome as you better your basic troubleshooting skills.)

Now… back to the personalities thing I mentioned up front in this blog. For whatever reason, any forum you join from restoring classic campers to Ram pickup trucks to networking, you’ll find people that simply want to help, and others that see the world through blinders, and it’s their way or no way. That being said… don’t be GodComplex5.

A Little PCAP Reader for iOS+ Meraki Remote Capture = Handy

I had been pecking away at a problem at a remote site, where phantom ringing was driving staff nuts on their Ring Central VoIP phones. I’ll spare you all the nasty things I want to say about the frailty of Ring Central phones and try to stay on topic… These devices are clients on a Meraki network, which means that you can capture their packets remotely, while doing analysis locally.

image0

It’s a nice feature, as it really helps you to exercise a common network troubleshooting task that traditionally requires you to be within the network environment to carry out. I had left the office, and my Wireshark-equipped workstation behind for the day, but found myself with free time, my iPad, and the phantom ringing problem on my mind.

Hmmm. I wonder if there are any PCAP-related apps for iOS? I doubt it, but what the hell… Let’s take a look and see if there is anything I can break down those remote capture files with… If I had my PC with Wireshark on it I wouldn’t need this… But all I have is my iPad… Let’s see.. 

Whoa- what’s this?

It’s an app for iOS called Telluric, and it reads (to a certain extent) packet capture files. It doesn’t do 802.11 radio header stuff. It doesn’t actually CAPTURE packets. You can’t really do display filtering or fancy stuff like Wireshark can. But it does do a decent job when no other tools are available, provided you have access to remote packet capture and local download (or can have someone send you a pcap file).

Sure, it’s a niche app of limited value. But it helped me find the source of my problem when I had no other real options:

image1

It’s time for a firewall rule. Sorry, Mr. Vicious.

(I do know that there are online resources for dumping and analyzing packet capture files. Don’t ruin the mood.)

 

 

Another Example of How Important Wire is to Wireless

A house built on a shaky foundation cannot endure. And a WLAN built on a shaky wiring foundation likewise cannot endure, I tellya. My friends, is your foundation shaky? Is it? CHECK YOUR FOUNDATION NOW. (I happen to sell foundation-strengthening herbal supplements on the side, if you need that sort of thing…)

I’ve long been a proponent of recognizing installed UTP as a vital component in the networking ecosystem. Too many people take Layer 1 for granted, and forgivable sins of of our 10 Mbps and Fast Ethernet pasts won’t fly in a Gig world. Toolmakers like Fluke Networks sell cable certification testers that take the guesswork out of whether a given cable run can be relied on to perform as expected. Don’t use one of these testers at time of cable installation, and you are only assuming you have a good station cable.

I just had an interesting situation come up that I helped a very skilled field tech with. He was working in several different small buildings, each serviced by a Cisco Catalyst Switch and a handful of 3802 802.11ac access points. The switches and cable had been in place for years, and the APs for many months, all with no issues whatsoever.

Then, we changed out the old 3560X switches for shiny new 3650s (curse you Cisco for your bizarre fascination with part numbers so close together), and suddenly some APs weren’t working any more. Between us, we checked all switch settings, POST reports, CDP tables, logs, etc- everything you can dream up on the switch. We put the APs that weren’t working back on the old switches, and they came right up. Hmmm… thoughts turned to PoE/code bugs, but then I went a-Googlin’ before consulting TAC.

I found this document that put me on the path to righteousness. Though we weren’t having “PoE Imax Errors”, a couple of nuggets jumped out at me about our new switches.

PoE Imax

Holy guacamole- We got us a situation! But wait… THERE’S MORE!

PoE Imax2

Shazam! Which, of course, translates in Esperanto to “maybe your cable is actually kind of iffy, and all the CDP stuff that happens at the milliwatt level before PoE gets delivered worked OK with your old switch but not with the new one that has the enhanced PoE controller”.

If you don’t know that the newer switch does PoE differently, you might wrongly assume that your cabling is “good” because the APs worked on it when those APs used the old switches connected to that wiring. By now, you can probably guess where I’m headed…

Our tech tested the cabling on the new-switch-problem APs and in each case found that they needed help to work with the new switch. He re-terminated and tested each, with the APs then coming up with no issues. I have no doubt that this cable was certified 10-12 years ago, but in that time a lot can happen to either end of those cables depending on the environments where they are used.

Live and learn!

 

 

Open Mesh Brings Major Disruption to SMB Space, Goes Full-Stack

Another router coming to the SMB market generally isn’t that exciting, but this one is different for a number of reasons.

OM1

For one thing, it comes from Open Mesh. Those ports are part of the G200, which is the first router ever released by Open Mesh. It has a list price of $249 dollars, and it also brings the Open Mesh product line into the proverbial “full stack” domain.

OM2

Now customers can use access points, switches, and the G200 all from Open Mesh, and all cloud-managed in the excellent CloudTrax dashboard with no license costs.

Yes, you heard me right… I said “with no license costs”. If you are not familiar with Open Mesh, the operational paradigm is easy- you buy your components (routers, switches, and access points), you register them in the CloudTrax dashboard, and off you go with configuration and operation. CloudTrax is a pretty decent network management system in and of itself, and it is the only way you manage Open Mesh components. It’s simple, it’s feature rich, and given what Open Mesh hardware costs, the entire paradigm is an absolute steal compared to pricing and complexity of enterprise solutions that masquerade as SMB-friendly.

The G200 is a significant milestone to not only the Open Mesh product line, but also to the SMB market in that it seriously drops upfront costs and TCO while providing what may be the easiest to use interface among any of it’s competitors.

But what do you get for under $250 for features with the G200? A lot, actually. From a resource perspective, Open Mesh promises gigabit throughput compliments of a quad-core processor and dedicated crypto engine. The G200 has two passive PoE ports for Open Mesh APs to connect directly, and also has an SFP port for fiber uplink to an Open Mesh switch or 3rd party vendor switch. All the typical “router stuff” is onboard, from VLAN support, DHCP server and firewall to decent traffic classification, QoS, NAT functionality, user VPN, and even usage statistics. Not bad for an initial edge-router at this price point, that won’t hit you up in 12 months for a fat license fee to keep using it. Mine has been reliable as I could ask for in the couple of weeks that I’ve been testing it. One gripe- no site-to-site VPN, although that is coming.

g200

I can’t stress how important price is for the SMB space, and I know some of my own customers are dealing with sticker shock that comes from other cloud-managed solutions that charge big and small environments the same way when it comes to licensing (or worse, they penalize the small networks for not having volume purchasing leading to better pricing). If Open Mesh continues to evolve their edge functionality and hardware offerings, this vendor could deliver a sales smack-down to the bigger players who have become license-happy to the point of ridiculousness over the last few years.

A New Access Point and Switch, Too!

I’m a huge fan of the Open Mesh A60 dual-band indoor/outdoor 802.11ac access point. It has been the top-dog of the Open Mesh access point line for several months, with a list price of $225 (again, no licensing and free CloudTrax support). Now, as part of the same product announcement that features the G200 router, Open Mesh is also bringing out it’s new A62 access point. It’s still dual-band and indoor/outdoor, but this Wave 2 AP also sports two 5 GHz radios, support for up to an estimated 150 streaming clients, and the same $225 price tag as the A60.

The latest S24 switch also breaks new ground for Open Mesh with 10 Gbps SFP+ uplink ports and a higher PoE power budget than it’s predecessor.

Let’s Do Some Math

Open Mesh has over 100,000 network customers around the world. When I think of one of my own small sites that’s up for renewal with another cloud vendor, I’m looking at trying to explain to my customer why a 3-year renewal license on old AP costs almost as much as purchasing the latest license-free AP from Open Mesh, and why a 3-year renewal license on an older security appliance costs almost twice the price of a new Open Mesh G200 router that would never need another license. These are real dollars for small businesses, and you pay the big price for the other guys whether you ever use actual support or not.

It’s time for a shake-up at this end of the market, and I think Open Mesh is the vendor to do it.

___

Related posts:

A Little Quiet Please- The Jabra Evolve 75

Jabra 75 EvolveFew things are more annoying when you’re troubleshooting in a noisy environment while trying to work with technical support or a coworker over the phone and you can’t hear them (or they can’t hear you). There’s usually enough stress in play to begin with, and having to repeat yourself and say “what? I just can’t hear you…” over the exhaust fans of network equipment in packed racks only makes it worse. Enter the Jabra Evolve 75.

Quiet Down!

Marketed as “the best wireless headset for concentration in the open office”, the Evolve 75 from Jabra is impressive for a number of reasons.

I don’t work in an open office environment so I can’t validate Jabra’s claim in that regard, but I can tell you that the Evolve 75 impresses in the noisy data center and telecommunications closets frequented by network technicians and administrators. I recently took an evaluation unit for a spin, and the timing was perfect for a real-world test under extremely loud background noise conditions. Not only could I hear the support engineer on the other end perfectly, but he also said my audio was clear despite the Evolve 75 mic being  inches away from blowing server fans. It’s hard to ask for more than that, but there is more to talk about.

Classy

Jabra has always made fairly elegant Bluetooth headsets for cellphones, in my experience. This was my first outing with a professional grade Jabra product, and it blew the doors off of other headsets I’ve tried to use in similarly noisy situations. The Evolve 75 is fairly light, but substantive. It feels good on my big head, and the mic raises and lowers smoothly, and in both positions is unobtrusive. The overall look and feel whether it’s on your head or in it’s beefy charging cradle is one of quality.

Functional

I’m finding that the Evolve 75 has really, really good battery life for my usage patterns- but I also do not wear it as long as a call-center person or the like might. The ANC (Active Noise Cancellation) functionality works like a dream, and these may be the absolute quietest phones I’ve ever used. I manipulated the volume buttons and mute functions on WebEx, Skype, and simple phone call without having to fumble around, and it doesn’t take long to feel at a disadvantage when you take a call without the benefit of this headset.

I also made use of the unit’s ability to pair with two Bluetooth devices at once, which proves handy more often than you might think. When paired simultaneously to my computer and my smartphone, the new-found mobility and seamless transitioning between the two was just a joy to experience. I’m not one to gush, but I kinda fell in love the Evolve 75 and don’t mind saying so.

Nice Bonuses

I’m not sure that I qualify as an audiophile after all the beating my ears took working around loud fighter jets in my Air Force days, but I do enjoy music and the occasional podcast. Jabra makes both activities more enjoyable with the sound quality, volume, and purity that the Evolve 75 delivers. It also comes with a really nice travel case.

The only downside I see with the Evolve 75 is that it’s one of those really, really nice tools  that you miss greatly when you don’t have it.

 

NETSCOUT’s Next-Gen LinkRunner Tester Is Ready For The Changing Network Landscape

Just when you thought that maybe all of the cool testing innovation was reserved for Wi-Fi and the likes of the AirCheck G2, NETSCOUT brings out an equally impressive wired networking tester. The new LinkRunner G2  (shown on left below) sports the same color scheme and physical profile as the AirCheck G2:

IMG_20171127_140442975

But a closer look topside reveals some tell-tale features:

IMG_20171127_142046889_BURST000_COVER_TOP

What isn’t obvious from these images is that NETSCOUT made the LinkRunner G2 with a whole new user-customizable philosophy in mind that applies to both hardware and software. See the fiber SFP on the left and the USB WLAN adapter on the right side in the second image? You don’t HAVE TO get those from NETSCOUT if you find a better price on similar interfaces elsewhere. Unlike some vendors, NETSCOUT opted to be very accommodating of 3rd party adapters. And the LinkRunner G2 is actually a hardened Android computing platform that you can tweak in a number of ways- but let’s come back to that in a bit.

Recognizing the LAN-Scape For What It’s Becoming

In talking with LinkRunner’s product management, I was able to hear the inside scoop on where the company sees the product fitting into the connected world. It’s no secret that the number and type of network-connected devices “out there” is skyrocketing, but investment in support staff and their capabilities isn’t for many organizations. That being said, the LinkRunner G2 is viewed by NETSCOUT as the “smart network tester for the connected world”. Now, I’m as buzzphrase-adverse as anyone, but the deeper you dig into the LRG2, the more you realize that NETSCOUT is not over-hyping the new tester’s capabilities. With strong physical layer support capabilities, LRG2 is handy before the network even goes live. On active networks, Ethenet and core services are tested and characterized nicely. Then there are the true differentiators- and Power over Ethernet (PoE) functionality is a prime example.

Robust PoE Measurement and Charging

NETSCOUT points out that today’s LAN environment is no longer just a client-access domain, it’s also a power-distribution system for a growing number of devices. Beyond VoIP phones, CCTV cameras and wireless access points, we now have lighting systems, locks, and Bluetooth sensors among the many devices using PoE for operating power.

Netscout describes the LRG2 as the only current tester fully able to support all versions of PoE including Universal PoE (UPOE) that sources 60 watts at the switchport and up to 51 watts at the field jack. And this LinkSprinter also charges off of PoE!

Screenshot_20171127-184138

Link-Live

Depending what other NETSCOUT (or Fluke Networks) tools you use, you may already be a Link-Live user.

 

Link-Live makes the storing, sharing, and reporting of test results and absolute cakewalk. When you have a number of staff with varying skillsets, NETSCOUT’s Link-Live-capable tools can bring a uniformity of testing that reduces errors and faulty troubleshooting, and makes the results available for reference and escalation. It’s a free service, and LinkSprinter G2’s tests are as well formatted as the likes of the AirCheck G2’s. I’m a fan.

The Multi-Function Tester That You Can Customize

This tester is still a LinkSprinter, so you’d expect to see views like this on the crystal-clear touchscreen as you test copper and fiber links:

 

But there is sooooo much more. Remember, I said that this is a full-blown Android device. It also happens to have many “phone-like” features including a built-in flashlight, camera (attach pictures to Link-Live reports or email them from the LRG2), web browser, screen shot capabilities, Micro SD slot for file storage and portability, and even access to other apps that can be installed on the tester.

Screenshot_20171127-184637

This is so handy, and almost too good to be true. The caveat to the starting app paradigm? You don’t have access to the full Play Store. This is a hardened Android device after all, and you do not log in to the Play Store with your own account. But you do have access to a wide range of vetted network/documentation/productivity apps via NETSCOUT’s partnership with Google. If you find an app that you’d like, NETSCOUT provides an easy path to request it.

Within five minutes after discovering this capability, I had my test unit nicely loaded with some of the same Wi-Fi and network apps I use on my own Android phone and tablets, and the ability to run them all off of a robust network tester feels incredibly empowering.

Screenshot_20171127-184155

There is just a lot here to appreciate in the LinkRunner G2. It’s clear that NETSCOUT was shooting for versatility and expandibility with this network tester, and they hit both targets nicely.

Learn more at product web site.

 

Future-Proofing Networks with Fabric-Attached Wi-Fi: Q&A with Extreme Networks’ Director of Wireless Product Management & Strategy

It’s easy to become desensitized to the onslaught of marketing that surrounds networking concepts like “fabric” and “unified networks” when every vendor has their own version of them. Naturally, each marketing department promises that their solution is the best, but reality shines through when you start to look past the buzzwords for substance. I was recently  introduced to (and impressed by) Extreme Networks’ own fabric accomplishments, and wrote about my impressions here. Soon after, I had the chance to talk with Extreme’s director of wireless product management and strategy, Mike Leibovitz, about where WLAN specifically fits into the company’s fabric approach.

Leibovitz is one of those people that I’m always glad to catch up with. I’ve spent time with him at different Tech Field Day events and  IT conferences, and have had opportunities to socialize with him. Beyond just being an all-around nice guy, Leibovitz has a passion for his job and believes strongly in Extreme’s products, methods and his company’s future. Our most recent conversation evolved into an informal Q& A about the Extreme Automated Campus solution and Wi-Fi. Here are the highlights from that discussion (I’m in italics).

Mike, Extreme has been busy integrating the likes of ExtremeWireless WiNG from Zebra/Motorola and Avaya’s fabric portfolio (from recent acquisitions) with Extreme’s own wireless product lines. How’s all that going?

It’s been a great run, for us and our customers. We’re fully supporting all product lines, and it’s only getting better for the end users, regardless of which hardware they use. Looking forward, the best of all our product lines will be fused into new feature options that customers of either ExtremeWireless WiNG or ExtremeWireless can take advantage of without forklift upgrades.

We’ll get to fabric and Wi-Fi in a bit, but first- is there anything on the horizon that is particularly driving Extreme’s WLAN-specific evolution, and do you have any examples of where ExtremeWireless WiNG might bring something new to Extreme’s story that customers can appreciate?

Aside from our fabric architecture taking deeper root, we see the coming of 802.11ax as significant, and that does figure into our current product evolution. As the radio side of the equation gets higher in performance, we’ll continue to leverage things like Motorola’s unique excellence in access point design for challenging and high-ceiling environments, for instance. Also, we have the successful integration of the Azara Cloud into ExtremeCloud as an example of how we make what’s good even better.

It seems that Extreme goes to great lengths to make sure that new customers gained through acquisitions are treated just as well as long-time Extreme customers. Is that a fair characterization?

Absolutely, and that’s something we work hard at. You’ve experienced and written first-hand about being a customer on the losing end of an acquisition, when the purchasing company doesn’t get it right when it comes to integrating support for its new customers. Despite being well-established, Extreme has more of a start-up mentality in that all of our customers matter. We take none of them for granted. No one should have to guess at what’s going to happen when they need support just because their vendor was acquired.

Amen to that, Mike. Now onto fabric, Extreme Automated Campus, and wireless specifically. I know that you are pumped up about this area. What’s the first thing that potential customers should know about Extreme when it comes to fabric and WLAN?

I’d say first that people should realize that our fabric offering is mature, proven, and is shipping now. That includes how our Wireless solution connects to the fabric. Other market leaders have their fabric stories ahead of their deliverables to a certain degree, but Extreme doesn’t use customers as guinea pigs while we figure out how to keep promises.

Give me a sense of how that integration of Wi-Fi to the fabric works. Do you have any  examples?

Sure. Let’s start with ExtremeControl, which competes with ISE and Clearpass for functions like onboarding, authorization, and role-based policies. ExtremeControl has always excelled at extremely granular policy constructs used to program per-session behavior of the access point, the data plane, and the likes of QoS and analytics. That’s what we’ve been doing for years. Now add in the Avaya fabric contribution. Instead of just bridging traffic to a controller or to an AP you can now bridge wireless sessions to different fabric segments, uniquely for each connected device. That’s a new level of micro-segmentation that basically means you can traffic engineer wireless user traffic literally anywhere in the enterprise campus with the policies you set for RBAC, Layer7 control, QoS, and analytics carried all the way through.

So… we’re used to thinking of wireless access points or AP/controller pairings as bridges that have 802.11 on the radio side, and 802.3 Ethernet on the wired side. Am I reasonable in suggesting that now we can replace Ethernet with fabric on the wired side when we think about access at the WLAN edge?

That’s a good way of picturing it for functional discussion.

Can you give a specific scenario where fabric-attached Wi-Fi yields obvious, easy-to-highlight benefits that solve real-world problems?

We’re already leveraging fabric-connected WLAN in healthcare environments. As a wireless networker, you know the technical importance of reducing the number of SSIDs in a given wireless environment. Think about having one single SSID for everything, with a slew of different security and policy constructs going on behind it with no dependence on VLANs. From doctors’ unique security requirements to guest access to IoT devices and their various limitations – all are configured via ExtremeControl and micro-segmentation on the fabric. We can bridge traffic anywhere it needs to be for any user or use case. It’s really impressive, and no other vendor is even close to this level of functionality yet.

 Does the new magic come at the cost of CPU or memory utilization anywhere?

 That’s a great question, but actually the opposite is true. You can even add new policies on the fly, non-disruptively, directly on our access points. The flow technology that came way back from our Enterasys purchase works wonders in keeping resource utilization low.

This is great information, Mike. It’s awesome to learn of real-world, low-hype network fabric technology that is proven, shipping, and mature. What else do you want people to know as we close?

It sounds silly to say that “fabric is the future” because for Extreme Networks, fabric is now. At the same time, our fabric today does future-proof customer environments by providing unparalleled flexibility in security, segmentation, simplicity, control, and analytics that will only evolve for the better. Extreme will be ready to add 802.11ax into our fabric-connected Wi-Fi strategy when it comes, and we’re a natural fit for IoT in its many incarnations. Our roadmap is exciting, and I encourage our customers and analysts like you to watch us as we evolve.

FTC-required disclosure: I was compensated to comment on the Extreme Networks Automated Campus referenced in this blog, by PR company Racepoint Global. I have no direct business relationship with Extreme Networks, and in no way claim to be an Extreme Networks customer or representative of Extreme Networks. At the same time, I have known Mike Leibovitz for years.