Now that I’ve wrapped up my trip out west and the dust is settling in my mind, it’s time to reflect on what I heard and took away from Mobility Field Day 10. Here’s at least some of my perspective on Cisco”s session, as written up as a LinkedIn article. Hint: Cisco and Meraki are now same-same.
More to come.
I’m a long-time user of the Meraki MX security appliance product line. Going way back to the MX-70, I have found tremendous value in what the MX products can do for my far-off sites. (Here’s an old- and I mean old- case study that gets into the early appreciation of the MX line.) I’ve probably set up maybe 65ish total MX devices through the years in multiple states and countries, doing site-to-site VPN, stand-alone, and also some pretty creative configurations. Despite my experience, I was recently reminded that I don’t know it all about a product that I feel extremely comfortable calling myself an expert on.
In one remote site that connects to the main network with site-to-site VPAN, an NTP vulnerability was flagged on a couple of audio visual devices. The device vendor was of absolutely no help (go figure), and our security team asked if we could help from the Meraki side. “Oh sure…” says I. “We got a firewall to leverage.”
We needed to cabash NTP between the remote site and the main network. I pulled up the Firewall page on the MX and set to work. This is an area in the MX I’ve probably manipulated maybe a couple of dozen times, for everything from stopping phantom ringing on 3rd-party hosted IP phones to simple outbound protocol blocks.
That image represents like three stages of desperation in getting rules right- as nothing I did worked. I simply could not tame the NTP beast to/from the two hosts, and it was making me feel silly. My first inclination was to blame Meraki- surely this stupid box must have issues! Except it didn’t… about the only thing Meraki could have done is perhaps mentioned on the L3 Firewall Page that there is a seperate firewall rule set on the VPN configuration page for site-to-site rules. That looks like this:
I had just never did firewall rules for the site-to-site tunnel. I didn’t know after many years! But I did leverage the Meraki “search our documentation” repository to get educated, with this document that explains it. There’s nothing complicated about it, you just have to know where to find it the first time you need to configure rules for the tunnel versus the Internet edge.
And now you know, too.
After opening a support case, Meraki was able to verify what I describe below in their own lab testing- some MX ports were not doing LLDP right. And…. the fix: “The issue of the transmission of LLDP packets from the MX is fixed in versions starting MX14.2” – which so far is proving to be the case. (Now we need to solve client MAC addresses showing up on the wrong switch ports, but that’s another tale for another time.) Read on if you’d like, but keep in mind that they DID fix this problem at long last.
The only thing worse than an important feature that’s missing in a network management solution is an important feature that works wrong. Those of us who pay for these systems expect the features that we trade our dollars for to actually, well, work. And bad information is just terrible in that it sends us down time-wasting bad roads.
Meraki- you have a problem. Your topology view doesn’t refresh for hours, days, or weeks after changes are made. And for whatever reason, you have not given us decent LLDP tools that we can invoke on demand. That’s the polite description, which can be abbreviated down to your topology view sucks at times.
Before I show and tell my latest frustration in this regard, let me share Meraki’s summary on it’s topology view and evidence that what I’m about to describe is not my frustration alone. See this post from the Meraki community. Now on to the defective topology feature.
It Just NEVER Updates
In a faraway branch location, we had two switches that daisy-chained off of another switch due to physical layer constraints. Here’s the old topology:
Pay attention to VillaRosa-3, Villino-1,and Vilillino-2. V1 and V2 were daisy-chained off of V3. THEN THEY WERE MOVED TO EACH DIRECTLY CONNECT TO THE MX. And then I got an email… “The switches got moved to ports 6 and 7 on The MX, but the topology isn’t updating. Maybe I should restart the devices…” As we see in the community forums, we may be waiting quite a while for this map to update…
The Implications
Few things are more basic in network support than simply knowing what connects to what- both physically and logically. Commands like <show cdp neighbors> and <show lldp neighbors> are pretty important, but not available in the Meraki dashboard paradigm. Instead, we need to rely on incomplete or inaccurate graphical information.
As I mentioned above, V1 and V2 moved to direct connects on the MX. So what does the MX say about the switches that are connected to it? Unfortunately, nothing. For whatever reason, Meraki has never seen it as important to give full lldp information for devices connected to an MX:
This is utterly nonsensical for enterprise-grade networking equipment. But it gets worse. Let’s look at one of the switches that has had it’s uplink moved to the MX. Days after the move, the switch still says it’s connected to it’s old uplinked V3 switch:
This is pretty bad. It’s wrong, it’s outdated, it’s misleading, and it’s inexcusable. Let’s have a look at “VillaRosa-3 / 51.
To recap:
That this “feature set” can be this off-kilter defies logic- especially when you consider the cost of the gear and it’s licensing and the importance of lldp to network support.
Please remember, this blog is mostly about wireless- but not always. I recently brought a customer site online where I did the LAN, Wi-Fi, simple background music speakers with satellite radio feed, and IP-CCTV, all in a brand new building. The Ethernet and Wi-Fi environments are just footnotes to the rest of this blog, but the Meraki MX84 that serves as the network head-end will factor large in the story.
The video system is of Hikvision components, with a couple of POE camera types and a networked DVR. I don’t do a lot of video applications, but the Hikvision stuff is pretty easy to work with end-to-end. The new building is fairly large for a small business, and we ended up with 15 cameras total in what can only be described as just another well-networked environment.
Can we do… THIS?
This client has multiple business sites, and I have built the networks in all of them. It turns out that in the past, a case of arson and significant loss of property has them constantly thinking about security. One day after the new site opened, I was asked if we could somehow backup the video at the new site to one of their other sites. If the unthinkable should ever happen and the new-site DVR were to be destroyed along with it’s contents, the idea would be that the same video would live at a backup site to hopefully shed light on what happened.
I was a bit stumped on this one, and did some research on cloud-backup options (In this case very expensive per-camera subscription prices made this unappealing ) and other avenues that didn’t seem real supportable or reliable. But in reading more on the Hikvision capabilities, I realized the solution would be pretty simple if we got another networked DVR.
VPN, Manual Camera Assignments, Success
Both the new site and the backup site have decent Internet connections with adequate amounts of headroom, so I felt comfortable going down the road I’m about to describe. Each site has a Meraki MX servicing the local network, and creating a site-to-site VPN with the MX appliances couldn’t be easier (something I’ve done dozens of times now). You do want to be mindful of MX capacity for this stuff when working at the enterprise scale, but my small-business deployment is an easy fit for where this is going.
The new site has several VLANs in 10.x.x.x address ranges, while the backup site uses 192.168.x.x on the inside. After I brought up the VPN, I made sure that both sites could ping each other in the right IP ranges, and with just a couple of clicks, the sites were joined (this is soooo easy on Meraki gear).
At the backup site, we added an identical DVR, directly off of the MX. I gave it a fixed IP address, and added it to the list of hosts I want Meraki to notify me about if it ever drops offline. And again, I made sure the new site could ping the new DVR at the backup site. The latency between sites is pretty low as well, given that the same ISP feeds each in a fairly tight geographic area. So far so good.
The Hikvision cameras have a “primary” stream and a lower-load “sub-stream”. My strategy here was to keep the high-quality primary stream confined to the new site, where cameras and DVR are on the same network. Then for the backup, I’d use the sub-stream with the second DVR to not overwhelm either site’s ISP link. As I mentioned, each link has headroom, but this application was not foreseen when the ISP connections were bought and that headroom can disappear quickly if not managed right.
To finish it up, I manually added the handful of cameras from the new site that would likely be of interest in an investigation to the new backup DVR, while watching the primary DVR and network utilization at both sites for any signs of trouble or dropped video. End result- it’s working well so far in all regards.
Takeaways
I’m guessing an IPTV Pro would read this and say “uh, OK… big deal.” For me, it was an opportunity to learn more about the capabilities of the Hikvision equipment, which happens to be almost as straight-forward to work with as Meraki on the network side. At the same time, there are lots of ways to screw something like this up if you don’t take your time and proceed with caution. I can still remotely manage and monitor all of the parts involved, which is very important in the support model for this customer. We’ll see if it stands the test of time, but so far it’s looking good.
When you read this blog, you get value. As the Chief Data Scientist for Wirednot LLC, I crunch numbers and I draw conclusions so you don’t have to. I’ll pie you up a chart so fine you’ll wonder what happened… but that’s besides the point. I’m here today to declare that minimally 18% of all vehicles on the road with a Wi-Fi signal might be equipped with in-built WLAN capabilities. And I stand behind behind that uncertain claim with conviction, I tellya.
Find yourself a busy two lane road, like Route 5 in Elbridge, NY.
Build yourself a business alongside that busy two-lane road, maybe something like this one. When you do the networking for that business, pay attention to what you see in the Air Marshal dashboard in it’s Meraki wireless system.
Seeing that this fairly isolated establishment (no neighbors) “saw”over 1,000 outside SSIDs in a week, I was struck by how many were OBVIOUSLY automobiles and likely built-in. I went through, and counted applied an advanced algorithm to arrive at around 180 of these being safe bets as automobile-mounted Wi-Fi.
That’s 18%, where I come from, baby.
In reality, at least twice of the guaranteed-auto SSIDs looked like they were “likely” car-mounted Wi-Fi, as opposed to Mi-Fi or personal hotspots not mounted in the vehicle. We’ll take half of that additional 18%, or 9%, and add it the original 18% to speculate that in reality, like 27% of all vehicles that passed the Bailiwick Cafe and Market in the last week were equipped with on-board Wi-Fi.
Disclaimer
Of course, I could be wrong. And, I’m not really a Chief Data Scientist. But these two facts aside, feel free to use this analysis any way you’d like.
(Edited after Shay pointed out a phrasing deficiency.)
One of my small rural customers was frustrated. A site I’d not yet been involved with has a single PC that runs a specific agricultural application that occasionally checks into a web database used by all of their sites. And since the problem location is in the boonies, they had no options beyond 4G for Internet service. The frustrations:
I researched the agricultural application and found that it shouldn’t be using but a few MB at a time when it synchronized, yet usage was well into the GB per day. It was time to visit the site, and to do some sleuthing.
More Than Just One PC After All, Other Oddities
The notion of Network Policy can be hard to formalize in small businesses where everyone knows everyone, and it’s as much like family at times as it is a business. When I first got to this site to do my investigation, I confirmed with the site chief that yes, there was only a single computer. And a time clock, behind the 4G connection. That was all that was officially in service operationally. When I got into the 4G modem though, I could see multiple Wi-Fi clients connected to the 4G hotspot… <the plot thickens>. It also turns out that the fairly lightweight application- the only reason the 4G link was being funded to begin with- had it’s own story. And… the data plan itself was pretty pricey as it had not been freshened up in years.
The Fix(es)
To get the costs under control, and to remove all mystery about what was going on here, I did the following:
With the Z1, I was able to accomplish a number of things:
After the changes, I’m seeing total site usage of only around 80-90 MB per day in an operational paradigm where I’ve budgeted for around 400 MB per day. As I see recreational traffic pop up, I can quietly block it remotely, without the owner constantly needing to re-enforce the rules (staff here have specialized skills, they can’t just be replaced). And I’ve given the owners a 3rd-party they can turn into a bogey man if they need to should anyone complain (this in itself has value).
Bottom line- this was a fun one to solve. We were able to contain costs, remove any mystery, and provide remote monitoring and alerting. Also- by using the Z1, the time clock can benefit from site-to-site VPN back to the main site where another Meraki MX is in use with the Time and Attendance server.
Though I have used many Meraki wired and wireless products, this was my first outing with the Z1. It’s an impressive little gem, and very much “feels” like it’s big brothers, the MX line.
My own love affair with Meraki started way earlier than Cisco’s acquisition of the cloud networking biggie. Though we use Meraki in targeted locations at my “day job”, I’ve followed their evolution in my long-running role as a gonzo freelance IT journalist since the days when they only offered Wi-Fi, then through the addition of the MX series of security appliances and Ethernet switches. I’ve had the rare frustration with Meraki’s features, but I do mean rare compared to the pain caused by other vendors consistent shitcode. For my own small consulting company, there’s one Meraki feature I’m incredibly fond of as an administrator- and that’s Client VPN. It’s easy to setup (but you still have to understand a few things), and incredibly empowering to the remote administrator.
Down on the Farm
My favorite customer is a prestigious large dairy farm that needed a network overhaul. When I took on the account, there was a mishmash of consumer-grade routers and switches in use, multiple 4G ISP connections, and lots of odd little islands of individual networks. I was able to tame the beast, making it a single decent network with point-to-point wireless bridges connecting far away buildings (using 5 GHz where possible, 900 MHz through trees), UPS, managed switches, and Meraki APs. Along with keeping the network healthy, I find myself doing a bit of desktop and device support. My philosophy is to never visit the site unless something new is physically being added. I’d much rather do everything remotely, which brings me back to Meraki’s client VPN.
Setting it up: the farm network is 192.168.1.0/24 on the inside (part of what I inherited), with a single public ISP address on the outside of the Meraki MX. Here’s where you set up client VPN in the MX:
Then, you need to configure the VPN client on a PC, and here’s Meraki’s how-to. The guidance is straight forward, but I was first tripped up by a Windows 7 machine that absolutely wouldn’t work despite proper VPN settings (I’ve done a lot of VPN administration through the years, have never seen anything like this one odd Win 7 laptop). Once you get the PC set up and connected to the MX with client VPN, you have to be mindful of what you’re doing between networks.
NOTE: My home network also happens to be 192.168.1.0/24- just like the farm. This creates a routing problem going from my home network to the farm network over VPN, as I need to “come in” to the farm network from a differently numbered network (you’ll see why in a minute). I could solve this multiple ways- like by re-addressing my home network, adding a second VLAN/IP space to use for administering far-away 192.168.1.0 networks, or tethering to my 4G phone that uses a different IP space on the “inside” when I’m at home). Just know that 192.168.1.0/24 can’t client-VPN off to another site and then be used to administer the same 192.168.1.0/24 IP space on the far end (not easily, at least).
The last step in the process that allows me to reach into the private farm network with client VPN is to configure a static route that points my traffic to the farm’s 192.168.1.0 network via my connected VPN interface (in this case 192.168.19.148). The following shots show me 1.) connecting to the farm from a public network with VPN address 192.168.19.148, 2.) adding the static route in Windows and 3.) then both ping and trace route to farm network router at 192.168.1.1.
If this seems complicated, it’s not. It takes minutes. From here, anywhere in the world, I can administer and monitor the devices on the farm as if I were standing there in the front office. Of course, PC configurations like Remote Desktop still need to be correct if that service is needed, but I’ve used the method described here to change printer settings, check on bridge links from the bridges themselves, and to find devices on the network that had been moved- all remotely and without travelling to customer sites. I know that this isn’t exactly cutting edge or exclusive to Meraki, but I haven’t seen a client VPN setup as easy as with the MX, myself. Well done, Meraki.
When an administrative account is used to access a Cisco wireless controller, one of two things have happened. Either a legitimate admin has logged in to do config work or to view settings or whatever, or someone has gotten hold of an admin credential and your organization has bigger problems than simply protecting the pre-share key on PSK Wi-FI networks.
My question for Cisco: why can’t paying customers with proper admin credentials see their own PSK keys? Whatever “protection” is in play here is far outweighed by the nuisance of not being able to visually verify or recall what these keys are, says I.
You’ll notice there is no toggle to show the key. You either write it down somewhere, or do a lot of re-creating PSKs if the entered value gets lost. Sounds like maybe no big deal? I disagree. Given the dumbing down of client devices in the name of IoT and BYOD, PSK networks aren’t going away anytime soon and WLAN is only getting more popular. For some customers, hundreds of PSK networks are in play, so it can be a very big deal.
It’s time for Cisco to trust us to see our own PSK strings (and no, they don’t show in CLI, either) and to not worry that bogeymen might be standing behind us waiting to write these down.
I can’t recall another UI that doesn’t let you see the PSK. Here’s Meraki:
Whatta ya say, Cisco? Can we please get a view to our own PSK strings? Can we?
Or am I off-base here? Please comment- and thanks for stopping by.
Things are always shaking in Wi-Fi Land. New stuff, company goings on, regulatory drama… it’s never boring. Here’s a quick bundle of interesting hits to consider.
I don’t typically favor scraping press releases into a digest blog, but this mix of topics struck me as a bit profound in showing just how dynamic the Wi-Fi world is at many tiers. Exciting, thought-provoking stuff that can be hard to keep up on. Don’t blink, things change quick around here!
“Guest Access” means different things to different people, and organizations. Certainly if you’re a traveler using hotel or conference Wi-Fi, you have a general set of expectations and desires. If you’re a company or a school, the guest wireless service you provide is likely shaped by organizational policy. And for many of us, the guest environment also tends to act a s a catch-all for client devices that don’t fit on our secure WLANs- a place for “free passes” and MAC exceptions. But the devil is in the details, and I have found finding the right guest access feature set can be difficult.
What you WANT may not be what you can HAVE
Having designed a number of guest environments for large and small networks, I’m always astounded to engage a WLAN vendor on the topic and to find how far their guest offering is from what I’m looking for (more on that in a bit). Worse, seldom do I hear “what are your requirements?” as it tends to be more like “this is what we think everyone should want and accept”.
Simplicity? Fat chance…
Guest access can also have a lot of moving parts, depending on how it’s implemented. Overall functionality tends to be broken up and scattered across access points, controllers, RADIUS servers, credential stores, web servers, and sometimes switches. It all has to click, or you have problems. And for me, despite the typical complexity of guest services, I still find myself frustrated at features that are not included.
What worked for my environments
Years ago, for my big honkin’ 3,000 AP environment (and our small branches alike), we arrived at a desired feature set that went more or less like this:
Going back those several years, our WLAN vendor (Cisco) didn’t come close to being able provide what we wanted. In their defense, nor did any other market leaders at the time. We heard that Colubris Networks had a gateway that might fit the bill, but they had just been bought by HP and try as we might, we couldn’t locate anyone that could talk with us about what we were looking for.
Then we found Bluesocket (now Adtran) and their BSC Controllers. When I first contacted Bluesocket, we came to the mutual realization that they could do about 75% of what I wanted. They weren’t really initially open to developing the self-sponsored texting and “anyone authorized can sponsor a guest” features. So… we thanked each other for our time, and I kept searching. Then a week or so later Bluesocket called back, and said they were game for a bit of development, and saw the value in what would become a feature set that they were able to market to others. They were able to do everything I was looking for in a single, kick-ass box in a matter of hours.
What Bluesocket was able to deliver after actually listening to our requirements has been in play for us for lots of years. We’ve served thousands and thousands of guests with it, along with using it as a mechanism for supporting wonky devices like Google Glass (turn head, spit) that weren’t built with enterprise security support, and so can’t be on the WLANs we’d rather they used.
It’s been absolutely great, and I know of at least three other schools that pursued the same guest access model after experiencing ours.
Looking forward
Our old Bluesocket boxes are getting, well… old. They are appliances, and Adtran seemingly has no desire to virtualize what we need into an OVA or the like. In fact, on newer Adtran wireless products, what we appreciate about the BSC has been moved to Adtran APs that we’ll never buy, so the research for a suitable replacement starts again.
The thing is, we absolutely love what we get out of our aging guest solution, and in a perfect world, I’ll find a similar third-party, one-box bolt-on for our big Cisco WLAN. (I will give Cisco another chance to catch me up on how their native guest access services have improved, but I also know that my requirements are firm). I have also inquired to Adtran one last time about the possibility of somehow preserving this wonderful magic, but the silence thus far is pretty telling.
Which brings me to Meraki. The features I need for my guest environment are pretty much included in the WLAN side of the Meraki product line, and we use it with great success in our Meraki-enabled branch sites. But… to bolt the Meraki capability up to my Cisco WLAN in a way that would replace Bluesocket, I’d need the guest features made available in the Meraki MX security appliances and not just in the AP feature set. I’m hoping to get Meraki’s ear on this anyway, because guest access needs also do tend to pop up on the wired side occasionally, too. Right now, wired guest needs are a gap in the MX.
If Meraki can accommodate, a big MX would snap in nicely where my Bluesocket sits now for guest access. If not, I’ll have to consider things like pfSense, Packetfence and other one-offs that I’d rather not get into after being happy with a commercial solution. Or, I’ll have to rethink our requirements, which would really suck, as they really are what we consider requirements, not just nice-to-haves.
There will obviously be more to follow to this evolution. I am curious if anyone else is facing a similar situation, and how you might be approaching it.
(Please- I’d love your comments, just don’t blast me with pointless “you should switch to vendor X for your WLAN!” type feedback.)
wirednot 