Tag Archives: Meraki

Don’t Forget About Those OTHER Meraki MX Firewall Rules

I’m a long-time user of the Meraki MX security appliance product line. Going way back to the MX-70, I have found tremendous value in what the MX products can do for my far-off sites. (Here’s an old- and I mean old- case study that gets into the early appreciation of the MX line.) I’ve probably set up maybe 65ish total MX devices through the years in multiple states and countries, doing site-to-site VPN, stand-alone, and also some pretty creative configurations. Despite my experience, I was recently reminded that I don’t know it all about a product that I feel extremely comfortable calling myself an expert on.

In one remote site that connects to the main network with site-to-site VPAN, an NTP vulnerability was flagged on a couple of audio visual devices. The device vendor was of absolutely no help (go figure), and our security team asked if we could help from the Meraki side. “Oh sure…” says I. “We got a firewall to leverage.”

We needed to cabash NTP between the remote site and the main network. I pulled up the Firewall page on the MX and set to work. This is an area in the MX I’ve probably manipulated maybe a couple of dozen times, for everything from stopping phantom ringing on 3rd-party hosted IP phones to simple outbound protocol blocks.

L3 Firewall

That image represents like three stages of desperation in getting rules right- as nothing I did worked. I simply could not tame the NTP beast to/from the two hosts, and it was making me feel silly. My first inclination was to blame Meraki- surely this stupid box must have issues! Except it didn’t… about the only thing Meraki could have done is perhaps mentioned on the L3 Firewall Page that there is a seperate firewall rule set on the VPN configuration page for site-to-site rules. That looks like this:

Site-to-Site FW

I had just never did firewall rules for the site-to-site tunnel. I didn’t know after many years! But I did leverage the Meraki “search our documentation” repository to get educated, with this document that explains it. There’s nothing complicated about it, you just have to know where to find it the first time you need to configure rules for the tunnel versus the Internet edge.

And now you know, too.

 

Unfortunately, Seeing Isn’t Always Believing With Meraki Network Topology View (updated)

update After opening a support case, Meraki was able to verify what I describe below in their own lab testing- some MX ports were not doing LLDP right. And…. the fix: “The issue of the transmission of LLDP packets from the MX is fixed in versions starting MX14.2” – which so far is proving to be the case. (Now we need to solve client MAC addresses showing up on the wrong switch ports, but that’s another tale for another time.) Read on if you’d like, but keep in mind that they DID fix this problem at long last.


The only thing worse than an important feature that’s missing in a network management solution is an important feature that works wrong. Those of us who pay for these systems expect the features that we trade our dollars for to actually, well, work. And bad information is just terrible in that it sends us down time-wasting bad roads.

Meraki- you have a problem. Your topology view doesn’t refresh for hours, days, or weeks after changes are made. And for whatever reason, you have not given us decent LLDP tools that we can invoke on demand. That’s the polite description, which can be abbreviated down to your topology view sucks at times.

Before I show and tell my latest frustration in this regard, let me share Meraki’s summary on it’s topology view and evidence that what I’m about to describe is not my frustration alone. See this post from the Meraki community. Now on to the defective topology feature.

It Just NEVER Updates

In a faraway branch location, we had two switches that daisy-chained off of another switch due to physical layer constraints. Here’s the old topology:

Pay attention to VillaRosa-3, Villino-1,and Vilillino-2. V1 and V2 were daisy-chained off of V3. THEN THEY WERE MOVED TO EACH DIRECTLY CONNECT TO THE MX. And then I got an email… “The switches got moved to ports 6 and 7 on The MX, but the topology isn’t updating. Maybe I should restart the devices…” As we see in the community forums, we may be waiting quite a while for this map to update…

The Implications

Few things are more basic in network support than simply knowing what connects to what- both physically and logically. Commands like <show cdp neighbors> and <show lldp neighbors> are pretty important, but not available in the Meraki dashboard paradigm. Instead, we need to rely on incomplete or inaccurate graphical information.

As I mentioned above, V1 and V2 moved to direct connects on the MX. So what does the MX say about the switches that are connected to it? Unfortunately, nothing. For whatever reason, Meraki has never seen it as important to give full lldp information for devices connected to an MX:

This is utterly nonsensical for enterprise-grade networking equipment. But it gets worse. Let’s look at one of the switches that has had it’s uplink moved to the MX. Days after the move, the switch still says it’s connected to it’s old uplinked V3 switch:

This is pretty bad. It’s wrong, it’s outdated, it’s misleading, and it’s inexcusable. Let’s have a look at “VillaRosa-3 / 51.

To recap:

  • Topology views don’t refresh as they should (if ever)
  • The MX doesn’t tell who it’s lldp neighbors are- which is a glaring deficiency
  • The lldp neighbors information on the Meraki switches can’t be trusted
  • There are no obvious ways to invoke on-demand refreshes for topology and lldp views, we are at the mercy of some undeclared loooooooong refresh timer

That this “feature set” can be this off-kilter defies logic- especially when you consider the cost of the gear and it’s licensing and the importance of lldp to network support.

Leveraging Meraki Site-Site VPN For Backup Security Video

Please remember, this blog is mostly about wireless- but not always. I recently brought a customer site online where I did the LAN, Wi-Fi, simple background music speakers with satellite radio feed, and IP-CCTV, all in a brand new building. The Ethernet and Wi-Fi environments are just footnotes to the rest of this blog, but the Meraki MX84 that serves as the network head-end will factor large in the story.

The video system is of Hikvision components, with a couple of POE camera types and a networked DVR. I don’t do a lot of video applications, but the Hikvision stuff is pretty easy to work with end-to-end. The new building is fairly large for a small business, and we ended up with 15 cameras total in what can only be described as just another well-networked environment.

Can we do… THIS?

This client has multiple business sites, and I have built the networks in all of them. It turns out that in the past, a case of arson and significant loss of property has them constantly thinking about security. One day after the new site opened, I was asked if we could somehow backup the video at the new site to one of their other sites. If the unthinkable should ever happen and the new-site DVR were to be destroyed along with it’s contents, the idea would be that the same video would live at a backup site to hopefully shed light on what happened.

I was a bit stumped on this one, and did some research on cloud-backup options (In this case very expensive per-camera subscription prices made this unappealing ) and other avenues that didn’t seem real supportable or reliable. But in reading more on the Hikvision capabilities, I realized the solution would be pretty simple if we got another networked DVR.

VPN, Manual Camera Assignments, Success

Both the new site and the backup site have decent Internet connections with adequate amounts of headroom, so I felt comfortable going down the road I’m about to describe. Each site has a Meraki MX servicing the local network, and creating a site-to-site VPN with the MX appliances couldn’t be easier (something I’ve done dozens of times now). You do want to be mindful of MX capacity for this stuff when working at the enterprise scale, but my small-business deployment is an easy fit for where this is going.

The new site has several VLANs in 10.x.x.x address ranges, while the backup site uses 192.168.x.x on the inside. After I brought up the VPN, I made sure that both sites could ping each other in the right IP ranges, and with just a couple of clicks, the sites were joined (this is soooo easy on Meraki gear).

At the backup site, we added an identical DVR, directly off of the MX. I gave it a fixed IP address, and added it to the list of hosts I want Meraki to notify me about if it ever drops offline. And again, I made sure the new site could ping the new DVR at the backup site. The latency between sites is pretty low as well, given that the same ISP feeds each in a fairly tight geographic area. So far so good.

The Hikvision cameras have a “primary” stream and a lower-load “sub-stream”. My strategy here was to keep the high-quality primary stream confined to the new site, where cameras and DVR are on the same network. Then for the backup, I’d use the sub-stream with the second DVR to not overwhelm either site’s ISP link. As I mentioned, each link has headroom, but this application was not foreseen when the ISP connections were bought and that headroom can disappear quickly if not managed right.

To finish it up, I manually added the handful of cameras from the new site that would likely be of interest in an investigation to the new backup DVR, while watching the primary DVR and network utilization at both sites for any signs of trouble or dropped video. End result- it’s working well so far in all regards.

Takeaways

I’m guessing an IPTV Pro would read this and say “uh, OK… big deal.” For me, it was an opportunity to learn more about the capabilities of the Hikvision equipment, which happens to be almost as straight-forward to work with as Meraki on the network side. At the same time, there are lots of ways to screw something like this up if you don’t take your time and proceed with caution. I can still remotely manage and monitor all of the parts involved, which is very important in the support model for this customer. We’ll see if it stands the test of time, but so far it’s looking good.

At Least 18% of All Vehicles With Wi-Fi Are Integrated (Maybe)

When you read this blog, you get value. As the Chief Data Scientist for Wirednot LLC, I crunch numbers and I draw conclusions so you don’t have to. I’ll pie you up a chart so fine you’ll wonder what happened… but that’s besides the point. I’m here today to declare that minimally 18% of all vehicles on the road with a Wi-Fi signal might be equipped with in-built WLAN capabilities. And I stand behind behind that uncertain claim with conviction, I tellya.

The Methodology

Find yourself a busy two lane road, like Route 5 in Elbridge, NY.
route 5

Build yourself a business alongside that busy two-lane road, maybe something like this one. When you do the networking for that business, pay attention to what you see in the Air Marshal dashboard in it’s Meraki wireless system.

AirMarshal

The Findings

Seeing that this fairly isolated establishment (no neighbors) “saw”over 1,000 outside SSIDs in a week, I was struck by how many were OBVIOUSLY automobiles and likely built-in. I went through, and counted applied an advanced algorithm to arrive at around 180 of these being safe bets as automobile-mounted Wi-Fi.

That’s 18%, where I come from, baby.

Now the Fudge Factor

In reality, at least twice of the guaranteed-auto SSIDs looked like they were “likely” car-mounted Wi-Fi, as opposed to Mi-Fi or personal hotspots not mounted in the vehicle. We’ll take half of that additional 18%, or 9%, and add it the original 18% to speculate that in reality, like 27% of all vehicles that passed the Bailiwick Cafe and Market in the last week were equipped with on-board Wi-Fi.

Disclaimer

Of course, I could be wrong. And, I’m not really a Chief Data Scientist. But these two facts aside, feel free to use this analysis any way you’d like.

(Edited after Shay pointed out a phrasing deficiency.) 

 

Oh Say Can You See- What’s Driving Up Your Small Site Data Costs?

One of my small rural customers was frustrated. A site I’d not yet been involved with has a single PC that runs a specific agricultural application that occasionally checks into a web database used by all of their sites. And since the problem location is in the boonies, they had no options beyond 4G for Internet service. The frustrations:

  • Huge data bills that weren’t making sense for a single PC
  • No sense of what was going on at the site over the network
  • Getting to the site isn’t exactly a quick drive

I researched the agricultural application and found that it shouldn’t be using but a few MB at a time when it synchronized, yet usage was well into the GB per day. It was time to visit the site, and to do some sleuthing.

More Than Just One PC After All, Other Oddities

The notion of Network Policy can be hard to formalize in small businesses where everyone knows everyone, and it’s as much like family at times as it is a business. When I first  got to this site to do my investigation, I confirmed with the site chief that yes, there was only a single computer. And a time clock, behind the 4G connection. That was all that was officially in service operationally. When I got into the 4G modem though, I could see multiple Wi-Fi clients connected to the 4G hotspot… <the plot thickens>. It also turns out that the fairly lightweight application- the only reason the 4G link was being funded to begin with- had it’s own story.  And… the data plan itself was pretty pricey as it had not been freshened up in years.

The Fix(es)

To get the costs under control, and to remove all mystery about what was going on here, I did the following:

  1. Calculated what the application should need, along with Windows updates, etc. then found a newer, more generous plan than what they were on. I recommended 12 GB/month plan for $80, which should provide fixed cost and at least 300% headroom on my estimated usage. (The bonus, Verizon throws in an extra 2 GB per month on this plan.)
  2. Had the application vendor audit the application behavior. What was taking 600 MB per day was dialed down to around 60 MB by changing from continuous sync to a 4-hour interval (which still met the owner’s needs).
  3. Reigned in the 4G rogue client use. On this modem, the Wi-Fi can’t be disabled. But I changed the SSID and password, lowered the number of allowed users to 1 (the minimum) and instructed the owner to tell the staff that this network is off-limits even if they can figure out how to get back on,  along with a message that “the IT guy monitors everything!”
  4. Both eliminated any mystery and took control of the bad habits associated with the PC by installing a Meraki Z1 Teleworker appliance between the 4G modem and the PC and time clock. Weedsport3

With the Z1, I was able to accomplish a number of things:

  • Use traffic analysis to remotely see what else was going on with the PC, besides the ag application
  • Use firewall rules and application controls to put an end to all non-authorized applications
  • Provide a client VPN-endpoint so I can access the environment for troubleshooting if need be
  • Monitor data usage and get automated reports on what’s going on in the small environment
  • Get alerted should either the PC or time clock go offline
  • Make myself the heavy in the situation, and take that title off of the owner

After the changes, I’m seeing total site usage of only around 80-90 MB per day in an operational paradigm where I’ve budgeted for around 400 MB per day. As I see recreational traffic pop up, I can quietly block it remotely, without the owner constantly needing to re-enforce the rules (staff here have specialized skills, they can’t just be replaced). And I’ve given the owners a 3rd-party they can turn into a bogey man if they need to should anyone complain (this in itself has value).

Bottom line- this was a fun one to solve. We were able to contain costs, remove any mystery, and provide remote monitoring and alerting. Also- by using the Z1, the time clock can benefit from site-to-site VPN back to the main site where another Meraki MX is in use with the Time and Attendance server.

Though I have used many Meraki wired and wireless products, this was my first outing with the Z1. It’s an impressive little gem, and very much “feels” like it’s big brothers, the MX line.

 

Loving Meraki Client VPN For Remote Administration

My own love affair with Meraki started way earlier than Cisco’s acquisition of the cloud networking biggie. Though we use Meraki in targeted locations at my “day job”, I’ve followed their evolution in my long-running role as a gonzo freelance IT journalist since the days when they only offered Wi-Fi, then through the addition of the MX series of security appliances and Ethernet switches. I’ve had the rare frustration with Meraki’s features, but I do mean rare compared to the pain caused by other vendors consistent shitcode. For my own small consulting company, there’s one Meraki feature I’m incredibly fond of as an administrator- and that’s Client VPN. It’s easy to setup (but you still have to understand a few things), and incredibly empowering to the remote administrator.

Down on the Farm

My favorite customer is a prestigious large dairy farm that needed a network overhaul. When I took on the account, there was a mishmash of consumer-grade routers and switches in use, multiple 4G ISP connections, and lots of odd little islands of individual networks. I was able to tame the beast, making it a single decent network with point-to-point wireless bridges connecting far away buildings (using 5 GHz where possible, 900 MHz through trees), UPS, managed switches, and Meraki APs. Along with keeping the network healthy, I find myself doing a bit of desktop and device support. My philosophy is to never visit the site unless something new is physically being added. I’d much rather do everything remotely, which brings me back to Meraki’s client VPN.

Setting it up: the farm network is 192.168.1.0/24 on the inside (part of what I inherited), with a single public ISP address on the outside of the Meraki MX. Here’s where you set up client VPN in the MX:

client VPN

Then, you need to configure the VPN client on a PC, and here’s Meraki’s how-to. The guidance is straight forward, but I was first tripped up by a Windows 7 machine that absolutely wouldn’t work despite proper VPN settings (I’ve done a lot of VPN administration through the years, have never seen anything like this one odd Win 7 laptop).  Once you get the PC set up and connected to the MX with client VPN, you have to be mindful of what you’re doing between networks.

Client VPN 2

NOTE: My home network also happens to be 192.168.1.0/24- just like the farm. This creates a routing problem going from my home network to the farm network over VPN, as I need to “come in” to the farm network from a differently numbered network (you’ll see why in a minute). I could solve this multiple ways- like by re-addressing my home network, adding a second VLAN/IP space to use for administering far-away 192.168.1.0 networks, or tethering to my 4G phone that uses a different IP space on the “inside” when I’m at home). Just know that 192.168.1.0/24 can’t client-VPN off to another site and then be used to administer the same  192.168.1.0/24 IP space on the far end (not easily, at least). 

The last step in the process that allows me to reach into the private farm network with client VPN is to configure a static route that points my traffic to the farm’s 192.168.1.0 network via my connected VPN interface (in this case 192.168.19.148). The following shots show me 1.) connecting to the farm from a public network with VPN address 192.168.19.148, 2.) adding the static route in Windows and 3.) then both ping and trace route to farm network router at 192.168.1.1.

CLient VPN 3 Client VPN 4

If this seems complicated, it’s not. It takes minutes. From here, anywhere in the world, I can administer and monitor the devices on the farm as if I were standing there in the front office. Of course, PC configurations like Remote Desktop still need to be correct if that service is needed, but I’ve used the method described here to change printer settings, check on bridge links from the bridges themselves, and to find devices on the network that had been moved- all remotely and without travelling to customer sites. I know that this isn’t exactly cutting edge or exclusive to Meraki, but I haven’t seen a client VPN setup as easy as with the MX, myself. Well done, Meraki.

It’s the Little Things… Hey Cisco Wireless!

When an administrative account is used to access a Cisco wireless controller, one of two things have happened. Either a legitimate admin has logged in to do config work or to view settings or whatever, or someone has gotten hold of an admin credential and your organization has bigger problems than simply protecting the pre-share key on PSK Wi-FI networks.

My question for Cisco: why can’t paying customers with proper admin credentials see their own PSK keys? Whatever “protection” is in play here is far outweighed by the nuisance of not being able to visually verify or recall what these keys are, says I.

No PSK show

You’ll notice there is no toggle to show the key. You either write it down somewhere, or do a lot of re-creating PSKs if the entered value gets lost. Sounds like maybe no big deal? I disagree. Given the dumbing down of client devices in the name of IoT and BYOD, PSK networks aren’t going away anytime soon and WLAN is only getting more popular. For some customers, hundreds of PSK networks are in play, so it can be a very big deal.

It’s time for Cisco to trust us to see our own PSK strings (and no, they don’t show in CLI, either) and to not worry that bogeymen might be standing behind us waiting to write these down.

I can’t recall another UI that doesn’t let you see the PSK. Here’s Meraki:

Meraki PSK

Whatta ya say, Cisco? Can we please get a view to our own PSK strings? Can we? 

Or am I off-base here? Please comment- and thanks for stopping by.