Respect Your Physical Layer

At the risk of sounding cheesy, I have a confession to make. I freakin’ love the physical layer of networking. I learned it well over 20 years ago, and I learned it from the best (thankfully I still work with those gents, though a couple of us have changed roles). The tools are elegant, the test equipment is fascinating, and you have to be daft to not appreciate the importance of a high-performing network foundation. I see each UTP and fiber run as it’s own network component, and I have the utmost respect for the men and women who make a living getting this stuff right for the rest of us. Here in my neck of the woods, we have some challenging old buildings, and the physical layer gets a lot more difficult than lifting tiles in a drop ceiling and heaving a coil across the grid.

Get your cabling right, lest it bite you in the afterior.

I don’t run a lot of cable anymore, as my networking career has taken me into the engineering and architect realms in my day job. But “on the side”, I have a small company and the occasional need to revisit my physical layer skills.

 

Murphy’s Law

Last week at a customer site, I ran some new UTP. We got an ISP upgrade from coax service to fiber and despite explicit instructions, good old Spectrum dropped the new fiber service in the wrong room. Thankfully there was pathway between the wrong room and our network area upstairs, but the seemingly simple task  of connecting the two would prove to be challenging for a number of reasons.

I formulated a plan to run the cable and cut over the network to the new ISP connection on a weeknight evening. The business closed at 8 PM, and I figured I’d run the cable at 7 PM and be cutover and out of there by 8:30… easy-peasy lemon squeezy. Ah, silly me. The first of a handful of issues hit when I was gathering my tools. It would seem that my vintage 200′ fish tape had decided it was tired of my abuse on rural plumbing projects and being put away wet as it was broken in ways I’ve never seen from a fish tape, just sitting there on the shelf. Hmmm- I also have a 50-footer, a cheap plastic thing. It’s a short run! This should work. You know where this is going, but stay with me.

It turns out the 3/4″ plastic pathway I had to work with was, well, let’s just say shit. It must have had 42 bends in it, and after getting maybe 20′ of my Plan B fish tape in the Pipe From Hell, I’m pretty sure they both would have broke if I pushed any further. Fine, whatever. I’m resourceful. I found another unused cable that was in a parallel pathway that was extremely hard to see given the odd layout of the room. Sweet! It turned out I had already terminated the other end years ago, when the building was new and the electrician roughed in the UTP. This was one that I could not find back then…

Throw a jack on the end, tone it out, put it in a single-gang box connect it all up and off we go, right? Yeah, if only… I mentioned getting into this at 7 PM- I had gotten up at 5:30 AM that day and was tired. Things were already not going according to plan, so now I’m scrambling to hurry. My good tester got left at home, so I went the lazy route and checked the new cable with my venerable Siemens STM-8. It’s a wonderful tool for wire-mapping, toning copper runs out, etc- but it’s hardly qualified to properly test a cable’s capabilities. But it’s what I had on hand. The cable “tested” good (wink wink) and it was time to take on the ISP thing. The fiber service was changed to copper in a funky little transceiver box, and I connected my laptop to it in test, via the new cable I just assembled (it was 61′, BTW- my cheap fish wouldn’t have got me there anyways).

The new ISP worked great on my laptop. Then failed miserably on the actual network equipment- a Meraki MX84.

What Gives? The Physical Layer Can Screw With Your Mind

It’s now after 9 PM. I rechecked the cable, end to end- it pinned out fine. I verified my laptop could use the new service. I verified the networking gear could not. DAMMIT IT MUST BE A SPECTRUM ISSUE. Those idiots must have some sort of Advanced Security screwing up my Meraki MX connection to the cloud. Call the vendor!

Nope.

DAMMIT IT MUST BE A MERAKI ISSUE, THE STUPID SECOND WAN PORT ON THIS MX IS BAD. Call the vendor!

Nope. Ruled that out.

It’s 10:30 PM. Customer is tired. I’m tired. I leave them on the old service, we both cuss our way to our vehicles and go home. As I drift off to sleepy goodnights, my mind is on that cable. (I also assumed my laptop was negotiating some oddball speed/duplex combo that the MX84 could not do).

Next day I get to the customer site very early, before opening. I know what I have to do. For sanity’s sake, I use yet a different simple pair mapper to show that yes, I have 8 pins going end to end. Then I throw on The Big Tester. It laughs at me. It tells me that not only will this 61′ cable NOT do Gigabit, it also won’t do 100 Mbps. Oh yeah- it mentions something is wrong with the pin 3 wire. Wowsers. Bad jack? I’m desperate. Let’s re-terminate the patch-panel end and put a new jack on. But start with the field end, hoping to not have to disturb the patch-panel.

I took apart the just-installed jack, and everything LOOKED good. Crappy circuit board in the jack? Gotta be, no pins are bent. And then… I found Mr. White/Green was hanging on by the threadiest of copper threads under it’s plastic coating about half an inch from the end of the wire. Visually it was fine, and the plastic coating was undisturbed, but the copper inside had somehow broke but not so bad that it couldn’t fool the cheap testers. Cut it back a couple of inches, start over… viola!

Ten minutes later, the physical layer was good, PROPERLY TESTED, and the new ISP connection was put into service.

Common Impacts of Poor or Damaged Cabling

My outing was a little trippy, and only reaffirmed my faith in good tools and techniques. (I also questioned the wisdom of getting into this 14 hours into the day.)

Know that cabling also gets weird long after it’s installed, even when it’s installed right. Wiremold covers gets manipulated, new cables can disturb old ones, other trades can ruin life for the data folks and countless other things happen. I didn’t have to look very hard across the environments I support to find the three graphics I used for this blog- two wireless access points and a CCTV camera that ought to be running at Gig, but that are currently operating on a lesser connection because of unknown cabling issues (I know that each of these was Gig once upon a time).

Respect that physical layer, ya’ll. It’s easy to take for granted, but it can also turn on you.