Code Bugs Do Have Real World Consequences

I’m not sure if my expectations are just too high for today’s world. When I buy a new vehicle, I don’t want to see surface rust forming two weeks after it leaves the lot. I don’t like the current presidential election and the horrible choice that voters have to make. And I actually expect that network vendors will put out decent code, or at least be very up front and open when significant faults are found. 

You see, those significant faults have real-world consequences. They bring operations to a screeching halt, and diminish organizational credibility. And ill-conceived “work arounds” and cavalier vendor attitudes to the customer’s bug-induced plight just make matters worse.

Here’s a real-world example.

I had a carefully worked-out maintenance window to upgrade both ends of a site-to-site VPN topology that spans Syracuse to London, using my favorite cloud-managed vendor’s gear. I’ve done this procedure at least a half dozen times, and have installed at least 30 of this particular security appliance. My Syracuse work was coordinated with a gent on the other end, and we’d do one end at a time. But… we never got past my end.

I configured the new appliance with what few settings it needed: IP address, gateway, subnet mask, and DNS servers. I saved them, then I waited for the indications that the box had made contact with the cloud and pulled down it’s updates. But those indications never came.

Like many a networker would do, I went to verify that the settings that I entered were correct. Curiously, there were NO settings saved. OK- maybe I forgot to save… The second try yielded the exact same result as the first. It was time to open a support case- as my maintenance window ticked away and my partner in London waited patiently.

I opened the case, then immediately called the support line (for the sake of expedience). I was told that this particular appliance has a firmware bug straight from the factory and that I’d need to find a DHCP-served network to use because it won’t actually save anything you enter with out-of-box firmware. When I asked if this was documented anywhere, I was told very matter-of-factly “we don’t share that information with customers” and that it shouldn’t be a big deal to just use DHCP.


Most places I’ve installed these appliances don’t have DHCP services readily available, because ultimately the appliances use a static IP and eventually ARE the DHCP servers for inside clients. And, I don’t tend to lug around an extra SOHO router just on the off-chance I’ll have to jam something in that can act like a DHCP server to get around a code bug that my vendor doesn’t feel customers need to know about before they actually try to use the product.

Let’s skip to the end:

  • I got to use some of my best “military” language after I realized the gravity of the situation
  • The maintenance window was busted, and the scheduled change didn’t happen
  • I probably lost credibility with my London partner as I was the Guy in Charge for this
  • My vendor has absolutely lost my confidence given the bug, and the “you should just be okay with this” attitude. I’m just not sure I can trust them at this point
  • This vendor had my respect and trust for years, and those have pretty much been undone with this one incident

So… I dragged the appliance off to where I could hook it up to a DHCP server and it could get a firmware upgrade. We’ll have to do the same on the London end, and then reschedule the outage and maintenance.

Sadly, the examples don’t end here. Same vendor- different hardware set. Also dealing with a long-running problem with a feature set that absolutely adds to the appliance’s stratospheric price tag. The work around? Don’t use the feature. The feature that I bought- to use. It’s insanity, and it’s way too frequent.

And I can just deal with that, because code bugs are pretty much a way of life anymore with certain vendors.


4 thoughts on “Code Bugs Do Have Real World Consequences

  1. John Cochran

    It’s become frustratingly clear that we’re all just beta-testers (sometimes I think alpha or early alpha!) living in a Vendor’s world. *insert grumpy old man voice ranting about “Back in my day…..”

    It’s sad, the state in which big business (and sadly ever increasingly with small businesses) treats their customer base/s today. Twenty years ago and this kind of crap wouldn’t fly and you’d be treated with respect and could expect most retailers/vendors to be honorable and do the right thing.

    1. wirednot Post author

      Thanks for reading, John. I’d be fine with the whole industry taking a year of from development of features to focus on fixing bugs, at this point.

  2. Manon Lessard (@Mae149)

    I definately agree—with the cost of gear, quality should be a preoccupation. In a world where competition is greater every day, what’s happening is that lots of vendors are cutting corners to “win the contract” where they know they shouldn’t even be bidding—and you can see it in code, in their pre-sales and even with their support. Who pays a premium for support, when it takes 40 minutes to get a p1 engineer? Apparently we still do—
    go figure…

  3. MainFragger

    Quality is definitely important. Buf for some people, the best aspect of quality is getting that device or software up and running yesterday. When you are presented with a bug fix that could be coming next week or could be coming a year from now, that is devastating. That may mean that you can’t get things done that are part of your every day/hours/minutes/seconds of operation. You didn’t even want it to be down for a half hour, and now you may not get a resolution for a year or even, in some cases, never? That is horrific for a business. It also means that in many cases, you may not be able to plan future projects, because this bug derailed the foundation you were laying for those. In some cases, it affects front facing web services that your customers use. Your customers now think YOU are an idiot because you can’t fix the problem on your web site that has existed for a year or more.. (granted, a little flexibility and planning can probably find away around displaying the error publicly, but you get my over all point). Every software company needs an “A-team” or “A-programmer” that can do, “We need it NOW!” type fixes. Downtime is evil these days. No one wants excuses about why something doesn’t work any more. They want it to work or they want you to put your head on the guillotine for them.


Tell me what YOU think.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s