Net Neutrality = Diet Soda, Part 4: the wrap-up

What the hell does this have to do with diet soda?

If you waded through all of that, you deserve an answer.

“Net Neutrality” sounds great. Democratic and fair. Who could be against it?

Full and final disclosure: I do not work for an ISP and will be quite happy to go to my grave never working for one again. You work like a dog, the compensation isn’t that great. The skill level required is very high (at least at the core; there are only a few hundred people in the US qualified to do it). Businesswise, your costs can change wildly, you’ve got a big bulls-eye on your back with dozens of people aiming at it:

• The family-values creeps who often are making money leasing you land, (especially in areas where churches have significant holdings). With a terabyte of netflow logs and the ability to correlate IP blocks to location, I could have written a PhD thesis on the level sexual deviance correlated with church membership, and a statistically-significant favoring of various sexual paraphilias as related to specific branches of various denominations.
• The mpaa/riaa/publishers who see ISP’s as basically a bunch of bored people who stand around passing out copies of Shrek IV: The Search For Happy Meal Tie-ins
• Municipalities, which might just decide that you only get that one extra foot of right-of-way for fiber if you’ll wire Mayor McFarkwad’s school with fiber. Not that he’s twisting your arm.
• Other regulatory costs. This cost has gone up since, oh, roughly October of 2001. It’s difficult to quantify these costs or recover them, and on a macro-economic level it seems overall like a piss-poor investment, but that’s just my opinion.
• Other ISP’s. Us techies might all like each other, but everybody wants job security. If I can figure a way to undercut another company, steal their customers, and have their car repo’ed at the same time…

Diet soda came on the market decades ago as consumers decided they liked soda, and wanted to drink more, but didn’t want to get fatter. The companies considered the possibility of saying “well gee, how about you just drink less?”, decided against it, and an entire industry was saved. The psychology of it is fascinating; how you can charge more per serving for a product with the same cost, by putting it in, say, an 8-pack instead of 12. there’s also significant evidence that the nature of the product itself leads to obesity. So while it sounds great and there’s a documented advantage to that one single can, overall it’s a pretty crappy deal for the buyer. (but don’t try telling ‘em that: they have a brand loyalty second only to cigarette smokers)

Network neutrality is the same crap. Even the most published study of it, ironically, pointed out that the carriers are, quite truly, content-agnostic. Far from being interested in censorship, or promoting a viewpoint, or silencing dissent, it’s just some basic traffic management.

• Bandwidth costs money.
• Customers don’t like tiered pricing. They like a fixed price, and they’ll happily give up features they don’t use.
• Customers want to be able to download something quickly.
• Some customers are a liability. A few customers are a major liability.
• If you can saddle your competitors with more liabilities than yourself, you win. (to the point that if I were in the business I’d personally pay for ads for my competitor on a few selected sites *chuckle*)

The irony of free-speech advocates demanding government monitoring of ISP traffic to ensure compliance with legislation is simply mind-blowing, face-palming, forehead-smacking stuff, proof that if you package it right, you truly can sell a refrigerator to an eskimo – with a service contract, no doubt.

Corporate customers won’t be affected. We’re used to paying a premium for higher uptime and guaranteed throughput. Residential users just want something cheap. First ISP to figure out (and successfully market) a “Casual Broadband” package is going to make some good money.

Screaming censorship every time someone asks you to be quiet in a movie tends to dilute the seriousness of the accusation. Let’s try and save it for the times – and they clearly exist – when it’s real.

Net Neutrality = Diet Soda, part 3

Yep, you caught me. I pocketed that $15 a month and … well, no, not really.

ISP’s make money buying very expensive bandwidth and selling it a lot cheaper. Really.

Remember I told you that bandwidth costs a little over $100 per month per megabit? That’s not too far off.

“But I heard all these ISP’s get bandwidth from each other for free, you big liar!”

Sometimes, they do. I personally made a lot of deals for what’s called “settlement-free peering” with other carriers. But the fact that I’m not writing a check to the company I’m plugged into, doesn’t make it anywhere near free. First, I have to buy more equipment. And this is the heavy iron. Six-digit price tags very typical. Rack space costs money. In a popular data center, expect to pay $1,200 per month for a single rack. Plus power – that big Cisco box with the dual DC power feeds? $700 a month to keep it running. Want a dial-up line to manage it? $300 a month (no, I’m not making that up) If you nab a high-volume peer, you’re going to need a cross-connect to them, figure $500 a month. And you’re going to need an OC-OMG backhaul to this peering point, which isn’t going to be cheap, especially if you want it to be redundant.

I sat down and did the math and figured out that “free” peering typically cost me about $70 per mb/month, once I realized all the other costs. So it *can* be cheaper, but it’s not a guarantee, and you have to plan and use it very efficiently for it to make any sense at all.

So much like the traditional landline telephone, you can sell that $500-a-month circuit to 90 people for $40 each – just as long as they don’t all try to actually use it.

Hence the problem. It’s difficult for even a dedicated gossip to stay on the phone 24 hours a day, 7 days a week. But I can leave my computer on all the time and it’s quite capable of tearing through immense amounts of bandwidth. Phone companies panic when an episode of “American Idol” airs. Really. ISP’s panic the day after a new “Galactica” episode airs.

Remember those figures about bandwidth? A typical household only costs the ISP about four bucks for bandwidth. But they have, let’s assume, a 6mb/s connection. If they run it at 100%, for six hours a month, that one account ends up being a net liability.

If they can flat-top that circuit 10% of the time, that $14 in profit has just turned into $120 in costs, or essentially negating the profit of almost 10 customers.

(ok, a confession is in order here: While in my tenure as the head of engineering at an ISP, I used to keep a good eye on forum postings in a few reputable sites such as dslreports.com as an early-warning system for problems. More than once, I spotted posts that helpfully provided enough specifics – a symptom, a location, a time range, and even a graph sometimes. While it would have been improper to respond in public, I can personally say that two well-informed posters helped me justify an emergency network upgrade affecting over 210,000 households. So you guys aren’t talking to air, the people who actually will fix your problem are listening, even if they can’t talk back.)

Then there’s support. Remember truck rolls? Well, the family that uses the web to send email and occasionally download a movie from itunes, they’ll never notice momentary packet loss due to network congestion. But the guy who’s got six torrents running, he’ll notice it, even if it happened during a 2am planned maintenance window. He’s going to complain. He’s going to need three or four truck rolls. He’s going to call the support center much more often.

It’s harder to do than you might think. Even with totally indiscriminate use of streaming video, voice, lots of downloads, and being completely sloppy and leaving VPN connections up, I can’t even break an average of 300kb/s on my personal connection, averaged over a month. And I’ve tried, mind you, even with a commercial-rate account that has high upload and download speeds, at several times residential rates. Yeah, I can run the connection at 8mb/s for an hour solid with a couple good downloads, but.. that’s only one hour. It takes me 7 minutes to download one hour of high-quality video. Even if I turn into Jabba the Hut on the couch, there’s only so much you can do.

Ah, but torrents. They’re a different animal. First, they’re sloppy. The high error rates due to congested or dead peers mean often that the traffic may be several times the actual data transmitted. Deliberate poisoning by content owners makes the problem worse. The router upstream from you needs to track 50 times as many ip “pairings”, meaning more CPU, more memory, and more likelihood of problems affecting other users.

Note that I don’t give a flying f*ck what’s in the torrent. Family movies, tv shows, tentacle porn, it’s all the same to me, it’s just bits. (personally, I like porn, but generally the mostly plain stuff. I find it hard to empathize with an eight-armed creature of unknown origin and figure out whether they’re having fun, or are just performing some tedious and unpleasant task.)

Ok. Cool. So the telco’s and ISP’s are usually making a profit, even if it’s not exactly the piles of cash we all hoped it would be back in 1998. What the hell does this have to do with diet soda?

Net Neutrality = diet soda, part 2

So we were left with $500 a year. Good money, right? Well, maybe.

A typical residential customer is going to need one ‘truck roll’ a year. Many will need none, some will need several. He’ll work for an average of two hours. He’s not exactly at the top of the pay scale, but still, his salary and benefits for those two hours plus a half-hour of getting to the next call, cost about $80, with salary and benefits. Let’s toss in another $40 for vehicle depreciation, repairs, fuel, insurance, and so on.

So we’re down to about $380 a year. Still, not too shabby, but make note of this: Support rolls cost big #$*%# money.

Equipment costs are a little harder to pin down. A cable modem or DSL modem costs (wholesale) probably under $50, and has a lifetime of at least 24 months. Not a major factor here. We’re still at roughly $360 a year.

Upstream network costs fall into two categories: Plant (the fiber running down the street and around town) and infrastructure (expensive boxes that sit in *hopefully* air-conditioned rooms)

Plant, I can’t really put a handle on it. It does have a lifespan – 15 years might be typical between major upgrade cycles. Assuming each household costs $1200 to wire/re-wire, you’re paying about $100 a year give or take – a little good financial planning probably helps, but of course major customer demand for a new technology means you write a big check sometimes. Still, let’s say that leaves you with $260 a year.

Infrastructure, I can put pretty good numbers on. The distribution equipment costs roughly
$120 per household to purchase, and has about a three-year lifespan. Maintenance, power, and spares add about $70 to that number if you plan well and don’t buy a lot of one-offs. The gear for concentrating users is very pricey – you could buy an S-Class mercedes with the bells and whistles for what a single add-in sometimes costs – but, you’re supporting maybe 50,000 customers with it – so it’s not too bad. Let’s call indfrastructure $35 per year and cross our fingers. We’re left with $225.

Now come some annoying bits. Invoicing, collections, and account management: $4 per month. I know you pay your bill on time and never have a question, but the deadbeat who changes his service every month costs, so you’re still paying it. We’re left with $177 a year.

Advertising. I hate it. You hate it. But it is the only way to get new business. If you add up print, TV, radio, billboard, sponsorships, and what-not, it costs about $110 to bring in each new subscriber. And they leave – someone undercuts you, or they move, or they get married and now what used to be two accounts it only one.. An average account lasts 42 months. We’re left with $150/year, or $12.50 a month. Some of that goes to taxes, some goes to pay for the guys who are brainstorming on how to raise your cable rates, but a few bucks has to find it’s way to shareholders and investors and bankers, otherwise, well, there’s not a lot of point to staying in business, is there?

“But you glossed over that other 20%, you weasel”

Ok. I’m getting to that.

Net Neutrality = Diet Soda, part 1

It seems like a stretch, but stick with me for a minute.

First, my credentials: I’m a 20-year veteran of various IT functions, almost all related to networking. I’ve been actively using the internet since 1982-ish, and I’ve used several of the predecessor and offshoot networks (BITnet, Fido, and yes, even CompuServe and The Source – bet you forgot that one). While I don’t work for a carrier now, I have worked for a couple, where I have had primary design and management responsibility for networks supporting very large user groups. I won’t say exactly which ones, but it’s safe to say that depending on which company I’m talking about, an “oops” would be noticed by most of the population of at least one large state and would affect at least one other country.

Here’s a few not-so-secret items that might help when considering this flowery idea of network neutrality:

The typical residential internet user generates about 28kb/s average traffic over any given month.

When one carrier buys bandwidth from another, a reasonable price would be in the area of $110 per mb/s per month.

There are a lot of other costs, too. Let’s look at the customer side first. We’ll use a hypothetical $60 a month invoice.

80% of that might go to the local company that you’re buying service from. 20% likely is allocated to an internal or external partner that runs the network for them.

So of that 80%, or $45 a month, we’ll round it off to $500 a year. Where does it go? Stay tuned.

Next: Part 2 – where does that money go?