Content-Aware DPI. Nirvana or dead end?

I've been seeing some buzz in the industry regarding content-aware DPI solutions. While I think it's a bit of an oxymoron - show me DPI that isn't content aware and does any sort of meaningful recognition - it's really a fancy name for P2P content awareness in this case.

The general idea is that providers will use this to differentiate between freely distributable and non-freely distributable material over P2P. This is interesting in several ways.

To set the stage - DPI boxes are pretty capable nowadays and relatively cheap - maybe not in euros, but on a cost-per-subscriber basis. So it's quite possible to do fairly advanced (and in some cases, invasive) stuff if you want to munge the data. If you combine a box seeing all the traffic for a given provider with an external service that traverses P2P networks, you can piece together quite an accurate image of what data the subscribers are hauling. If you care about this for whatever reason, do not just assume that encryption is the holy grail for avoiding it.

Who would like this?

From a provider point of view, it'd be possible to treat the 'illegal' traffic differently - shaping, prioritizing, blocking or sending it down a different/cheaper network path. If this is a good thing or a bad thing is entirely up to the observer. In terms of euros and cents, P2P accounts for 75% of the total traffic transferred in some european markets. The majority of that would be movies. Make a dent in this and you can postpone network investments for a while. In a world where hardware gets cheaper and cheaper down the line, postponing an investment for a year can mean a lot of money saved. There's also possible bandwidth savings on top of this.

From a governmental point of view, this is technology that somewhat reliably allows for stopping copyright infringement. It's also cheap enough so that providers can be mandated to implement it (I'm sure providers would see this differently for obvious reasons)

From the copyright sensitive and lawyer-happy trade group point of view, this is nirvana. Paint yourselves as victims, ask the government to protect this important GNP-improving revenue stream. "It's not only the right thing to do, it's also the reasonable thing, right?"

What would it lead to?

Better legal music and video services is one thing. If we see something like RIAA-friendly-P2P filters implemented, it will mean that JaneRegularUser will have trouble using her torrent/gnutella client to fetch music and movies and either will have to update to a newer version (which might not be available right off the bat) or use something else. We're seeing an emergence of services that fill this 'something else' gap. Judging by Spotify usage statistics, there's already quite a few people who are using that and it's a steadily increasing number. I'd estimate roughly 1.5% of the Swedish internet users, compared to 0.3% six weeks ago.

There might be a rise in Darknet activity - but darknet limitations in terms of latency and outright virtual hooliganism (to mess the network up) - won't make this attractive to JaneUser - especially not compared to the legal means of sourcing the material.

P2P clients will adapt. There's quite a few ways to screw DPI gear (most of which I haven't seen in the wild yet) and we'll be seeing more of that. There'll be a spot of an arms race, but the end result will probably result in P2P looking quite different from what we're used to - from a networking perspective, at least. I'm quite sure that P2P would still be identifiable as P2P - identifying the content in transit, however, might not be doable.

Options?

Simple. Don't go down that path. As mentioned, we're seeing legal means of distributing music (Spotify, others) and video (Hulu, iPlayer, many more) gaining popularity as is. P2P is largely a convenience thing for the majority of users and as the legal means of distribution gets more convenient than the P2P, well.. you do the math.

From a governmental perspective, we have a small but fairly deep-pocketed number of trade groups who really would love stricter control over what's being transferred over the Internet. Most of their problems stem from their reluctance to get on with the times. Sure, consumers are breaking laws, but seeing that pretty damn large swathes of the population does it and that they seem quite keen on going for a legal option where one exists makes it more of a non-issue. Less regulation is a good thing in this case, methinks.

The majormulticide has begun

We've seen a number of indie MMO's come and go, but a majorly hyped and expensive production - Tabula Rasa - is closing its doors early next year.

MMO's have been seen a good way forward for gaming - sure, the upfront expenses to actually be a mainstream player are somewhat staggering, but the gains once and if you get there are pretty neat, and it's one very efficient way of bypassing the entire problem of piracy.

With the early success of EverQuest and the breakthrough of World of Warcraft, World + Dog decided that now's a good time to develop an MMO and get a cut. 'Now' being 2005-2006, with some major launches last year and this year.

Judging by european and north american traffic though, we'll be seeing a few more of the major releases go titsup sooner or later. I'd judge Age of Conan to be a good candidate for this, due to a very low player count and a (probably) fairly expensive development cycle.

Games with very strong IP ties - Lord of the Rings, Warhammer and - naturally - Warcraft seem to be doing allright, alongside EVE and Second Life. Warhammer's a bit too new to judge yet, but if they manage to keep quality somewhat high, I have a hard time seeing them do a sharp nosedive anytime soon. Various small indie games are also doing well - Tibia is actually right up there competing with Warhammer and LOTRO in some regions.

A number of the big, expensive productions in the SciFi and Fantasy space without very good IP ties will have a hard time recouping development costs and justifying further development, though. And, as such, will look less attractive compared to the few that do sport a larger userbase and to the absolute multitude of Free2Play games that tend to sport content that's limited but hard to beat on price. Many will die.

This is not to say that there isn't room for original ideas in the MMO space - but it seems that in order to sell anything new, you need either well known IP or a way of stealing large chunks of community from World of Warcraft. Thus far, few seem to have found a way of executing the latter. Design elements from some of the more novel, yet successful, games such as EVE Online and Second Life could well be key to this.

It's also quite possible that the next big mainstream title could come as a dark horse out of a Korean software house as a Free2Play game that appeals to western audiences.

White spaces, DPI and net neutrality

With the recent White Spaces news (which I think is pretty neat - more competition is needed in the states, and the US is a pretty damn big area to cover by the usual means), I'll go out on a limb and make a prediction: traffic management will be required. We're talking somewhat larger cell sizes than WiFi and the medium is decidedly shared.

I've seen nothing about a public access mandate for this space, so let's assume it's all about commercial ISP's. Any provider entering the fray here would be competing against the incumbent Telco or Cableco and would need to compete on price and service, pretty much.

It's certainly doable, but don't expect the Network Neutrality Marvel at work here. It won't be fast and every-packet-is-equal. Either of, perhaps, but not both. And if you're selling service, I'm pretty sure I know which one goes away first.


This might perhaps be even more applicable if the AWS bit goes through. Citing that article:

"The FCC now says that the ultimate winner of its AWS spectrum auction must use up to 25 percent of its capacity to provide free, two-way broadband Internet service at data rates of at least 768 kilobits per second in the downstream direction. "

Right. And this, again, is over a shared medium? Let's say that 25% of the capacity is 25Mbps per frequency and cell. With at least 768 Kbps per user, they'll have a hell of a hard time covering peak usage. I suppose it boils down to how many frequencies the radios can muster and how many users they're getting, but it's a service that'll suffer more as it gets more popular - and being free, I can see how a lot of casual Internet users would use it.

You'd probably see ~0.15 Mbps per active subscriber at peak, giving us ~160 users per frequency and cell before you actually start seeing bad congestion. And that's assuming that the users follow a pretty average usage demographic - a few YouTube addicts or filesharing users will skew that somewhat badly since we're seeing a pretty low number of total users per cell/frequency.

I admire the notion, but I think they'll be needing to do both host fairness, ideally some smart queueing and perhaps bulk services shaping in order for it to be very usable at peak. Pretty much the same deal as 3G providers and WiFi providers today.

DPI and US Telco execs

While I actually got the time to write for once in a blue moon, I might as well throw in some other news as well. It's a presentation written by Jon Linden, a veep at procera. (Disclaimer: Procera is also where I happen to work, so please apply a grain of salt)

What makes it interesting is that it's a presentation aimed at higherups at telcos and presented at USTelecom Business Executive Forum. I doubt that there's a unified angle for the (traffic management) DPI industry when it comes to pushing features and agendas, so I couldn't really call this a view of the industry.

Nonetheless, this is one take on it and it's a pretty short read. Go for the speaker notes first, the presentation doesn't make as much sense without them.

http://www.proceranetworks.com/ustelecom.html

Sorry, HOW much?

While reading the new TLD application announcement at ICANN, most of the announcement looked like one would expect it to. One thing, however...

The total fee per applicant takes into account close to $US13 million invested by ICANN since October 2007 to put the design of the implementation program in place. It includes allocated staff time, direct consulting expenses and other fixed costs. This cost will be allocated across the new gTLD applications until it is reclaimed and amounts to $US26,000.

13 million USD for that? Okay - there's more than you see beneath the surface here, for sure. 200 pages, six languages, lawyers, translators and whatnot involved, no doubt - but 13 million?

I'm in the wrong line of work, obviously.

SIG 2008

Sorry about the leave of absence from blog duty - been traveling a lot. Long days and lots of work don't go well with writing text in the evenings. Also, there's been a lack of any seriously interesting news (besides for the Comcast bandwidth cap, but that's been debated to hell and back anyhow)

One of the benefits of travel, however, is that you get to go to cool places. One of these cool places would be the Security in Government 2008 trade show in Canberra, Dropbearland. If you're into things like next-to-unbreakable boxes, very mean looking electric fences, seven sorts of barbed wire, access control systems and things like that, it's a pretty nifty place. Definite geek appeal - but not very IT.

They had two things that were both IT and nifty though, and they cover the same general idea from different angles.

nuix
nuix is a company specializing in what could best be described as data mining email on a massive scale. I got a quick, yet fairly thorough rundown of how their stuff works. Extremely cool visualization makes it pretty easy to follow the path(s) a mail has taken through an organization. Combine this with e-mail retention policies (or even just storing a copy once it passes the server) and you've got a pretty capable tool. They have a video on their site that gives you the highlights (sure, it's a bit sales-heavy, but it's on their website, so I can't blame them really..)

Note the bit where the guy says "we have organisations who actually want want to plug us into the internet network going in and out of that country" - granted, we're likely talking about a small'ish country here - but it gives you a bit of a scale to things.

All in all cool stuff. If I were a CIO or anything with more users than I knew the face of, I'd definitely opt for something like this.

endace
These guys make some pretty hardware and the appliance with the worst name ever, NinjaProbe (geek points, but seriously..) - this isn't news, they've been going at it for a while. What's news for today though, is their new analysis center software. In short: Their appliances are capable of storing a metric shitload of network data to disk. Their center software is capable of mining these metric shitloads for useful data and extract only what you're actually interested in for further processing. It's a niche, but it's one that makes a lot of sense to fill.

I didn't see a demo of this, so no idea how well it works, but all in all they too seem to have their shit together.

The punchline
The common denominator here is of course data mining. In both cases we're seeing massice data mining simplified to a level where pretty much anyone can be trained to use it effectively and it's - comparatively speaking - extremely affordable. What does this mean then?

1. Your local intelligence agency might not be able to code stuff like this themselves, but they sure got a lot of money to throw at the problem. Assume that they have access to these tools for better or worse (I rather like the thought myself, but I have some faith in law enforcement)

2. These are tools that are within the budgetary reaches of your local university as well. I don't find it at all a stretch to see that some of them would want to 'ensure the safety of their students'. Which might well mean that mailing nasty horrble things such as texts containing the word 'fuck' to other students would be seen as an offense. Or something equally silly. (I don't have as much faith in academia. Techies tend to be allright, but they're not the ones who are calling the shots when it comes to that)

3. This is just scraping the surface. I'm sure we'll see quite some more interesting uses for data mining internet services and comms soon enough.

Net neutrality and strawmen

Reading The Precursor Blog by Scott Cleland (a blog labelled 'policy, markets and change'), it has some interesting commentary on Comcast & the FCC. On a factual level, I can't agree with him. On the other hand, I can't disagree either. It should be said, first and foremost that the guy is biased - linking to an industry issued Q&A on Net Neutrality from every page on the site is a bit of a giveaway.

Starters, I think it's not very nice to say "Net neutrality is thisandthat. As such, anyone in favour thinks this - and thus, by extension, are wrong. By the way, the FCC pooped on you. Neeners, neeners.", no matter if you wrap it in long words with many syllables and three levels(!) of bullet points. It's a pretty ugly strawman and I hope you're proud of yourself, Cleland.

Why a strawman? Simple. "Net neutrality" means different things to different people or interest groups. Is this an ideal situation? No. Does it make it easier to even discuss the issue? Definitely not. Does it allow for pretty cheap rhetoric? Yes. And in my opinion you better stay off that particular bottle if you actually want to commit anything but canned line noise.

That said, the guy does have some valid points - the main one being that there's some semblance of approval for reasonable network management from the generation direction of FCC.

"The system works and that there is no need for legislation or regulation;"

In a word, no. I think it's fair to say that the threat or uncertainty of eventual legislation has affected the planning or policies of major ISP's in the US. In a way, the threat of legislation is what makes the system work.

Don't get me wrong - I see business cases for extremely restricted and cheap Internet service, I see business cases for wide-area Wireless ISP's and both cases would probably need to stray pretty far from network neutrality (by any definition of the term) in order to even work out.

The difference is that 'reasonable' would be applicable in those cases. It's reasonable to prioritise interactive traffic slightly to ensure that the shared media is feeling somewhat responsive. Killing BitTorrent in one of the messiest ways conceivable - I'm looking at you, Sandvine - is another ballgame. The FCC seem to be in agreement. As for the rest of the world, no other Sandvine customer tried anything just as messy and most other traffic management/Deep Packet Inspection kits out there offer way better facilities for managing congestion (Disclaimer: I work for Procera Networks, Sandvine is their competitor. Opinions here are my own.)


"Given that: over half of Internet traffic is P2p and ~90% of P2p traffic is illegal piracy per the US PTO; given that 40% of email is spam per the Spam Filter Review; and given that 28% of pay per clicks of the large search engines are fraudulent per Click Forensics; the majority of Internet traffic is not protected by the FCC's principles and can be legally blocked."

Great - so let's block all P2P, e-mail and web browsing. Problem solved.

Statistics are good and all, but I don't think they're very valid in this case: Let's look at Instant Messaging - that's also (predominantly) P2P. In terms of transferred bytes, I'd say that a decent chunk of the traffic is copyrighted material. Same goes for FTP.

Light throttling is one thing - it makes sense for file transfer apps in some cases (P2P file transfer protocols are usually very good at hogging a decent sized chunk of the shared media. This can be at the expense of other applications) - but blocking is another deal altogether.


"Unable to sustain radical hardline non-discrimination absolutism in the face of facts and a real governmental process, the debate has shifted to what is reasonable -- and on that point obviously reasonable people can disagree."

The debate really hasn't 'shifted' in any direction at all, I'd say. Again, net neutrality proponents have a wide array of opinions - yes, some people really do whip the "all bits are created equal" drum, but you can't categorically say that's what network neutrality is.

Or, if you so prefer, "My unequivocal findings in the line of researching and opining on this issue is that the claim that there exists either a collectively ratified or de-facto definition of the aforementioned terminology is moot. As such, a sweeping characterization of its proponents is premature at best."

If you excuse me, I'll go wash my keyboard out with soap now.