Free Press wants the FCC to ban any form of Internet discrimination even if engineers argue that some types of discrimination are good and necessary to make all application work well.  But to make their point, Free Press enlisted M. Chris Riley and Robb Topolski who have shown a shockingly poor understanding of how networks function.  Riley and Topolski have published a paper “The Hidden Harms of Application Bias” which claims that prioritized networks are harmful to overall efficiency of the network and that they are harmful to low priority applications.  The paper gets every key assertion wrong and it is ironic that these two non engineers would criticize proper engineering theory that prioritized networks are more efficient as “flatly incorrect”.

Who do we believe?

The problem with these types of arguments is that the issue of network engineering is fairly complex and it’s easy to get into a dilemma of who to believe.  Regulators at the FCC have been tasked with figuring out how all of this technology works so that they can come up with some sensible rules of the Internet.  As someone with an extensive background in designing, building, and supporting enterprise class networks, I have worked tirelessly to explain why prioritized networks are an efficient and fair way to manage the network.  I have published “A Policymaker’s Guide to Network Management“, created simple to understand analogies, and created educational animations that make very complex concepts of network engineering relatively easy to understand.  Furthermore, these positions are held by many other network engineers and protocol architects who will also refute the Riley-Topolski paper.

The central premise of the Riley and Topolski paper is flawed

The central premise of the Riley-Topolski paper is that applications shall not be treated differently by the network, and that networks should adhere to the First In First Out (FIFO) model of data transmission.  The irony of this is that Riley and Topolski would actually lock us into Internet circa 1983 yet they’re mistakenly worried that a prioritized network would lock us into 2009. The two wrote:

“Central to net neutrality is nondiscrimination – the principle that content and applications on the Internet should not be treated differently by the network operator.”

“The Internet generally operates along a First-In, First-Out (FIFO) model, in which packets are forwarded in the order in which they are received”

The first statement is a matter of opinion so it technically can’t be wrong, though network engineers would view it as incredibly misguided.  The second statement is simply false.  There is no Internet commandment that states that the Internet shall operate as a FIFO network, and prioritization is very common inside the Internet with various virtual private networks and along the edge of the Internet with IPTV deployments.

The Riley-Topolski paper on application bias is laced with so many inaccuracies that it actually deserves a line-by-line rebuttal to correct all of the flaws.  But it may be more useful just to sum up the key assertions and dismantle them.

Key assertions of “The Hidden Harms of Application Bias”

• Prioritized networks degrade low priority applications
• Prioritized networks degrade overall network performance
• Locks the Internet into typical usage patterns of 2009
• Application bias has not shown to be necessary or even substantially beneficial

Myth – Prioritized networks degrade low priority applications

To prove their first key assertion that prioritized networks degrade low priority applications, Riley and Topolski use the following arguments.

“But with congestion, prioritization forwards higher priority packets ahead of other traffic, and lower priority packets are negatively affected until there are no higher priority packets to send.  Prioritization operates by degrading and harming lower priority traffic, because (by definition) more low priority packets are delayed or dropped.”

This assertion is fundamentally wrong because the mere fact that the high priority application e.g., Voice over Internet Protocol (VoIP) is using the network means that there is less available bandwidth for a low priority application such as peer-to-peer (P2P) file transfer. Whether the network routers intelligently and fairly alternates packets between the two applications or treats the packets on a First In First Out (FIFO) basis has zero bearing on how much bandwidth each application gets.  So it is NOT the prioritization that causes lower bandwidth availability to the lower priority traffic; it is the concurrent usage of applications that causes it so it is misleading to attribute it to prioritization.

The VoIP application with or without prioritization will typically occupy no more than 3% of a typical broadband connection while the P2P application will occupy the remaining 97% of available bandwidth whether it’s deprioritized or not.  More specifically, the FIFO network might at times transmit 100 P2P packets before it transmits a single VoIP packet which is blatantly unfair and destructive to the VoIP application as shown in figure 1 below.  The FIFO method means that many of the VoIP packets have to wait so long that they expire and cause dropped audio. The network that prioritizes VoIP offers perfect jitter-free VoIP but affords the same bandwidth to the P2P application.

Figure 1 – BitTorrent P2P induces destructive jitter for applications like VoIP

Source: Analysis of BitTorrent uTP congestion avoidance

Moreover, it can easily be argued that deprioritized P2P operates even faster than it does in a neutral FIFO network.  That’s because in the real world where P2P users are also VoIP users, a FIFO network forces them to shut down P2P because they don’t want the destructive effects against VoIP.  A network that protects and prioritizes VoIP by definition results in better P2P performance because the user isn’t forced to shut P2P down.  So the reality is exactly opposite of what Riley and Topolski suggests, that prioritized networks result in better performance for all applications.

We can take another example of HTTP (used by web browsers) versus P2P file transfer where we compare a FIFO network to a prioritized network. In this example, a prioritized network that alternates packet transmission between HTTP and P2P as opposed to a FIFO network that simply operates on a first come first serve basis would actually result in lower performance for P2P file transfer, but it would be 100% justifiable. That’s because one of the key features of P2P is its ability to hog bandwidth at the expense of other applications because it uses multiple flows to handle a single data transfer.

Figure 2 – BitTorrent P2P hogs over 90% of the bandwidth at the expense of HTTP

Source: Analysis of BitTorrent uTP congestion avoidance

Figure 2 above shows the results of a tests conducted using the latest BitTorrent application called uTorrent and it showed that the BitTorrent P2P protocol consumed over 90% of the bandwidth on a broadband connection leaving HTTP with less than 10% of the bandwidth.  A prioritized network using separate transmit queues could fairly alternate between P2P packets and HTTP packets resulting in a 50/50 distribution of bandwidth between the two applications.  So the network in this case isn’t actually “picking winners and losers”, it’s merely providing the kind of fairness that a dumb FIFO network could never offer.  So to argue for a neutral FIFO network is to argue for the law of the jungle approach to network management where the most aggressive applications like P2P always stomps on less aggressive applications like HTTP or VoIP.

Myth – Prioritized networks degrade overall network performance

To prove their second key assertion that prioritized networks reduce overall network performance, Riley and Topolski make the following assertion.

“Application bias is sometimes portrayed as efficient, which is misleading at best if not flatly incorrect, as prioritization decreases overall network performance. Multiple distinct elements of application bias diminish performance. First, deep packet inspection (DPI) would typically perform the work of application identification needed to enable application bias. DPI may add significant delay to the total routing time for a packet”

The problem here is that their fundamental assumption that prioritization requires DPI is wrong, and anyone who has ever built a network with priority queuing in the real world would laugh at this assertion.  The fact is that no one uses DPI to make priority classifications because it simply isn’t necessary.  The easiest and most common method used by network engineers is to simply identify the commonly used port headers.  Even applications like Skype that uses dynamic (frequently changing) ports has a fairly predictable range of ports and telltale behaviors that help indicate that it is a Skype VoIP application, but the vast majority of applications use fixed and readily identifiable port numbers.

The port numbers can theoretically be forged by the application or the user but port spoofing rarely occurs in practice because there isn’t anything to be gained by it.  Network routers have simple enforcement mechanisms that limit the amount of priority packets any end point can transmit so there’s little incentive to spoof port numbers.

Even if the ports can’t be identified and even if the traffic is encrypted, just the traffic pattern and consumption level would reveal the application type or at least tell us that it is an application of some kind.  So long as a network is only trying to balance the amount of bandwidth each application gets and alternate packets between applications for transmission, it doesn’t really need to know what the exact protocol is or what the content is.  This is not to say that this method of network prioritization is perfect, but it doesn’t need to be so long as it’s only trying to distribute network resources equitably.  Moreover, explicit prioritization methods where the application or end-user labels their own packets with the desired priority level can give more precise levels of control.

Lastly, Riley and Topolski repeats many of the myths regarding Deep Packet Inspection (DPI) technology and I clear up many of these misconceptions in my paper “Understanding Deep Packet Inspection technology“.  Not only does Riley and Topolski mistakenly believe that DPI is necessary to classify priority levels, but their assertion that DPI increases packet delay is fundamentally wrong.  No respectable DPI device on the market increases packet delay because they always work on the side of the network monitoring a copy of the traffic so the original traffic goes unimpeded, and network operators hate deploying anything that might decrease performance or increased unreliability in their networks.

Note: It is interesting is that Mr. Topolski made similar claims of increase packet delay in a paper on copyright finger printing technology and it turned out that he never even bothered to contact the hardware vendor to learn how the hardware they are criticizing actually works.  Moreover, just some rudimentary research on the web would have revealed that DPI and finger printing technology operate in out-of-band environments that don’t induce delay and they have all operated this way for many years.

Riley and Topolski go on to claim that variable latency has negative consequences for buffered video streaming.

“most applications such as video streaming can function quite well by varying quality and buffer size, but inconsistent latency can throw off these techniques and produce a worse user experience than a slower but consistent connection would have otherwise supported.”

Wrong again.  Even the smallest video buffer of 1 second is equivalent to 1000 milliseconds which is well outside of the norm of even the worst variations on latency (called jitter).  A typical video buffer of 10 seconds would offer 10 times overkill in jitter protection.

Furthermore, Riley and Topolski are making the incorrect assumption that eliminating jitter for high priority applications is tantamount to creating jitter for low priority applications and that it is somehow a zero sum game.  Nothing could be further from the truth because the most delay you could ever add to a low priority application is the time it takes to transmit a single high priority application packet which is tiny to begin with (typically 8 times smaller than a P2P packet).  To be more precise, favoring the VoIP packet usually adds zero delay to anything else because the VoIP application isn’t transmitting anything 97% of the time.  During the 3% of the time it does transmit packets, it adds at most 4 milliseconds jitter to a typical broadband connection which is the time it takes to transmit a single packet.  On the other hand, not favoring the VoIP packet in a FIFO network means that we’re forcing VoIP to suffer 20 to 1000 milliseconds of jitter caused by the P2P application.

Myth – Locks the Internet into typical usage patterns of 2009

The final claim that Riley and Topolski makes is that a prioritized network will lock users into typical usage patterns of 2009 because current classifications of low priority may not be accurate in the future and may not even be accurate today.  They argue;

“For example, many peer-to-peer (P2P) file transfers may indeed be low priority and tolerant of significant delay – but P2P applications are also used for high priority tasks, including VoIP and video streaming applications, along with esoteric but important uses such as transmitting satellite imagery from NASA to scientific researchers.”

There are several problems with this statement.  For one thing, it makes the common mistake of conflating different usages of the term peer-to-peer (P2P).  P2P used in the context of VoIP communications has nothing to do with P2P file transfer nor are they similar in function.  Skype’s policy people like to use the talking point that their VoIP protocol is a “P2P” protocol and therefore they might be the victim of priority misclassification, but Skype’s VoIP protocol operates in a nearly identical way to every other VoIP protocol and it is nothing like P2P file transfer applications.  Case in point, ISPs like Cox Communications have explicitly stated that Skype is treated as a high priority VoIP application as it should be.

The other mistake is the claim that P2P file transfer is used for video streaming, but that is false.  P2P file transfer is normally used in video downloads and never in video streaming because P2p content arrives out of order so it can’t be played back in order until the entire file is downloaded.  P2P file transfer can be used in a hybrid configuration where traditional streaming technology is used to supply the in-order content and P2P is only used to buffer ahead and offload some of the work from the traditional video streaming servers.  This buffering ahead does not require any kind of priority and a fair bandwidth sharing scheme is perfectly legitimate.

Image file transmission shouldn’t be treated any differently by the network as any other file.  If the user of the network wishes to make an image transfer faster, then can simply shut down every other application for the time being.  But if they’re trying the transmit multiple files, and no explicit packet priority labels were used by the application, the network should simply try to give every application what it wants until it reaches a state of equal bandwidth sharing between multiple applications.

As for video streaming in general, it’s always buffered going over the Internet which means it can tolerate jitter and I explained this in detail in the previous section.  But even if it wasn’t buffered, a prioritized network that alternates packet transmission between applications will not cause the video application any harm.  The only time we run into trouble is when the user is demanding more bandwidth than the network can offer, but that’s when the capacity problem needs to be tackled because prioritization technology only makes better use of available capacity but it doesn’t increase total capacity.  The point is that while prioritization can’t solve all of our problems and can’t be a substitute for capacity, it makes the network run much better without creating new problems.

Riley and Topolski go on to claim that prioritization can’t work with Virtual Private Networks (VPN), but they are ignorant of decade long advancements in VPN technology.

“A similar problem will likely impact users of virtual private networks (VPNs), as communications through a VPN may be either “high” or “low” priority, and a network operator is poorly positioned to tell the difference; as a result, VPN traffic will likely be classified as entirely high or low priority, regardless of its actual use, based on whether it is more typically high or low priority.”

These problems have been solved at least a decade ago in the IPSEC VPN standards by allowing the Differentiated Services (DS) prioritization fields into the IPSEC headers.  So VPN encrypted packets will still get the correct priority classification that the application intended.  VPN encrypted bulk file transfer packets will be handled with low priority, and VPN encrypted VoIP packets will be handled with high priority.  This is old news to any network engineer working in the field, but Riley and Topolski never heard of it.

Riley and Topolski then claimed that ISPs might have prioritized some older video applications like RealVideo but neglect newer applications like YouTube and hurt YouTube.  Not only does YouTube not care about prioritization because it is relatively low bandwidth (initially 320 Kbps) and buffered, this is a false choice to begin with.  Riley and Topolski is assuming that an ISP even needs to choose between one video application over another when the reality is that the ISP doesn’t need to decide which application deserves more bandwidth than another.  The only fair assumption is to assume that all applications deserve as much as they need up to an equal share of the bandwidth and an equal chance of getting its packets forwarded by network routers in a timely manner.

Myth – Application bias has not shown to be necessary or even substantially beneficial

Riley and Topolski go on to claim that prioritization isn’t necessary or substantially beneficial and go on to cite YouTube as an example of something that can work without prioritization, but that’s sort of like saying it never gets too hot in the arctic.  It’s true, but it doesn’t give us any enlightening information because YouTube or any kind of buffered video streaming especially when they are low bandwidth never needed prioritization to begin with.  The claim that Skype functions well without prioritization is simply false.  Skype call quality degrades just like any other VoIP application if a P2P file transfer is run over the same broadband connection at the same time.  Skype or VoIP applications can buffer for a fifth of a second to smooth over sub-200 millisecond jitter, but it adds an annoying amount of delay to the voice conversation and it can’t do anything about jitter that often exceeds 200 milliseconds.

Furthermore, Riley and Topolski never even mention good examples of applications that benefit greatly from prioritization today such as online gaming, IPTV (unbuffered video streaming), VoIP, or video conferencing.  There are millions of IPTV deployments over Fiber to the Node (FTTN) broadband in the U.S. alone and tens of millions more globally that rely on prioritization technology to bring reliable television service to consumers today.

Conclusions of the Riley-Topolski paper

Riley and Topolski concludes that “User-based congestion management is eminently more reasonable than application bias”.  Riley and Topolski make the argument that it is acceptable for ISPs to protect their subscribers from the heaviest usage subscribers that are hogging the network, yet they argue that it is not acceptable for ISPs to try and protect some applications from other applications that hog the network.  After debunking every key assertion made in the Riley-Topolski paper, it is clear that their rationale for this conclusion is based on a shoddy understanding of network engineering.

Conclusion

The Riley-Topolski paper on “The Hidden Harms of Application Bias” is riddled with serious factual errors from beginning to end, and they make the fundamental mistake of assuming that the Internet must be some sort of dumb First-In-First-Out (FIFO) network.  They wrongly assert that prioritization will harm many applications of today and make wild guesses about the problems with prioritization in the future.  The reality is that the real application bias actually lies in the FIFO network that they have become obsessed with because the FIFO network allows some applications to thrive at the expense of others.  The prioritized network actually eliminates application bias by ensuring that all applications get up to an equal share of the bandwidth and an equal chance to get their packets in a timely manner.

Since the FCC is likely to make new rules that will regulate the Internet, it is crucial that the rules are based on the facts.  The challenge is to separate fact from fiction and this article makes every attempt to do that.  It is our hope at Digital Society that the FCC examines all of the evidence and ask the tough questions and challenge every assertion to see if they can withstand scrutiny.  Only then can we settle on the right set of facts and get things right.

Update 11/18/2009 – Scientific data confirms that Riley and Topolski are wrong.