Data shows CDN prioritization more harmful than router prioritization
One of the things I’ve discussed before is the fundamental bias in the Internet against long distance Internet file transfer and the fundamental speed limit affecting web servers that are far from their customers. A web server that is 100 milliseconds (ms) away (round trip time) will face a speed limit of 5.1 Mbps per file transfer while a server 25 milliseconds away will have a speed limit of 20.4 Mbps. But one additional point that I didn’t address in my first article was the issue of bandwidth contention between a nearby and distant web server.
I had ignored the bandwidth contention issue previously because I had incorrectly assumed (which goes to show the importance of experimentation) that the contention advantage would be trivial. I figured that as long as the two servers were capped by the user’s bandwidth limit (which is 2.84 Mbps actual measured data throughput in my case since I live about 13,000 feet away from the nearest DSLAM), then the TCP speed limit shouldn’t be much of a factor and the two servers would have a roughly equal shot at sharing bandwidth. I decided to test that assumption with actual experimentation shown in Figure 1 and proved that the contention advantage of the nearby server is actually quite notable.
Figure 1 – Near versus distant server bandwidth contention
Figure 1 shows the bandwidth contention between two servers. One server is www.digitalsociety.org and the other server is v23.lscache5.l.google.com which is a very nearby caching server Google uses to serve YouTube content. Red represents the bandwidth delivered via the DigitalSociety server and blue represents the bandwidth for the YouTube server. The DigitalSoceity server had an average round trip of 105 ms away and the YouTube caching server averaged 47 ms away which is a mere 3 ms from the first AT&T router I hit. At 3 ms, Google is probably parked inside the same Internet Exchange Point (IXP) that AT&T resides in which gives them super low latency to AT&T customers (and probably all the other major ISPs).
Since both the YouTube and DigitalSociety server can run faster than my broadband connection and I verified that each server operating alone can max out my broadband connection, I can try to download a file from both servers concurrently and compare how each server performs. To my astonishment, the YouTube server that is 47 ms away averages 2.3 times faster than our DigitalSociety server which is 105 ms away! It probably isn’t a coincidence that 105 ms turns out to be 2.23 times slower than 47 ms so it would seem that the Internet’s congestion control mechanism (Jacobson’s algorithm) disfavors content to a degree that is proportional to the latency. Had the latency difference been a 5-fold difference (which is often the case since I’m normally 20 ms away from Google), a reasonable hypothesis (which I will test and update when I get a chance) would be that Google’s server would run roughly 5 times faster than the distant server.
This means is that CDNs not only offer higher user capacity due to its distributed caching architecture, but it effectively affords its own traffic a higher link priority which is functionally no different than a higher priority class in DiffServ router prioritization.
How is this significant to the debate on Net Neutrality?
Free Press and other strict Net Neutrality advocates believe that differentiation facilitated by Content Deliver Network (CDN) services are harmless and “not a zero sum game” and they also believe the fantasy that all websites run at the same speed. Free Press wrote:
Free Press has never, and does not currently oppose these forms of “geographic” prioritization, because geographic prioritization unlike routing-based prioritization, is not a zero sum game. Because it is not a zero-sum game, ISPs can sell as much caching as they like without causing degradation of other traffic on the best efforts Internet.
Oh really? This nonscientific opinion of Free Press is clearly inconsistent with the scientific data presented in figure 1 because it’s clear that CDN services offer a substantial competitive advantage to its customers at the expense of its non-CDN competitors. In other words, those who can’t afford to purchase CDN service get the meager leftover bandwidth. Now everyone has the opportunity to buy CDN service, but we all know that not everyone can afford it. But that is how the Internet has always operated and always will operate and it is an even playing field with uneven players.
Furthermore, fast servers like the edge cached servers from Google YouTube have the ability to generate a lot of jitter that is extremely harmful to online gaming and Voice over IP (VoIP) applications. Figure 2 quantifies the harm in jitter induced by Google’s server “v23.lscache5.l.google.com” (more analysis on this here). So not only do nearby servers hog more traffic, they can also generate more jitter because of the higher bandwidth.
Figure 2 – Jitter induced by video stream from nearby YouTube caching server
On the other hand, Free Press and other strict Net Neutrality advocates want to see an abolition of router based prioritization and service differentiation because they claim it harms other applications. They even claim that router prioritization violates the architecture of the Internet even though it is defined in the Internet Standard RFC 2475 so they essentially want the government to outlaw existing Internet standards and business models. The overview of RFC 2475 says:
This document defines an architecture for implementing scalable service differentiation in the Internet. A “Service” defines some significant characteristics of packet transmission in one direction across a set of one or more paths within a network. These characteristics may be specified in quantitative or statistical terms of throughput, delay, jitter, and/or loss, or may otherwise be specified in terms of some relative priority of access to network resources. Service differentiation is desired to accommodate heterogeneous application requirements and user expectations, and to permit differentiated pricing of Internet service.
The reality is that router based prioritization is harmless in most implementations when we actually measure the effect. In fact, jitter sensitive applications like online gaming or VoIP cause almost no jitter for other applications as shown in figure 2. Furthermore, no amount of prioritization in favor of VoIP or gaming can cause any measurable bandwidth deprivation on other applications because VoIP and gaming applications use less than 0.1 Mbps.
Figure 3 – VoIP cause almost no jitter for other applications
Even when we have a high bandwidth application like multiple IPTV streams being prioritized, it doesn’t generate any jitter and this is shown in the data from figure 4. When a router prioritizes an IPTV stream even if its 8-16 Mbps of bandwidth, it does so by releasing packets in an even non-bursty manner that almost entirely avoids inducing jitter. When a CDN delivers content, it slams as many packet into the network as it can get away with to maximize its own performance and everyone else is left with the consequence of high jitter. This is why there is a staggering difference between the jitter shown in figure 2 and figure 4.
Figure 4 – High bandwidth prioritized IPTV causes very little jitter
So based on the scientific data, we can conclude that Free Press and other strict Net Neutrality advocates have their facts backwards. The router prioritization that they claim is harmful to others is actually not harmful and the CDN “geographic prioritization” that they claim is harmless is actually the most harmful.
I am not arguing that CDN based services should be outlawed. What I am saying is that service differentiation has always been the norm on the Internet and that the Net Neutrality arguments for outlawing router based prioritization and differentiation is devoid of the facts. If CDN services are normal and permitted despite being extremely harmful to other applications, then it makes no sense to outlaw router prioritization which is far less harmful and defined as an Internet standard. Lastly, it is the router prioritization technology that will cure and save us from the jitter induced by CDN-based services like YouTube by minimizing the harmful effects of jitter and ensuring true neutrality in the network where all applications run well. To suggest that router prioritization be banned while praising CDN is like condoning the ailment while outlawing the cure.
I don’t know if it’s fair to say the TCP latency issues don’t matter. In the first half of your first chart, the orange peaks are visibly lower than the blue ones. You might try re-running the test using many simultaneous connections instead of just one, first for each server separately, then both together, and see what results you get.
As I said in the article, each server can max out my connection when standalone. The maximum performance of a single TCP stream is roughly 2.24 Mbps on my 3 Mbps sync-rate DSL connection though my maximum aggregate for two streams is 2.84 Mbps. Based on how Jacobson’s algorithm works, this is quite normal behavior.
Since each individual server can max out my single-stream speed, the main question that needs to be answered is how the two servers contend for bandwidth and that’s what this article answers. It’s clear that nearby servers have a substantial contention advantage in addition to having a proportionally higher absolute speed limit.
Thanks. A very useful analysis. Now the challenge is to translate this into language that can be understood in policy circles in Washington! Unfortunately the debates surrounding “net neutrality” have centered on “bumper sticker” phrases and statements of “facts” that, as you have done here, often prove not to be entirely accurate. That, alone, should mitigate against swift regulatory or legislative action. The challenge is whether it is “catchy” enough for the media to be told that the real answer to a lot of the questions now being posed is that even on a technical basis, we’re still just learning!
I appreciate your series of articles. However, most of the articles have to do with refuting one argument or another. Do you have a policy paper?
Would you propose router prioritization regulation to enforce play-nice? To enforce balanced load despite geographic differences?
Who profits from the death of the current model? Do ISPs and content providers start charging each other for customers and content?
Thank you again.
@Joshua Stough
Joshua, I appreciate your questions very much so I’ll take a few minutes to answer them.
1. We do have papers and we do file them with the FCC. This one will be filed soon enough. I’ve written some long papers like http://archive.itif.org/index.php?id=205 but not everyone wants to read a paperback book on the issue. The blog format offers smaller bite size chunks of information in a real-time format.
2. In the network management paper I liked above, I’ve proposed good network management practices that are fair and reasonable to everyone. Balancing geographic and multi-flow differences would be something extra an ISP has the option of doing, but that would clearly make CDN users nervous if you neutralized their geographic advantage in TCP. Neutralizing the multi-flow advantage of TCP (http://www.zdnet.com/blog/ou/fixing-the-unfairness-of-tcp-congestion-control/1078) makes the P2P and BitTorrent proponents nervous since that eliminates one of their cheats.
3. I’m not talking about killing the current Internet model. That’s the big myth of Net Neutrality which claims that there is no price differentiation on the current Internet. Paid peering with elevated packet priority is an existing business model and existing practice. Net Neutrality proposals before the FCC and congress seeks to eliminate current business practices by outlawing ISP-enabled differentiation for content providers. It doesn’t eliminate CDN enabled differentiation or server-enabled differentiation or developer-enabled differentiation.
4. “Do ISPs and content providers start charging each other for customers and content?”
Content has been able to “shake down” ISPs but not the other way around. ESPN is the most obvious example where ESPN charges ISPs a per-subscriber fee to make ESPN360 available to the ISP’s entire subnet. ESPN has the market power and there isn’t much political or media opposition to this. ISPs lack the market power and the political cover to pull something like this. If an ISP tried to shake down a website for money or else they get blocked, it would be suicide from a Public Relations and political viewpoint.
ISPs or CDNs currently charge content providers for peering, or content providers like Google can pay for transit which is usually more expensive than paid peering. Google can often negotiate free peering because they have the clout but Google’s competitors don’t. Google would benefit greatly if paid peering was outlawed because it would prevent smaller sites from buying cheaper/faster server bandwidth while giving Google more negotiating clout to demand free peering. http://www.digitalsociety.org/2009/11/fcc-nprm-ban-on-paid-peering-harms-new-innovators/. Smaller content providers won’t have the clout to negotiate free peering so they’ll be forced to pay more for slower transit access.
ISP’s contractually can’t block sites today because it would violate their peering agreements with other transit providers on the Internet. What ISPs can do is offer a better bang/buck and offer an alternative to transit access. Net Neutrality wants to outlaw these voluntary agreements between ISPs and content sites and that is a big mistake that will benefit the largest websites like Google. Net Neutrality proponents argue that smaller sites and startups can’t afford these services but that is silly because they are forcing them to pay for more expensive and slower transit service. Furthermore, bandwidth costs are not the real barriers to startups http://www.digitalsociety.org/2010/05/the-real-barriers-to-innovation-isnt-the-cost-of-bandwidth/.
[...] [...]
[...] the Internet’s TCP congestion control protocol isn’t always fair when it comes to distance unfairness and multi-flow [...]
[...] [...]
[...] [...]
[...] [...]
[...] It’s no different from Google or other large content providers building or leasing expensive CDN infrastructure to get a huge advantage over everyone else’s content and [...]
[...] [...]
[...] [...]
[...] [...]
[...] [...]
Leave your response!
Twitter Feed
About Us
Digital Society is a digital think tank that believes culture and commerce are inseparable, that the digital economy flourishes when people are free and rights are secure, and that free markets free people.
Digital Society is an independent 501(c)3 non-profit organization, funded by donations from Jon Henke and from Arts+Labs. We advocate for a pro-culture, pro-commerce digital society through research, analysis and debate on emerging technology issues.
Reply Comments
Transparency and interactivity are trademarks of the Internet era, and we aim to foster them here at Digital Society. It is inevitable that some people will disagree with the technology policy positions we take. We want to have that constructive debate.
The Reply Comments feature gives our critics a chance to respond to our viewpoints and the Digital Society audience convenient access to competing arguments. Any time we directly challenge the views of an individual or a group on this site, the party in question may substantively respond in a guest post.
Please contact executive director Jon Henke by e-mail.
Subscribe
Daily Digest Email
Recent Posts
Recent Posts
Most Commented
Most Viewed