Home » Internet

FCC NPRM ban on Paid Peering harms new innovators

By 10 November 2009 36 Comments

The current FCC NPRM would prohibit paid peering agreements and harm small content providers while ensuring Google’s dominance on content distribution.  In paragraph 106 of the NPRM quoted below, broadband Internet access providers will no longer be permitted to offer application and content providers cheaper direct access.  This means that smaller application, content, or services providers will no longer be able to purchase cheaper direct access to broadband consumers and they will be forced to purchase more expensive indirect transit bandwidth which is subject to much more congestion.

Meanwhile, Google has already built a substantial private Internet bypass network along with the necessary server farms which has allowed them to negotiate many free peering agreements with ISPs.  By consigning the smaller competitors to more expensive and inferior transit bandwidth through new regulations or legislation, Google would ensure that their dominance on the Internet will not be challenged.

Banning any kind of enhanced or prioritized delivery

FCC NPRM Paragraph 106 - We understand the term “nondiscriminatory” to mean that a broadband Internet access service provider may not charge a content, application, or service provider for enhanced or prioritized access to the subscribers of the broadband Internet access service provider, as illustrated in the diagram below.  We propose that this rule would not prevent a broadband Internet access service provider from charging subscribers different prices for different services.  We seek comment on each of these proposals.  We also seek comment on whether the specific language of this draft rule best serves the public interest.

FCC NPRM diagram 2 – Regulations on broadband access Internet Service Providers

While it isn’t entirely clear what services the NPRM prohibits because of the vagueness of the language, it is clear that “enhanced or prioritized” content delivery is now prohibited.  The term “enhanced” and “prioritized” are broad and they can and will be used in a torrent of formal FCC complaints whenever someone somewhere doesn’t like something.  What immediately comes to mind is that NPRM paragraph 106 will prohibit Paid Peering which is a service where an ISP offers cheaper and higher quality direct access to its broadband consumers, and this will have some very undesirable consequences for the Internet.

Transit versus Paid Peering versus Free Peering

Figure 1 – How Transit, Paid Peering, and Free Peering works
free versus paid peering

Internet Peering is a very secretive subject and it is comprised of complex commercial agreements that are all under Non Disclosure Agreements (NDA).  Even FCC inquiries on peering in the past have been rejected.  The following information is based on William Norton’s Dr. Peering website which has some of the most extensive insider knowledge of peering agreements on the Internet, and information from the Arbor Networks study of Internet traffic.  Norton also wrote this blog explaining why Google wants to end Paid Peering.

Transit: The traditional form of network access content providers use to offer web based content to consumers over the Internet.  This is the simplest method of buying bandwidth where a company wanting a presence on the Internet leases space in a datacenter and pays monthly fees for bandwidth by the megabit per second (Mbps) per month.  You only need to drop your bits off at a single location, and the transit service delivers your bits to any part of the Internet.  The cost of this flexibility is that it is the most expensive form of bandwidth and it is the slowest and least reliable because packets have to traverse multiple networks which are shared with every other Internet user and is often congested.  Uncapped (no limit on usage) transit bandwidth in 2009 typically cost between $2 and $9 per Mbps per month depending on quantity and commitment level.  Some unusually large content providers like Google can get their transit costs down to $0.50 per Mbps per month.

Paid Peering: A newer form of network access that offers content and application providers a cheaper and more direct way of delivering bits to consumers over the Internet.  This is a more complex method of buying bandwidth because it requires additional infrastructure to get close enough to various ISPs.  This infrastructure can come in the form of a private network connection to the Internet Exchange Points (IXPs) where many ISPs and businesses congregate, or it can simply be caching servers located at the IXPs where data can be sent over expensive transit connections just once but it gets cached and retransmitted thousands of times more at cheaper Paid Peering rates.  Paid Peering only grants access to the ISP that is being peered with but all data sent or received in this manner is diverted away from costly and congested transit connections.  Not only is Paid Peering the least expensive form of bandwidth (next to free), it also offers the least congested and least contended content delivery to consumers and it offers the lowest latency access.  Prices in 2009 for  uncapped Paid Peering cost between $1 and $3 per Mbps per month depending on quantity and commitment level, and that makes it roughly 3 times cheaper than transit bandwidth while offering less congested throughput and lower latency.

Free Peering: Whenever two entities on the Internet feel that they can offer each other equal value, they can enter into a Free Peering agreement.  Free Peering is the same as Paid Peering in terms of how the networks exchange traffic and it offers similar performance benefits, but the difference is that no money is exchanged between the two peers.  This has traditionally been limited between Internet Network Providers of equal size but according to the Arbor Networks study of Internet traffic, Google is now valuable enough that they can negotiate free peering agreements directly with many ISPs.  Where Google can’t negotiate Free Peering deals, they can still get extremely low transit rates because of their quantity and commitment level.

Net Neutrality and Google’s efforts to ban Paid Peering

Ever since former SBC/AT&T CEO Ed Whitacre’s infamous interview in 2005 where he appeared to give an ultimatum that application and content providers weren’t going to use his network for free, Net Neutrality proponents have seized the opportunity to make any kind of enhanced or prioritized content delivery illegal.  Initial support for Net Neutrality was broad because Mr. Whitacre’s comments were widely perceived to be extortion where an ISP would either block or degrade a website unless the website owner paid the ISP even though Whitacre made no explicit demands.  Years later, the fears have largely subsided since no ISP has attempted this type of extortion.

Furthermore, it turns out that ISPs didn’t have this kind of market power and in fact it was the opposite that was true.  In 2006 when I was Technical Director at ZDNet, I first reported that ESPN may have been engaging in “reverse Net Neutrality” where ESPN blocks ISPs from accessing the ESPN360 website until ISPs paid ESPN a per-subscriber fee.  That means every customer of that ISP whether they use ESPN360 or not have to subsidize the cost of ESPN360.  Those reports were later confirmed to me by ESPN employees and later confirmed by the ISPs that paid per-subscriber fees to ESPN for access to the ESPN360 website.

Once it became clear that Ed Whitacre’s threats were more paper tiger than legitimate threat and that content was king, major Net Neutrality backers like Microsoft withdrew their support for Net Neutrality.  More importantly, the ramifications of Net Neutrality legislation and now regulation in the form of the FCC NPRM means that it would be illegal for application and content providers to purchase cheaper and more direct access to consumers from the ISPs.  So in the name of protecting application and content providers from services that were mistakenly viewed as extortionist, Net Neutrality regulations would actually cripple application and content providers by forcing them to buy more expensive and more congested transit bandwidth.

By coincidence or by design, Google as a major backer of Net Neutrality regulations like the FCC NPRM or legislation like H.R. 3458 will ensure an environment that will be hostile to Google’s competitors in the content or application space.  Google has outgrown Paid Peering because they are one of the few application/content providers that has enough leverage to get Free Peering, but their smaller competitors have to rely on Paid Peering products.  Taking the Paid Peering option off the table would mean much higher cost and much lower performance for Google’s competitors.  Another benefit of banning Paid Peering is that it puts Google in a better position to negotiate Free Peering deals with the few ISPs that are trying to get a Paid Peering arrangement with Google.

Conclusion

While many Net Neutrality proponents may be well intentioned and they want to spur the next new innovators, it’s becoming clear that the unintended consequences of Net Neutrality regulation would produce the opposite effect.  It appears that the NPRM regulations operate under the misguided vision of how the Internet works as illustrated in diagram 2 of the FCC NPRM, and it may be assuming that these ISP charges to content providers are levied on top of existing transit costs which would appear to be a form of extortion and double charging.  The reality as illustrated in figure 1 is that these paid peering charges are levied in place of the transit costs providing better service at a lower cost.  If we truly want to foster innovation on the Internet, the last thing we want to do is kill Paid Peering.

Richard Bennett also wrote a good explanation here at GigaOm on the changing landscape of the Internet and why it is a bad idea for the FCC to prohibit paid peering in the name of anti-discrimination.

36 Comments »

  • Robb Topolski said:

    George, is it possible that the NPRM actually allows paid peering but bans additional prioritization over that link? The traffic would get the benefit of being more local (better speed naturally rather than super-naturally)?

  • George Ou (author) said:

    Robb,

    That’s part of the problem with the NPRM and it proves what David Farber has said all along. These things are much too complicated to be micromanaged at the regulatory level.

    The NPRM despite its length is still extremely vague and it will be a field day for lawyers to file formal FCC complaints. It’s not exactly clear what the NPRM bans and what it doesn’t ban, what we do know is that bans ISPs from charging content providers for ENHANCED OR PRIORITIZED delivery. The word enhanced clearly goes beyond just network prioritization and it covers services like caching. Why ban these services and reduce competition in the CDN space and reduce choices for content providers?

    But even if we assume your interpretation that this only bans prioritization to be accurate (and it isn’t an accurate assumption), why ban additional prioritization over that link? If the end-user only paid for best-effort and corresponding infrastructure, why would we prohibit the content provider from paying for additional delivery characteristics with improved infrastructure? It would only be double charging if the ISP charged the content provider for best effort access to the customer when the customer already paid for best effort access to any website on the Internet. Charging one side for best effort and charging the other side to boost delivery to prioritized delivery is not double charging.

    Also, we need to think beyond just prioritization because it could be coupled with other benefits as well. If the end-user only paid for 1.5 Mbps but they want to access 6 Mbps content, why can’t the content provider pay for the additional 4.5 Mbps and pass on the discounted shipping onto the consumer? A content provider could offer a 6 Mbps HD movie for $5 to the consumer and a small part of that movie could pay for the additional bandwidth. This is precisely how devices like the Amazon Kindle works where the content provider pays for end-user shipping where Amazon can negotiate cheaper bulk shipping rates.

    If the purpose it to increase innovation, why kill these perfectly legitimate business models?

  • The real purpose behind Net Neutrality is anything but honorable | Technology for Mortals said:

    [...] FCC NPRM ban on Paid Peering harms new innovators [...]

  • Robb Topolski said:

    You as a network guy knows the answer to your first question — you don’t prioritize the content on a short link because you don’t need to. There is much less intervening traffic to slow it down.

    As for the content end paying for prioritization, you’re forgetting that any prioritization harms the non-prioritized traffic. A subscriber has already bought into the network share with a certain expected contention, so now after-the-fact some other buyer is going pay for prioritization, essentially degrading my traffic further? That’s a raw deal!

    Finally, with all this paying for all this prioritization means that we have all this congestion and no particular incentive to build out of it and every incentive to keep it congested (prioritization does nothing on a non-congested network). That’s pretty twisted!

    Robb

  • Digital Society » Blog Archive » The difference between transit and paid peering said:

    [...] reading about the FCC’s NPRM banning Paid Peering, someone asked me a great question yesterday that wanted to [...]

  • George Ou (author) said:

    Several problems with your comment Robb.

    1. You don’t need to prioritize “short link” edge cached content that is BUFFERED so long as the user doesn’t over use the broadband connection while attempting to watch video. You DO need to prioritize real-time content.

    2. Your claim that prioritization harms non-prioritized traffic is dead wrong, and you’re showing your ignorance. I have completely debunked your paper on this issue from start to finish. http://www.digitalsociety.org/2009/11/debunking-the-myth-that-prioritized-networks-are-harmful/. You and Mr. Riley have demonstrated that the two of you don’t even understand the simplest of networking concepts. The two of you have demonstrated that you have zero qualifications to be making expert assertions in this field.

    Your assertion that prioritization kills incentives to build out capacity is ignorant of the facts. Prioritization exists on all next generation broadband networks in one form or another. Verizon FiOS keeps their video content on completely separate wavelengths in their fiber. Every FTTN provider in the world has deployed prioritization for Voice and IPTV services, yet they’re building out MORE capacity.

  • FCC Unintended Consequences Could Lobotomize the Internet « NetCompetition said:

    [...] 11, 2009 George Ou has a great new post — “FCC NPRM ban on paid peering harms new innovators” — that should be [...]

  • Chris Riley said:

    First of all, George, it’s Dr. Riley, not Mr. Riley. I do have a Ph.D. And in Computer Science, in case you didn’t notice.

    Second, you did no such thing as “debunk” our assertion that prioritization harms non-prioritized traffic. Your claim that prioritization won’t limit the bandwidth of the deprioritized traffic was offered with no evidence whatsoever. You also seemed not to recognize that prioritization occurs in congestion, and in fact only provides any benefit in congestion, and provides that benefit by (as you yourself note) letting the prioritized traffic get more packets through than the deprioritized traffic. The idea that prioritizing gives a benefit without having any corresponding detriment is ludicrous. If you want to argue that the slowed-down packets don’t matter because the traffic is truly not latency sensitive – that’s one thing. But to say the packets aren’t delayed or dropped more often is ignorant.

    Third, putting voice and video on separate wavelengths is not the same thing as prioritization. So it’s not remotely accurate to say that “every FTTN” deployment prioritizes voice and video.

    Fourth, you’re not a lawyer, so let me explain the NPRM to you. When the FCC draws a little diagram that points to the end-user connection, and clearly indicates that paid prioritization over the end-user connection is what they want to prohibit, then *paid peering* (which is not the end-user connection) is not affected. Period. End of story.

  • George Ou (author) said:

    Dr. Riley,

    1. Apologies for not referring to you as Dr., and I’ll do so from this point forth. However, I should point out that Network Engineering is a very specific field within computer science and that most computer scientists aren’t experts in this specific field. The fact that you have gotten so many facts wrong indicates to me a lack of Network Engineering expertise on your part.

    2. You said “You also seemed not to recognize that prioritization occurs in congestion, and in fact only provides any benefit in congestion”

    No Dr. Riley, this is another fallacy that you have bought into and are perpetuating. Prioritzation is often needed in low congestion states where average network utilization is low. Congestion occurs any time a bursty application (HTTP or P2P) tries to maximize packet throughput until 100% link utilization, and that’s just how those applications are designed. So when you see one of those network utilization charts that might only show 10% network utilization, what you are looking bandwidth graphic where time is measured in seconds or often 5-second intervals. This unfortunately shows the average utilization during that 1-second or 5-second interval as being 10%, but it doesn’t show that there are in fact 100% utilization states that occur at the millisecond level often lasting 0.2 seconds to 1 second long. That may not sound bad on the surface until you consider how long a .2 or 1 second disruption to a VoIP or gaming application is. So this idea that you keep pushing that we can simply eliminate congestion by adding capacity is a fallacy, because applications will always burst to 100% capacity (100% defined as goodput) no matter how much capacity you throw at the problem.

    What I claimed was that prioritization (when using DiffServ-based priority queuing techniques inside the network and not some cheap throttling technique that is deployed in end-user software) does not limit the bandwidth of non-prioritized ANY MORE than a dumb FIFO network would. Just the mere use of another application on the same network is obviously going to displace and limit the performance of another application. The point is that VoIP or online gaming traffic won’t take any more bandwidth in a VoIP or gaming prioritized network than a dumb FIFO network. That is a fact because VoIP and gaming have very low and fixed bandwidth requirements. That debunks your assertion that prioritization harms non-prioritized applications.

    I also mentioned that in the case of HTTP versus P2P, the network that prioritizes HTTP would in fact reduce P2P performance more than a FIFO network, but that it was perfectly justifiable. That’s because the FIFO network would allocate 90% capacity to P2P and 10% to HTTP due to the multi-flow aspect of P2P and the way that TCP congestion control worked. The prioritized network would allocate 50% to HTTP and 50% to P2P and that would be a desirable outcome.

    Lastly, a dumb FIFO network doesn’t offer fine-grain resource sharing between applications. In some cases, the dumb FIFO network allows a single application to forward hundreds of packets back-to-back while starving other applications for 1000 milliseconds. A prioritized network alternates the packets being forwarded between applications and it reduces jitter for all applications.

    3. Please do not confuse FTTH and FTTN. Puting Voice and Video on separate wavelengths applies to FTTH technology; not FTTN technology. Every FTTN deployment does in fact deploy DiffServ type prioritization on the IP network. Moreover, allocating separate spectrum or radio frequencies (over wires or wireless) is a form of QoS which is ultimately what we are talking about. DiffServ prioritization is simply a way of attaining circuit switching level QoS on a flexible packet switching IP network. There’s no reason to prohibit prioritization on the IP level and if it were done, it would have some unintended consequences of forcing FTTN operators to deploy some kind of circuit switched separation.

    4. As a lawyer, I’m sure you appreciate the vagueness of the NPRM as it ensures future opportunities for filing complaints. The NPRM language is vague, and it prohibits ENHANCED or prioritized network services. “Enhanced” packet delivery could certainly be construed to cover paid peering or Content Delivery Network (CDN) offerings. Now I would love to see the FCC to come out and say my interpretation that paragraph 106 bans paid peering and CDN services as wrong, but it really does need to be clarified.

    However, I’m also opposed to the idea that ISPs should be banned from offering prioritized services to content providers. I do not view this as a form of double charging or double dipping so long as the ISP isn’t asking that the content provider pay for best effort delivery when the end user already paid for best effort. If the ISP is charging the content provider for higher bandwidth or priority than the end user paid for, that’s perfectly legitimate.

    In some cases such as the Amazon Kindle, the end user paid for nothing and it’s perfectly legitimate to have the ISP charge Amazon for Kindle network connectivity. H.R. 3458 (Rep. Ed Markey’s 3rd Net Neutrality bill in the house) goes as far as banning ISPs from charging content/app providers at all which would ban devices like the Amazon Kindle.

  • Chris Riley said:

    A two-point response:

    1) You assert that the NPRM is vague – which isn’t precisely correct as much as it would be to say “open-ended”. Final rules can be fairly called vague, and can fairly be said to open up a realm of unanticipated complaints; an NPRM does not permit a complaint, because it’s merely a proposal. So as a legal matter, this statement is incorrect: “I’m sure you appreciate the vagueness of the NPRM as it ensures future opportunities for filing complaints.” If you were to say “the proposed rules” instead of the NPRM, however, yes, I would agree that the proposed rules ensure future opportunities for filing complaints. But that’s the point of proposed rules. They also don’t guarantee liability for companies engaging in behavior that seems to violate the proposed rules. The FCC has two legal opportunities to say that the companies’ actions are fine – one, that they’re not discriminatory, and two, that they’re reasonable network management.

    My essential point is, you cannot honestly assert both that the NPRM is vague, and that it conclusively says or does anything – yet, you do. You say “the NPRM is vague” and at the same time you say “the NPRM bans paid peering.” That’s illogical. If you at least insert *some* language of caution, I feel as if we could have an honest conversation. If you were to say “the proposed rules could ban paid peering if interpreted in this way”, I would say “I don’t believe that would be a valid interpretation for XYZ”, and we would probably agree to disagree, but at least we could be more cordial. As it is, you assert that the NPRM will ban paid peering, even though you yourself say that it is vague (and, as you don’t note, at least not frequently, it’s an NPRM, and the final rules are months away).

    2) “Every FTTN deployment does in fact deploy DiffServ type prioritization on the IP network.” Can you give me some examples of such deployments? I’m not familiar with any.

    3) “Moreover, allocating separate spectrum or radio frequencies (over wires or wireless) is a form of QoS which is ultimately what we are talking about.” Maybe that’s what you’re talking about, but it’s not what I’m talking about. I’m talking about prioritization at the packet level – giving a preference to some applications over others. I’m not talking about separating traffic on wavelengths; I’m also not talking about peering arrangements, or about business SLAs or anything that gives users different treatment. I am *only* talking about giving some applications priority over other applications. The point of the conversation is not allowing any skewing of consumer choice and innovation over applications by not allowing artificial network operator influence over application performance. That’s also why I’m not talking about CDNs – which I don’t have a problem with. That’s not an example of artificial network operator influence.

    4) You again frustrate me with your language here: “The point is that VoIP or online gaming traffic won’t take any more bandwidth in a VoIP or gaming prioritized network than a dumb FIFO network. That is a fact because VoIP and gaming have very low and fixed bandwidth requirements. That debunks your assertion that prioritization harms non-prioritized applications.

    I also mentioned that in the case of HTTP versus P2P, the network that prioritizes HTTP would in fact reduce P2P performance more than a FIFO network, but that it was perfectly justifiable.”

    I did not assert that prioritization *always* harms non-prioritized applications. Certainly, without network congestion, prioritization doesn’t do anything to benefit traffic, so it doesn’t harm non-prioritized traffic. What I said was that in an environment of congestion – heavy congestion not 10% utilization micro-congestion, which happens a lot, as I’m sure you know – a lot of packets get dropped and a lot of communications get disrupted.

    In heavy congestion, non-prioritized traffic WILL be harmed. You say so yourself. But then inexplicably you assert – without qualifier – that you have “debunked” the claim that prioritization harms non-prioritized applications. Even though you *just offered* an example of a situation where it does, in fact, harm non-prioritized applications.

    Again, we could have a conversation about whether the harm is justified. That’s a separate conversation. But you cannot seriously contend two opposite statements. Prioritization does – in times of heavy congestion – have a negative impact on non-prioritized traffic. Please stop making up unique examples of situations where the prioritization may have only a negligible impact, and asserting that based on those few situations, prioritization never causes any harm to non-prioritized traffic.

  • George Ou (author) said:

    Dr. Riley,

    1. Thanks for admitting that the NPRM is open ended and could invite complaints and legal battles. As I pointed out, the word “enhanced” could cover everything from paid peering to CDN. In the context of H.R. 3458 (Markey’s 3rd Net Neutrality bill) which goes as far as banning all forms of ISP charges against content providers, it is reasonable to conclude that Net Neutrality proponents would like to ban all such charges.

    Again, I would like to raise awareness of this issue and at least get some clarification. Where exactly does the FCC stand and where does Free Press stand. Do they want to prohibit any or all of the following service options under the no enhanced or prioritized services clause in the NPRM.

    a) Paid peering services
    b) ISP charges content provider for end-user network access e.g., Amazon Kindle
    c) ISP charges content provider to boost end-user bandwidth beyond what end user is paying
    d) ISP charges content provider to boost end-user priority

    2. It surprises me that you have done so little of the most basic research. Every FTTN provider such as AT&T U-verse or Deutsche telekom that has implemented IPTV relies on packet prioritization. If they deployed managed VoIP services, that also gets prioritized.

    3. Yes I understand you only oppose prioritized packet switching IP networks, and that’s where you’re being ridiculous. The whole reason IP networks have to be prioritized is to give them more of the reliable/predictable performance characteristics of circuit switched networks while maintaining the flexibility/efficiency of an IP network. If you insist on a dumb network, you’ll force the use of less flexible/efficient circuit switching models. See figure 1 and 2 here http://www.digitalsociety.org/2009/10/dont-outlaw-successful-business-models-on-the-internet/.

    4. Seems like you’re changing your assertions here. You clearly asserted in your paper that prioritization reduces performance of unprioritized applications MORE SO than dumb First In First Out (FIFO) networks without providing any data while demonstrating you haven’t even done the most basic research. You also assert in that paper that prioritization raises jitter for unprioritized applications to the point that buffered video streams would suffer.

    Furthermore, you seem thoroughly confused about the relationship between congestion and packet dropping. OF COURSE congestion causes packet dropping, that’s by design! Any time an application attempts to exceed the maximum throughput of a network, TCP congestion control (Jacobson’s algorithm) kicks in and drops packets which is an implicit signal to the application to cut its speed in half and then slowly increase speeds until it hits another packet loss which causes the application to halve its speed again. Applications exceed the network’s capacity all the time and it is the normal bursty application that attempts to transmit data at ever increasing speeds e.g., HTTP, email, FTP until the network signals implicitly to slow down. You’re acting as if there is something wrong when the network to drop packets when this occurs naturally tens or hundreds of times a second and you wrongly attribute this packet loss to prioritization, and you somehow believe that prioritized networks penalize non prioritized applications MORE SO than dumb FIFO networks.

    Then based on this gross misrepresentation of how networks work, you assert here and in your paper that prioritization causes a drop in overall network efficiency and it penalizes nonprioritized applications MORE SO than a dumb FIFO network. But your assertions are based on a very poor understanding of networking.

    I have refuted your assertions, but you’re demanding proof. I’ve already provided a lot more data than you have, but I’ve gathered more proof with hard data from measurements, and it will be coming in a follow up post.

  • Broadband Politics | Guest Blog at GigaOm said:

    [...] another good treatment of the subject at Digital Society, inspired by the same conversation with peering maven Bill [...]

  • Digital Society » Blog Archive » The bias against long distance Internet file transfer said:

    [...] allows content providers to negotiate cheaper short distance connectivity to a single ISP (called Paid Peering) and bypass the higher expensive and higher congestion levels of traditional Internet transit [...]

  • SiliconANGLE — Blog — Netflix won’t destroy the Internet any more than prioritization said:

    [...] but that shouldn’t preclude broadband providers from offering direct wholesale bandwidth (e.g., paid peering services) at a quality and price level that is attractive to Netflix and other content or application [...]