BitTorrent – Now with improved destructiveness
I ran an analysis of BitTorrent’s Micro Torrent Protocol (µTP or uTP) last November (using uTorrent 2.0 beta build 16850) and found that the claim of “network friendliness” was dubious. Now that the official BitTorrent client with uTP has moved past the beta stage, I decided to test it again using the latest uTorrent 2.0.2 (build 19648) client. What I found was that not only did BitTorrent not improve in its network friendliness, it’s more destructive than ever to other applications.
Figure 1 – Jitter from BitTorrent µTP (uTorrent 2.0.2)
Figure 1 above shows just how destructive the latest version of BitTorrent is. All of the spikes that touch the top were ping timeouts. BitTorrent’s uTorrent client was installed on an MSI Netbook using an 802.11g Wi-Fi connection and the continuous ping tests were conducted on my wired desktop computer to the first DSL gateway. What the chart measures above is the roundtrip ping time above the baseline of 12 milliseconds (ms) which varies no more than 1 or 2 ms during idle times. So if the measured ping is 400 ms, then the jitter value is 388 ms.
The average measured jitter was 312 ms but the real shocker was the 10% packet loss rate where the ping simply timed out. There were solid blocks of 4 seconds where there was simply no service at all so if I were trying to use my Lingo VoIP service (a competitor to Vonage), I would have experienced 4 seconds of silence both directions. As little as 30 ms jitter is enough to cause packet drops in a VoIP call using typical Analog Telephony Adapters (ATA). These jitter levels would make gaming nearly unplayable even though the game requires less than 100 Kbps when my DSL connection has almost 1 Mbps of unused bandwidth. I’ve pointed out that single stream applications like YouTube and Netflix can cause some fairly serious jitter problems but the latest version of uTorrent brings unimaginable damage to jitter sensitive applications like VoIP or online gaming.
I’ve struggled to point out that BitTorrent isn’t network friendly as the company claims even though the media and blogosphere continue to repeat the BitTorrent press releases without doing any kind of fact check, but it’s important to debunk these mistruths. Those who propagate these myths are often the same people who propagate the myth that networks are supposed to be dumb and that only end points should be intelligent. But end points can’t solve the problem because those who control the end points face the classic problem of externalities. I don’t have a problem with the right of BitTorrent to exist and use the network since I’ve been a fan of uTorrent before BitTorrent even bought the software, but it’s important that networks can deal with BitTorrent congestion issues in an intelligent and fair manner so that all applications can live in harmony.
George–
I’m new to your writing, so I had to acquaint myself with your previous stuff on TCP fairness. Wish I’d seen some of this earlier. Disclosure/disclaimer: I work for a large telecom equipment company, but my views here are my own.
First, a prosaic instrumentation question/comment: Is that DSL GW that you’re pinging to in the path of the torrent? If so, note that a ping server is often implemented as a fairly low-priority process, so your results may be somewhat misleading if the GW is bumping up against its compute limits. Packet-switching always has priority over vanilla-flavored processes, so a big chunk of your ping delays could be computational artifacts, rather than delays caused by true congestion.
A better strategy would be either to place a dedicated ping server behind the GW or, better still, to measure jitter in an RTP stream with a sniffer somehow. (Both of these are obviously hard to do on a real ISP’s network…) You might even be better off using a ping server that’s several hops away and much deeper into the internet, as you’re most interested in measuring edge effects and the core usually has minimal congestion problems.
Now, on to policy. I am in substantial agreement that the app neutrality variant of the net neutrality religion only makes sense if you know nothing about real networks. However, I am pretty sure that any form of fairness implemented at the endpoint is a fool’s errand. uTorrent is a fine example of this: These guys didn’t go to UDP out of the goodness of their hearts; I suspect they did it because it prevents them from being subject to RED or WRED, which tend to work only on TCP streams. As a general rule, any congestion avoidance scheme can be defeated with enough custom software in the endpoint. The only solution is to implement proper policy and policing at the provider edge.
TCP is far from perfect, and I was frankly a little surprised that there was any sort of consensus that it was fair. All that the Jacobsen algorithm does is guarantee that TCP can’t cause congestive collapse. If you want to make TCP fair, you have to do it with policy, and many of the tools already exist, or could be made to exist using existing mechanisms.
Here’s an example of something that would be fairly easy to do: At the provider edge, a policy system tracks the smoothed throughput from all TCP sources. If a particular IP address significantly exceeds that throughput, it gets marked with a lower-priority DSCP. After that, any early congestion occurring along the path will trigger WRED, which will cause early drops preferentially in the lower-priority traffic. You could do a similar thing with ECN, although nobody wants to turn ECN on unless they control the entire end-to-end path (in which case they can engineer their network so that they don’t need it).
My next point is one that you’ve touched on peripherally, but which needs to be emphasized wrt app neutrality: Even if you were to make TCP fair (see above), real time applications behave very poorly where they have to coexist with TCP traffic unless they have some policing advantage via diffserv. (We’ll leave intserv alone for now, OK?) This cuts both ways–TCP traffic near the congestion limit will cause so many RTP packet drops that the RTP application probably won’t work, but RTP, due to its incredibly weak congestion avoidance regime (which nobody really implements anyway), can grow to swamp the router and cause congestive collapse. That this doesn’t happen a lot today is merely because voice traffic is relatively low bitrate and relatively uncommon. I suspect that things will start to get interesting when conversational HD video gets popular.
The only solution to managing RTP traffic is via appropriate use of DSCP, with appropriate policy associated with it. If the dumb network folks have their way, the internet will be good only for TCP-based traffic, and not very good even for that. The good news is that the FCC seems to sorta-kinda understand the problem. (For example, nobody is saying that ISPs can’t operate MPLS networks for their corporate VPN customers.) Given the choice between the FCC having jurisdiction over the issue or Congress legislating technical standards, the FCC is clearly the lesser of two evils. Meanwhile, it’s important to stay engaged in the debate to make sure that the app neutrality crazies don’t win by default.
@TheRadicalModerate
TRM: “First, a prosaic instrumentation question/comment: Is that DSL GW that you’re pinging to in the path of the torrent? If so, note that a ping server is often implemented as a fairly low-priority process”
That DSL GW is the very first IP I hit (first layer 3 device beyond my own router and DSL modem) regardless of where I am going. I’ve noticed that it can change from time to time, but even when I ping a different gateway during congestion times, the jitter still affects me.
I understand that replying to pings is a low priority process, and I’ve noticed that the pings going to the routers in the path of where I am going can occassionally be higher than the ping I get from the end point. However, this additional latency generally stays within 0-3 ms on the routers that I use as a measuring stick. But in general, the first router I pick usually has a very stable response time and my baseline ping is extremely reliable at 12-13 ms when there is no jitter-inducing traffic. But it is quite possible that the ping times I measured are 0-2 ms higher than what I measured due to latency from the router I’m measuring against but that really pales in comparison to the measurements I am getting. We’re talking roughly a 0-2% error at most.
TRM: “A better strategy would be either to place a dedicated ping server behind the GW or, better still, to measure jitter in an RTP stream with a sniffer somehow. (Both of these are obviously hard to do on a real ISP’s network…)”
Agreed, but you answered your own question :).
TRM: “You might even be better off using a ping server that’s several hops away and much deeper into the internet, as you’re most interested in measuring edge effects and the core usually has minimal congestion problems.”
I’ve done this before measuring from the west coast (where I am) to near the east coast. The total jitter was between 0-8 milliseconds which is about 4x higher than the instability from the first SBC router I hit. So my current method has the lowest error rate.
TRM: “As a general rule, any congestion avoidance scheme can be defeated with enough custom software in the endpoint. The only solution is to implement proper policy and policing at the provider edge.”
I’m in absolute agreement here.
TRM: “That this doesn’t happen a lot today is merely because voice traffic is relatively low bitrate and relatively uncommon. I suspect that things will start to get interesting when conversational HD video gets popular.”
I’ve measured the jitter induced by VoIP and it is generally zero :). Those VoIP packets are so small and the fact that they only come one at a time every 50th of a second means that you will at worst be delayed a few microseconds behind a VoIP packet. I’ve also measured the jitter inducing effect of two IPTV HD video streams that consume a total of 12 Mbps and I was quite pleasantly surpised that there was nearly no measureable jitter. The cause of this behavior is because of its isochronous behavior such that all the packets are spaced out and other applications can work their way into the queue. HD video conferencing typically runs between 1.2 and 6 Mbps and is also isochronous so it should behave in the same friendly way.
I actually suggested back in 2008 in a blog post (http://www.formortals.com/why-bittorrent-causes-so-much-jitter-high-ping-and-how-to-fix-it/) that BitTorrent should try to send its traffic in an isochronous manner with the packets spaced out. They told me that they actually took that suggestion seriously and researched it. When they found out that this also meant losing a little performance because it was less aggressive, they decided that it wasn’t worth it and they’d rather be aggressive and destructive than nice and yielding.
TRM: “The good news is that the FCC seems to sorta-kinda understand the problem. (For example, nobody is saying that ISPs can’t operate MPLS networks for their corporate VPN customers.)”
It’s bad enough they want to abolish last mile intelligent networks and abolish ISP-provided enhanced peering services for content providers.
TRM: “Given the choice between the FCC having jurisdiction over the issue or Congress legislating technical standards, the FCC is clearly the lesser of two evils.”
Most of us once thought that. But with the current FCC, we’re not so sure. We have 3 regulators that are trying to rewrite telecom law and grant themselves powers not given to them by Congress.
There’s still something bothering me about your test. So, you’ve got two paths:
uTorrentNetbook–>wifi–>YourRouter–>DSLModem–>ADSL–>DSLAM–>RestOfNet
and
pingPC–>xBaseT–>YourRouter–>DSLModem–>ADSL–>pingServerOnDSLAM,
right? (Note that I’m making the assumption that the first-hop DSL GW and the DSLAM are one in the same, which is usually the case.)
So, given that you’re pretty sure that the ping isn’t being compute-starved by more than a couple of ms, what you’re really seeing is congestion either on your home router or in some artifact of the DSL MAC layer. Assuming that the DSL is offering you a clear upstream path with no flow control (hey, it’s the wonders of a synchronous, albeit slow, TDM network, dude), I’m guessing that the most likely behavior is that you’re buffer-bound on your home router, since a much higher aggregate offered load is coming in over wifi than is going out of DSL. I have no clue what kind of congestion control you’ve got in your home router, but that might be an interesting place to experiment. Given that uT is now UDP, RED won’t touch the traffic, and you might have something akin to tail-drop synchronization going on amongst the various uT channels.
Based on that, the jitter you measure may just be head-of-line blocking or plain old buffer drops of the ping packets in your home router, rather than some more serious behavior at the provider edge. That’s definitely a problem, but it’s hard to go from that to a statement like “uT is causing congestion problems at the provider edge.” After all, you’re really more worried about your neighbor running uT and screwing up your service than you are about screwing up your service inside your own home.
I don’t know how sophisticated your home router is, but there are a couple of things you might try:
1) If you’ve got a non-routing wifi base station lying around, trying using that instead of your router I’m guessing that the base station will respond to 802.3 carrier sense (which is what the DSL modem ought to be putting out as soon as its puny little packet buffer gets filled) by asserting 802.11 flow control (which I don’t think is particularly CSMA/CD-like, but there’s something there, obviously).
2) If you can turn on weighted fair queuing or even dump the ICMP ping traffic into a priority queue on your home router, that would more accurately simulate a couple of endpoints sharing a DSLAM.
3) Use two DSL lines, preferably going into the same DSLAM.
4) See what happens on DOCSIS. My guess is that you’ll get similar behavior, given that the upstream channels in DOCSIS are using TDMA or CDMA methodologies, which don’t have much real contention associated with them.
The last thing to note here is that you really, really, really need to be pinging deeper into the net. As long as the provider’s DSLAM is anything close to properly configured, you should never see congestion on it, irrespective of the offered load, because ADSL is effectively offering you–and all your neighbors–a clear channel to the DSLAM. Even if the DSLAM does drop stuff, it’s unlikely to drop packets that are destined for processes residing on it, like the ping server. So, to see the real congestion effects at the provider edge, you’ve got to be pinging something on the other side of the provider edge. Try doing a traceroute to find something two or three hops deep and see if you can find something that will accept your ping.
I’m not saying that there isn’t a problem here, but your methodology has to be squeaky-clean in these kind of arguments.
I’ve done very specific download-only tests here (http://www.digitalsociety.org/2010/06/how-video-streaming-can-ruin-voip-and-gaming). I get more downstream jitter (which is out of my control since the queue builds up on the ISP’s equipment) than I get on upstream jitter.
I do get some upstream jitter, but that occurs on the DSL modem which is not in my control (though I could configure it that way by enabling the router in the modem, which I plan to do soon to test). It’s not my router that’s causing the jitter since that has a 100 Mbps path to the DSL modem and a 100 Mbps path to my desktop computer and an 802.11g path to my Netbook. It’s funny how the netbook on a slower congestion can starve out the Desktop computer on the faster FastEthernet connection.
“The last thing to note here is that you really, really, really need to be pinging deeper into the net. As long as the provider’s DSLAM is anything close to properly configured, you should never see congestion on it, irrespective of the offered load, because ADSL is effectively offering you–and all your neighbors–a clear channel to the DSLAM”
I’ve found that pinging deeper into the net to be less accurate than pinging the nearest router along the path. Pinging the nearest router produces a jitter variation of 1 to 2 milliseconds. Pinging something on the East coast from the West coast will typically result in 1 to 8 milliseconds of jitter variation. Try the experiment yourself and you’ll see the same thing. I’ve tested this from many locations. So using my method, the error rate is 1-2 milliseconds out of the typical measurements of 200 to 1000+ millisecond induced jitter from BitTorrent. That is essentially an error rate of less than 1%. Pinging deeper into the Internet would result in a higher error rate.
ADSL/VDSL and DOCSIS will shield you from inter-subscriber jitter. It does not shield you from intra-subscriber jitter so it can’t protect you from the computers on your home network.
There’s a few things (rather complex) that I can do about upstream jitter, but nothing I can do about downstream jitter other than to kill the P2P or HTML downstream.
[...] [...]
[...] [...]
[...] alarming to think that regulators might force carriers to carry network-destructive applications like BitTorrent or other streaming video services that force wireless networks to support video on demand when [...]
Leave your response!
Twitter Feed
About Us
Digital Society is a digital think tank that believes culture and commerce are inseparable, that the digital economy flourishes when people are free and rights are secure, and that free markets free people.
Digital Society is an independent 501(c)3 non-profit organization, funded by donations from Jon Henke and from Arts+Labs. We advocate for a pro-culture, pro-commerce digital society through research, analysis and debate on emerging technology issues.
Reply Comments
Transparency and interactivity are trademarks of the Internet era, and we aim to foster them here at Digital Society. It is inevitable that some people will disagree with the technology policy positions we take. We want to have that constructive debate.
The Reply Comments feature gives our critics a chance to respond to our viewpoints and the Digital Society audience convenient access to competing arguments. Any time we directly challenge the views of an individual or a group on this site, the party in question may substantively respond in a guest post.
Please contact executive director Jon Henke by e-mail.
Subscribe
Daily Digest Email
Recent Posts
Recent Posts
Most Commented
Most Viewed