Chris Riley of Free Press responded to my thorough debunking of his anti-prioritization paper by posting some comments on an unrelated article.  Riley’s main objection to my analysis is that I provided no proof to refute his paper which is ironic given the fact that there was no data in the original Riley-Topolski paper on the “Hidden harms of application bias“.  But my initial analysis paper did in fact contain supporting data, but its primary purpose was to pointed out the technical inaccuracies in the theories that Chris Riley and Robb Topolski used.  However, I do realize that the case can be made even stronger if specific data was crafted to refute unsubstantiated assertions of the Riley-Topolski paper.

I have conducted additional experiments so that data measured from the field could be used to refute Riley and Topolski’s claims and present them below.

Myth that prioritization increases jitter for low priority applications

One of the key claims made in “Hidden harms of application bias” is that packet prioritization technology reduces overall efficiency of networks and that it increases jitter (the variation and erratic increase in latency) for low priority applications.  Riley and Topolski even claim that this is enough to be problematic for buffered video streaming which is ludicrous on the face of it given the fact that jitter won’t even approach a fraction of the buffer size, but actual test data from a live production broadband deployment that implements extensive packet prioritization technology show that increased jitter for low priority applications is not a byproduct of network prioritization and this is illustrated in figure 1 and 2 below.

Every Fiber to the Node (FTTN) VDSL2 deployment in the world e.g., AT&T U-verse in the United States and Deutsche Telekom in Germany that offers managed Voice over IP (VoIP) or Internet Protocol Television (IPTV) deploys extensive packet prioritization technologies to guarantee good service quality for those managed services.  This is something that is widely known in to anyone familiar with the telecom and communications industry, yet this fact somehow escaped Chris Riley who stated that he was “not familiar with any” prioritization deployments on FTTN networks.  If he was familiar with this basic fact, perhaps he might have avoided making so many flawed assertions about prioritization technology.

Test setup 1

To get these measurements, I ran some ping (tool that measures round-trip latency) tests on a friend’s FTTN VDSL2 broadband service on Deutsche Telekom’s network under idle IPTV conditions when packet prioritization was not engaged to get some baseline measurements.  My own home doesn’t have VDSL2 service and I obviously can’t walk into a stranger’s home to run some tests, so my friend’s home setup was the perfect test subject.  Since the first router outside of the customer premise (test subject’s home) did not respond to ping, I measured the second upstream router outside of the customer premise at a baseline latency of 20 milliseconds.  The jitter graphs in figure 1 and 2 is the amount of latency above 20 milliseconds.

Test results 1

In figure 1 below, average the jitter hovered between 0 and 4 milliseconds which is fairly normal variation.

Figure 1 – Measured jitter (variation in latency) on idle FTTN VDSL2 broadband

The question is what happens if we engage the packet prioritization mechanism needed to ensure IPTV quality.  Any network engineer or person familiar with network engineering theory knows that this should not cause any additional jitter of any significant quantity for the unmanaged low priority network, yet Riley and Topolski claims that it would induce incredible amounts of jitter that would have to be in the 1000+ millisecond range if it can break buffered video streaming.

So to settle this question scientifically with experimental data, my friend turned on his IPTV service onto an HDTV channel that probably draws between 5 and 10 Mbps of variable bandwidth, and that would ensure that the packet prioritization mechanism is engaged.  More importantly, any non IPTV network traffic is treated as “best effort” which means it is low priority.

Update 3:33PM – Figure 2 indicates that the jitter varies between 0 and 5 millisecond jitter which is virtually identical to the idle FTTN VDSL2 circuit in figure 1.  The average increase in jitter was 0.3 milliseconds and maximum increase was 1 milliseconds.  That is what we would expect to see just from having the heavy IPTV flow of traffic going through the network, and it is not some adverse “harm” caused by the use of prioritization technology.

Update 3:33PM – Some of the comments to this post made the mistake of trying to suggest that this was a huge percent increase in jitter, but I want to make very clear that this is a big mistake.  After all, going from 0.1 millisecond jitter to 10 millisecond jitter would be a 9900% increase in jitter which sounds horrible, but that 9.9 millisecond increase is insignificant.  Nobody cares about percent difference in jitter and it is an irrelevant metric.  If average jitter was increased from 1 millisecond to 3 millisecond, that is a 200% increase but it is a meaningless statistic.  Moreover, this would actually be an increase of latency from 40 milliseconds to 42 milliseconds if we used the typical baseline latency of 40 milliseconds which would actually constitute a 5% increase in delay, and even the most hardcore gamer with extreme reaction times won’t notice that 2 millisecond change.  So any talk of percent jitter increase is just statistical smoke and mirrors.  The fact is that we don’t get into trouble with online gaming until we see jitter above 30 milliseconds and 60 millisecond jitter can result in some packet loss for VoIP applications which means audio loss.

This is convincing proof that there is no problematic jitter or variation in latency and it thoroughly debunks one of the key assertions in the Riley-Topolski paper which claimed that there would be enough jitter to break buffered video streaming.

Figure 2 -  Measured jitter (variation in latency) with HD IPTV active on FTTN

Applicability to real-world applications

We know that the ping tool is fairly good at predicting problems for real-world applications.  If the ping is volatile and high, applications that are latency and jitter sensitive will suffer.  Furthermore, we know that FTTN VDSL2 service has millions of users and none of them complain about high jitter when the packet prioritization is engaged.  I have verified this myself during the countless times I play online with my friend while his family watches the prioritized IPTV service, and I can see that his latency shown on the score board inside the game remains very stable.  He also reports that IPTV doesn’t cause any latency problems for him.  Since online gaming applications are the most sensitive to jitter than any other application, it is the perfect proxy to measure the impact on other applications which are less jitter sensitive.

Update 4:00 PM

Myth that prioritization degrades network efficiency

Another major assertion in the Riley-Topolski paper is that prioritization degrades performance of lower priority applications and the overall performance of the network compared to a dumb First In First Out (FIFO) network that does not employ packet prioritization technology.  Riley and Topolski asserted that:

“Prioritization operates by degrading and harming lower priority traffic, because (by definition) more low priority packets are delayed or dropped.”

“prioritization produces more dropped and delayed packets, causing more packet retransmission, and thus requiring greater traffic to perform the same communication”

But this is a very misleading assertion because it attributes the degradation in performance to prioritization technology when the reality is that it is simply the existence of the other application’s data flowing on the network.  Riley and Topolski goes further to blame an increase in packet drops on the use of packet prioritization, but this assertion ignores even the most basic concepts of network engineering and TCP congestion control.

Figure 3 – Jacobson’s TCP congestion control in action

Source: George Ou – A Policymaker’s guide to Network Management

All networks drop packets many times a second when dealing with bursty applications e.g., email, ftp, web (http), and especially peer-to-peer (P2P).  These applications like to keep increasing speeds indefinitely until the network sends an implicit message to the application to cut its transmit rate in half by dropping one of its packets.  This cycle of additive increase and multiplicative decrease in application throughput occurs hundreds of times per second which means hundreds of packets are dropped per second and this is a normal behavior of TCP congestion control.

The more applications that use the network, the higher the packet drop rate and the higher the overhead in packet retransmissions.  Some applications like P2P which open up dozens or even hundreds of TCP flows essentially behave as dozens or hundreds of normal single-flow applications, and they cause the network to drop even more packets.  Other applications like online gaming, VoIP, or IPTV which have fixed rather than bursty transmission patterns cause zero packet drops because they never attempt to exceed the capacity of the network in the first place by design.  So it is the number of bursty applications that attempt to exceed the network’s capacity that determines the rate of packet drops and not the use of packet prioritization as Riley and Topolski asserts.

Test setup 2

Using the same test subject as our first test, we can measure the effect of prioritized IPTV traffic on low priority traffic (the best effort Internet traffic that is everything other than the IPTV traffic).  If Riley and Topolski’s theory that prioritization over penalizes low priority applications and reduces overall network performance is correct, then we should see excessive and disproportionate harm on all non IPTV traffic and we should be able to measure something that proves or disproves this theory.  To perform this test, we simply need to measure peak throughput with and without the IPTV service running, and we can use speedtest.net to perform the measurements.

Test results 2

Figure 4 below are the actual test results that compares Internet download and upload throughput of a VDSL2 broadband service while an HD IPTV service is operating and while no IPTV service is running.  As we can see from the results, HD IPTV was allocated an average of 7.57 Mbps which is exactly what we expect it to be, and it falls within the 5 to 10 Mbps range for the variable bitrate encoding it uses.

Figure 4 – Comparison of throughput with and without HD IPTV usage

HD IPTV is being allocated less bandwidth than than the 15.27 Mbps of capacity being allocated to Internet access which goes to show that prioritization only allocates what it needs to allocate and it does not necessarily mean that less bandwidth is being given to lower priority traffic.  This is especially true when we are talking about the prioritization of VoIP traffic which requires less than 0.1 Mbps fixed bandwidth which means that no more than 0.1 Mbps of capacity is reserved.

Furthermore, we can see that overall network efficiency has not declined as Riley and Topolski asserts.  If network efficiency had declined as a result of packet prioritization, we would expect to see far less than 15 Mbps of remaining capacity on the VDSL2 service when IPTV was being prioritized.  But the data shows that there is no excessive or disproportionate harm on low priority traffic.

Conclusion

The theories put forth by Chris Riley and Robb Topolski are contrary to the realities of network engineering.  The tests here confirm that the assertions made in “The hidden harms of application bias” were wrong.