Yochai Benkler of the Berkman Center for Internet & Society at Harvard University responded to my criticism of their broadband study.  The curious thing is that they acknowledge many of the flaws (but ignored some) but they insist that they can still use the data simply because it’s the best they can do.  Here is my response to Benkler’s response.

UPDATE 11/15/2009 – George Ford wrote this paper showing that the economic modeling was flawed as well and that their own models contradicted their own conclusions.

On hotspots

Yochai Benkler (who headed the Berkman study) acknowledges their data is flawed.

“We agree that this data is problematic, and in our study we explain in much detail that accounting for nomadic access is hard, non-standardized, and imperfect.”

But they go on to say that in the absence of better data, they tried to grab different sources of data and do the best they can, and decided to use the information anyways despite its imperfections.  They also say that their imperfect data actually lifts US rankings, but that misses the point that the rankings are silly to begin with.  For example, if we went with OECD’s methodology and the US deployed 100% broadband to every home and business, we’d actually fall to 20th place in the International rankings.

The use of flawed latency data

Benkler admits that their report had to use the data with “great caution”, but that they still factored it into their rankings.  Yet they completely missed the point that the use of latency is fundamentally flawed because it should never be used as a basis for comparison between nations due to the fact that latency is largely dependent on the size of the nation, and the fact that latency is a very poor measure of technological progress because latency values haven’t really changed over the last 10 years.

The problem with using advertised speed rankings

Benkler responds by acknowledging the fact that different nations do in fact inflate their advertised numbers at different rates.  However, he chooses to cherry pick the fact that Sweden’s numbers are even less inflated than the U.S.  despite the fact that the U.S. is the second least inflated.  He also narrows in on the fact that the fastest country in the world Korea is still has 3.8 times more bandwidth when looking at Akamai measurements as opposed to 8 times better with OECD numbers.  But this really misses the point that rankings are not very useful to begin with.

For example, 20 countries could be within 30 percentage points of each other but one country could be ranked 10th place and another country could be ranked 30th.  30th certainly sounds horrible, but is 30% difference in broadband bandwidth capacity going to break that nation or affect their ability to innovate?

Figure 1: Akamai comparison of international broadband speeds – 2009 Q2

We can see this illustrated in figure 1 where the US ranks 18th but it’s within 28% difference from number 5 ranked Sweden.  Does Sweden’s slightly faster broadband capability make it a better technology innovator than the United States?  It certainly doesn’t appear that way.  The reality is that anyone who has ever spoken to European policy experts know just how extremely envious Europe is of America’s ability to innovate in technology.  Furthermore, it also makes a world of difference if Japan averages out to 7 Mbps of usable bandwidth than the OECD’s claim that they average out to 93 Mbps.  Yes they’re faster, but is it that different?

Figure 2 below further illustrates is how silly rankings are to begin with and why it’s irrelevant to compare small European member states to the whole of the United States.  According to speedtest.net data, the top 10 U.S. states all rank significantly higher than the European Union.  Does that mean the Delawareans and the Washingtonians should be singing “We will rock you” and “We are the champions” to their neighboring states and laughing at the Europeans?  Of course not, but this is in essence what the Berkman study is doing (in reverse) by touting the OECD data.  The United States even averages better than Europe.  More notable is the fact that the top 10 ranked states in the U.S. aren’t the states are not necessarily the states that produce the most innovation.  Seen in this context, the rankings are largely a statistical artifact that has little bearing on innovation.

Figure 2: Rankings are silly

What does all that bandwidth do for Korea and Japan?

There’s no doubt that Korea and Japan have the fastest broadband access in the world, though a lot of that has to do with government subsidies and the fact that they are extremely urbanized and their populations are very concentrated, but the real question is whether all that broadband capacity is useful.  In my extensive research conducted for my coauthored paper on broadband performance “Need for speed“, I searched long and hard for evidence of the benefits of faster broadband and spoke to many Japanese and Korean analysts and experts.  I found that the Koreans and Japanese broadband customers weren’t really doing anything that different than their American counterparts and there really were no big bandwidth applications.

What shocked me was the fact that popular Japanese content (video) sites actually offered lower bandwidth lower fidelity content than their U.S. counterparts.  After some further investigation, it turned out that server bandwidth cost many times more than the United States.  So while bandwidth on the consumer side was plentiful and cheap, it was super expensive on the content side which means the content was forced to be lower quality.  So while the U.S. looks weak on the broadband side compared to Japan, it is a more balanced and effective network that evolved through free market forces rather than a network with artificially jacked up access tier performance.  The lesson learned is that we can offer the cheapest bandwidth on the client side in the world but if the bandwidth costs for content isn’t changed, it’s effectively a  Ferrari on bicycle tires.

Flawed pricing data

Benkler spent significant time rehashing a lot of fancy graphs and charts.  Ironically, he criticized my San Francisco example where $30/month for 100 Mbps broadband access may not actually be 100 Mbps of effective throughput and where a lot of the cost may be rolled up and hidden in the high rent.  But that was precisely the point I was trying to make and that we can’t compare apartment complexes where “last mile” infrastructure is hidden by the facilities charges and where the bandwidth may be more exaggerated.  I deliberately referenced the San Francisco example as a silly example much like figure 2 above to illustrate a point, yet Benkler thinks that it is perfectly acceptable to use the same dubious data from Japan and Korea which are filled with these apartment complexes with similarly inflated performance and hidden costs.

One key criticism Benkler ignored was how flawed the pricing data on Japan was.  To reiterate my original criticisms, the Berkman study claims that very high speed broadband access in Japan (100 Mbps) only costs $32 a month.  Yet I cited specific pricing data showing that the actual costs were $65.76 to $73.90 for single family homes.  But instead of answering this specific criticism, we get more migraine inducing graphs and charts that obfuscate the glaring flaws in the Berkman broadband report.