The Saïd Business School at the University of Oxford and the University of Oviedo’s Department of Applied Economics have released a new study on the state of broadband networks around the world and some [such as here and here] have already determined that the “Canadian rankings are lousy”.
Once again, it appears that logos on the page from such institutions as “University of Oxford” and “Cisco” are clouding the willingness or ability to apply a critical eye in examining the findings.
In a blog posting, Suzanne Blackwell at Giganomics questions the reliability of the broadband speed rankings which were derived from Speedtest.net.
There are a number of other questions raised by the Oxford report. The report uses penetration as a percentage of households, but it is unclear where the numbers come from.
Australia was reported to have a penetration rate of 85%, which does not appear to align with the Australian government – let’s face it, if the Australians were really at 85% penetration, would they be so keen to spend $43B on their broadband? We think that Australia is really closer to 53% penetration.
That is a pretty big discrepancy.
The Netherlands is listed by the Oxford / Cisco study at 83% but the EC recently reported it at 74%. Denmark and Norway are similarly divergent. The U.S. penetration rate is not 75% but only 63% according to the Pew Internet and American Life survey from earlier this year.
Since penetration rates are used to derive “leadership” scores, any inaccuracies or inconsistencies can cause a country to rise or fall.
Will a more academically sound paper be released describing the details of the methodology, the sampling methods and the sources of the data?
"Suzanne Blackwell at Giganomics questions the reliability of the broadband speed rankings which were derived from Speedtest.net."
What's her point? Unless she states that Speedtest.net only records the speed and ping that occurs at the *end* of each test, she really doesn't have an argument. As well, you failed to mentioned the sample size of the oxford paper:
"The study includes 24 million tests, from 66 countries, and was analysed by researchers in Oviedo and at Oxford's Said Business School. Speedtest measures upload speed, download speed, and latency (ping time), which de Bernabé argues make a good-enough measure of quality."
They have over twenty four million tests from 66 countries in this paper. That's really important information that challenges your view of te oxford paper lacking in academic standing. That sample sounds reliable to me. Now , you could argue that there is no way to determine if every test was by done a single individual person, that more than one could have been repeated by the same person. At the same time, i could argue that more than one test for each person, each day, would be more representative of their connection than just one test.
Not to be rude, but you should have included the oxford sample figure in your article…
Sample figure:
http://www.eweekeurope.co.uk/knowledge/broadband-policy-makers-need-to-balance-quality-and-reach-1966
There are a number of problems with using aggregate Speedtest.Net results, as observed by the Oxford study's sponsors, who acknowledged the bias:
the study has an innate bias which de Bernabé acknowledges:it draws on broadband quality data gathered from the widely-used site speedtest.net. It is not a statistically-chosen sample, just a study of users who choose to check their speed at that site: "One of the limitations of this study is auto-selection,"
24 million tests, if not selected properly, are not necessarily a good "sample".
Further, Speedtest doesn't measure infrastructure in anyway. It tries to measure delivered speed. If I subscribe to a 10Mbps service, that sets a cap on the speed that will be achieved, regardless of the speed that can be supported by the infrastructure.
Read our full report for a more complete review of the issues.
Most Canadians subscribe to 10mb plans or a plan that is near its equivalent. If they test their connection and end up only getting 2mb (for example, then that user is only getting a fifth of the service they are paying for. This can be acceptable; however, many users, myself included, don't even get a fifth of the service we pay for. For example, I subscribe to a 1.5 mb wireless service from rogers, and i'm lucky if i even get 100kbps download, much less 200kbps like i use too. Most of the time, i get anywhere from 500 bytes (yea, BYTES) to 20kbps, and i get latency through the roof. Nowhere near an acceptable level when i'm paying 50 bucks a month.
Twenty four million samples is huge, and there were 66 countries involved in this study. 24 million/ 66= 363,636.36… tests per country. That is still a huge number, and can be pretty representative of Canadian internet in general. I am unsure of how to access speedtests geography data to get a picture of the quality of service across Canada, but 360,000 per countries is huge, and my opinion, relevant.
Many Canadians are in the same boat (ie: lousy service), and external research has proven the lackluster state of Canada's internet. No report from the major cable companies, the one's responsible for this situation, will convince me otherwise… especially when it comes a week after an Oxford report.
Radar E 33: I don't know what the source of your subscriber information is that has you conclude "Most Canadians subscribe to 10mb plans or a plan that is near its equivalent."
If you look at Table 5.3.3 in the CRTC's recent Communications Monitoring Report – which was our source – you will see than only 11.1% of Canadians have subscribed to a service >10Mbps.
You also assumed the Speedtest sample was evenly divided among the 66 countries, for which we would also like to see your source. As we noted, there was no information about how any of the samples were selected – other than an acknowledgment that there was self selection. Which we all know.
24 million poorly selected samples doesn't make the survey results better – as any statistician will tell you.
It was generalized statement. What i forgot to mention was that most people subscribe up to 10mb, a fact confirmed by your table you linked to in your last post. Though i must point out, your sample is only 8000 approximately (2009), hardly representative of Canada in general, compared to my assumed speedtest figure of 363,000 people. You argue that there where poorly selected samples involved. Well, where's the evidence? I too, would like to see if these people went after crap samples, or if they averaged out their sample to a mean score.
The comment i made was in regards to your 10mb speed cap comment. You mentioned infrastructure and stated that this 10mb limit would not be representative of the networks ability to produce higher speeds.
However, 10mb plans are not capped at 10mb; merely, these plans are representative of the base speed you are paying for, and should be getting. It does not mean your speed can't go past 10mb. I pay for 1.5 mb download, and in the past, i've seen my speed go as high as 3mb download… impressive for rogers, to deliver me internet at 200% efficiency (nothing like now though).
You frequently highlight that the Oxford sample had its own bias in regards to how data was obtained; This bias was admitted in their report. However, i have yet to see you acknowledge your own bias with your report sir. Your report is funded by the big Canadian internet companies, so there is a conflict of interest there. They have a vested interest in disproving the oxford report, and various other reports that have exposed them. You repeatedly reference your contributors in your report and never seem to consider the trustworthiness of your source (remember, quite a few of these companies *rogers, bell* where not honest about throttling its customers, haven't been honest about the whole local tv crisis, and haven't been honest about traffic shaping). As well, you reference Wikipedia in this report… i'm sorry, but for those of us in University, THAT is laughable. That's the kind of thing that gets you a big, fat, F. So, how am i going to trust? The company that gouges us and cannot deliver the service we need? Or an independent academic body?
The prices for 10mb and up in Canada are ridiculous at this point, exceeding what our neighbors down south have to pay for an equivalent service. Its one thing to say "hey, we offer 50mb connections, we have infrastructure". Its another to offer them at reasonable prices.