Questionable statistics

Over the past couple weeks, I have had a couple occasions to dust off my old college statistics books because of sampling methodologies that lead to questionable results being published.

In the first instance, there was the result from a Harris/Decima survey published in a May 20 press release from Rogers. At the bottom of the release, there was language that may have been missed by many because it reads like boilerplate:

The data was gathered between May 6th through May 9th, 2010 through Harris/Decima’s weekly teleVox, the company’s national omnibus survey. A representative sample of 1,000 Canadians were surveyed, of which 653 own a mobile phone and 283 send at least one text message a month on average. The corresponding margin of error for mobile phone owners is +/-3.8%

Rogers was trying to look at how text messaging enthusiasts use the service and whether these types of customers want to do more with SMS messaging. Harris / Decima thought that they could get a representative sample from their weekly Televox poll. Problem is that their pool of candidates for sampling only includes listed phone numbers and the company has acknowledged that this would exclude the pool of Canadians that have gone wireless only. I would think that wireless only households are different than the average – in use of features, volume of minutes, demographics. All sorts of ways that make it clear that the Harris / Decima study was not a representative sample when it was wireless phone behaviour being studied. 

The other study that caught my eye was a recent release of a Net Index from Ookla.  The usual folks are again bemoaning Canada’s ranking in the study – 32^nd. The statistician in me said to look at the numbers.

When you click on the country name, you get additional details. Canada shows up with an average download speed of 7.92 Mbps. The tests were drawn from a pretty sizable pool: 

Tests from 4,801,628 unique IPs have been taken in this country and of 25,424,152 total tests, 959,172 are being used for the current Index.

That is an impressive result. Nearly 5 million unique IP addresses have used Speedtest to create a pretty good dataset to examine. That is in the order of half of all internet connections in this country.

For the United States (ranked 28), nearly 31M unique IPs are in the pool, a smaller proportion of all of their connections, but still pretty good size – in fact, a pretty awesome pool. But look at number 1 ranked South Korea. Only 201,000 unique IP addresses in the sample – one 25^th of the number for Canada, despite having 50% more total broadband subscribers. Just 33,000 were used for the index.

Number 4 Japan has only 336,000 unique IPs in the sample pool, one 20^th of the representation for Canada despite Japan having 3 times the number of broadband subscribers. Only 67,000 were used.

Why?

Why is the pool of tests so much broader in Canada? Does the breadth of the sample skew the data? 

Unfortunately, there don’t seem to be enough people who bother to look behind the numbers.

The Digital Economy consultation recognized the limitations of many of the international studies when it said “international comparisons should be treated with caution.” But that appropriate caveat was unfairly ridiculed in a recent opinion piece in Telemanagement.

It’s already a tough job to figure out which way you want to go when we develop our digital strategy; it is especially hard if we’re being given questionable data as to a starting point.

Len Waverman, Dean of University of Calgary’s Haskayne School of Business and author of the Nokia Siemens Networks Connectivity Scorecard will be speaking at The 2010 Canadian Telecom Summit next week.