Statistical insignificance

The Ottawa Citizen used some pretty extreme terms – public outcry, sharp jump – after Michael Geist commented on the release of the CRTC’s quarterly report on Internet Traffic Management complaints. The Citizen quotes Professor Geist as saying “I think the public is more aware of the rules and the value of filing complaints because over the last three months in particular, we’ve seen a pretty big spike in complaints.”

Really? Let’s look at the numbers to see if the figures merit such terms.

According to the Citizen, the CRTC reported that it received 41 ITMP-related complaints in the fourth quarter of 2011, compared to 67 complaints in the prior two years. The article ignored the fact that the CRTC ruled that a third of the complaints in 4Q11 were misdirected (the CRTC classified 14 of the 41 as non-ITMP related and, in some cases, referred the complaints to the CCTS). So the real numbers are 27 ITMP complaints compared to 59 in the prior 2 year period. We do not know the timing of the complaints within that 2 year period, so people have implicitly assumed that the complaints were evenly distributed in that period.

More importantly, we need perspective on the magnitude of these numbers.

There are close to ten and a half million residential internet connections in Canada. That is right: 10.5 million. Twenty seven complaints works out to 0.00026%. That is in the same order of magnitude of the odds of being struck by lightning. Twenty seven complaints in a three month period from 10 million customers can hardly be called a sharp spike, let alone be evidence of a public outcry.

If 6 people gathered to protest in front of Toronto City Hall, would we call that a public outcry? If a year earlier, only 1 person had been at the City Hall protest, would we say that the 6 people represented a “pretty big spike in complaints”? Those are the kind of numbers we’re talking about.

The fact that a third of the complaints were non-ITMP related is hardly a sign that the public is more aware of the rules. Indeed, the percentage of non-ITMP complaints appears to be rising, which may indicate that recent education efforts are failing.

The significance of the report from the CRTC wasn’t worth much more than a tweet. It is the start of the Commission’s reporting; I look forward to starting to see more quarterly data before I would suggest there are any trends.

Scroll to Top