When flawed data leads to flawed conclusions

With the Shaw – Rogers merger announcement, and Parliament debating a bill that threatens to bring a lot of internet content under CRTC regulation, it seems everyone has been forming an opinion on regulatory reform. Unfortunately, there is a lot of misinformation being spread, and opinion pieces being written using deeply flawed reports. Garbage in; garbage out.

I have written much on two sets of reports that seem to keep being cited, despite egregious errors that should be obvious with even a cursory examination. As I wrote last November, studies from Rewheel Research have been thoroughly discredited, with 24 leading academics and telecom economists and lawyers from around the world referring to their reports as a “careless mish-mash of data points from which no reliable conclusions can be drawn.”

I wrote about some obvious problems with the methodology for a UK-based mobile study from cable.co.uk in a piece last summer, “Look at the data”. As I wrote at that time, “It’s very easy to look at a chart on social media, nod one’s head, and retweet or reply without bothering to look beyond the headline. It is tougher to apply a critical eye, look at the data, and determine policy based on deeper analysis.” Sadly, not enough people take the time to look beyond the eye-catching headlines before regurgitating them.

A more recent report from cable.co.uk suffers from the same methodological problems as I documented about the mobile report. One glaring data point jumped out at me. If you are to believe cable.co.uk, the average monthly cost of broadband in Canada is US$76.14, up from US$34.86 last year – an increase of US$41.28. Does anyone believe prices more than doubled last year? Anyone?

That is why I say, “Look at the data”.

A week ago, Toronto Star columnist David Olive appears to have relied on the flawed cable.co.uk data and rankings as the basis for his article “The time is right for Ottawa to fix Canada’s disgraceful telecom system”. By failing to detect the fairly obvious errors in the report, there is no support for Olive’s conclusions.

CWTA president Robert Ghiz had this response published in Saturday’s Star:

Canada’s telecoms among world’s best in overall value
Questionable studies report misleading findings on price of country’s wireless services

Canadian telecom providers offer some of the best and most affordable telecom services in the world.

The Economist’s Intelligence Unit’s Inclusive internet Index ranks Canada third out of 100 countries in internet affordability. Similarly, global accounting firm PwC recently ranked Canada first in the G7 for affordable wireless services.

Unfortunately, these studies are too often ignored. Instead, more attention has been given to one-dimensional and misleading studies that paint an inaccurate picture of telecom prices and affordability in Canada. These include Cable.co.uk’s fixed broadband price study and price comparison reports by Finnish consultancy, Rewheel.

The Cable.co.uk study looks at the median of surveyed broadband plans to develop its country rankings. Using this median price, it claims that broadband fees in Canada are 27 per cent higher than the U.S. But the actual dataset used by Cable.co.uk tells a different story.

Of the Canadian and U.S. broadband plans measured by Cable.co.uk, the cheapest was offered in Canada. By focusing on the median plan, irrespective of which plans consumers choose, Cable.co.uk gives the false impression that Canadians pay more for broadband service.

The questionable quality of the Cable.co.uk methodology is made crystal clear when its 2020 report is compared to its findings in 2019.

For 2020, Cable.co.uk concluded that the average fixed broadband monthly cost in Canada was $76.14 (U.S.) while the average monthly cost in 2019 was $34.86.

Clearly, prices for fixed broadband internet in Canada did not more than double in one year. In fact, in its most recent pricing study, the government of Canada observed that “over the last five years, Canadian broadband prices have trended downwards” and were lower than the U.S.

Meanwhile, Rewheel has been widely criticized for its flawed approach to mobile wireless price comparisons. Last year, 24 leading telecom academics, policy experts and economists released a highly critical review of Rewheel’s methodology, including the recommendation that Rewheel’s reports should come with the same kind of warning labels that social media platforms apply to suspicious information. Similarly, the International Center of Law and Economics published an article cautioning that a Rewheel study examined by the article’s authors amounted to little more than a “careless mishmash of data points from which no reliable conclusions can be drawn.”

In addition to the Economist and PwC reports, other studies have found that Canada’s telecom industry offers superior value. For example, a U.S. industry association-commissioned study found that Canadian wireless subscribers receive more value for their dollar — or “more bang for their buck” — than customers in all other G7 countries plus Australia.

While the studies cited above do not generate the same eye-catching headlines as those which misrepresent telecom prices in Canada, they offer a more meaningful consideration of affordability and value.

Also under-reported is the fact that the three federal government agencies charged with carefully monitoring the telecommunications industry have all concluded that wireless prices in Canada are dropping quickly. According to the CRTC, the average price of wireless plans declined by 37 per cent between 2016 and 2019.

Statistics Canada’s Cellular Service Price Index has declined 23 per cent since January 2019, in contrast to its all-item Consumer Price Index, which shows the cost of all goods and services increased. Finally ISED’s quarterly price monitoring has found that most wireless plans surveyed have decreased between 10 and 18 per cent compared to benchmark prices collected in early 2020.

Even as prices decline and usage soars, Canada’s telecom network operators continue to invest billions each year in expanding Canada’s digital infrastructure to underserved communities and ensuring that Canada maintains its global leadership in quality of service by deploying next-generation technologies such as 5G. Largely due to these investments, Canada’s telecom industry contributes more than $74 billion in GDP and supports more than 630,000 jobs across Canada.

As the COVID pandemic has highlighted, Canada’s economic well-being, safety and quality of life depend on high-quality digital infrastructure. Making world-class telecommunications services available to all Canadians at affordable prices remains the focus of our members.

Flawed data leads to flawed conclusions.

Canada’s telecom policy needs to be based on high-quality reports and studies.

3 thoughts on “When flawed data leads to flawed conclusions”

  1. Michael Connolly

    The Telecommunications Policy Review Panel in its 2006 report lamented the dearth of policy research capability at Industry Canada and in Canada generally. Like many of the Panel’s recommendations this one was ignored:

    “Industry Canada should make a multi-year commitment to fund ongoing policy research to support improved policy making and regulation in the telecommunications and information and communications technology sectors. Research grants should be awarded by a qualified, independent panel, and the research results should be made publicly available in a timely manner.”

    The politically-minded would rather strike populist policies that play on the importance that Canadians attach to using their devices and the internet in an effort to make points. It’s a credit to the industry that it has done as well as it has. Unfortunately the outlook for evidence-based rational policy looks dim.

  2. Mark,

    Personal experience says something different. From the perspective of mobile, when traveling for our regular extended trips I use a dual SIM phone and purchase a local SIM card. I just went over the last six years of travel (2014-2020) and found that I paid on average $30 CAD (All in including taxes etc.) for a month of travel service. That included all you can eat data, wide area long distance (Europe wide if you use Vodaphone, US-Canada-Mexico with AT&T), Text ad nauseum… In Canada I pay about $90 per month for my mobile, am limited to 1GB data, and have unlimited Canada wide calls and text. And Rogers has a nasty habit of throttling mobiles to 3G service without informing you.

    For fixed broadband Rogers costs have doubled in same period (though I might add so have their speeds). Bell (at our cottage) is no better in terms of cost and its provision of service is minimal (10Mbs down 1Mbs up). They have not made a single improvement on speeds in nearly 7 years.

    While no doubt there are probably flaws in all types of wide collection cost studies I have to tell you that personal experience says we are paying way to much for the service we get in Canada at least on a personal basis.

  3. Personal anecdotes are no substitute for proper research. If you are paying $90 for a mobile plan with just 1 GB of data, you are a few years out of date.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top