When flawed data leads to flawed conclusions

With the Shaw – Rogers merger announcement, and Parliament debating a bill that threatens to bring a lot of internet content under CRTC regulation, it seems everyone has been forming an opinion on regulatory reform. Unfortunately, there is a lot of misinformation being spread, and opinion pieces being written using deeply flawed reports. Garbage in; garbage out.

I have written much on two sets of reports that seem to keep being cited, despite egregious errors that should be obvious with even a cursory examination. As I wrote last November, studies from Rewheel Research have been thoroughly discredited, with 24 leading academics and telecom economists and lawyers from around the world referring to their reports as a “careless mish-mash of data points from which no reliable conclusions can be drawn.”

I wrote about some obvious problems with the methodology for a UK-based mobile study from cable.co.uk in a piece last summer, “Look at the data”. As I wrote at that time, “It’s very easy to look at a chart on social media, nod one’s head, and retweet or reply without bothering to look beyond the headline. It is tougher to apply a critical eye, look at the data, and determine policy based on deeper analysis.” Sadly, not enough people take the time to look beyond the eye-catching headlines before regurgitating them.

A more recent report from cable.co.uk suffers from the same methodological problems as I documented about the mobile report. One glaring data point jumped out at me. If you are to believe cable.co.uk, the average monthly cost of broadband in Canada is US$76.14, up from US$34.86 last year – an increase of US$41.28. Does anyone believe prices more than doubled last year? Anyone?

That is why I say, “Look at the data”.

A week ago, Toronto Star columnist David Olive appears to have relied on the flawed cable.co.uk data and rankings as the basis for his article “The time is right for Ottawa to fix Canada’s disgraceful telecom system”. By failing to detect the fairly obvious errors in the report, there is no support for Olive’s conclusions.

CWTA president Robert Ghiz had this response published in Saturday’s Star:

Canada’s telecoms among world’s best in overall value
Questionable studies report misleading findings on price of country’s wireless services

Canadian telecom providers offer some of the best and most affordable telecom services in the world.

The Economist’s Intelligence Unit’s Inclusive internet Index ranks Canada third out of 100 countries in internet affordability. Similarly, global accounting firm PwC recently ranked Canada first in the G7 for affordable wireless services.

Unfortunately, these studies are too often ignored. Instead, more attention has been given to one-dimensional and misleading studies that paint an inaccurate picture of telecom prices and affordability in Canada. These include Cable.co.uk’s fixed broadband price study and price comparison reports by Finnish consultancy, Rewheel.

The Cable.co.uk study looks at the median of surveyed broadband plans to develop its country rankings. Using this median price, it claims that broadband fees in Canada are 27 per cent higher than the U.S. But the actual dataset used by Cable.co.uk tells a different story.

Of the Canadian and U.S. broadband plans measured by Cable.co.uk, the cheapest was offered in Canada. By focusing on the median plan, irrespective of which plans consumers choose, Cable.co.uk gives the false impression that Canadians pay more for broadband service.

The questionable quality of the Cable.co.uk methodology is made crystal clear when its 2020 report is compared to its findings in 2019.

For 2020, Cable.co.uk concluded that the average fixed broadband monthly cost in Canada was $76.14 (U.S.) while the average monthly cost in 2019 was $34.86.

Clearly, prices for fixed broadband internet in Canada did not more than double in one year. In fact, in its most recent pricing study, the government of Canada observed that “over the last five years, Canadian broadband prices have trended downwards” and were lower than the U.S.

Meanwhile, Rewheel has been widely criticized for its flawed approach to mobile wireless price comparisons. Last year, 24 leading telecom academics, policy experts and economists released a highly critical review of Rewheel’s methodology, including the recommendation that Rewheel’s reports should come with the same kind of warning labels that social media platforms apply to suspicious information. Similarly, the International Center of Law and Economics published an article cautioning that a Rewheel study examined by the article’s authors amounted to little more than a “careless mishmash of data points from which no reliable conclusions can be drawn.”

In addition to the Economist and PwC reports, other studies have found that Canada’s telecom industry offers superior value. For example, a U.S. industry association-commissioned study found that Canadian wireless subscribers receive more value for their dollar — or “more bang for their buck” — than customers in all other G7 countries plus Australia.

While the studies cited above do not generate the same eye-catching headlines as those which misrepresent telecom prices in Canada, they offer a more meaningful consideration of affordability and value.

Also under-reported is the fact that the three federal government agencies charged with carefully monitoring the telecommunications industry have all concluded that wireless prices in Canada are dropping quickly. According to the CRTC, the average price of wireless plans declined by 37 per cent between 2016 and 2019.

Statistics Canada’s Cellular Service Price Index has declined 23 per cent since January 2019, in contrast to its all-item Consumer Price Index, which shows the cost of all goods and services increased. Finally ISED’s quarterly price monitoring has found that most wireless plans surveyed have decreased between 10 and 18 per cent compared to benchmark prices collected in early 2020.

Even as prices decline and usage soars, Canada’s telecom network operators continue to invest billions each year in expanding Canada’s digital infrastructure to underserved communities and ensuring that Canada maintains its global leadership in quality of service by deploying next-generation technologies such as 5G. Largely due to these investments, Canada’s telecom industry contributes more than $74 billion in GDP and supports more than 630,000 jobs across Canada.

As the COVID pandemic has highlighted, Canada’s economic well-being, safety and quality of life depend on high-quality digital infrastructure. Making world-class telecommunications services available to all Canadians at affordable prices remains the focus of our members.

Flawed data leads to flawed conclusions.

Canada’s telecom policy needs to be based on high-quality reports and studies.