Evidence-based policy making

Over the years, I have tried to draw attention to the need for better, and deeper statistical analysis to help guide policy making in Canadian telecom. I’m not just talking about public opinion polling, especially internet-based surveys that are open to ‘gaming’. We need ongoing econometric analysis of all segments of the communications marketplace.

Frankly, I wonder what kinds of conclusions will be drawn from the CRTC’s current consumer survey, drawn from a completely non-random sample, with such questions as:

• How much data is included in your cell phone plan?
• Do you bundle your cell phone service with other services from your provider?
• How satisfied are you with your current cell phone provider?
• Have you ever switched cell phone providers?
• How difficult or easy was it to switch cell phone providers?
• At the end of your contract with your current cell phone provider, how likely will you be to consider switching to another provider?
• Have you ever told your cell phone provider that you plan to switch to another provider in order to lower your bill?
• How much do you agree or disagree with each of the following statements?
  • I’m happy with the upload and download speeds I get with my cell phone provider.
  • I get good value for money from my cell phone provider.
  • I have a good selection of cell phone providers in my region.
  • The cost of cell phone plans has decreased in the last three years.
  • My cell phone calls are almost never dropped. [A dropped call occurs when you are disconnected from the person you are speaking with due to a cellular interruption]
  • I rarely experience dead zones with my cell phone provider. [A dead zone is an area in which there is no cell phone service available]
• In your view, are Canada’s cell phone prices better, worse, or about the same as what you would find in other countries?

The CRTC’s online survey might have been more useful with a few modifications. For example, the survey asks “In which Province or Territory do you live”, but does not examine the geographic location with any finer resolution, such as the first 3 characters of the respondents’ postal code. It might have also been interesting to ask for the first 6 digits of the respondents’ mobile phone number, to see if any people living in one location are subscribing to a service from a different province.

Many of the questions correlate highly to objective, quantitative data that could be obtained from the mobile service providers. Will the CRTC analyze the responses with such a view?

Just a month ago, in “We need more data”, I wrote “How is Canada supposed to be engaged in evidence-based policy making when there is so little information being gathered about who is online, how Canadians are using the internet and perhaps most importantly, who isn’t online yet and why not?”

The kinds of questions being asked might have more analytic value in quarterly tracking, based on representative sampling. For more than a decade, various arms of the government have been intervening in the wireless market, with spectrum set-asides, consumer codes, sharing rules and more. What kinds of econometric analysis has been undertaken to examine the effectiveness of the competitive measures? Before new measures are introduced, do we properly understand what the current measures are doing?

As the Competition Bureau notes in the conclusion to its November submission to the CRTC, “this competition has not yet reached its full potential and a mandated MVNO policy applied broadly risks undermining the steps taken by wireless disruptors, without much certainty that the MVNO policy will significantly decrease pricing.”

An opinion piece by partners of Boston Consulting Group appearing in the December 3 Globe and Mail observed, “Policy changes can be quick to implement, but take years to undo.”

With Canada’s digital future at stake, we need to ensure policy-making is based on evidence. Maybe it’s time for one of Canada’s public policy schools to undertake the ongoing research needed to help inform the discussion.