In my college days, I recall using a textbook called “The Art of Asking Questions” by Stanley Payne, originally published in 1951 [Princeton University Press]. I was pleased to discover that the book continues to be cited by institutions as a guide for questionnaire design. This news provided a modicum of satisfaction for me: I’m not sure that any of my chemistry texts still have value.
You may ask why the interest in my survey research books? Last week, Leger Marketing released a study commissioned by eBay Canada that claims that “Canadians rebuff restrictions on their Internet access.”
Michael Geist had a write-up on the study last week. I just received the study report yesterday and I wanted to wait until I had a chance to see what the survey actually asked before commenting on the results.
In sum, I was disappointed with the poor quality of the study. So poor, that I don’t see how the study helps draw any conclusions. I’m going to ignore the typo in one of the figures that apparently slipped past 4 months of internal reviews. There are fundamental problems with this study that limit its value to lining a rabbit cage.
In many ways, the report shows why real leadership requires policy makers to go far beyond simply responding to public opinion polls. In some cases, I’m not sure it is possible to decipher what was being asked and therefore, how do you interpret their answers.
To start with, the study acknowledges that only a third of the respondents acknowledged familiarity with the concept of net neutrality. The interviewer then went on to define the term:
Net neutrality is the principle that Internet users should be able to access any web content they choose and use any applications they choose, without restrictions or limitations imposed by their phone or cable company that provides the internet service.
All in favour of that definition? Let’s face it, if you heard about net neutrality for the first time (and two thirds of the respondents were in that category), wouldn’t you agree with this definition of the principle? Remember, this is a telephone interview. You don’t get to stop and think about each clause. The last thing you hear is “without restrictions or limitations imposed by their phone or cable company.”
Heck, I think even I would have said that I agree. Maybe even strongly agree.
In fact, I’m surprised there are 20% of Canadians who disagreed – I hope none of them own puppies. Either that, or I have to wonder if these 20% were part of the 34% of Canadians who said they actually understand net neutrality.
In the interest of fairness, how about asking people if they think that the term “access any web content they choose” should include illegal content such as child exploitation images. Of course, blocking illegal content would violate this survey definition of net neutrality. Do you still support it?
How about asking people if they think phone and cable companies should be able to protect other subscribers from users running malicious applications (whether intentionally or unintentionally). That would also violate this definition of net neutrality. Are you still supportive?
So much for the overwhelming support of their definition of net neutrality.
Let’s turn to outright misleading statistics. According to the press release:
Almost three in four Canadians support the current government policy that requires Internet providers to offer net neutrality. Net neutrality is supported equally by men and women.
That is a pretty strong result. So, what was actually asked? Here is how the question was phrased:
Government policy in Canada currently requires all Internet service providers to offer net neutrality. Some Internet service providers would like to change that policy so they can collect fees to direct people toward certain web sites. Would you say that you support the current policy that requires net neutrality or a new policy that lets searches be directed to certain sites?
Whoa!
We need to parse this just a little bit. Right from the start, the interviewer is stating that government policy requires all ISPs to offer net neutrality. What does it mean that the Government requires ISPs “to offer net neutrality”? Is it an option? What definition of net neutrality? What government policy are they talking about that applies to all ISPs? Most ISPs are resellers and are largely unregulated – so is this a policy or a requirement?
I am confused from the start. I haven’t seen any current policy that specifically talks about net neutrality, although for my own definition, no new regulations are needed.
The second sentence, the hypothetical of ISPs wanting to change the mythical current policy to collect fees to direct people to certain websites? Is the question suggesting that jointly operated websites like Rogers Yahoo, or Sympatico with MSN are actually not allowed, and would require a change to policy in order to operate? There is no policy that I am aware of that prohibit your ISP from collecting fees and including a browser pre-configured to go their home page. Other than being confusing, what is meant by that sentence?
But look at the actual question – the final sentence: “Would you say that you support the current policy that requires net neutrality or a new policy that lets searches be directed to certain sites?”
Where the heck did search redirection come from? Did you walk to school or carry your lunch? This is the question that Leger Marketing used to determine that three in four Canadians support the current government policy that requires Internet providers to offer net neutrality?
This is actually the kind of question that statistics professors should be using to demonstrate how not to design a survey. Stanley Payne warns “the penchant of many respondents for answering questions which have no meaning for them poses a major problem for public opinion researchers.”
Do 72% favour keeping the existing mythical legislation or is it really that 72% are frustrated that every application they load keeps trying to change their home page or search engine? Or did respondents really have any idea about what they really answered? Maybe they just wanted to get back to their dinner.
There are other contradictions that should also be highlighted. Here is one of the more glaring sets:
- Only 21% of respondents agreed with a statement: The federal government should allow Internet services providers to charge for Internet access and impose additional fees for access to specific web content. (69% disagreed)
- But, 63% of respondents agreed with: The market should determine the price consumers are willing to pay for internet access. (25% opposed)
I’m confused about the discrepancy in results, considering what these two questions seem to be saying. Should the market determine the price or should t
he government regulate the fees? Notice the word “impose” in the first question? As in, should the government allow ISPs to impose fees? What change would there have been if the expression had been “have optional additional fees”.
Anybody still doubt that the way you ask a question can influence the outcome of a survey?