Abacus 101: Making Sense of Polls
Headlines like “Conservatives up 2; Liberals down 3″ or “Consumer Confidence Takes a Hit” appear almost daily in the media. But how do you know if a poll is good or bad? There are few things to consider: what biases could be in the results and how confident are we that the results represent what is actually happening in the population?
Bias in Surveys
Errors and bias can be introduced into a survey the following ways:
1. Question wording – Surveys that include leading or biased questions should not be considered objective research. But sometimes, question wording can bias results without any intention of doing so. For example, there is a debate about how to ask the ballot or vote choice questions on surveys. Some pollsters prompt the respondent and randomly list the parties while others do not prompt and just ask the respondent which party they would vote for. The amount of information included in a question can have small effects on the responses to the question.
2. Non-response bias – If your response rate is low, there could be a chance that the results do not accurate reflect the study population. The best way to think about non-response bias is like this: if the group who answered your survey have different attitudes and opinions than the group who refused to answer, then there is non-response bias. The higher the response rate, the less likely non-response bias will affect the results.
3. Question order – Along with question wording, the order of questions in a survey can influence responses. If a survey measuring support for gun control starts off by asking whether a respondent knows that gun deaths have increased and guns are the leading cause of crime-related death, the respondent is probably more likely to be in favour of gun control than if those comments were not mentioned.
4. Interviewer bias – Interviewers can introduce bias in the way they ask questions, what accent they have, or sometimes whether they are male or female. Moreover, in online research, some studies have found that the way a question is presented can influence responses.
Random Sampling and Margin of Error
One of the primary misconceptions about surveys is that you need a larger pool of respondents for a larger geographic area. In fact, the margin of error is largely dependent only on the sample size, not the population size. In other words, it doesn’t matter if we are trying to represent the views of all adults in Canada, or the views of residents of a single federal constituency – a poll of 500 respondents, randomly selected will produce a margin of error of + 4.5%, 19 times out of 20.
When you see the statement: “The margin of error was + 3.1%, 19 times out of 20″ the pollster is telling you that in 95 times out of a 100 or 19 times out of 20, he or she expects the results of the survey to be reflective of the true answer within the margin of error. In other words, if a survey measuring vote preference finds support for the Conservative Party at 40% and support for the Liberal Party at 30%, a 3.1% margin means that 95% of the time we are confident that Conservative support could be as high as 43% or as low as 37% and Liberal support could be as high as 33% or as low as 27%.
A quick way to calculate margin of error is to divide 1 by the number of respondents and take the square root of the number.
For example, if 550 respondents answered a survey, the margin of error is roughly, + 4.5%.
Abacus Data Inc. is a member of the Marketing Research and Intelligence Association (MRIA) and adheres to the standards and best practices of the association. To find out more about these standards, visit MRIA’s website on:
