August 31, 2010
It is no wonder that with the cost of telephone surveys rising and response rates declining, automated or IVR polling is growing as an alternative method for quantitative research.
A recent story in the Wall Street Journal profiled Scott Rasmussen, the founder of Rasmussen Reports a firm that conducts automated polls. Other firms such as SurveyUSA and Public Policy Polling (PPP) have been using automated polling for years, despite repeated criticism from more established firms and academics on the weakness of the methodology. In Canada, EKOS has been using IVR technology to conduct its much debated national surveys for the CBC.
Automated or IVR survey technology is a telephone survey method that uses a recording instead of a live person to conduct the interview. The survey is conducted inbound or outbound and the respondent presses their telephone key pad to answer the questions.
There are, as with every survey methodology, advantages and disadvantages.
- Response rate – people are less likely to respond to a recording. However, with response rates declining with traditional live telephone interviews, this disadvantage less pronounced. The question is: are respondents to automated surveys any different than those who don’t respond? If yes, we have a problem. If no, then it’s ok.
- Complexity – automated surveys are not meant for long, complicated surveys. Since a respondent cannot ask questions of the interviewer or ask for clarification, they may get confused. This may introduce bias into their responses. IVR is meant for short, simple surveys.
- Representativeness – Most times when conducting a survey using a live interview, the interviewer asks to speak to a specific person based on some formula (last birthday, etc). This increases the randomness of the survey. IVR this cannot be done and so whoever answers the phone fills out the survey. Since men and older people are more likely to answer the phone, the results can be biased towards those two demographics.
- Cost – Since you do not need a call centre full of warm-blooded people manning the phones, IVR surveys can be much cheaper to conduct.
- Socially desirable responses – Since a respondent does not speak with a live person, they may be more willing to answer truthfully to controversial or personal questions. For this reason, a lot of medical research is done using IVR. Tom Jensen of PPP has noted this in explaining the trend that their work has found a higher proportion of people reporting they dislike politicians. “[P]eople are just more willing to say they don’t like a politician to us than they are to a live interviewer because they don’t feel any social pressure to be nice. That’s resulted in us, Rasmussen, and Survey USA showing poorer approval numbers than most for a variety of politicians.”
- No Interviewer Bias – Since each respondent hears the interview, in the same voice and in the same way, there is less of a chance for interviewer bias being introduced into the process. I’ve heard some remarkably bad interviewers pushing respondents to a response by agreeing with a comment they made or introducing new material or events into a conversation. With IVR, there is no chance of poorly trained interviewers biasing the results.
The more I read about IVR the more I see it as an alternative to live telephone surveys. Those who dismiss it cannot argue with the fact that it has been very accurate in the United States. A recent Politico article discusses the recent primaries in Florida and Arizona and IVR polling.
At Abacus Data, we are looking at IVR as an alternative survey method to conduct short, quick surveys.