Hi!
We’ve been having the issue with fraudulent surveys responses and I was wondering what safeguards others have taken to protect data quality. This is primarily happening with surveys that have an incentive for participation.
We’ve discussed with partners how we advertise surveys and when possible used local incentives, but keep running into this issue.
Any recommendations for prevention (and data cleaning!)?
what do you mean by fraudulent responses? how do you know the responses are fake?
This is a classic problem, I was just reading about historical fraudulent responses from the Census in the early 1900s. One good way is to have the form you’re using to collect responses track submission time and to have respondents use their first name, last name, and email address to respond. For submission time, you can flag high volume times (many responses within a short period), for email addresses you can identify fraudulent inputs by flagging emails from abnormal domains, and you can identify duplicates in name submissions. Hopefully these help!
Hello Alexandra!
We use Qualtrics for survey data collection, so I can only speak to that. We have also definitely noticed an increase in bots and scammers when it comes to our surveys with incentives.
One thing we’ve done is use the respondent’s latitude and longitude to rule out any surveys not in the region we’re doing research. Almost all of our surveys are either US-only or limited to a state/district/region, and that gives us some things we can use to filter out responses.
Another thing is we’ve used a two-step approach. An individual fills out an initial survey with their contact information and a few questions we think could help filter out bots, and then we review those results. If the response seems legit, we then invite the person to complete the survey directly using the contact info they provided.
We have NOT found captcha’s to be helpful. Unfortunately the bots have gotten much savvier and that no longer seems to be an effective way to filter results.
Hope that’s helpful!
–
Julie
We have been finding this with surveys (and focus groups!) that offer incentives. To add to Julie’s comment above, I’m sharing a blog post we put out a few months ago, with the caveat that we listed captchas in the blog post – looks like it’s time for an update! Stop the Survey Bots! — Sharp Insight, LLC
I also recently took a survey from NPR and it had 3 separate fake questions in it (e.g., “choose the word ‘banana’ from the response options below” or “which of these is not a device on which you can listen to NPR [radio, computer, blanket, smartphone]”).
Good luck!
Kristen
to combat the bot armies, we have started including more free-response questions, which are harder for them to complete successfully without copy-pasting responses
Similar to Kristen’s response, I have a lot of experience administering surveys in psychology research at universities, which often fulfill course requirements for students (and so have incentives attached, as in your case).
We frequently include attention check questions that are intended to be very easy if the participant is doing the task per the instructions. Often these will fit into the theme of the survey so that participants don’t notice they are different, which could prompt them to selectively respond correctly to those questions while clicking randomly through the others. Then we exclude survey responses with less than ~75% accuracy on those attention check questions as likely bots or spammers.
What those questions would look like exactly for your survey depends on the content of the survey, of course. I hope this answer is helpful!
Thanks for all of the thoughtful comments and recommendations!