Review Of Three Powerful Online Survey Tools

Summary: Comparison of three online survey tools: Alchemer (formerly SurveyGizmo), Qualtrics and Decipher (formerly Beacon).

10 minutes to read. By author Michaela Mora on March 19, 2012
Topics: Market Research, Online Survey Tools, Survey Design

Powerful Online Survey Tools

After posting the comparison of three popular online survey tools’ free versions, many have requested a similar analysis of paid online tools. It seems there is a new tool coming up every day and comparing them all would be a rather difficult task. I decided to compare three online survey tools I have used most recently for different projects.

I have been using Alchemer (formerly SurveyGizmo) for a while now. As I said in my previous post, it is a great value given the number of features it offers at a very affordable price. The tool has been improving with time so it was a natural choice for comparison purposes.

Not long ago, I saw a demo of Decipher (formerly Beacon) and was pleasantly surprised by the advanced features if offered and the business model that supports it. Most online survey tools require either a monthly or annual subscription, but Beacon does not. You only need to pay for completed surveys. I decided to give it a shot and used it to program a pretty complex survey. I had the chance to become very familiar with the tool and thought it was worth including it in this comparison.

Finally, since my above-mentioned post about free online survey tools, I have had many requests asking me to review Qualtrics, which is also becoming quite popular. I happened to use it in two recent projects with very different requirements, which gave me the opportunity to test it in depth.

For a comparison between Alchemer (SurveyGizmo), Qualtrics, and SurveyMonkey check the post: Three Popular Online Survey Tools – What You Get For Your Money.

The list below includes some of the most often-used features in moderate to advanced survey programming and, as you can see below, all three online survey tools are pretty powerful and comparable in many aspects. That said, none of these online survey tools is perfect, and they all would benefit from continuous improvement in capabilities and usability. Most of the time they do the same thing in different ways that have an impact on users’ learning curve.

 

 
General Overview Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Number of surveys Unlimited Unlimited Unlimited
Number of questions Unlimited Unlimited Unlimited
Number of responses Unlimited Unlimited Depends on license
Number of email invitations Up to 500k Unlimited Unlimited
Multi-page surveys
Account users Up to 20 users Unlimited Depends on license
Question types 32 60+ 90+
Graphic survey templates 37

22

+ custom templates

55
Real-time reporting
Data exporting
Data deletion Never Never If account is closed
Supports all languages
Price $75/mo.
or $810/annual subscription

No subscription needed. Pay $0.75/complete. No minimum

From Free trial. $5,000/year
Question Types Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Textbox – Short text
Textbox – Numeric Format Validation
Textbox – Email Format Validation
Essay – Long text
Multiple text fields
Table of text fields
Checkbox
Table of checkboxes
Radio button
Table of radio buttons
Bipolar, semantic differential  
Drop-down menu
Tables of dropdown menus
Ranking – Table format
Ranking – Drag & Drop
Ranking – Top X# of items
Ranking – Stars
Constant sum
Continuous sum
Constant sum table  
Descriptive Text/Image
Image choice (radio button/checkbox)
Video Free if using a streaming service. Hosted for a fee
Contact information form
Date format  
Heatmap
Image/Text Highlighting  
MaxDiff format  
Card Sorting
Stack Sorting  
Question/Page timer  
File upload
Captcha    
Drill down    
Combination of different question types in one table
Survey Formatting Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Customizable font family
Customizable font size
Customizable font color
Customizable background colors
Add/Edit Options in Bulk
Change orientation of labels and options
 
Change position of labels
Editable Next, Submit, and Close buttons text
Editable Next, Submit, and Close buttons graphic elements (color, size, borders)    
Optional question numbering
Fully Customizable XHTML/CSS templates
Can create own CSS templates Simple form
Mobile survey compatible templates
Auto-format as currency, percentage, or a whole number
Editable error messages
Change length, height, and width of textbox or table
Editable survey width
Editable survey after deployment
Min and Max options for numerical answers
Reorder Questions
Randomize the answers
Randomize columns in a table/matrix question
 
Randomize questions within a page
Randomize pages
Set how many checkboxes someone can check
Optional survey tool’s logo
Depends on license
Add your own logo /White label
Create max word size
Separate answer titles and reporting values
Question grouping in blocks
 
Logic Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Question-based skip logic
Page-based skip logic
Skip logic based on hidden variables
Unconditional skip logic
Show and hide questions based on logic
Question trigger on the same page
 
Pipe answer titles or reporting values from a question into another
Repeat questions based on answers to previous questions
Repeat pages based on answers to previous questions
Add values from a URL variable as question options
Add custom script / Javascript
Quiz score with tally
 
Quiz score with pass/fail
 
Make questions required
Make the questions soft-required
 
Random assignment of respondents to different paths/cells/blocks
Quota-based logic
Loops
 
Report Creation Options Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Real-time online report
Crosstabs Counts, Percentages Counts, Percentages, Means, Top 2 Box Counts, Percentages
Statistical Testing   T-Test, Test of Proportions Chi-Square, T-Test, Test of Proportions
Graphing – Use a bar graph, pie chart or spreadsheet to view answers
Export data to Excel, CSV
Export data to SPSS
Browse all responses individually
Can change titles and internal report values
Clear all responses
Share results with others via a link with or without a password
Share results with all respondents by including a link on the Thank You Page
Add a sharing/bookmarking button    
Export report to PDF
Export report to Word
Export report to Excel
Export report to E-mail
Export report to Power Point  
Survey Deployment Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Generate test data
Send email using survey tool
Send reminders using survey tool
Set a cutoff date and time
Set maximum response count
Enable IP blocking
Redirect browser
Link to panel providers
Pass data through survey link
Add a unique survey identifier to each link
Embed surveys into a Webpage
Embed surveys using an iFrame
Collect data in offline mode    
Collect data in Kiosk mode
Convert link into QR code  
JavaScript option embed surveys using JavaScript
   
Question-based quotas
Close survey based on total quota
Remember respondents’ place in survey if completion interrupted
Other Survey Options Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Recode values of answer options
Pipe in values from answered questions into later questions’ answers
Pipe in values from answered questions into later questions’ text
Upload images to internal library
Embed images and media via external link
Prevent search engines from indexing survey
Thank You page
Add a “Close” button to Thank You page  
Back Button Optional
Progress Bar Optional
Create hidden variables
Compliance with Section 508 guidelines of the Rehabilitation Act
 
Export survey to PDF
Export survey to Word
Import SurveyMonkey surveys    
Create mobile-compatible surveys
Import data from other data sources  
Polls Alchemer formerly SurveyGizmo Focus Vision Decipher Qualtrics
Poll templates 37 + Custom   Customizable CSS
Poll Types Single choice, Image Choice, Ranking   Single choice, Multiple Choice
Poll result sharing  
Vote protection By IP address, by cookies, or none   On/Off
Poll Widget sizing  
3rd Party Integration
Open API access   Available for certain accounts
Facebook integration  
Twitter integration  
MailChimp integration    
Salesforce.com integration  
iModerate integration  
ClickTale integration    
ExactTarget Surveys integration    
Freshbooks Surveys integration    
Jive Surveys integration    
Eventbrite integration    
GoDaddy integration    
Twilio SMS integration    
Cint Panel integration    
WhatUsersDo Testing integration    
Support
Support via email
Support via phone
Support – FAQ, Tutorials

Updated December 2020

 

I find Alchemer (formerly SurveyGizmo) to be the most user-friendly and you can do pretty advanced programming with it. They have been adding new question types and integration partners. However, the data export to SPSS and result reporting feature need improvement. For some reason, multiple choice questions get some random variable names that don’t match the variable name specified in the survey, when they get exported to SPSS. Variable names don’t show up in the online reports either. Also, I wished they included statistical testing in their crosstabs.

Qualtrics is the second most user-friendly survey tool of the three. Some things are not obvious. I like the large number of question types. Many are variations of standard questions with horizontal and vertical layouts, with and without images. Among the question types, you will find Constant sum tables, Heatmaps, Hotspots (The latter is pretty useful in concept testing), and MaxDiff (assuming you already have an experimental design). They recently added a new reporting tool, which is still in beta. I like that you can add banner points to the crosstabs and drill down through different question layers, unfortunately, it is still not very intuitive when it comes to create different table types.

Decipher (Formerly Beacon) is a powerful tool, but the learning curve is a bit steep. Thankfully, they have a very responsive technical support, and I was able to get answers to my long list of questions and issues, very quickly. What I liked the most about Beacon is the flexibility it offers to create very complex skip patterns and validations. I also liked its crosstabs. They include counts, percentages, means, and Top 2 Box with statistical testing. Initially, I was mostly bothered by a somewhat cumbersome process for previewing and testing surveys, since I think these are basic features, but they have recently simplified the process and made previewing and testing much easier and user friendly. A minor thing I don’t like, that is more about my own taste than anything else, is that all questions have a table format and there is no way to change that. The tool seems to have been developed by and for programmers since the environment is not very intuitive for neophytes. You can spend a lot of time trying to figure out how certain basic features work.

Regarding cost, each online survey tool uses a different model. Alchemer”s subscription can be monthly or annual (the latter gives you a 10% discount). Beacon doesn’t require a subscription but charges $0.75/complete. This may be a good deal compared to SurveyGizmo as long as your project’s sample or the total sample across different projects is 100 or less. If you go over 100 respondents, SurveyGizmo would be a more economical alternative with unlimited completed surveys.

Qualtrics is the most expensive of the three. The number of completed surveys will depend on license cost and you have to commit for a whole year, so consider your research volume before making such a commitment. However, if you have need of some of their unique question types (e.g. Hotspot), it may be worth it.

As I said before, no online survey tool is perfect. All have quirks when you get into the details. The decision to use one or the other should be based on how well they meet the needs of the project you are working on and what the budget is.