Workplace Web surveys are increasingly used with - or instead of -print surveys to measure employee motivation, program effectiveness, and staff performance. But few of the companies embracing them are aware of a fundamental problem: The same question posed on the Web and in print can yield very different answers.
It's not that Web surveys are always unreliable. Done correctly, they can produce dependable, even superior, results. But done poorly, as they usually are, they can dramatically distort results, leading management into bad decisions and even derailing careers. Web surveys typically yield higher scores than print surveys, lower response rates, a more restricted range of responses (fewer very high or very low scores), and a host of other distortions. But in our experience designing, executing, and troubleshooting surveys for large companies like Disney, EDS, Xerox, Fallon Clinic, and GM, we've found that each of these problems can be corrected if they are properly understood.
Five types of problems can undermine the validity and reliability of Web surveys.
Opting Out. Response rates for Web surveys can be as much as 80% lower than those for their print counterparts. We've found that employees - reluctant respondents at best- resist Web surveys for a number of reasons, including difficulty accessing the survey, inability moving forward and backward through the questions, difficulty completing an interrupted survey, and fears about confidentiality.
Sugarcoating. Poorly designed Web surveys usually produce implausibly favorable responses. In many cases, the problem stems from employees' reluctance to complain because they're not confident their identity will be protected . Though the causes of this bias are complex, the bottom line is clear: One unreliable Web-based employee performance appraisal we were recently called in to redesign for a national retailer artificially elevated scores by ,6%. That was enough to mislabel "good" performance as "superlative" on an assessment used to set compensation.
Skimming. In the workplace, printed surveys and Web surveys usually attract distinctly different respondents. The typical Web survey user has private access to a computer, holds greater responsibility, and is better paid. When a company offers both print and Web surveys, this self-selection bias means that the Web survey tends to skim higher-level respondents off the top, while lower-level employees stay with paper. Because high-level employees often have a unique, and uniquely favorable, view of their firm, skimming can dramatically skew results.
Clipping. Web surveys tend to elicit responses that are "clipped" - they artificially compress the range between high and low scores. Accordingly, clipped responses can seriously impede analysis by excluding important information, just as a car speedometer would if it only showed your speed between 40 and 60 miles per hour. This is critical in surveys used to measure product quality because they can't distinguish between the fabulous and the pretty good. And it's crucial to performance appraisals, because they can't distinguish between the C and the B- players or the B+ and the A- players, so that people may be unfairly fired or unwisely promoted.
Reshuffling. Web surveys almost always reshuffle rankings of scores. That is, when you compute the average response for each question in your survey and rank those averages from highest to lowest, the ranking from the two formats will most likely be different. This is serious because when the ranking of averages is disrupted, correlations between questions are also disrupted. And it is these correlations that determine the outcome of any analysis examining the links between "50ft" survey responses and "hard" performance metrics- for example, the link between employee motivation and staff retention.
Given the potential drawbacks, Web surveys may seem unduly risky. But a well-designed Web survey can be cheaper, easier to use, faster, better received by participants, and actually more accurate than its paper equivalent. We just completed a study involving nearly 1,000 employees at Duke Energy, an international utility based in North Carolina, that demonstrated Web survey best practices in action. Half the participants used a paper version of a 360-degree leadership evaluation we designed especially for Duke; the other half used a Web-enabled version of the same assessment. The questions were identical in the two formats, but the Web survey incorporated over 40 features designed to neutralize the problems we've just discussed . When we analyzed the survey responses, we found that the data from the two formats matched precisely. Here's how we did it:
Enhancing Access and Ease of Use. To limit the skimming problem and raise response rates, it's imperative to make the interface easy to use, even for those who are marginally computer literate. We recommend putting computers in private settings for employees who don't already have them and eliminating unusual characters from the survey's Web address (such as - and / and \ ) so that typos don't stop users even before they get started. We also suggest adding a navigation tool that lets respondents move forward or backward between questions easily. Other important features include a quick-exit-and-save button on each screen, so users can leave and return easily without losing data; a progress indicator, so respondents know how much more they have to do; and a simple undo capability, so they can revise answers without fuss.
Improving Accuracy. We've found that simple formatting adjustments can improve accuracy and reduce effects like clipping and reshuffling. It's important to center response scales (for 1-to-5 ratings, for instance) on the user's screen and to provide a "Don't Know/ Not Applicable" option that is clearly visible but not overly prominent. Other accuracy adjustments include a lockout feature, which requires participants to provide an answer before allowing them to move on, and an automatic resizing feature so that, regardless of the size of the window on the user's desktop, an entire page of the survey will appear on the screen.
Unlike their print counterparts, Web surveys can provide instant error-checking. For example, respondents sometimes misread negatively worded questions (that is, questions where a "yes" response or a high rating means that something is not good) and mistakenly give a response that is inconsistent with all their other answers. Filters on Web surveys can spot these outliers (and other important, more subtle, anomalies as well) and prompt respondents to confirm or fix them. That's why Web surveys with filters can actually be more accurate than print surveys.
Finally, because concern about confidentiality can undermine accuracy, it's important to move Web surveys off the company's intranet and onto a secure third-party server; it's also advisable to have users create their own passwords.
It's not possible to list here all the problems associated with Web surveys and their solutions. But understanding that Web surveys have particular short comings, ready fixes, and unique strengths can help you use them wisely. After all, as any good carpenter knows, it's important to be clear about the differences between a wooden yardstick and a steel ruler - especially if you intend to use both to build your house.
Dr. Palmer Morrel-Samuels is a Research Psychologist with extensive training and experience in Statistical Analysis and Assessment Design. He has done a considerable amount of research and applied work on communication, testified to the U.S. congress on employee motivation and its linkage to objective performance metrics, published several articles on survey design in Harvard Business Review, among others, and wrote several patents to assist in the administration and analysis of workplace assessments. Dr. Morrel-Samuels currently teaches graduate courses on survey design and research methodology at the University of Michigan.
©Copyright - All Rights Reserved
DO NOT REPRODUCE WITHOUT WRITTEN PERMISSION BY