Companies are replacing paper surveys with Web-based versions that can dangerously distort the results.
In the course of affirming the district court's decision in Kraft Foods Group Brands LLC v. Cracker Barrel Old Country Store, Inc., 2013 WL 6017396, Judge Posner went a step further. While upholding the injunction, he ended his assessment with some comments "for future reference" when it comes to consumer surveys offered to demonstrate consumer confusion in support of a trademark infringement claim:
"Consumer surveys conducted by party-hired expert witnesses are prone to bias. There is such a wide choice of survey designs, none fool-proof, involving such issues as sample selection and size, presentation of the allegedly confusing products to the consumers involved in the survey, and phrasing of questions in a way that is intended to elicit the surveyor's desired response-confusion or lack thereof-from the survey respondents...All too often "experts abandon objectivity and become advocates for the side that hired them"...it's clear that caution is required in the screening of proposed experts on consumer surveys."
There is certainly the potential for surveys to be misused and/or misinterpreted. Consistent with Judge Posner's advice, care must be taken to make sure your survey results are accurately representative of the true nature and beliefs of the underlying population. Otherwise they may be disregarded due to bias (real or perceived) or simply poor design. Survey inaccuracies generally fall into the following five categories:
The technical aspects of data collection (i.e., the first four items above) require planning and design that is too often mishandled in customized surveys done in support of either marketing or litigation claims. An even larger bias concern exists with question wording and question order (i.e., context/placement). Small question wording/order differences can result in significantly different results between seemingly similar surveys.
There is substantial research that attempts (i) to measure the impact of question wording differences and (ii) to develop methods that minimize differences in the way respondents interpret what is being asked. Some of the items to consider when formulating survey questions include:
As demonstrated above, the courts recognize the potential for surveys and their evaluation to be seriously flawed. A careful practitioner should always take steps to ensure proper methodology and wording/order of the questions. Similarly, when evaluating litigation surveys, careful analysis of the sampling techniques, survey instrument, and data analysis should demonstrate that the results are not biased in favor of any particular position.
Fulcrum Inquiry performs economic and statistical consulting. We prepare and analyze surveys as a means of obtaining data for our consulting assignments when needed information is not otherwise available.
David Nolte is a principal at Fulcrum Financial Inquiry LLP with over 30 years experience performing forensic accounting, auditing, business appraisals, and related financial consulting. He regularly serves as an expert witness.
©Copyright - All Rights Reserved
DO NOT REPRODUCE WITHOUT WRITTEN PERMISSION BY AUTHOR.
THERE'S NO DOUBT that companies can benefit from workplace surveys and questionnaires. A GTE survey in the mid-1990s, for example} revealed that the performance of its different billing operations, as measured by the accuracy of bills sent out, was closely tied to the leadership style of the unit managers.