Merged Datasets: An Analytical Tool For Evidence-Based Management

Effective evidence-based management requires analyzing data from a broad array of sources and conducting carefully designed pretest-posttest comparisons. However, our experience suggests that few businesses take that process to the next level by building merged datasets that can be used for rigorous pretest-posttest comparisons and meaningful statistical analyses. When data are merged from diverse independent sources across a business, researchers can then make evidence-based decisions and run pilot tests with a precision, speed, and breadth that have not been practical until now. Evidence-based management becomes especially useful when researchers build large merged datasets that are progressively linked with each other over time and that include a time series of measurements reflecting past, current, and subsequent performance. This article provides the guidance and background to aid researchers who want to build these merged datasets without further outside assistance.

Researchers in psychology, medicine and health care, education, public health, Computer science, business, and numerous interdisciplinary fields have used and advocated aspects of evidence-based decision making for decades; often, but not always, while citing the respected traditions from which that approach has emerged: quasi-experimental analysis in the behavioral sciences, and evidence-based medicine. A brief word about the theory behind evidence-based management, as well as its origins, will put our expansion of the work into its proper context.

A Brief Note on Theory and Origins

The theoretical underpinnings of our approach trace an interesting story that is rarely acknowledged. Evidence-based management derives its name and method from evidence-based medicine- a field usually attributed to a loosely organized consortium of physician-educators. The consortium's initial report is frequently cited as the starting point for evidence-based medicine. However, few realize that evidence-based medicine was developed to improve the education of physicians, and that the method's assumptions come from a theory of adult learning articulated by Neame and Powis. Although much of the work in evidence-based management is unapologetically atheoretical (as is the related work in quasi-experimental analysis, analytics, and business intelligence) it is occasionally helpful to recall that evidence-based management, like its precursor in the medical area, rests on adult learning theory.

A Merged Dataset: The Essential Tool

The critical tool in evidence-based management is a large merged dataset that welds together a multitude of "hard" performance metrics and "soft" survey data measuring the corporate culture. The familiar and rudimentary uses of such a dataset are measuring performance across the organization and documenting change. The more advanced and less common uses are measuring the impact of programs, identifying and quantifying linkages, capitalizing on positive deviance, discerning emerging trends masked by "background noise" from irrelevant factors, evaluating the effect of specific leadership styles, measuring the impact of communication on profit, or computing the Return on Investment (ROI) of complex interventions where many variables exert their influence simultaneously. The primary process consists of rigorous, methodical pretest-posttest comparisons that many readers typically associate with medical research, public health, or behavioral science.

The first step in building a merged dataset is to locate and combine ("merge" in some computer languages, "join" in others) all the important databases that track a corporation's performance, resources, profits, and expenditures, regardless of their scope, location, and focus. That is, these data are moved across the enterprise into one repository, where each database is aggregated (averaged) by a common indexing variable based on one common unit of analysis (e.g., the organization's ID number) and cross-indexed by time (viz., hour, day, week, month, or year) so that all information can be indexed to a date and a business unit within the company.

Additional rows of new data are concatenated onto the bottom of the dataset at regular intervals (e.g., every week). The dataset also grows by adding new columns containing lagged data that track performance during the previous month and the next month. Accordingly, four data manipulation procedures (merging, indexing, concatenating, and lagging) are used to build the unified dataset. Note that lagging is a two-part process wherein current data lags backward in time (enabling a comparison between this month's and the previous month's performance) as well as forward in time (enabling a comparison between this month's and the next month's performance). These four procedures are really quite straightforward, as Figure 1 illustrates.

People unfamiliar with quantitative methods may believe they are already merging data in their Profit & Loss statement- which seems true enough at first glance. However, the dashboard or scorecard from a conventional P&L summary is purely descriptive, generates no unified dataset, and cannot be used to measure causal linkages. Unlike a conventional P&L smnmary, a typical analysis of a merged dataset provides rich diagnostic and prescriptive information that comes from every domain of the corporation. Proper use of a merged dataset makes it possible to diagnose root causes, measure impacts, evaluate the effectiveness of corporate initiatives, prescribe interventions, and forecast performance. The principles behind this kind of analysis are not new, and often, for example, hinge on accessing and applying the proper covariates for a multivariate statistical analysis. However, the utility, precision, ease, and scope certainly are new especially in the business world, where analytic methodology has lagged behind similar work in the behavioral sciences.

Merged Datasets: Two Brief Examples

. . .Continue to read original article (PDF).