John Putzke, PhD, MSPH
http://www.sciencetrax.com

INTRODUCTION

As interest in accountability and end results rapidly increases, outcomes

research has become a multi-billion dollar industry. Both public and private

sector organizations utilize outcomes research as the primary vehicle to

generate new knowledge across a wide range of outcomes.

Outcomes research involves identifying, measuring and evaluating various

effects. The process of performing outcomes research is challenging,

economically costly and labor intensive. Thus, it is important to identify,

understand and to minimize the impact of some of the common pitfalls

associated with conducting outcomes research.

As stated earlier, outcomes research is applicable to a wide-range of

industries. For illustrative purposes within this document, the medical

industry will be used. Indeed, outcomes research is the focal point of

integration between the micro-level and macro-level influences on health.

For clinicians and patients, outcomes research provides evidence about

benefits, risks, and results of treatments so they can make more informed

decisions. For health care managers and purchasers, outcomes research can

identify potentially effective strategies they can implement to improve the

quality and value of care.

The research-related workflow from the initial conceptualization of an

outcomes research project to the final publication of study results involves

a number of often complex steps. At each point along the way various pitfalls

may occur that disrupt the process and thereby introduce increased costs or

delays, or possibly contaminate study results.

Although each step of the research-related workflow deserves its own

discussion, there are four common pitfalls that investigators typically

encounter throughout their career. As an example, when describing the current

process of managing and storing study data, Dr. Stein put it this way:

"...the bioinformatics equivalent of neighboring but isolated medieval nation

states, each with different systems of weights and measures."
Stein, L. (2002). Creating a bioinformatics nation. Nature, 417, 119-120

Now, let's get right to the first of four common pitfalls that will be

discussed in this document.

Common Pitfall #1
REDUNDANT EFFORT

One common pitfall within outcomes research is redundant effort that is

applied from one study to the next.

Consistent with the nature of the scientific method, outcomes research is

generally proposed and performed in a sequential manner moving from one

scientific problem to the next. Indeed, the hallmark of securing funding

resources (e.g., grants, sponsored trials) involves allocation of both monies

and personnel on a study-by-study basis.

The sequential process of scientific investigation serves to narrow the scope

of activities to the immediate needs of the most current study. This process,

then, sets the investigator up for one of the most common pitfalls associated

with conducting outcomes research. That is, redundant effort from one study

to the next.

Limiting the scope of development to one study has contributed to a 'use once

and discard' approach. For instance, very little, if any, of the effort to

produce a study database can be utilized by the next study. As a result,

substantial redundant effort is expended with each new study in designing

systems for collecting, storing, and analyzing data. Moreover, a new learning

curve is encountered with each application developed.
Rarely does the investigator have the luxury to step back, think through, and

develop solutions that can be utilized across studies. As a result, the same

or similar issues are revisited with each new study, serving to substantially

delay the process and reduce time spent on research and analysis.
Moreover, other factors, such as pressing performance timelines and engagement

in other activities, provide further disincentive to develop more

sophisticated and standardized solutions that can be implemented across

multiple studies.

Common Pitfall #2
LIMITED ATTENTION ON PRACTICAL ASPECTS

A second common pitfall within outcomes research is limited attention on the

practical aspects of running a study.

Understandably, investigators get excited about research ideas. Indeed, it is

the ideas, or more appropriately the questions and wanted answers that

provide the drive and energy required to engage in outcomes research.

However, it is tempting to stay within the world of 'ideas' and not work

through a plan to monitor and ensure the practical mechanics associated with

actually running a study and ensuring it is accurately performed.

As a result, errors take place such as subjects falling through the cracks,

inaccurate or incomplete data collection, etc., and begin to add up and

sabotage the study.

The investigator is left with trying to salvage the study, often leaning

heavily on statistical attempts to address methodological limitations. Poor

study management of even the best ideas can delay or limit the findings of a

study or possibly result in falsely rejecting an accurate hypothesis (ie, a

type II error: failure to find an effect when in fact it's there).

Certainly, a central component of conducting clinical research involves

working with people, both those involved with running the study and the

subjects. Thus, to some extent such errors can not be entirely eliminated.

However, poor management of the practical aspects associated with running a

study unnecessarily magnifies the impact of these factors.

Common Pitfall #3
FAILURE TO BRIDGE DATA STORAGE AND DATA ANALYSIS

A third common pitfall within outcomes research is the failure to bridge the

gap between data storage and data analysis.

Advances in information technologies (e.g., electronic data capture devices)

and database applications have served to dramatically increase the volume of

study data that is captured and stored.

As investigators utilize these technologies to address more complex scientific

problems, data management and analysis requires an increasing level of

expertise, often consuming time and resources investigators could spend

focusing on scientific problems.

In many cases, research databases are not developed by investigators or

research personnel, but rather information technology (IT) staff with little

or no training in data analysis.
As a result, both 'what' and 'how' data are stored often has little to do with

how the data is ultimately analyzed. In such cases the investigator is left

with a database that stores a multitude of data that only a few highly

trained staff can actually analyze, resulting in numerous delays and

frustrations when attempting to extract usable data.

Furthermore, statistical packages such as SAS® and SPSS® require a "de

-normalized" format for analysis, whereas the structure of stored data is

typically in a "normalized" format to ensure data integrity. As a result, a

series of complex, time consuming transformations are required to analyze

study data.

These transformations have to be performed on a field-by-field basis. Thus,

data analysis involves multiple, successive formatting hurdles that serve to

significantly delay execution of analytical plans and dissemination of

results.

It should be clear that data analysis, not data storage, is the real challenge

with regard to transforming data into usable information. Anticipating and

facilitating the steps involved in the analytical process and wrapping these

routines within a user-friendly interface that does not require technical

expertise are critical factors with regard to bridging data storage and data

analysis.

Common Pitfall #4
TRYING TO DO TOO MUCH AT ONCE

A fourth common pitfall within outcomes research, and the last one we will

discuss here, is trying to do too much all at once.

The scientific method involves small, incremental contributions of information

by a series of studies that sequentially build upon one another that, when

taken together, contribute to a larger body of literature regarding the

understanding of a given phenomenon.

This "small, step-by-step" process can be quite frustrating for investigators.

The decision with regard to what outcomes to include is a critical component

of the study design.

There is a broad range of outcome measures from which to include, ranging

along a continuum from biological markers (e.g., C-reactive protein) to

Overall Health Related Quality of Life (HRQOL).

A common mistake is the 'shotgun' approach to measurement selection whereby an

investigator chooses to collect 'a little bit of everything' about study

subjects. At first blush this may seem reasonable, but nearly always results

in a number of problems, particularly specificity errors.

That is, failure to include important constructs related to the study's

dependent outcome of interest. In such cases, the investigator is often faced

with trying to explain to a journal reviewer why a particular measure was

left out (ie, not collected) of the analysis.

Even in the case where a 'shot-gun' battery is collected on a longitudinal

basis, the database is typically relegated to "a prospectively collected,

retrospective database." Theory driven research whereby the measures are

specifically chosen on an apriority basis to include all constructs of

interest is the best way to overcome this problem, as well as a bit of

humility that comes with a small, but focused research topic.

SUMMARY

Four common pitfalls within the area of outcomes research have been outlined

here including: (1) redundant effort, (2) limited attention to the practical

aspects of running a study, (3) failure to bridge data storage and data

analysis, and (4) trying to do too much at once. Avoiding these pitfalls

should enhance the quality of study results and increase the efficiency of

conducting outcomes research. Within the health care industry, advances in

information technology will continue to integrate the process of conducting

outcomes research and the results of outcomes research (ie, evidence-based

medicine) into clinical practice. Ideally, large-scale research software

solutions will enable investigators and all research staff to focus on

activities that best utilize their training and leverage the previous and

ongoing work of the entire research community.