Journal of Knowledge Management Practice, March 2001

Data Quality Assessment Methods in Healthcare Information Systems

Robert Jameson, CDS Research; Professor Daniel P. Lorence, Pennsylvania State University;
and Rick Churchill, Virtual Management Institute

ABSTRACT:

Historical attempts to impose rigid data-based practice and performance standards in the delivery of healthcare have met with resistance from providers, though the personalized nature of healthcare practice required little need for rigorous data quality assessment in daily practice. The rapid adoption of evidence-based decision support systems at the provider level, however, now suggests that the application of data quality improvement is less likely to be found objectionable in the establishment of standards for information application and management in health care.  Such an information-intensive environment requires a system of formal, continuous data quality assessment in service delivery and quality management.


Data Quality Assessment Methods in Healthcare Information Systems 

On analysing types of methods favored by early adopters of formal data quality assessment in healthcare organizations,  one sees that historically, a reactive, problem-based approach has been the norm for the assessment and measurement of information accuracy. With the emergence of evidence-based, data-driven medicine in the U.S. however, the representation of the patient, and ultimately the organization’s measured quality of care, is primarily represented by data rather than intensive, personal interaction with the patient (1994). As such, the quality of data maintained by the organization becomes a critical factor in the ultimate delivery of care, and the need for more rigorous quality assessment methodologies becomes apparent, requiring the inclusion of assessment of data comprehensiveness, consistency, currency, relevance, and timeliness (Agmon, 1987).

Past attempts to impose rigid data-based standards in the application of clinical judgement have met resistance from healthcare providers, and thus the need for rigorous data quality assessment was of little consequence. Today, with the increasing deployment of decision support systems at the provider level, application of data quality improvement is less likely to be found objectionable in the establishment of standards for information application and management in health care, in both the care and treatment of patients as well as the management of the system where they are treated .  Such an information-based scheme now requires a system of continuous data collection, evaluation, feedback, and adjustment of the health care process (Ballou, D. 1985).

A number of quality improvement measures have been attempted in response this development, including the use of health outcomes data in clinical practice, which are objective measures of post-treatment health, based on defined criteria . Commonly, large outcomes databases of similar procedures are maintained and analyzed statistically to provide the patient with an objective prognosis for success or failure (Fennell, 1988). So a patient contemplating heart surgery, for example, can compare himself to other patients with similar characteristics who have undergone the surgery, and obtain a statistical prediction of success with the surgery, as well as chances for survival without it. Outcomes databases are being compiled by surveying patients after surgery on their quality of life, which examines indicators other than medical symptoms, such as mobility or social activity. This information is becoming more accessible to office-based providers for use as reference points or data quality benchmarks in providing day-to-day treatment. 

The availability of rich data sets makes it likely that U.S. medicine will continue to move to data-drive, evidence-based medicine. Sophisticated Electronic Data Interchange (EDI) networks, as well, are being more widely adopted as both automated claims processing systems and financial analysis resources.  Data quality indicators have been developed related to the standardization of patient information and the formatting and submission of data, along with payment standards through electronic funds transfer and communication with providers through electronic mail. Likewise, standards for uniformity of clinical records are actively being pursued . Such systems are designed to create a paperless claims system, and create a comprehensive database of healthcare practices and outcomes involving Medicare recipients. The American National Standards Institute (ANSI), among others, are actively developing system standards for this process (Huh, 1990).

The growing adoption of evidence-based medicine further promotes the use of data-driven decision-making, or the use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine relies on integrating individual clinical experience with the best available external clinical research evidence.

In day to day practice, reliable data quality can enhance the judgement that individual clinicians acquire through training and practice. Increased clinical performance is thus achieved in many ways, but especially in better-informed diagnosis and in more complete identification of treatment options and application of an expanded knowledge based in day-to-day practice.

High quality external treatment data, which applies clinical research in routine practice,  is integrated into patient-centered clinical research, incorporating such concepts as   accuracy and precision of diagnostic tests (including the clinical examination), the power of outcomes-based diagnostic indicators, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens.

All types of evidence, ultimately, are derived from raw data, and the quality assurance  of such data is a prerequisite for the adoption of evidence in clinical practice.

Data and Methodology

Data from a nationwide survey of healthcare information managers, the AHA Annual Survey, the U.S. census, Interstudy publications, state and regional health service departments, and The Market Statistics Report were used to examine the organizational and environmental characteristics of data quality in a variety of healthcare settings. A comparison of selected data quality characteristics was made across practice settings, geographic areas, and selected healthcare demographic characteristics.

The survey data include measures of the quality of data obtained via automated encoders, the impact of organizational mergers and acquisitions on data quality, and the existence of an organizational data quality committee or team. The survey further identify the existence of a data quality manager, the most common methods used for identifying data quality problems, and the use of edit checks in automated data fields. Finally, the data provide measures of the existence of organizational master data dictionaries, the existence of organizational policies and procedures related to the timeliness of data capture, and the existence of an organizational process to identify and prioritize data acquisition.  Preliminary findings from the survey were summarized for analysis and are reported elsewhere (1999).

Design and Methodology

The survey, completed in May of 1999, was a response to the need of the healthcare industry for more timely and frequent practice information in the field of health information management No comprehensive study of this kind has been available in the past.

The goal of this undertaking was to establish a source of practical, comparative information that can be used in both strategic planning and day-to-day practice. This project was designed to allow ready access to ideas and innovations that other professionals have used in solving management and technology problems common to many organizations.

This study was designed to provide periodic information on a range of information management topics, such as clinical coding, reimbursement, compliance, health record computerization, and data quality. In addition, a portion of the survey was devoted to collecting information about the professional issues facing practicing managers.

Sample Design

The initial survey was initially fielded in June 1998, with follow-up assessments accomplished through May 1999. It was designed to provide representative information on the population of health information managers. The sample included respondents from a variety of practice settings and job titles and excluded students. The survey obtained data from 16,591 health information managers, for a 50.4 percent gross response rate

Samples for surveys were selected from a database of certified health information managers provided by the Foundation for Record Education , and contained current and historical information on RRA/ART credentialed information professionals in the United States.  The data included on the population surveyed were obtained primarily from membership renewal forms and from an annual member profile mailed to all active members. Preferred mailing address data were obtained from the member population from those members with changes in address or professional status. These changes may be signaled by input from periodic mailings or by other correspondence.

Questions for the main questionnaire were developed from a review of past surveys and focus group results. Questions in the survey were designed so that managers from a variety of practice settings and work roles were asked questions that were generally relevant to the profession overall.

Work settings captured in this study included the following:

Ø      Hospitals and medical centers

Ø      Group practices

Ø      Ambulatory care clinics

Ø      Managed care offices

Ø      Long-term care and rehabilitation facilities

Ø      Colleges and universities

Ø      Consulting firms

Ø      Government agencies

Ø      Software product companies

Ø      Pharmaceutical companies

Ø      Self-employed HIM professionals

Ø      Other work settings

Topics were formulated and pre-tested by convened groups of practicing information managers to represent a broad range of activity areas. Questions that had not been used previously in any known surveys of health information professionals were pre-tested prior to the survey fielding to evaluate the wording and ordering of questions and to determine the ability of respondents to provide the desired information.

Field Procedures

Prior to the survey mailing, announcements were made through professional publications and meetings to inform members of the upcoming survey. A preprinted questionnaire was mailed to all credentialed health information managers who had identifiable mailing addresses. An instruction letter accompanied the form and explained the purpose of the survey and instructions for completion. At the six-week point following the last wave of the initial mailing, a second mailing was sent to those who had not responded.  Follow-up on specific issues identified after the second mailing was accomplished as a series of more focused studies, reported elsewhere.

The questionnaires were processed by an independent testing and research firm, National Computer Sytems of Minneapolis, MN, which processed forms weekly over the length of the study. Region-specific response rates were tracked to ensure that the mailings were received in a timely manner.

Strict adherence to confidentiality standards was maintained in this study. Data were entered via a computerized scanning system and released only in aggregate form, without individual respondents identified within the reported results.

In addition, a number of data quality control measures were employed to provide the cleanest possible data. A detailed review was made of all sample response dispositions, both pending and final, on a weekly basis. From this evaluation, the time schedule was reviewed and necessary recommendations were made to the contractor to enable the survey to be completed during the allotted field period.

As part of the post-survey program review, the design and methodology were examined to identify areas needing improvement. After data entry was complete, an evaluation was made of the impact of survey and item non-response rates and various potential methods for adjusting results to correct for non-response.

Respondents in this study were asked:

1.      What is the most common way you identify data quality problems?

·        Errors reported from analysis of aggregate data

·        Data completeness edits when data is transferred from one system to another

·        Errors reported on individual records

·        User complaints caused by unavailable information

·        Data quality edits when data is transferred from one system to another

·        External data audits

·        Internal data quality audits

2.      What percentage of automated data fields captured employ edit checks to improve data quality?

·        0-5%

·        5-10%

·        10-20%

·        20-30%

·        30-40%

·        OVER 40%

Results

The most popular method of identifying data quality problems is the internal audit. Thirty-two percent of respondents report that this is their preferred quality check, followed by examination of errors reported on individual records (22 percent) and errors reported from analysis of aggregate data (21 percent). The least preferred method is the use of auditors from outside the organization (5 percent).

Fig. 1   PREFERRED METHODS FOR IDENTIFYING DATA QUALITY PROBLEMS

 

 

 

 

 

 

 

 

TOTAL

(A)

 

BASE:  THOSE RESPONDING

 

100%

 

 

INTERNAL DATA QUALITY AUDITS

 

32.1

 

 

ERRORS REPORTED ON INDIVIDUAL RECORDS

 

21.9

 

 

ERRORS REPORTED FROM ANALYSIS OF AGGREGATE DATA

 

21.3

 

 

OTHER

 

24.7

 

 

The use of computerized automated edits appears to be a seldom-used data quality technique. About 60 percent of respondents indicated that 10 percent or fewer of their computerized data accuracy measures use computer edits to analyze data quality. About 15 percent of respondents claimed that more than 40 percent of their computerized data is checked this way.

Internal data quality audits are the most common way “other” practice setting respondents (37 percent) identify data quality problems. This finding is significant compared to hospital (31 percent) and clinic (23 percent) setting respondents. Clinic practice setting respondents (38 percent) reported a significant percentage compared to hospital (19 percent) and “other” (23 percent) practice setting respondents in regard to errors reported on individual records as the most common way they identify data quality problems.

Significant variation exists across several key demographic and organizational settings. 

Non-metro area respondents with a <25K population density (32%) are more likely to identify data quality problems by errors reported on individual records than respondents with a 25K-49.9K population density (25%).  Non-metro area respondents in the 25K-49.9K population density category (26%) are more likely to use an “Other” method as the most common way to identify data quality problems than respondents with a <25K population density (21%).

Fig. 2   POPULATION VARIATION IN PREFERRED METHODS FOR IDENTIFYING DATA QUALITY

            PROBLEMS

 

 

 

 

NON-METRO

 

METRO

 

 

 

 

 

 

 

 

TOTAL

(A)

 

< 25K

(B)

25K-

49.9K

(C)

 

<250K

(D)

250K-

1MIL.

(E)

 

 

> 1MIL.

(F)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

 

INTERNAL DATA QUALITY AUDITS

 

32.1

 

27.2

 

30.1

 

31.2

 

32.4

 

33.8

 

 

ERRORS REPORTED ON INDIVIDUAL RECORDS

 

21.9

 

32.3

C

 

25.4

 

24.5

EF

 

20.5

F

 

17.9

 

 

ERRORS REPORTED FROM ANALYSIS OF AGGREGATE DATA

 

21.3

 

19.4

 

18.6

 

20.2

 

22.4

D

 

22.4

 

 

OTHER

 

24.7

 

21.0

 

25.8

B

 

24.1

 

24.7

 

25.9

 

* letters within cells indicate t-test significance at (p<.05)

 

Respondents in population areas tend to favor aggregated data review for identifying data quality problems. Metro area respondents with a population density of <250K (24%) are significantly different from respondents in the 250K or more population when reporting errors reported on individual records as the most common way to identify data quality problems.  Metro area respondents in the 250K-1 MIL. population category (22%) report a significantly higher percentage than respondents with a <250K population density (20%) for errors reported from analysis of aggregate data as the most common way to identify data quality problems.

High managed care areas favor individual record error tracking for identifying data quality problems. Respondents with 1-10% HMO enrollees (27%) are significantly different compared to respondents with 11% or more HMO enrollees when using errors reported on individual records as the most common way to identify data quality problems.  Using an “Other” method as the most common way to identify data quality problems is significantly different for respondents with 21% or more HMO enrollees compared to respondents with 1-20% HMO enrollees.

Fig. 3   HMO ENROLLMENT VARIATION IN PREFERRED METHODS FOR IDENTIFYING DATA    

            QUALITY PROBLEMS

 

 

 

PERCENT HMO ENROLLEE

 

 

 

 

 

1% -

10%

(G)

11% -

20%

(H)

21% -

30%

(I)

OVER

30%

(J)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

 

INTERNAL DATA QUALITY AUDITS

 

29.4

 

32.8

G

 

32.6

G

 

33.3

G

 

 

ERRORS REPORTED ON INDIVIDUAL RECORDS

 

26.8

HIJ

 

22.0

J

 

19.6

 

18.2

 

 

ERRORS REPORTED FROM ANALYSIS OF AGGREGATE DATA

 

21.1

 

21.9

 

20.5

 

21.6

 

 

OTHER

 

22.8

 

23.3

 

27.3

GH

 

26.9

GH

 

 

Overall, internal data quality audits are the most common way to identify data quality problems for respondents with over 98K hospital inpatient visits (33%), compared to respondents with 1-10.5K hospital inpatient visits (30%).  Respondents with 1-10.5K hospital inpatient visits (27%) report a significantly higher percentage than respondents with 10,501+ hospital inpatient visits, for using errors reported on individual records as the way to identify data quality problems.

High outpatient organzations favor internal audits for quality assessment. Respondents with 10,501+ hospital outpatient visits are more likely to use internal data quality audits as the way to identify data quality problems than the respondents with 1-10,5K hospital outpatient visits (29%).  Errors reported on individual records as the most common way to identify data quality problems is significantly different for respondents with 1-10,5K hospital outpatient visits (29%) compared to respondents with 10,501+ hospital outpatient visits.

Fig.4   PATIENT VISIT VARIATION IN PREFERRED METHODS FOR IDENTIFYING DATA QUALITY

           PROBLEMS

 

 

 

HOSPITAL INPATIENT VISITS

HOSPITAL OUTPATIENT VISITS

 

 

 

 

 

 

1-

10,5K

(K)

10,501

-30K

(L)

30.1K

-98K

(M)

OVER

98K

(N)

1-

10,5K

(O)

10,501

-30K

(P)

30.1K

-98K

(Q)

OVER

98K

(R)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

 

INTERNAL DATA QUALITY AUDITS

 

30.2

 

32.1

 

32.8

 

33.4

K

 

28.6

 

33.2

O

 

33.5

O

 

32.6

O

 

 

ERRORS REPORTED ON INDIVIDUAL RECORDS

 

27.0

LMN

 

21.4

N

 

20.8

N

 

18.2

 

28.9

PQR

 

20.4

 

21.2

 

19.6

 

 

ERRORS REPORTED FROM ANALYSIS OF AGGREGATE DATA

 

19.3

 

21.8

 

22.3

K

 

22.0

K

 

19.6

 

20.4

 

22.1

 

22.2

O

 

 

OTHER

 

23.6

 

24.8

 

24.1

 

26.3

K

 

22.9

 

25.9

O

 

23.2

 

25.7

O

 

 

High managed care expenditure areas tend to prefer audits over individual record review.   Respondents with over $150 in managed care expenditures are significantly different from respondents with managed care expenditures of $0-$125 when using an “Other” method to identify data quality problems.

How are automated edit checks adopted by information managers?

Generally, larger population areas favor automated system edits over record audits. 

The average percentage of automated data fields captured that employ edit checks to improve data quality is significantly higher for non-metro area respondents with a 25K-49.9K population density (14.9%),  than for respondents with a <25K population density (12.4%). 

Metro area respondents with a population density of 250K+ (17.9)% are significantly different, on average, from respondents with a population density of <250K (15%) when reporting the percentage of automated data fields captured that employ edit checks to improve data quality.

Fig. 5   MANAGED CARE EXPENDITURE VARIATION IN PREFERRED METHODS FOR IDENTIFYING

            DATA QUALITY PROBLEMS

 

 

 

MANAGED CARE EXPENDITURES

 

 

 

 

 

 

0

(S)

1 -

125

(T)

126 -

150

(U)

OVER

150

(V)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

 

INTERNAL DATA QUALITY AUDITS

 

31.0

 

34.6

S

 

32.6

 

31.8

 

 

ERRORS REPORTED ON INDIVIDUAL RECORDS

 

25.3

UV

 

22.7

 

20.1

 

19.5

 

 

ERRORS REPORTED FROM ANALYSIS OF AGGREGATE DATA

 

20.2

 

20.6

 

21.9

 

22.4

 

 

OTHER

 

23.5

 

22.2

 

25.4

T

 

26.3

ST

 

 

Likewise, higher managed care areas tend to favor automated system edits over manual record audits. Respondents with over 21% HMO enrollees (17.6%) are significantly different from respondents with under 21% HMO enrollees (15.4% )for the average percentage of automated data fields captured that employ edit checks to improve data quality .

The average percentage of automated data fields captured that employ edit checks to improve data quality is significantly different for respondents with over 30K hospital inpatient visits (17.7%) compared to respondents with 30K or under hospital inpatient visits (15.2%). 

Areas with higher outpatient patient volume also tend to use system edit checks for data quality assessment purposes. Respondents with over 30K hospital outpatient visits (17.7%) are significant compared to respondents with 1-30K hospital outpatient visits (14.9% )for the average percentage of automated data fields captured that employ edit checks to improve data quality.

Fig 6   PERCENTAGE OF AUTOMATED DATA FIELDS  EMPLOYING EDIT CHECKS TO IMPROVE

           DATA QUALITY, BY POPULATION AREA

 

 

 

 

NON-METRO

 

METRO

 

 

 

 

 

 

 

 

TOTAL

(A)

 

< 25K

(B)

25K-

49.9K

(C)

 

<250K

(D)

250K-

1MIL.

(E)

 

 

> 1MIL.

(F)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

 

0-5%

 

42.7

 

54.4

C

 

47.1

 

46.1

EF

 

39.9

 

38.6

 

 

6-10%

 

16.6

 

16.7

 

16.2

 

17.1

 

16.3

 

15.9

 

 

11-20%

 

10.5

 

8.2

 

10.2

 

10.1

 

10.0

 

12.4

DE

 

 

21-30%

 

9.9

 

7.1

 

8.6

 

8.7

 

10.5

D

 

10.7

D

 

 

31-40%

 

5.1

 

4.0

 

4.6

 

4.9

 

5.4

 

5.2

 

 

MORE THAN 40%

 

15.3

 

9.6

 

13.2

B

 

13.0

 

17.9

D

 

17.2

D

 

 

MEAN

 

16.4

 

12.4

 

14.9

B

 

15.0

 

17.8

D

 

17.7

D

 

 

 

MEDIAN

 

8.2

 

5.4

 

6.9

 

7.1

 

9.0

 

9.5

 

 

Fig 7   PERCENTAGE OF AUTOMATED DATA FIELDS  EMPLOYING EDIT CHECKS TO IMPROVE

           DATA QUALITY, BY HMO ENROLLMENT

 

 

 

PERCENT HMO ENROLLEE

 

 

 

 

 

1% -

10%

(G)

11% -

20%

(H)

21% -

30%

(I)

OVER

30%

(J)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

 

0-5%

 

47.4

HIJ

 

42.9

J

 

41.4

 

38.3

 

 

6-10%

 

16.4

 

17.1

 

16.3

 

16.6

 

 

11-20%

 

10.7

 

10.3

 

9.3

 

11.7

I

 

 

21-30%

 

9.1

 

9.4

 

10.6

 

10.1

 

 

31-40%

 

4.3

 

5.4

 

4.8

 

5.6

 

 

MORE THAN 40%

 

12.1

 

14.9

G

 

17.6

GH

 

17.7

GH

 

 

MEAN

 

14.5

 

16.2

G

 

17.4

G

 

17.9

GH

 

 

MEDIAN

 

6.8

 

8.0

 

8.6

 

9.5

 

 

Fig 8   PERCENTAGE OF AUTOMATED DATA FIELDS EMPLOYING EDIT CHECKS TO IMPROVE DATA

           QUALITY, BY PATIENT VISITS

 

 

 

HOSPITAL INPATIENT VISITS

HOSPITAL OUTPATIENT VISITS

 

 

 

 

 

 

1-

10,5K

(K)

10,501

-30K

(L)

30.1K

-98K

(M)

OVER

98K

(N)

1-

10,5K

(O)

10,501

-30K

(P)

30.1K

-98K

(Q)

OVER

98K

(R)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

100%

 

 

0-5%

 

49.0

LMN

 

41.7

 

40.1

 

39.3

 

50.6

PQR

 

42.9

R

 

41.0

 

39.5

 

 

6-10%

 

17.2

 

17.1

 

16.0

 

16.3

 

16.5

 

19.3

OQR

 

15.6

 

16.0

 

 

11-20%

 

9.7

 

11.0

 

10.1

 

11.5

 

8.8

 

12.0

O

 

9.8

 

11.1

O

 

 

21-30%

 

8.1

 

9.6

 

10.4

K

 

10.6

K

 

8.2

 

8.3

 

10.4

O

 

10.6

OP

 

 

31-40%

 

4.4

 

5.5

 

5.5

 

5.1

 

4.4

 

5.2

 

5.3

 

5.3

 

 

MORE THAN 40%

 

11.6

 

15.1

K

 

17.8

KL

 

17.2

K

 

11.5

 

12.2

 

17.8

OP

 

17.4

OP

 

 

MEAN

 

14.0

 

16.4

K

 

17.7

KL

 

17.6

K

 

13.8

 

15.0

O

 

17.6

OP

 

17.7

OP

 

 

MEDIAN

 

6.3

 

8.4

 

9.0

 

9.2

 

5.8

 

7.8

 

8.8

 

9.2

 

 

Fig 9   PERCENTAGE OF AUTOMATED DATA FIELDS EMPLOYING EDIT CHECKS TO IMPROVE DATA

           QUALITY, BY MANAGED CARE EXPENDITURES

 

 

 

MANAGED CARE EXPENDITURES

 

 

 

 

 

 

0

(S)

1 -

125

(T)

126 -

150

(U)

OVER

150

(V)

 

BASE:  THOSE RESPONDING

 

100%

 

100%

 

100%

 

100%

 

 

0-5%

 

46.3

TUV

 

41.9

 

40.9

 

40.4

 

 

6-10%

 

17.6

U

 

16.7

 

15.4

 

16.7

 

 

11-20%

 

9.9

 

9.0

 

11.7

T

 

10.7

 

 

21-30%

 

8.4

 

11.0

S

 

10.6

S

 

9.8

 

 

31-40%

 

4.8

 

5.2

 

5.5

 

5.1

 

 

MORE THAN 40%

 

13.0

 

16.2

S

 

16.0

S

 

17.4

S

 

 

MEAN

 

14.8

 

16.9

S

 

17.1

S

 

17.4

S

 

 

MEDIAN

 

7.0

 

8.4

 

8.9

 

8.8

 

 

Discussion

Overall, where data quality assessment does occur, traditional audits are still preferred as the formal means of measurement in healthcare organizations.  

While information systems now routinely have built in error checking and editing functions, managers may lack the expertise and/or motivation to use them. Likewise, the widespread availability of standardized language dictionaries provide the resources to implement more standardized information management practices, but to date are not being universally adopted (Corman, 1988).

Is there need for further study in this area? The variation in adoption of data quality management practices in healthcare organizations in an emerging data-driven healthcare system suggests that payers, provider organizations, and employers now face the very real possibility that they will be required to maintain coverage of employees long after they leave the company, often under health plans that are inefficient, unaudited, or overpriced (Fennell, 1984).

There is a growing need to train managers in the application of aggregated primary data in their organization’s management structure. One proposed tool employers may utilize in analyzing their health program data, in assessing the quality of their health plans, is the newly-developed Healthplan Employer Data and Information Set (HEDIS). HEDIS is a set of standardized performance measures designed to assure that purchasers and consumers have the information they need to reliably compare the performance of managed health care plans. It is sponsored, supported and maintained by the National Committee for Quality Assurance (NCQA), a not-for-profit organization that evaluates and reports on the quality of managed care plans.  Unlike previous standards, The most recent version of HEDIS allows health plans and employers to measure how well their patients are able to function in their daily lives, rather than simply providing medical classification of treatment.  It also provides subjective measures of patient satisfaction and their perceptions regarding the effectiveness of a given treatment.

While showing great promise, one shortcoming of HEDIS standards is that they are global in nature, developed by broad-based committee consensus which relies heavily on large, nationally-compiled information sources.  This may pose problems in evaluating local trends.

At a more local and regional level, variations in data quality assessment may lead to  gross inefficiencies and significant cost issues under an evidence-based healthcare system, since any comprehensive measure of data quality is incomplete without such  variation accounted for. 

Some studies suggest that data quality assessment under the current system is plagued by wide variations in the methodologies used by health information managers in assessing the quality of information within their organizations.  There may at times be striking differences in the care provided in regions across the country, underscoring both inefficiencies in management practice and the difficulty experienced by providers, payers and policy‑makers struggling to make sense of often-inconsistent information (Kaluzny, 1974).

This has implications for both clinical quality as well as reimbursement, since information on costs for a specific procedure can vary greatly from one city to another. The key question: is this due to actual variation in cost or charges, or simply inaccurate data being reported?  In many cases, the total amount paid for medical services per beneficiary is more than twice as much in some markets as it is in others, even after adjusting for differences in the local prices of services. This is often based upon cost data collected and maintained with questionable information management practices. Overall Medicare payments medical services per beneficiary in the Southeast, for example, has been found to  less than half as much as in the Midwest, even after adjusting for price.  In home‑care services, regional Medicare reimbursement adjusted for price may cost as much as 12 times more per person in Tennessee, compared to areas of Minnesota (Wennberg, 1997).

Regions with relatively high reimbursement for outpatient and home‑care services provided to Medicare patients also provide high inpatient hospital services.  Often, analysts wrongfully assume that if you spend more for ambulatory care, you should cut down on inpatient costs. Frequently, however,  spending in these sectors is completely independent. For example, the rate of coronary‑bypass surgery per 1,000 Medicare beneficiaries in Northern Illinois  has been found to be more than four times the rate in southern Colorado. Similarly, rates of balloon angioplasty procedures for opening diseased heart vessels ranged from 1.6 per thousand in Buffalo, New York, to 12.8 per thousand in Stockton, California. (Wennberg, 1997)

Such comparisons demonstrate sharp differences in data even after adjusting for the actual costs of services in each market. The differences are present even though there is no evidence that patients in one region are sicker or more costly to treat compared to those in another region. This variation suggests that the quality of information, such as local costs, may be less a reflection of actual truth than the need to report data that maximizes reimbursement.

Conclusion

Evidence-based measurement is at the heart of efforts to promote more consistent, and effective data quality management as the US healthcare system grows increasingly reliant on data representations in lieu of extensive patient-provider interaction, This exists in an environment of concurrent utilization of information systems in day-to-day clinical practice. Variation in data errors within the Medicare system, which is based largely on traditional fee‑for‑service insurance reimbursement, are especially timely as Congress seeks to curtail spending growth in that program without cutting necessary services to beneficiaries.

References

Agmon, N., and Ahituv, N. 1987. Assessing data reliability in an information system. Journal of Management Information Systems (4): 34-44.

Ballou, D., and Pazer, H. 1985. Modeling data and process quality in multi-input, multi-output information systems. Management Science (31): 150-162.

Corman, L. 1988. Data integrity and security of the corporate database: the dilemma of end user computing. Data Base (19): 1-5.

Fennell, M.L. 1984. Synergy, influence, and information in the adoption of administrative innovations. Academy of Management Journal.(27): 113-129.

Fennell, M.L. & Warneke, R.B. 1988. The diffusion of innovations. Plenum Press: NewYork.

Huh, Y.,  Keller,F.,  Redman, T.C.,  and Watkins, A. 1990. Data quality. Information and Software Technology (32): 559-565.

Kaluzny, A.D. 1974. Innovation in health services: theoretical framework and review of research. Health Services Research, (9): 101-120. 

Lorence, D. Benchmarking quality under healthcare reform: the next generation. Quality Progress, April 1994.

Lorence, Daniel P.  Compliance & Coding Accuracy in Health Information Management, Journal of the American Health Information Management Association, April, 1999.

Wennberg, John E.  1997.  Dartmouth Atlas of Health Care, Center for the Evaluative Clinical Sciences, Dartmouth Medical School, Hanover, NH.


Acknowledgements

The authors wish to thank Ms. Lee Hersch for her assistance in editing and compiling this manuscript


Contact the Authors

Robert Jameson is President of Community Development Services, Chicago, IL. He can be contacted at CDS Research, Chicago, IL 60610; E-mail rjameson33@hotmail.com

Daniel Lorence PhD currently serves as an Assistant Professor, with appointments in the Department of Health Policy & Administration and in the School of Information Sciences & Technology. He can be contacted at PO Box 1154, State College, PA 16802, Telephone: 814.278.0446; Email: danlor467@yahoo.com

Rick Curchill M.S, is Senior Policy Analyst, Virtual Management Institute, Gettysburg, PA