Draft Final Report - Prevention Effects Delphi Study

Abstract
Abstract Introduction Methods Results Discussion

The Prevention Research Center (PRC) Program is a network of 23 PRCs and 3 Urban Research Centers (URCs) centers focusing on community-based prevention research centered around research themes that are specific to each Center. The overarching goal of this network includes conducting innovative prevention research; generating new knowledge; and translating knowledge into improved public health practice and policies. In 1997, the Institute of Medicine (IOM) conducted a formal assessment of the PRC Program and pointed out a weakness in documenting the translation of knowledge gained from research into public health programs and policy. To deal with this issues, a computerized knowledge management system, called "Prevention Effects" was envisioned that would be capable of yielding information across a wide range of health problem areas.

As part of a larger planning process for designing the Prevention Effects architecture, a Delphi Study was planned to identify consensus on a number of issues relative to the design of Prevention Effects. The Delphi Process involved a three-round survey of among 40 respondents to gather information on their perceptions of the importance of specific functions and content for Prevention Effects. The participants also made recommendations on prevention practitioners and exemplary prevention programs that might serve as models for gathering information for Prevention Effects. The current report documents the methodology and findings of the three round PRC Prevention Effects Delphi Study.

It was always considered that Prevention Effects would be developed over time - with both functionality and content added at different times in its development. To that end, more than 30 separate functional components were evaluated by the respondents, and by the end of the study 11 stood as candidates for the first version of Prevention Effects in three categories:

Similarly from more than 30 potential content elements, the group rated 10 as important for the first iteration of Prevention Effects. These fell into two broad categories:

Four principal conclusions were drawn from the study, including:

The results of the Delphi provide important input to the CDC and its advisors on Prevention Effects. The recommendations for the first version of Prevention Effects will help guide preparation of a Request for Proposals from potential vendors interested in collaborating with CDC on its development.

Introduction
Abstract Introduction Methods Results Discussion

The Prevention Research Center (PRC) Program is a network of 23 PRCs and 3 Urban Research Centers (URCs) centers focusing on community-based prevention research centered around research themes that are specific to each Center. (1) The overarching goal of this network is best characterized by the interaction of three specific tasks:

  1. Conducting innovative prevention research;
  2. Generating new knowledge; and
  3. Translating knowledge into improved public health practice and policies.

In 1997, the Institute of Medicine (IOM) conducted a formal assessment of the PRC Program. The IOM documented many strengths of the PRC program, while pointing out a weakness in documenting the translation of knowledge gained from research into public health programs and policy: "PRCs have not regularly and systematically reported their findings concerning research dissemination and implementation to CDC, and CDC does not have a mechanism for assembling findings from the PRCs in order to promote such activities." (2)

This conclusion is not an isolated observation. Bill Foege, former CDC Director, has often made the point that only a small portion of what we know in public health is routinely applied either in programs or policies.

Given these circumstances, there were a number of assumptions made about why an investment in knowledge management technology might benefit both the PRC program and the public health system in general:

The Vision of a Web-Based, Prevention Research Knowledge Management System

An electronic system, called Prevention Effects, is envisioned to organize and add value to existing prevention research knowledge. Such a system would be capable of yielding the following information across a wide range of health problem areas. Exhibit 1 contains a summary of the elements of this vision.

Exhibit 1 Information Categories Envisioned for the Prevention Effects System
  1. Scope of the health issue - its prevalence, severity, trends over time, and impact on quality of life.
  2. Methods of health promotion and disease prevention tactics and strategies, classified by effect desired and by relevant demographic indicators.
  3. Health effects likely to result from the effective application of such tactics and strategies.
  4. Costs of those tactics and strategies and their potential impacts beyond health.
  5. Health and social policy implications.
  6. Research gaps and needs.

It was further envisioned that Prevention Effects would include features that would, given appropriate protocols and quality assurance standards, enable users to augment the system. For example, practitioners could access the system and document their experience in applying specific tactics and strategies they drew from the system. This would provide a much needed measure of the utility of a given method or strategy. Likewise researchers, whose findings were used to describe specific tactics and strategies and who learned that their work has influenced practice or policy, could enter such documentation into the system. These application markers are precisely the kind of indicators that the IOM called for in their report, stressing the importance of tracking the extent to which research is indeed being translated into practice or policy.

In June 2000, a planning group was brought together to help identify some of the specific characteristics and functions that would be in the proposed system. This group included representatives from CDC and an Advisory Group of representatives from the PRC's and URC's. As a follow-up to that initial meeting, a Delphi Study was planned to move the group towards consensus on a number of issues relative to Prevention Effects.

Methods
Abstract Introduction Methods Results Discussion

Delphi Process

A Delphi process was chosen because it allows people who are not in the same room to come to consensus on some issue of importance. The Delphi process also ensures that all participants have an equal voice in the outcome because they do not experience the interpersonal dynamics that occur in an in-person meeting. This aspect was particularly important in this effort because the respondents ranged from community organizers to research academicians.

Participants

The initial group invited to participate in the Delphi process were attendees of a meeting to discuss the form and function of the proposed knowledge management system, including CDC representatives (N=10) and members of the PRC Knowledge Management Advisory Group (N=5). In order to get input from a broad range of people who were potential users of the system, the 15 meeting attendees and two advisory group members who could not attend the meeting were asked to nominate researchers and front-line community health workers to invite to participate in the Delphi process. The nominations were based on willingness to respond and capacity to represent some segment of the intended audience for Prevention Effects. A total of 35 people were nominated as well as 8 members of the PRC Steering Committee. Two of the nominees were also on the PRC Steering Committee; therefore 58 people were invited to participate in the Delphi process.

Online Data Collection

The Public Health Informatics Research Laboratory has conducted a number of online Delphi Studies. We prepared survey instruments using html-based "forms." The survey was deployed on a research-oriented website with links to further information about Delphi studies. Participants used a hyperlink to access the online form. Only those who used the correct username and password were able to view and complete the survey.

Rather than using a "mailto" strategy to return the forms via email, we used a cgi-bulkmail program so users did not have to be at a computer configured for Internet mail. Irrespective of computer or browser configurations, anyone having access to the World Wide Web could submit survey responses. Further, the cgi-based bulkmail program automatically stripped unwanted information from a return and entered the respondent's answers directly into a database on the server-side. This strategy also allowed anonymous survey completion.

Participants were asked to respond to any or all of the three rounds of surveys because the timeframe was limited. In follow-up rounds of Delphi studies, it is customary for participants to see how their responses compare to the responses of other participants. Because participants could respond to any or all of the three rounds, the follow-up questionnaire did not include individual ratings. Instead, participants could view their responses to the previous questionnaire by using a hyperlink with a randomly assigned participant number.

Instrumentation

Round One Survey Form

The first round of the PRC Online Delphi Survey was designed to gather basic information building on a brainstorming session at an in-person meeting described above. At that meeting the participants were asked to identify components that they felt would be important to the success of Prevention-Effects. In all, 32 unique ideas were proposed. These initial 32 were broken up into two different groups of system components: (1) Functions of the proposed system - i.e., capabilities the system would have to address the needs of its intended users; and (2) Output / Content - i.e., information that the system would contain, and / or ways the system would provide that information to its intended users. Exhibit 2 shows a copy of the Round 1 Survey Instrument.

Exhibit 2 Round 1 Survey Instrument

The instrumentation for Round One consisted of four components:

  1. Rating the importance of each of the original 15 functions proposed during the initial brainstorming session on a five-point scale as described below*;
  2. Making recommendations for additional functions to be considered for P-E;
  3. Rating the importance of each of the original 17 output / content types identified during the initial brainstorming session;
  4. Making recommendations for additional functions to be considered for P-E.

*Rating was done on a five-point Likert type scale:

Each item in components 1 and 3 had a default response checked, NR for "no response." Each respondent was allowed to leave an NR if they chose. When ratings were analyzed, means and standard deviations for each item were computed. NR was treated as missing data, but the pattern of NR responses was noted in the analysis of the raw data.

Round Two Survey Form

The first step in creation of the questionnaire for round two was to compile the findings from round one. See Exhibit 3 (below) for a copy of the Round 2 Survey Questionnaire. Further analysis was not undertaken, and the preliminary results were not interpreted beyond what is stated below. Delphi techniques are designed to bring groups to consensus. To that end, a second round was constructed which had seven components: *

  1. Re-rating the importance of each of the original 15 functions with knowledge of the respondent's original rating for each item as well as the group's mean rating and standard deviation;
  2. Rating the importance of the newly recommended functions identified from Round 1 responses;
  3. Making final recommendations for additional functions to be considered for P-E;
  4. Re-rating the importance of each of the original 17 output/content items with knowledge of the respondent's original rating for each item as well as the group's mean rating and standard deviation
  5. Rating the importance the newly recommended output/content types identified from Round 1 responses;
  6. Making final recommendations for additional functions to be considered for P-E.
  7. Making specific recommendations to the following questions:
a.  Who are 3 prevention research practitioners (name / contact information) we should interview or profile to develop valuable vignettes on the demands and solutions in prevention research practice?
b.  What are 3 applied prevention research projects (name / contact information) that should be profiled to reveal the important information that does not reach the published literature or evaluation reports (e.g., problems, barriers, unintended consequences, what does not work)?

*In all cases rating was done on the same five-point Likert type scale used in Round One.

Exhibit 3 Round 2 Survey Instrument

Because of the technology used, one day after Round One was closed, the Round Two survey instrument was ready to be fielded. A key component of this second round included providing feedback to each respondent, including their own Round One responses, as well as the group means and standard deviations for each item rated. The Round Two questionnaire displayed these group measures of central tendency next to the item statement.

A hyperlink was created in the Round Two survey to each of the round one responses so respondents who responded to Round One could review their individual feedback before responding to the second round questionnaire. On clicking the hyperlink, the respondent could review his or her ratings in a separate browser window. These responses were keyed to a respondent code known only to the investigators and contained no personal identifying information. If a Round Two respondent clicked on an inappropriate code hyperlink, they would see raw data but have no way to identify the respondent.

Round Three Survey Form

As before, the initial step in the creation of the round three form was to compile the findings from round two. Again, further analysis was not undertaken, nor were the preliminary results interpreted beyond what is stated below. Given these constraints, a third round questionnaire was constructed which had two principal tasks

  1. To prioritize both the functions and the output / content. This prioritization was completed with a straight vote by each respondent on the top ten elements from both lists. With knowledge of the means and standard deviations for each of the importance ratings generated in rounds one and two, each respondent was asked to vote on the top ten functions that should be developed in the first iteration of Prevention Effects, and the top ten content elements that the system should provide in its first iteration.
  2. To identify the principal questions one might ask of the expert researchers and practitioners identified in round 2 and kinds of information that should be gathered about the exemplary programs identified in round 2. To that end, each respondent was asked to submit questions they would ask if given the chance.

As in Round Two, each respondent had a user ID - and with that ID could reach through hyperlinks in the questionnaire itself their Round One and Round Two individual choices. An additional hyperlink was included in the questionnaire that would allow each respondent to examine the list of experts and exemplary programs generated from second round responses. Finally, the technology allowed us to close out the survey at a specified time. Further, the technology allowed each respondent to see, on demand, their earlier responses to specific questions from any previous round. Exhibit 4 contains a copy of the Round 3 Survey Questionnaire.

Exhibit 4 Round 3 Survey Instrument

Results
Abstract Introduction Methods Results Discussion

Overview of Respondents

We began with an initial pool of approximately 30 invitees. Because of heightened interest in both the project and the methodology used for the Delphi study, those originally invited continued to nominate new participants for both Round Two and Round Three. Overall, 40 people responded to at least one of the three rounds, with 28 respondents for Round One, 30 respondents for Round Two, and 29 respondents for Round Three. Exhibit 5 contains a summary of the response patterns among the 40 participants. On examination of Exhibit 5 it can be seen that 42.5% of the respondents participated in all three rounds, but it is important to add that among those respondents in Round One, 82.1% completed at least two rounds, and 60.7% completed all three rounds.

Exhibit 5 Response Patterns for Delphi Participants

Response Patterns

Percent of Respondents

 Percent of Invitees

To all 3 rounds

42.50%

29.31%

To only 2 rounds

32.50%

22.41%

To only 1 round

25.00%

17.24%

Total

100.00%

68.97%

System Function

Exhibit 6 contains a summary of the functions, listed in order of vote preferences from the third round. The list is in descending order - from the function that received the most votes to the function receiving the least votes. Provided in this table as well are the importance ratings (means and standard deviations) from Rounds 1 and 2.

Exhibit 6

A few general observations are in order. First, functions for which "N/A" is listed under Round 1 were not included in the Round 1 questionnaire but were proposed by respondents during Round 1. These then became part of the Round 2 questionnaire. Second, the consensus process would appear to work if the variability in importance ratings decreased in Round 2 compared with Round 1. An examination of the standard deviations among those items found in both Round 1 and Round 2 suggest that this tendency was borne out in almost all cases (12 out of 15 cases - 80%). Third, a high level of correspondence should exist between the Round 2 importance ratings and the voting in Round 3 on whether a function should be delivered in the first version of Prevention Effects. While there is a high level of correspondence, one should not consider a less than perfect correspondence unusual - in part because there is a distinction between "importance" and whether the function is feasible or desirable in the first iteration of the knowledge management system.

The results outlined in Exhibit 6 illustrate that at least 50 percent of the final round respondents identified 12 of the functions that should appear in Prevention Effects Version 1.0. The most highly rated function - that the system should be user friendly - is telling and a basic assumption of system design. The remaining 11 of these functions seem to be organized along three different lines:

  1. Organizational structure: descriptors based on standards of evidence, organizing content by descriptive information, taxonomy based on diseases / health behaviors;
  2. User centered: formatting to user needs, determining user needs, user download capability, user privacy / confidentiality, user browsing;
  3. Adding value to existing information: providing links to additional information, mechanism for user feedback, users rating value of information.

If we include the top functions recommended by about a third of the respondents, we add two more functions to this list: 1) Providing technical assistance and tutorials (which aligns clearly with user centeredness); and 2) Connecting a user with others with similar interest (aligns with adding value to existing information).

Comparisons Between Advisory Committee and Subsequent Invitees

If we look at the opposite end of the list for functions to be added Version 1.0, we find a group of functions that are all related to understanding who the user is and what the user intends to do with the information (determining what the user intends to do with the information through profiling user needs during a session). This raised a question in our minds because 4 of the last 5 voted functions were all recommended as potential functions during the initial planning meeting with the Advisory Committee. As a result, we examined the responses of the two groups (those who participated in the Advisory Committee meeting with those invited to participate in the Delphi following the meeting) in two ways. We first compared the final Round 2 importance ratings (t-test for independent groups) and then examined the voting patterns in Round 3.

Exhibit 7 contains a summary of these comparisons. When comparing the final mean ratings from Round 2 for the intended functions for Prevention Effects, all but two were similar. The only two that were significantly different among these groups were "Profiling user needs during a session" and "Determining what grant opportunities would apply to the user's interests." The former was rated significantly more important by the Advisory Committee, the later significantly more important by the Invitees.

Exhibit 7

Exhibit 8 contains a listing of the top 10 functions according to votes of the entire panel responding to Round 3. Included also in the Exhibit is the voting record for both groups - Advisory Committee members and subsequent Invitees to the Delphi panel. Each of the sets of voting results is based on at least 50 percent of the panel voting for a specific function. For example, "Allowing a user to 'browse' the system received at least 50 percent of the entire panel's Round 3 votes and at least 50 percent of the Invitees' votes, but did not receive 50 percent of the votes of the Advisory Committee.

On examination of Exhibit 8, one can see that the voting patterns were similar, though the cumulative vote order was slightly different between the two groups. For example, several functions that were supported for PE Version 1.0 by the Invitees were not supported by at least 50 percent of the Advisory Committee: content organized by descriptive information; user downloads; user rating of value of information. Only one of the functions supported by the Advisory Committee for PE Version 1.0 (taxonomy) was not supported by at least 50 percent of the Invitees.

Exhibit 8

Output / Content Elements

Exhibit 9 contains a summary of the output / content (elements), listed in order of vote preferences from the third round. The list is in descending order - from the element that received the most votes to the element receiving the least votes. Provided in this table as well are the importance ratings (means and standard deviations) from Rounds 1 and 2.

A few general observations are in order. First, elements for which "N/A" is listed under Round 1 were not included in the Round 1 questionnaire but were proposed by respondents during Round 1. These then became part of the Round 2 questionnaire. Second, as with the functions above, the variability in importance ratings were reduced in Round 2 when compared with Round 1, giving evidence that the consensus process is working. A reduction in the standard deviations among those items found in both Round 1 and Round 2 was was borne out in fewer cases than among the functions (here 12 out of 17 cases - 71%) but still among a majority of elements. Unlike the ratings of the functions, one of these elements (Describing the kinds of interventions that exist related to a theme) had an "increase" (though insignificant) in variability from Round 1 to Round 2. Third, as with the functions above there should be a high level of correspondence between the Round 2 importance ratings and the voting in Round 3 on whether an element should be delivered in the first version of Prevention Effects. However, a less than perfect correspondence should not be considered unusual because of the difference in rating "importance" and selecting elements for inclusion in the first version of the knowledge management system.

Exhibit 9

The results outlined in Exhibit 9 illustrate that at least 50 percent of the final round respondents identified 10 of the elements that should appear in Prevention Effects Version 1.0. To give us some preliminary guidance grouping these 10 elements we ran a principal components analysis on the final vote patterns. This analysis was done purely for descriptive rather than analytic purposes. The results returned two interpretable groupings among 8 of the ten elements. These groupings can be found in Exhibit 10.

Exhibit 10

As illustrated in Exhibit 10, two groupings appear to comprise 8 of the 10 elements. The first group generally included information regarding what might reasonably be expected to happen as a result of planned programmatic and research efforts, and the second group appeared to have more to do with unexpected consequences and barriers to success.

Comparisons Between Advisory Committee and Subsequent Invitees

As with the functions, we examined the responses of the two groups (those who participated in the Advisory Committee meeting and those invited to participate in the Delphi following the meeting) in two ways. First, we compared the final Round 2 importance ratings (t-test for independent groups), and then we examined the voting patterns in Round 3.

Exhibit 11 contains a summary of these comparisons. When comparing the final mean ratings from Round 2 for the intended output / content elements for Prevention Effects all but four were similar. The four that were significantly different among these groups were

  1. "Describing what has been tried,"
  2. "Describing intervening variables and events that influence success / failure,"
  3. "Providing information on who else is asking the same questions," and
  4. "Providing specific feedback on the context and process variables that underlie successful dissemination of programmatic interventions."

The first and third were rated significantly more important by the Advisory Committee, the second and fourth significantly more important by the Invitees. These data are summarized in Exhibit 11.

Exhibit 11

Exhibit 12 contains a listing of the top 10 output / content elements according to votes of the entire panel responding to Round 3. Included also in the Exhibit is the voting record for both groups - Advisory Committee members and subsequent Invitees to the Delphi panel. Each of the sets of voting results is based on at least 50 percent of the panel voting for a specific function. For example, "Describing unintended consequences of actions" received at least 50 percent of the entire panel's Round 3 votes and at least 50 percent of the Invitees' votes, but it did not receive 50 percent of the votes of the Advisory Committee.

On examination of Exhibit 12 it becomes clear that the voting patterns were more dissimilar than the voting on functions. Only five of these elements received at least 50 percent of both groups - with the Invitee group voting for all 10 of the overall top ten elements.

Exhibit 12

Prevention Practitioners and Applied Programs

Respondents in Round 2 were asked to provide suggestions of prevention practitioners who were in a unique position to add information to Prevention Effects. As part of the Round 3 questionnaire all respondents had the option of seeing the list of recommended prevention practitioners generated in Round 2. Respondents were then asked "What questions would you like to have these prevention practitioners respond to?". This was posed in the context of adding additional information to the database of knowledge objects to be created for Prevention Effects. Exhibit 13 contains the list of prevention research / practitioners generated in Round 2. Exhibit 14 contains the follow-up questions generated in Round 3.

Exhibit 13 - - - - Exhibit 14

Similarly, each respondent was asked in Round 2 to identify exemplary applied programs they would recommend for PRC follow up in creation of the knowledge database. Subsequently in Round 3, each respondent was able to see the list of program recommended when asked to supply follow-up questions that might be asked of those programs. Exhibit 15 contains the list of exemplary applied programs generated in Round 2, and Exhibit 16 contains the follow-up questions suggested by Round 3 respondents.

Exhibit 15 - - - - Exhibit 16

At this point, all identifiers other than name and affiliation have been removed from both the practitioner and applied programs list.

Discussion
Abstract Introduction Methods Results Discussion

The purpose of the Prevention Effects Delphi Study was to gather data that would be of use in moving forward with establishing the scope, requirements, and expectations for the Prevention Effects Knowledge Management System. Because of the magnitude of the development effort and the size of the potential knowledge database it was assumed that some form of staging or phasing would be required. The size of the effort is complicated by the number of constituencies to be served in both its development and deployment.

At the current time, it is assumed that there are three primary audiences for Prevention Effects: 1) front line prevention personnel referred to here as practitioners, 2) prevention researchers initially affiliated with the CDC Prevention Research Centers / Urban Research Centers Program, and 3) policy makers likely to influence either the field or practice of prevention.

The study resulted in the generation and rating of 34 functions. Some of these functions overlapped, but clarity and separation came as we moved through Round 2 and 3. At the end of Round 3 we identified a dozen functions that at least 50 percent of respondents felt should be included in the first version of the knowledge management system. These functions were groups into three broad categories:

  1. Organizational structure: descriptors based on standards of evidence, organizing content by descriptive information, taxonomy based on diseases / health behaviors;
  2. User centered: formatting to user needs, determining user needs, user download capability, user privacy / confidentiality, user browsing;
  3. Adding value to existing information: providing links to additional information, mechanism for user feedback, users rating value of information.

As with the functions the importance ratings and final voting produced a set of 10 recommended output / content elements for the first version of Prevention Effects. These appeared to group into two broad categories of elements:

  1. Information regarding what might reasonably be expected to happen as a result of planned programmatic / research efforts: what has been tried; what does / does not work; how challenges are dealt with; unintended consequences of actions
  2. The unexpected consequences and barriers to success: what happened following implementation; barriers encountered and their solutions, broad spectrum interventions; sustainability of interventions.

If we take this as guidance for the purposes of staging it would suggests several things:

There are several issues raised by the results. Perhaps the one that is intertwined with most of the intended functionality of the proposed Prevention Effects System is the issue of profiling the user. The results are clear that profiling is not perceived as one of the early priorities - and far from being viewed as the most important element of functionality. At least two competing explanations could benefit from follow-up examination:

  1. End users are troubled by the potential threats raised to their privacy / confidentiality. A great many horror stories have appeared in today's media outlets about the hazards of online databases. When this is coupled with the prospect that a government agency is compiling information about an end user the concerns might be exacerbated.

  2. The respondents most concerned about profiling might not understand the potential benefits and the prospects for protection of their privacy / confidentiality. As explained to the representatives at the initial Advisory Committee meeting to the participants, profiling is the process of understanding enough about the end user to customize the session and the output for the users' needs and characteristics. The Delphi questionnaire talked about "anonymous user profiling," "session profiling," and "real-time profiling." Each of these is a difficult concept in the abstract and it is possible the subsequent invitees to the Delphi did not have the benefit of the extensive discussions regarding these issues at the initial in-person meeting. This absence could account for the differences in the importance ratings found between the two groups.

Because of the importance of some form of "profiling" to the intended customization processes of Prevention Effects, a in-depth re-examination of this issue is essential to ensure that the two elements of the issue are addressed: First there must be a clear and compelling purpose for profiling, and the end users should be able to understand its importance. Second, irrespective of the potential value of profiling, any schema adopted to provide this function to the system must be able to protect the end user from inappropriate use of the information. This is an issue that should be re-examined with the Advisory Group and vendor as CDC moves forward with its plans.

References

1. For more information on the PRC Program.

2. Institute of Medicine. Linking research and public health practice: a review of CDC's program of centers for research and demonstration of health promotion and disease prevention. National Academy Press: Washington, D.C. 1997. (p. 55)