The first case study discusses an evaluation of public service announcement impacts from the National Lead Information Center's Lead Poisoning Program. The National Safety Council and the U.S. Environmental Protection Agency jointly launched in May 1993 a nationwide advertising campaign to raise public awareness about the dangers of childhood lead poisoning, and to provide information on how to prevent it. The evaluation examines the impact of PSA airplay by the number of calls to the national lead poisoning information hotline (1800LEADFYI) and notes the trends of calls within a 15minute interval following airplay in the cities of New Orleans, Louisiana, and Minneapolis, Minnesota. The analysis suggests that a continued and expanded collaboration between the national campaign effort and local health officials would enhance its effectiveness.
The second case study illustrates how the National Cancer Institute used the focus group method to assess physicians' knowledge and attitudes toward changes in mammography guidelines. Before announcing its decision to drop the mammography guideline for women under age 50, the Office of Cancer Communications (OCC) at the National Cancer Institute (NCI) conducted a series of 7 focus groups with obstetrician/gynecologists (ob/gyns) and primary care physicians.
The purpose of the study was (1) to assess the likely impact of a change in the NCI mammography screening guidelines on physician decision making with regard to mammogram referrals, and (2) to explore ways that NCI can best support physicians in their use of the new guidelines and in their communications with patients. Obtaining a better understanding of the environment in which physicians practice, including their attitudes and concerns about the new guidelines, was considered a prerequisite to developing effective communication materials.
In the third case study, a manufacturer of heart valves informed the Food and Drug Administration of its intent to organize an extensive media outreach and letter notification program to identify, locate, and notify patients whose implanted valves were defective. Statistical data had shown a higher than normal incidence of valve fractures, generally resulting in death.
Evaluation efforts were aimed at assessing the notification program's effectiveness and reviewing several versions of the patient and physician notification letters to clarify the risk communication message. Evaluative findings from the manufacturer notification program were integrated into an FDA document that addresses the usefulness of pretesting notification materials, use of a variety of communication channels to reach the primary audience, and the need to provide more complete risk and risk reduction information to the primary audience.
The fourth case study shows how EPA's Office of Pesticide Programs (OPP) evaluated education materials designed to teach farm workers about pesticide safety. Materials included a poster and a 45page safety booklet. When evaluating educational materials, OPP's risk communication strategy was focused on the literacy level, cultural diversity, and language capabilities of the target audience. Formative evaluation was conducted to gather qualitative information through pretesting methods, including focus groups and individual interviews. OPP found that evaluation is especially important when the EPA is targeting a culturally diverse public with low education, literacy, and income levels. These audiences have special information needs, and communication approaches that work well with one group may not be successful with others.
In the fifth case study, ATSDR evaluated the effectiveness of a series of Community Health Assessment Workshops in exchanging information between the agency and communities nationwide. The goals of the workshops were to provide information to the public on ATSDR's programs and to share ideas on how the agency could provide better service to communities. During the workshops, one-on-one and small group interactions were highlighted. Participants were involved in and provided input to ATSDR's programs, and they networked with other members of the public who shared similar environmental public health concerns. Each workshop was evaluated using a twoto threepage form; items were set up on a Lykert scale with room for written comments on each item. Findings from the evaluation prompted ATSDR to review its philosophy on meeting with community members, both on national and local levels. ATSDR now asks community members how they would like to be involved in the planning of activities that will affect them.
JANET A. PHOENIX, manager, Public Health Programs,
Environmental Health Center, National Safety Council, Washington
The National Safety Council and the U.S. Environmental Protection Agency jointly launched in May 1993 a nationwide advertising campaign to raise public awareness about the dangers of childhood lead poisoning, and to provide information on how to prevent it. The campaign includes educational materials; radio, TV, and print public service announcements (PSAs): and other targeted outreach efforts.
The author of this paper presents a preliminary evaluation of the campaign. It examines the impact of PSA airplay by the number of calls to the national lead poisoning information hotline (1800LEADFYI). This evaluation notes the trends of calls within a 15minute interval following airplay in the cities of New Orleans, Louisiana, and Minneapolis, Minnesota. The analysis suggests that a continued and expanded collaboration between the national campaign effort and local health officials would enhance its effectiveness. These partnerships could have a profound effect on the amount of informational material distributed to the public.
This report presents a preliminary evaluation of the program results and considers the following:
The following evaluation is modeled after the National Academy of Sciences' (NAS) recommendations for CDC's AIDS Education Project. NAS recommends that evaluation be conducted from the initial stages of a health campaign, and it divides the evaluation into three parts: formative evaluation, process evaluation, and outcome evaluation.
Formative Evaluation. Formative evaluation is designed to ensure that only those materials most effective are finally produced for the media. This includes several research activities, most notably concept testing and copy testing, which must be completed prior to the full-fledged inauguration of the campaign.
Formative evaluation was also applied to the public service announcements. A risk communication specialist recommended that the PSAs
Convincing an individual that his or her family is at risk often requires overcoming some common misconceptions:
Process Evaluation. The information on PSA airing included the precise time of airplay, the program schedule, the media market, and the station call letters, and affiliation. The three markets with the greatest number of PSA airings were Sacramento, California; Minneapolis/St. Paul, Minnesota; and New Orleans, Louisiana, respectively. (See graphic below.)
Top 25 Metro Areas airing TV PSAs on Lead
New Orleans and Minneapolis/St. Paul were examined for this report. These two communities exhibited the second and third greatest airplay from August 1993 to October 1993. Sacramento, which had the greatest airplay, was eliminated from the evaluation because of the type of station airing the PSAs (a specialty home shopping network) and because the overlap of the Sacramento and San Francisco viewing audiences made it difficult to discern how much response was generated from a single broadcast on a particular station.
Outcome Evaluation. The outcome evaluation determines if the message presented during the PSA had the desired outcome. In this case, the desired outcome is one step in the process toward our ultimate outcome of a change in behavior. Having people call the 800 number to receive information is a measurable means of learning how many people were reached by the advertisements.
New Orleans. From the PSA monitoring data, we know that the 30second PSA was aired in Louisiana at 3:42 p.m. and 6:29 p.m. on August 27. On that date, 27 calls were placed to the hotline from that area, a 9fold increase from the average of 3 calls per day for the month of August. Sixteen of the calls were placed between 3:43 p.m. and 3:46 p.m., and 4 calls were placed between 6:30 p.m. and 6:38 p.m. The calls all originated from Orleans Parish, Louisiana, which includes New Orleans.
Minneapolis/St. Paul. Similar information was obtained for Minneapolis/St. Paul. On August 14, the PSA was aired four times: 2:37 a.m., 4:56 a.m., 8:57 a.m., and 11:05 p.m. The hotline received 18 calls from Minnesota that day, an increase from an average of 3 per day for August. Of the 18 calls, 1 was apparently in response to the first airing, 3 in response to the 8:57 a.m. airing, and 8 in response to the 11:05 p.m. airing during Saturday Night Live. The calls were placed from Ramsey and Hennepin counties, which encompass the Twin Cities.
From the previous example, it is apparent that the timing of PSA airplay and frequency of calls are closely linked. Some of the PSA airings had no impact on the frequency of calls to the hotline. The increase in calls from the communities of New Orleans and Minneapolis/St. Paul has been related to effective placement of the advertisements.
One of the reasons for the increase in calls may be that both New Orleans and Minneapolis have very active, well established lead education and screening programs. In New Orleans, long standing programs on lead poisoning awareness have influenced the news media. One television station public service director claimed that lead poisoning was prevalent in the area and therefore it was an important public service for the station to play the PSAs indefinitely at least three times a week.
In Minnesota, the State health department capitalized on the national campaign by arranging lead awareness days concurrent with its launch. The health department also sent press kits to local media and conducted followup calls to local TV stations.
The success of the campaign in these communities does not mean that other methods of outreach are less effective. The most effective approach for a particular community is determined in part by the demographics of the people who live there. For example, in a community with a low average education level, written materials may be less effective. Getting to know the community through local contacts and census data is an appropriate place to start.
Another lesson to be learned from this campaign is that the nationallevel activities need to concentrate not only on production of PSAs or educational materials, but also on targeted dissemination and communication at the Federal, State, and local levels.
Communication with local health department officials permits them to "catch the wave" of any national activity. Local health officials are also in a position to be more persistent with local media and draw more attention to important local issues.
The National Lead Information Center believes that the local
efforts have substantially affected the strategic placement of
the ads so that their effectiveness is maximized. In addition,
local community activities can serve to reinforce the message
about lead that is heard in schools, health care centers,
churches, and through Phylicia Rashad's public service
announcements on television. The message is clear-lead is a
problem that may be affecting your children. The various means of
hearing about lead poisoning could be having a synergistic effect
on the public's attitude toward this problem, and therefore may
motivate them to call 1800LEADFYI. The placement of the
public service announcements is just one small part of what needs
to be a coordinated effort at the national and neighborhood
ELLEN EISNER, communications research manager, Office
Cancer Communication, National Cancer Institute, Bethesda. MD.
Before announcing its decision to drop the mammography guideline for women under age 50, the Office of Cancer Communications (OCC) at the National Cancer Institute (NCI) conducted a series of seven focus groups with obstetrician/gynecologists (ob/gyns) and primary care physicians. The purpose of the study was (1) to assess the likely impact of a change in the NCI mammography screening guidelines on physician decision making with regard to mammogram referrals, and (2) to explore ways that NCI can best support physicians in their use of the new guidelines and in their communications with patients. Obtaining a better understanding of the environment in which physicians practice, including their attitudes and concerns about the new guidelines, was considered a prerequisite to developing effective communication materials.
To achieve diversity in the seven focus groups, physicians were recruited based on a number of different variables: their practice setting (i.e., managed care and feeforservice), length of time in practice (minimum of 5 years), patient base (minorities, low income, etc.), and sex. Physicians who had mammography equipment in their offices were excluded from participation in the study. Three of the groups were composed of fulltime practicing ob/gyns or gynecologists; four groups were composed of primary care physicians who were either family physicians or internists. Fifty-five physicians participated in all; approximately one quarter were members of the minority group.
The focus groups were held in four locations: Baltimore, Maryland; Chicago, Illinois; Omaha, Nebraska; and Newark (Wyckoff), New Jersey. Overall, these sites were selected for their geographic, ethnic, racial, and socioeconomic diversity. During the course of the focus groups, participants were asked to react to a number of sample print materials designed to support the introduction of the revised guidelines.
Findings and Lessons Learned
CAROL VEITER, director, Division of Device User
Programs and Systems Analysis, Center for Devices and
Radiological Health, Food and Drug Administration, Rockville, MD.
In 1990, a device manufacturer developed and began implementing a patient and physician notification program to provide information on the risk of heart valve strut fractures. Strut fractures are life threatening and can be successfully treated only by immediate surgical replacement of the valve. At that time, the manufacturer estimated that there were approximately 23,000 individuals in the United States and Canada who still had the valve implanted. Overall, approximately one of three patients with reported valve fractures had survived. The research focuses on activities undertaken to provide for and evaluate the success of efforts to notify patients.
In September 1990, the manufacturer presented a plan to the U.S. Food and Drug Administration (FDA) to provide a grant to an outside organization to implement a program to identify, locate, notify, and enroll individuals with the heart valves in an implant registry. This was the first program implemented under the notification provisions of the Food, Drug, and Cosmetic Act in which a manufacturer sent information directly to patients about a device posing a substantial risk of harm to health. Historically, notification was provided to physicians who were relied upon to decide whether and how to notify their patients about a device problem and steps to take to address the problem. The FDA closely monitored all phases of program development, implementation, evaluation, and modification.
The risk communication effort focused on identifying, locating, and notifying as many heart valve patients as possible.
The manufacturer's communication efforts reached out to patients and their physicians by using public and targeted media, e.g., television, newspapers, consumer magazines, and health professional journals and newsletters. A key notification method was sending an information kit with a cover letter from the manufacturerone directed to patients and one to physicians. Information kits were sent first to physicians. The time lapse between sending letters to physicians and to patients was to allow physicians time to read the letter from the manufacturer and to prepare adequately for discussions with their patients. The physician kit included a sample of the information kit that would be provided to patients. The information provided to physicians included the risk of valve fracture for each of their patients and requested that they send the names and addresses of their patients with the valve to the implant registry. The letter also was designed to provide information to use in medically managing the patient as well as to prepare the physicians for counseling patients about issues that the letters to patients suggested they discuss with their doctors.
A major goal of the notification program was to provide adequate information in the letter to patients to prompt discussions with their doctors about what their risk of valve fracture means for them, the symptoms of possible valve fracture, and what to do if symptoms appeared. Patients also were urged to join the implant registry and were told that the costs of registry membership would be paid by the manufacturer. The implant registry provided patients with a wallet card, bracelet, or neck chain to identify them as having the valve to alert medical care practitioners and aid them in providing appropriate medical care in the event of an emergency. The implant registry also would send any important new information about the heart valve to patients and their physicians.
The FDA closely monitored the findings of the manufacturer's evaluation activities and resulting modifications to the program. The FDA also conducted an audit of the manufacturer's notification program as a check on the findings of the manufacturer's midcourse evaluation. Some key methods employed by the manufacturer and FDA are summarized below.
Key evaluation methods used by the manufacturer included (1) pretesting the letter directed to patients; (2) pretesting the letter to physicians; and (3) conducting a telephone-administered questionnaire directed to a representative sample of 951 patients 6 months into the program. Contractors were hired by the manufacturer to conduct evaluations, analyze the responses, and report the findings of the questionnaires.
The FDA conducted an audit of the notification program based on the agency's authorities under the Food, Drug, and Cosmetic Act. The audit was conducted following the manufacturer's midcourse evaluation. The FDA used a telephone or inperson administered questionnaire directed to a sampling of 318 patients. The interviews were conducted by FDA field investigators throughout the country who had been trained in patient interviewing techniques.
The findings from the FDA survey of patients were used to assess the effectiveness of the notification program with respect to patients' reactions to the program, primary sources of information, and the quality and completeness of information provided to patients.
Based on similar findings from both the manufacturer's midcourse evaluation and FDA's subsequent auditthat a significant percentage of patients said their physicians were not providing counseling on symptoms to look for and actions to take if symptoms appearedas part of a midcourse program correction, the patient letter was modified to include information about symptoms and what to do if they occurred. Also, the manufacturer made available to patients a list of hospitals in the United States and Canada that performed open heart surgery.
Some major findings from the evaluation of the patient notification program include the following:
Evaluative findings from the manufacturer notification program
were integrated into an FDA pointstoconsider document,
Patient Notification Process, for recommending and conducting
direct patient notification programs about medical devices posing
hazards to health. The document is intended to be used by device
manufacturers and by FDA staff for internal decision making and
review of manufacturers' patient notification programs.
Recommendations included in the document address the usefulness
of pretesting notification materials, the value of using a
variety of communication channels to reach the primary audience,
and the need to provide more complete risk and risk reduction
information to the audience.
JOHN LEAHY, environmental protection specialist, Health Effects Division, Office of Pesticide Programs, U.S. Environmental Protection Agency, Washington DC.
In May 1993, the Office of Pesticides Programs (OPP) conducted an evaluation of education materials designed to teach farmworkers about pesticide safety. Materials included a poster and a 45page safety booklet.
Farmworkers are mainly Hispanic, members of other ethnic minorities, or immigrants. In addition to being ethnically diverse, most farmworkers have low education levels and do not speak or read English. One half of seasonal agricultural worker families have incomes below the poverty level. Therefore, when evaluating educational materials, OPP's risk communication strategy was focused on the literacy level, cultural diversity, and language capabilities of the target audience.
Because the educational materials were expected to reach 3 to 4 million farmworkers with language and cultural differences, the EPA wanted to find out if the safety messages could be understood.
OPP funded a grant to the Association of Farmworker Opportunity Programs (AFOP) for a pesticide safety education, communication, and outreach project. AFOP is a national organization that provides job training and other services to farmworkers. For this effort, AFOP contracted with the Program for Appropriate Technology in Health (PATH), an organization that specializes in teaching the development and evaluation of health education materials, to conduct the evaluation and to train AFOP and OPP staff to create more effective outreach materials. Formative evaluation was conducted to gather qualitative information through pretesting methods, including focus groups and individual interviews.
PATH conducted a 2week workshop in Tampa, Florida, to teach AFOP and OPP staff how to conduct focus group discussions and do pretesting interviews with the target audience. With guidance from PATH, AFOP organized 3 focus group discussions with groups of 8 to 10 farmworkers each. The goal of these discussions was to learn more about workers' knowledge, perceptions, and concerns about pesticides.
AFOP and PATH then formed teams to pretest both the poster and booklet with individual farmworkers. Pretesting teams consisted of two peoplean interviewer and a notetaker. This approach keeps the interviewer focused on the respondent while the notetaker records answers and reactions. Since many farmworkers are Hispanic, each team included at least one Spanish speaker. Individual interviews were used to ensure that respondents' answers were not influenced by other people. This technique is particularly important when pretesting materials intended for audiences with low literacy skills. Interviews were conducted at a migrant clinic, a school, a migrant camp, and at other migrant project locations.
For each message, AFOP first pretested the illustration alone (without text) by asking open-ended and probing questions to determine the illustration's understandability and acceptability. AFOP then pretested the illustration and the text together. If the respondent could not read, the text was read to them. PATH recommends revising materials when fewer than 70 percent of respondents understand an illustration alone or when fewer than 90 percent understand the illustration and text together.
Fifty-two farmworkers were interviewed. Of these, 96 percent were Hispanic and only 32 percent spoke English. Most participants work on fruit and vegetable farms around Tampa but migrate to other areas depending on the season and availability of work.
Excluding administrative delays, the evaluation effort took about 2 months to complete and cost $38,000, which included the cost of (1) hands-on training for AFOP staff and the OPP project officer in conducting focus group discussions, pretesting, and material revision; (2) 3 focus group discussions with farmworkers on pesticide safety; (3) individual interviews with 52 farmworkers; and (4) a final report of PATH recommendations based on farmworker interview results.
The farmworkers provided many helpful observations and suggestions for changes to more than 60 graphics and written messages:
Farmworkers also made recommendations on the text. For example, Mexicans often use the term excusado for bathroom while Puerto Ricans are more familiar with inodoro. Because the majority of Spanishspeaking farmworkers are from Mexico, OPP used the term excusado followed by inodoro in parentheses.
The focus group discussions showed that many farmworkers fail to recognize any danger from pesticides. Some workers even said that they had been told pesticides are harmless to people. As a result, OPP decided it was important to stress that pesticides can be harmful to farmworkers.
While farmworkers participating in the study had observed that people who apply pesticides need to use protective gear, they did not see the need to wear protective clothing when working in areas that had been treated with pesticides. OPP decided to increase the emphasis on this message by repeatedly suggesting throughout the booklet and poster to wear appropriate work clothes.
The evaluation experience was rich with lessons that can be extrapolated to future evaluations, regardless of the audience.
CHRIS BRANDT, behavioral program specialist, Office of the Director, Division of Health Assessment and Consultation, Agency for Toxic Substances and Disease Registry, Atlanta, GA.
In 1990, the Agency for Toxic Substances and Disease Registry (ATSDR), a Federal public health agency, met with a group of national environmental activists who recommended that ATSDR "do whatever it takes to become a household word." ATSDR acknowledged that the general public knew little about the agency or what services it could provide and held a series of Community Health Assessment Workshops to exchange information between the agency and communities nationwide.
Workshop participants included people who live on or near sites on the U.S. Environmental Protection Agency's (EPA) National Priorities List (NPL), a listing of hazardous waste sites; representatives of national environmental groups; elected officials; physicians; staff from other Federal, State, and local health and environmental agencies; congressional staffers; representatives of industry; and the media. ATSDR provided travel funds to more than 200 grassroots representatives who otherwise may not have been able to attend.
The goals of the workshops were to provide information to the public on ATSDR's programs and to exchange ideas on how the agency could provide better service to communities. During the workshops, one-on-one and small group interactions were highlighted. Participants were involved in and provided input to ATSDR's programs, and they networked with other members of the public who shared similar environmental public health concerns. Agency staff interacted with the public in a nonsitespecific forum, and heard firsthand the issues of concern to community members.
Each workshop focused on an issue or a group of people who shared specific concerns (e.g., African Americans, Hispanic Americans, people living around U.S. Department of Defense or Department of Energy sites, etc.).
Each workshop was evaluated using a two-to-three page form; items were set up on a Lykert scale with room for written comments on each item.
On the basis of the evaluation data, the following changes were made to the content and format of subsequent workshops:
According to the workshop evaluations, opportunities for ATSDR staff who participated in the workshops included
According to the workshop evaluations, opportunities for members of the public who participated in the workshop included
During the final workshop ATSDR learned that many communities did not want invitations to agency meetings; they wanted to come together at a common or neutral table. They didn't want to be invited to participate in any public health activity planned by the government; they wanted to participate at the ground level to plan and carry out something belonging to both the government and the community. They didn't want to review and comment on documents that affected their communities; they wanted to participate in the decision making.
ATSDR has modified its philosophy on meeting with community members, particularly on a national scale, but also on a local level. ATSDR asks community members how they would like to be involved in the planning of activities that will affect them. ATSDR also invites individuals to collaborate in addressing their needs based on a shared set of guiding principles. ATSDR is not alone in its struggle to resist the temptation to invite communities to participate in developing an agency model, framework, or guideline.
Perhaps the most important lesson ATSDR learned was not as a
result of formal evaluations, but as a result of people willing
to speak their minds. In the future, perhaps we will not depend
entirely on the formal evaluation process (however good it may
be) because people (for whatever reason) don't complete or
elaborate or communicate their depth of feeling on an evaluation
form as they might during a conversation.
Return to Table of Contents
Return to Committee Reports Page