This page has been archived.

Estimating the Proportion of Health-Related Websites Disclosing Information That Can Be Used to Assess Their Quality

Final Report - May 30, 2006


We undertook the following six research tasks for this project:

  1. Review background materials from ODPHP's prior work on Objective 11-4 and from related online health quality initiatives
  2. Finalize a methodology for determining the denominator of health websites
  3. Develop technical specifications that identify discrete elements required to comply with the disclosure criteria
  4. Create protocols and instruments for reviewing and scoring health websites
  5. Conduct a preliminary test of the protocols and revise them, based on findings
  6. Collect and analyze data for the baseline analysis

In the following sections, we describe our methodological approach to these tasks.

Reviewing Background Materials

To assure that our working assumptions appropriately reflected the prior work of the Objective 11-4 Technical Expert Workgroup, we reviewed background materials the ODPHP Project Officer provided:

  • "Measuring Healthy People 2010 Objective 11-4: Disclosure of Information to Assess the Quality of Health Web Sites; Discussion Paper," prepared by Carol Cronin, Consultant to ODPHP, September 15, 2004
  • "Measurement Options and Approaches; Defining the Denominator - An Estimate of the Number of Health Web Sites," prepared by Carol Cronin, Consultant to ODPHP, September 23, 2004
  • "Measuring Healthy People 2010 Objective 11-4: Disclosure of Information to Assess the Quality of Health Web Sites," September 30, 2004, Technical Expert Workgroup Meeting Summary

We also examined the standards and protocols produced by the two organizations that have developed processes to review and assess the quality of health websites, to assure that our definitions of terms, identified disclosure elements and criteria, and language were consistent with those of these established and recognized quality initiatives. These included:

  • URAC Health Web Site Standards, Versions 1.0 (©2004) and 2.0 (©2006)
  • Health Improvement Institute's health information rating instrument, Version 2.0 (revised April 16, 2005), prepared for Consumer Health WebWatch

This review assured that the assumptions guiding our work and our definitions of terms and concepts were consistent with evolving standards in the field.

Defining the Denominator

Several assumptions guided our definition of the universe of health websites on the Internet (the "denominator") and the sampling strategy for this project. First, we assumed that many (if not most) Internet users search for health information using standard search engines, such as Yahoo or Google, and then visit several different websites identified. Second, we assumed that topics of interest, and therefore search terms, will vary, depending not only on individual interests, but also on health issues in the news. Thus, while a small number of heavily used consumer-oriented health information websites may address common areas of interest or serve as portals to specialized sources of information, any given search might in fact lead the consumer to a large number of websites designed for many different audiences and many different purposes. We therefore defined "health-related website" in inclusive terms consistent with those suggested by URAC, Consumer Health WebWatch, and the eHealth Code of Ethics, to include websites associated with a wide variety of sponsoring organizations that provide information for staying well, for preventing and managing disease, and for making decisions related to health, health care, health products, or health services.3

For the purpose of defining the denominator, we sought to enumerate an inclusive universe of health websites visited by Internet users. We therefore relied on information generated by a commercial vendor (Hitwise) that tracks Internet traffic, in lieu of smaller or more selective lists available from other sources. Hitwise uses information from Internet service providers to monitor traffic and categorizes websites with a minimum number of visits by market type, based on their subject matter and content4. Market share is then reported based on the number of visits to each website within the market, the number of pages viewed per visit, and the amount of time spent per visit.

We purchased data from Hitwise on websites classified in its Health and Medical market category, based on Internet traffic in the United States during October 20055. Although information was also available on subgroups within this market (including the Health Information subgroup), we defined the "universe" for this project as all 3,608 websites within the overall Health and Medical category, in order not to exclude websites that might contain health information consistent with our working definition. Of these 3,608 websites, 213 accounted for 60 percent of all visits within this market. The dataset we purchased from Hitwise included the following information for all websites: domain name, ranking (from 1 to 3,608) within the Health and Medical category (based on number of visits), percentage of market share by visits, percentage of market share by number of page views, and average length of each visit (expressed in minutes: seconds). We used market share defined by number of visits to identify the heavily trafficked sites, since page views and the average length of visits could reflect navigational difficulties rather than user interest. Hitwise also provided domain names and limited information on market share of the top 100 sites within each of 11 subcategories within the Health and Medical category6. We used this information to assure diversity in our sample.

3 The e-Health Code of Ethics defines "health information" as follows: "Health information includes information for staying well, preventing and managing disease, and making other decisions related to health and health care. It includes information for making decisions about health products and health services. It may be in the form of data, text, audio, and/or video. It may involve enhancements through programming and interactivity." Accessed June 19, 2005 at []. This definition was also adopted by Risk, A., and Dzenowagis, J. "Review of Internet Health Information Quality Initiatives." Journal of Medical Internet Research, vol. 3, no. 4, 2001.

4 Hitwise monitors visits to primary domain names or sub-domain names only (that is, the initial part of a URL address up until the first slash). Such URL addresses typically direct users to a website's homepage. Hitwise does not track visits to more specific websites within a domain or sub-domain.

5 Hitwise excluded from its listing any websites that accounted for less than 0.01 percent of Internet traffic within the Health and Medical category during October 2005.

6 Hitwise also reports information on subcategories within each of its larger market categories. The 11 subcategories in the Health and Medical category are Information, Research, Well-being, Primary and Specialist, Pharmaceutical, Pharmacies, Paramedical, Organization, Health Insurance, Hospitals, and Alternative Medicine.

< PREVIOUS     |      TABLE OF CONTENTS     |      NEXT >