6. Test Your Site with Users with Limited Literacy Skills

Introduction

There’s no surefire way to control who comes to your website—and why would you want to? The web is for everyone.

As mentioned previously in this guide, as many as half of U.S. adults have limited literacy skills—and nearly 9 in 10 have limited health literacy skills.^94,95 And while the information in this guide is a good place to start when designing for users with limited literacy skills, it can’t predict user behavior on a specific tool or website.

That’s why it’s so important to test your websites and digital products with individuals with limited literacy skills.

Usability testing can take many forms. In general, it involves “watching people try to use what you’re creating/designing/building (or something you have already created/designed/built), with the intention of making it easier for people to use or proving that it is easy to use.”⁹⁶

The goal is to get feedback from users to truly understand their motivations and behaviors online. Conducting usability testing will help you answer important questions like:

• Can users find the information they’re looking for?
• Can our users understand and act on this information?
• Are user interface elements intuitive and easy to use?

The specific approach you choose will depend on the goals of your research, how much access you have to your users, and the details of your project, timeline, and budget. Learn more about usability testing.

Below we outline important strategies for you to consider when conducting usability testing with limited-literacy users.

6.1 Recruit users with limited literacy skills—and limited health literacy skills.

Most screening tools designed to measure health literacy skills (like the Test of Functional Health Literacy in Adults and the Rapid Estimate of Adult Literacy in Medicine) have to be used in person and are meant for patients in a clinical setting.⁹⁷

These options may not be practical or very useful for web and health content developers, especially if you’re using a private company’s recruitment database. Instead, you can use a proxy for health literacy based on commonly collected demographic data.⁹⁷ For example, the criteria below could be used as a proxy for identifying web users with limited health literacy skills:

• High school education or below
• Low income—defined as 200% of the federal poverty level or below (or less than $48,500 for a family of 4 in 2015)⁹⁸
• Have not searched for health information online in the past year

You can also try a simple 1-question health literacy measure. As part of screening, ask your participants: “How confident are you filling out medical forms by yourself?”⁹⁹

Keep in mind that some kind of personal connection can make a big difference during recruitment—when you can say that a mutual acquaintance suggested the connection, people are more likely to consider participating in your study.¹⁰⁰

Use trusted community recruiters to help you recruit from community contexts—like adult learning centers, federally qualified community health centers, and senior centers. This can also help you get participants from your target populations.

6.2 Identify and eliminate logistical barriers to participation.

Choose a testing location that is familiar and accessible to participants, like a local library, community center, or even a health clinic. Be mindful of factors that may affect participants’ ability to get to your testing site—like public transportation options or parking availability and cost.

Avoid remote and online testing if possible. If you have to do remote testing, make sure someone is on site with the participant to provide technical assistance.

It’s also important to consider accommodations that your target population may need. For example, if you’re testing with older adults, have reading glasses available for people who might need them.³ Also make sure your testing location is accessible to those with limited mobility. If your target audience’s primary language is not English, you may need to use an interpreter.

Finally, use cash incentives instead of gift cards when possible. Participants may want to use the incentive for basic living expenses like rent or utility bills.

6.3 Create plain language testing materials.

Write your screeners, consent forms, and moderator guides in plain language. Find sample testing documents on Usability.gov.

Limit the number of tasks and questions when conducting usability testing with users with limited literacy skills. Be realistic about what you can accomplish in each session. Make sure the most critical tasks are early in your protocol—it’s best to keep the session brief.

Users with limited literacy skills can have difficulty understanding consent forms—you’ll often find that they’ll sign without reading the form.¹⁰¹ In addition to providing easy-to-read consent documents, consider using a consent process that doesn’t rely on participants’ health literacy skills or English proficiency. You can:

• Review consent forms verbally³
• Ask participants to initial each thing they consent to—for example, written notes, having the screen recorded, or being videotaped

See the Informed Consent and Authorization Toolkit for Minimal Risk Research [PDF – 300 KB] from the Agency for Healthcare Research and Quality for more information.¹⁰²

6.4 Test whether your content is understandable and actionable.

When designing your study, think about ways to find out if your content is both understandable and actionable. To evaluate whether participants understand the content, ask them to:

• Think out loud as they complete tasks
• Describe what they’ve read in their own words
• Describe what action they would take after reading the content

Learning whether users can understand your content is a very important part of usability testing. But it doesn’t stop there—ultimately, your goal is that users will act on the health messages you created. That’s why you also need to find out if your content is actionable.  

With this in mind, don’t limit what you test to measures of comprehension and usability. Include a mix of quantitative and qualitative measures—for example, you can ask users:

• What they would do after reading your content
• How confident they are that they could do what the content is asking

Qualitative measures like these also offer insight into self-efficacy, an important predictor of a user’s ability to adopt healthy behaviors.

6.5 Use moderators who have experience with users with limited literacy skills.

Whenever possible, use moderators experienced in conducting usability testing with people with limited literacy skills or with people who don’t have a lot of experience using the internet.

Moderators familiar with limited-literacy participants in a usability testing setting know what to expect—they’re sensitive to participants’ challenges and know how to effectively support participants through tasks. For example, experienced moderators are more likely to be patient with participants completing tasks very slowly—or they may immediately recognize the need to repeat a task prompt when a participant is getting frustrated.

Ultimately, choosing moderators who have experience working with limited-literacy users can go a long way toward making your testing sessions a success.

6.6 Pretest your moderator’s guide.

It’s very important to pretest your moderator’s guide with participants with limited literacy skills so you can fine-tune tasks and timing. These pretest—or “dry run”—sessions can help you:

• Clarify tasks that were confusing for users with limited literacy skills
• Figure out how much time to leave between sessions so you can accommodate users who may need extra time

6.7 Use multiple strategies to make sure participants understand what you want them to do.

During introductions, make it clear to users that they can reread the task at any time if they’re confused or if they’ve forgotten what to do. 

During sessions, have the moderator read tasks out loud and provide them in writing (only 1 task per sheet). This will remind users of the task you are asking them to accomplish. Also have the moderator repeat the task during the session.⁸⁰

Keep in mind that participants with limited literacy skills tend to focus on the specific task—sometimes to a fault. During sessions, remind users that you’re less interested in the answer and more interested in where and how they would try to find the answer.

6.8 Test on mobile.

Users with limited literacy skills are more likely to access your information on mobile, so always test your website on mobile devices. Testing on mobile will also allow you to collect information about your site’s responsive design. Remember, you want your website to work well on a variety of devices.

Testing on mobile is different from testing on a desktop computer. Here are some things to think about before you test on mobile with limited-literacy users:

• Record movements using a mobile testing set up if you want to review your testing session later or share it with other members of your team. Make sure you think through your technical needs in advance.

• Let participants use their own device. Users are accustomed to how their smartphone or tablet is set up—you’ll get more accurate information if they test your site on a device they’re familiar with.
• Be mindful of data consumption limits. Since you’re asking participants to use their own device, you need to consider data use. Always make sure participants can access free WiFi. And in general, think through what you’ll be asking them to do in advance—for example, will they need to download an app or stream multiple videos? Activities like these may use more data than simpler tasks.

Summary

Conducting usability testing with users with limited literacy skills requires special preparation. Strategies like working with an experienced moderator, eliminating logistical barriers for your participants, and creating plain language testing materials can increase your likelihood of success.