The Easiest Way to Dramatically Improve the Quality of your Documentation

For those of you writing software, medical, or marketing documentation, there’s something you’re probably not doing that is hurting the success of your documentation. No doubt, you’re a good writer and you’ve gotten technical reviews from subject matter experts to verify the technical accuracy of your writing. If that’s been your review process, you’ve made the mistake of assuming that a technically-accurate document is a usable document. Many times, customers find documents hard to use even though they’re technically accurate. You’ve reviewed your document for technical accuracy, but you’ve not reviewed it for usability. That’s where the documentation usability study comes in.

What follows is a guideline (and cheat sheet) for conducting documentation usability studies. It’s not rocket science but it’s a bit of an art. Follow these guidelines and you will be rewarded with remarkable surprises that will enable you to dramatically improve your document. The result will be a document that is both technically accurate and easy to use.

You can follow the procedure on this page or download the PDF of it.

For more information, see my upcoming book on Amazon.

How to do a Documentation Usability Study

Preparing

1.     Read this entire document first.

2.     Select the content you want to test in your usability study.

There are three kinds of usability studies:

·      Can the user find the content?

·      Can the user perform a procedure?

·      Will the user use a custom, innovative feature you’ve put in the document, such as a clickable tile?

Generally, you’ll combine two or all of them in one study.

a.     Make sure the user can do the task in an hour. Hint: usability studies take longer than you expect.

3.     Find one to five users for your usability study.

Ideally, their role matches the audience persona you’re writing for, e.g., get a system admin to perform the usability study for a system admin document.

a.     Even if you can’t get someone whose role matches that of the audience persona, getting anyone will suffice if they don’t know the product or the documentation. For example:

·      Solution consultant

·      QA engineer

·      Dev

·      PM

·      Fellow writer

b.     The exception is if the user must do highly technical procedures, like writing scripts. In that case, make sure the user has the technical skills.

4.     Invite them to the usability study. Reserve one hour.

a.     You can look over someone’s shoulder as they do the study. Or, you can watch them online using a tool like Zoom. In either case, you’ll record the study.

b.     Include the credentials for the instance the user will use and ask them to confirm that they have access to it well before the study. This step is important.

5.     Minutes before starting the usability study, send the user a link to the documentation you’re testing.

a.     Don’t send them the documentation any earlier. You don’t want the user preparing for the study.

Getting the user ready to perform the study 

You’re now sitting beside or in a Zoom with the user. You’ll want to see both the documentation and the instance they’re working on. If you’re viewing the user online using a product like Zoom, you need to help them set up their screen.

1.     Ask the user to share their screen and display the documentation and the instance in two windows. They can do that by manipulating the windows. Here’s the easiest way to do it on a Mac:

a.     Tell them to select the window with the documentation and then click Windows > Tile Window to the Left of the Screen.

b.     Select the window with the instance and then click Windows > Tile Window to the Right of the Screen.

2.     Now that their screen is set up, read the following to the user:

1.     Thank you for participating in this usability study. The goal is to improve the documentation and make it easy for our customers to use our products.

2.     How this study works is that you put your mind on loudspeaker and please follow the instructions in the documentation. Please say whatever you’re thinking no matter how trivial. For example, you might think, “I don’t like the color of the tile.” Or, “I have no idea what this term means.” Whatever you think, speak it out loud. That will help me get a deeper understanding of where the documentation works and doesn’t.

3.     This study is not a test of your intelligence. Nor is it a test of my intelligence or my ability as a writer. This study is a test of the documentation. For that reason, please be blunt in your praise and criticism. Your comments will not be taken personally.

4.     I will not say anything during the usability test. Except, I might remind you to speak out your thoughts. Otherwise, the environment is supposed to reflect a customer’s workplace in which no one from our company is there to help.

5.     When we begin, I will ask you to complete a task using the documentation.

6.     Do you have any questions?

7.     Then, let’s proceed with the usability study.

Performing the usability study

Basically, you’ll ask the user to use your document to accomplish a task on the instance you provided. Here’s how.

1.     Say,

I’d like to record this session to capture all your feedback. Is that okay with you?

a.     If it isn’t, proceed without recording the session.

2.     State the task for the user to accomplish and put it in the chat.
For example, “Please use <this documentation> to create a landing page in a workspace that shows a real-time list of tasks an agent needs to perform. I’ve entered this task in the chat in case you need to refer to it.”

a.     If your study includes finding the content, it’s crucial that you not lead the user to the text. Make sure you pose a problem the user has instead of giving them the name of a task topic. For example, say, “How would you know that someone has used your application?” instead of, “Can you tell me how to retrieve the number of table inserts in the Application table?”

3.     Ask if they have any questions. If not,

a.     Start the study by saying:
I’ll start recording. Please remember to put your mind on loudspeaker. I’ll be quiet now.

b.    Start recording the session.

 

During the study

1.     Put your microphone on mute. Say nothing. You are “not there.”

a.     If there’s an extended silence, ask, “What are you thinking?” Nothing else. Don’t coach the user. You must get comfortable watching the user struggle.

b.     If the usability study is in danger of completely failing because the user is stuck, you can give them a hint about what to do next. But take note of where the documentation failed.

2.     Write down observations and questions you’d like to ask later, e.g., “I notice that you had trouble with step four. Can you say a little more about what caused you trouble there?”

3.     Keep track of any data you’re specifically testing.
For example, “The user clicked the left tile but not the others.”

4.     End the usability study after 50 minutes.
You need to reserve time for questions and final comments.

5.     When the user finishes the usability study or time runs out, say,
That was great, <name>. I just have a couple of questions.

Asking questions after a usability study

1.     Ask clarifying questions about something the user said.
For example, “You said on the XXX screen that you’d never click the xxx tile. Can you say a little more why?”

a.     Never persecute the user, e.g., “Why didn’t you click the tile?” Instead, say, “What do you think would have happened had you clicked the tile?”

b.     Never lead the user with your questions, e.g., “Do you think it would have been a good idea to use the tile?”

c.     Ask open-ended questions, in other words, questions that can’t be answered with a yes or no. For example, “What was your experience doing the procedure?” instead of, “Did you like the procedure?”

2.     Ask if they have comments about anything else. Say:

Do you have any final thoughts you’d like to share?

3.     Stop recording.

Completing the study

It’s important to thank and reward the user for their time. You want to stress the impact their work will make on your audience and the success of the software product. You also want to reward them, so they’re open to performing usability studies in the future. Say:

I want to thank you for your time. Your feedback will help improve the documentation and make our customers’ experience with the documentation and <product name> better.  I will send you a “Win as a Team” recognition. I’ll copy your manager, so they know.

Again, thank you!

© George Eckel, 2022