‘Whizzo on Trial’ User Testing with specific reference to disabled people

humanITy presentation given at UEL Inclusive New Media Design workshop

Date: 16/02/2008
Venue: The Rix Centre, University of East London, London, UK

1. Introduction - Us and Them

Almost everything I am going to say is obvious. I don't think that there is anything controversial. The only reason why some of the things I am saying might sound radical is that there isn't very much user testing by disabled people either of the goods and services which enable us all to lead a full life or even those specifically designed for them. I know you are all experienced but, then, so is the Whizzo Corporation; but it still needs to be reminded of some basics.

1.1 Mainstream. The providers of general goods and services rely heavily on marketing to design products and on intensive user testing; but they tend to stick with what they think of as 'normal customers'.

Scenario 1. A longer handle

Salesman: I'm the Quick Trim lawn mower salesman; Can I help you?
Customer: I would like to buy a lawn mower, please
Salesman: Here's a lovely little one; on special offer?
Customer: But the handle is below waist level and I can't bend.
Salesman: You need a fit young fellow to help with the grass
Customer: You've missed the point. Fit young fellows don't cut grass; they leave it to oldies like me. My son doesn't cut my grass; I cut his!
Salesman: So you need something with a longer handle?
Customer: Yes
Salesman: But it wouldn't look so nice
Customer: Perhaps; but it would cut grass efficiently and you would sell more goods.

1.2 Special. The designers of special goods and services for disabled people aren't much better:

Scenario 2. Redundant Device

Inventor: I'm an inventor. I've brought this special device for wheelchair users
Assessor: I check how useful things are. What does it do?
Inventor: It helps them to change the sail in a transatlantic yacht race
Assessor: How many people are going to need this?
Inventor: I don't know; I've never met a wheelchair user; but if you just clip this to the mast
Assessor: How much does it cost?
Inventor: It's a bit pricey; but if you screw this into the deck
Assessor: Who is going to pay?
Inventor: O, I suppose the Government; they do so much for people these days.

Rule 1. Consult disabled people and the organisations that work with them at the specification stage for the goods or services.

2. Demographics - Who and Why

2.1 Epidemiology - Disabled People Are All Different. There are all kinds of disabled people. We can generally classify ourselves into four groups with the biggest at the top; people with:

Learning and related difficulties;
Physical disability;
Hearing problems;
Vision problems.

2.2 Demographics - Age and Activities. Most disabled people are elderly. The few people disabled from birth or in childhood enjoy high visibility because of their serious problems, courage and fund raising value; and they tend to distort the public perception.

2.3 Capability. Like all other groups of people, disabled people vary widely:

Scenario 3. Wrong Testers

Designer: Im a Whizzo gadget designer. You're just the kind of person we need to test and promote it; you're so pretty!
User: I don't like gadgets
Designer: But you have a lovely television here with a remote controller
User: With my husband and the kids I never get my hands on the remote controller except when they're all out and I only watch one channel
Designer: But you would love this, it helps you send email
User: I don't use email; I don't like computers. Here's my son.
Designer: Hello, would you like to test this starter gadget
Son: Oh Yeh! I've had computer for years init?
Designer: Well you could pretend to be a beginner
Son: How much? Yeh. I just need to see if it's compatible with my OS and how much RAM it takes. Then we can pretend, init?

Rule 2. Test goods and services with a representative sample of likely users, taking account of disability, age and capability.

3. Specifications - What Do We Want to Know?

So we have carefully picked a weighted sample of users likely to need the goods or services we are planning to offer; we haven't selected children to test the Zimmer frames or pensioners to test the games consoles. The next job is to sort out what we are trying to test.

3.1 The System or a Function? - What Are We Testing? We need to work out what we are testing. There is a difference between accessing a web site when there is support available and accessing the same site at home on our own.

Scenario 4. Wrong Place

Librarian: Welcome to the library. I've switched on the computer and loaded the software and used the search engine to find the site you want and now you can just enjoy yourself
User: But what do I do if I'm at home?
Librarian: We're always pleased to see you at the library
User: Yes, when you're open.

Then there is the opposite complication:

Scenario 5. Wrong Test

Designer: I built this Whizzo computer system. We would like you to see if it works?
User: Which bit?
Designer: Well, the whole lot really
User: With my own computer I know how to switch it on; and then it loads itself up
Designer: But now you need to alter the settings
User: What?
Designer: Just right click?
User: What's a right click? I have managed without it so far.

3.2 Test Factors. The important factors in test designing are:

The task;
The environment;
The time for task completion.

As we have just seen, there is a difference between testing the whole system and a single facet of it; and a difference between the domestic environment and the library environment.

3.3 The Task. Then, there is the task itself:

Scenario 6. Too Vague

Promoter: I'm here to see how you like the Whizzo internet engine
User: What does it do?
Promoter: It's like the others but easier to use
User: What do you want me to do, then?
Promoter: Just play about, see if you like it?
User: Well, it looks nice.
Promoter: (Ten minutes later) Having fun?
User: Well, it's very nice but after ten minutes I can't find what I want; I can't get a train time or buy a ticket; or buy a watch; I keep getting trapped in boxes and buttons. Still, it looks very nice.

Rule 3. When testing goods and services define the task to be completed, the optimal process sequence, and the time in which it should be completed; do not confuse testing a system with a function.

4. The Specification - What Are We Trying to Do?

This is just the opportunity to summarise what we have already said.

If we are going to test something we need a specification. here are the elements:

What do we want to test?
What market group do we want to use for the test?
What environment do we need for the test?
What benefit to the user?

Scenario 7. Designing a test

Organiser: I am responsible for Whizzo user testing. We want to see how people react to our new Whizzo internet system. Let me see. Do we know what we are trying to test?
Assistant: Yes, we want to see if people can find the train times from London to Manchester and then buy a Super Saver ticket
Organiser: That is clear enough. What is the maximum time allowed to find the time and buy the ticket?
Assistant: We have worked out that it takes five minutes by phone to get a time and book a ticket; so the maximum time we are allowing, before we say an attempt is a failure, is seven minutes.
Organiser: Why are you allowing two minutes more for the internet over the telephone?
Assistant: There is a £5 discount for booking by internet
Organiser: OK; we have worked out the task, the time and the benefit to the user; what's left?
Assistant: We need to work out where to do the test? I think it should be on the computers of our users and not in a central testing laboratory; then we can see how they get on in the real world, and help them fix things if they don't work
Organiser: Let's sum up; the advantages for the users we are testing: they can find trains and book tickets at home; those who have hearing difficulties will find it easy; it is cheaper than doing it by phone. Good. Remember to document everything properly.

5. The Trouble with Screen Readers

I now want to take a few minutes out of the general discussion to discuss the specific case of screen readers and how their role in testing can be somewhat complicated.

5.1 A screen reader is a special device which reads out or turns into braille what is displayed on a computer screen.

5.2 There are a number of serious difficulties when using a screen reader which I will describe in a logical sequence:

First, a screen reader turns a two dimensional display of information into a linear format; if we are listening, the words have to be put into an order and the chunks of information have to be put into an order; and we can't listen to two words or chunks at the same time even though we can look at two pieces of information at the same time. It's the same with braille, we can only read it along a line. And of course a screen reader can only interpret text, symbolic language, so it's no good at pictures.

Secondly, because the two dimensional element is thrown out, the screen reader has to make some sense of the order in which it delivers text. This is not too difficult if the person writing the code is clear but quite often people making up pages don't write code with screen readers in mind.

Here's a home page:

http://www.whizzo.com

Whizzo

Home | Links | Contact Us | Services | Subscribe to Newsletter

Whizzo Search: Go

Whizzo Profit Pack	Whizzo Software	Whizzo Clearance
Whizzo Services	Whizzo Special Offers	Whizzo Club
Whizzo Security	Whizzo Terms & Conditions	Whizzo News

Whizzo Registered office: 108 High St, Hurstpierpoint Tel: 01273 834321

2008 Whizzo

Now if we look at the Whizzo display we will see that ‘Special Offers' is bang in the centre of the grid so that it draws our eye towards it; and it's in a special, different colour from all the rest of the squares in the grid.

But unless the person making the code thinks: "Ah, screen reader; put the middle square first in the series" then the screen reader will read from left to right and the special offers will be fifth; the same thing will happen if the screen reader reads the left column from top to bottom and then the middle column from top to bottom; either way, special offers will be fifth.

It's also true that, after the middle space, the top left and bottom right items will receive more attention than the others; so it makes sense for Whizzo news to be in the bottom right corner; but if we were programming the sequence for a screen reader we would want the first three to be:

Whizzo Special Offers

Whizzo Profit Pack

Whizzo News.

Thirdly, I have already hinted at what the third problem is; it isn't really the screen reader's fault if people write inconsistent code. The most familiar example to most screen reader users will be the problem of boxes.

Boxes were invented when printing was performed by making letters out of lead type and dropping lines of type into a frame. If we wanted to make a page we couldn't have the different lines of lead all swishing about, so we bound them all into boxes. We don't need them on computer screens where we could simply move our cursor down while filling in a form; but for software writers that just isn't good enough:

Scenario 8 Inconsistent Code

Tutor: Im Whizzo's accessibility Tutor. How are you getting along with the test.
User: How do I get into this box with my cursor
Tutor: That's easy, you just press tab
User: But last time I pressed tab on a box nothing happened
tutor: No, on that one you needed to press return
User: That's confusing
Tutor: No it isn't; they're different kinds of boxes
User: They look the same to me
Tutor: Ahh, they look the same on the outside; but the first one has two choices and the second one four choices
User: what difference does that make?
Tutor: Well when you have two choices you can tick one or the other; when you have four choices you can tick all four
User: So now that I'm in this second box and I have ticked two out of four do I use the tab to get out
Tutor: No, you have to confirm
User: I didn't confirm last time
Tutor: No, that was when you only had to tick one of two choices; you've ticked more than one in this box so you have to confirm
User: How
Tutor: It's obvious; press enter
User: But sometimes enter means leave a box
Tutor: Not this time.

Because of this box problem I frequently get trapped in a box and can't get out because I can't move my cursor to see if there is any advice handy; in other words, I can't get my cursor out of the box to press the help button.

Fourthly, the code which the screen reader uses and the source code on a web site or operating system are not always compatible. The software provider blames the screen reader and vice versa. So when we are testing a process, we need to be sure that the screen reader and operating system or web site are compatible.
Finally, when we are trying to design a test for screen reader use, we have to imagine that the test is being conducted by people who read text a line at a time in the order we dictate; and while people are on a specific line they may not be able to get to the next line or the buttons at the top of the screen unless we write code that makes this possible. As people are using the keyboard to drive the system instead of a mouse, their navigation is limited; and, anyway, many users will have no real spatial concept; they won't be able to imagine what the screen is looking like.

5.3 I now want to think for a minute about people who use screen readers as reading out mechanisms to accompany language. This is when they are at their best, although we would be better off if web site builders gave us free audio on the site so we didn't need a screen reader.

5.4 When people are watching television we don't usually ask them to choose between the sound and the picture or the sound and the text; so a screen reader is a more natural way of looking at a page than just having the text. If we are operating a computer with a mouse and have the screen reader turned on to talk, there will be some problems, however, because the voice is always going to be going at a different speed from us and our eye might have left the part of the page that the screen reader is still working on; this can be confusing.

5.5 Now I want to spend a couple of minutes saying a few very special things about user testing by people with learning difficulties; they partly repeat other messages but I think it's worth concentrating on:

First, the objective of a test is to see if the product or process is helpful to the user; so if we find that the user has a difficulty, we have to tell the provider that we need improvements;
Secondly, most problems with goods and services arise because the designers, providers or producers think that something is obvious when it's not. We all have slightly different ways of thinking; so if we do a crossword by one compiler in five minutes it doesn't mean we will be as quick with a different compiler. Some people divide their emails up into folders for different topics;
- I can never remember which topic I chose for an email if it could be two or more; but I can always remember the date on which I wrote an email
Thirdly, Some people can only do something one way; this does not mean that a process should only allow us to do things one way because each one of us may have a different one way. There is no point taking a wide variety of people with learning difficulties and asking them to test one way of doing something.
Fourthly, in my experience, although I am not sure there is a really fixed rule about this, we have to be clear about working alone and working together. There is really no point testing a system with people with learning difficulties acting alone when they are almost certain to use the same system with a helper. It's like sending a totally blind person like me round an art gallery on my own; I actually always go with a friend. Equally, if the system is some sort of home help device which a person would expect to use late at night when alone, there's no point testing it with a helper taking part.
Fifthly, the real objective, as we've said before, is task completion, so if somebody comes up with a weird way of solving a problem, that's fine.
Finally, all of these observations lead back to specifications for web design that will help everybody but will have particular benefits for people with learning disabilities. Here's a short list of the basic requirements. A site should be:
- Granular. All web elements should be separately rendered so that they can be separately manipulated.
- Porous. Unless there is some absolutely specific reason why there should be coded boundaries, the user should be allowed to move freely through a website (you can create physical demarcation without coded demarcation).
- Customisable. A site is for the benefit of the user not primarily a product of authorial artistic expression.
- Multi modal. All elements should be rendered in all media but each medium should also be capable of carrying the essence of the information.

The last point is of particular significance. The use of sound is undervalued in web design and the use of pictures is often careless. It is taking us some time to travel from the idea of a website as an academic resource to the idea that it should actually resemble television broadcasting.

What the designer has to bear in mind above all is that the user needs to feel in control of the experience so that it enhances rather than diminishes self-esteem.

Some of these things might be difficult to test for which is why my simple goal is task completion. The analysis of why a task is not completed is much more complex than analysing a check list even though a check list is important.

6. Testing

Rule 4. The specification is provisional until the testers are consulted.

6.1 If we have found a random sample of the market segment needed for a test, that sample should be able to spot the elephant traps. It is not the job of testers to question whether the test should be carried out but they should have a say in the methodology:

Scenario 9. Choosing the Right Place

Manager: Hello, I am managing this test on the Whizzo home shopping service
User: I'm representing the users
Manager: We were thinking of doing the tests at our labs; we'll have a top team of engineers; and the facilities are really good; we'll organise transport
User: But the whole point of a home shopping service is whether you can use it at home, without engineers; but, more important, many of the users would be totally at sea away from the familiar surroundings of their houses; you would get artificially bad results
Manager: That's a good point. I thought in the lab we might get artificially good results.

Rule 5. Check the industry testing brief with the user sample.

6.2 Briefing - What Are We Going To Do?

Rule 6. It is vital that those who are to carry out a test are given the following information:

The terms and conditions for the testers (payment, breaks, safety, supervision; permission to withdraw);
Ethics (the way in which the testers are being properly looked after, not exploited; confidentiality);
The system and not the tester is being tested (reassurance, dignity);
The actual steps of the test;
How people benefit from the test (eg disabled people).

6.3 Terms and Conditions.

Most of the points in Rule 6 are reasonably straightforward and familiar but some need further discussion:

Although many will express surprise, there is no reason why people helping a commercial organisation with testing should do so free of charge. It is far too often assumed that disabled people have nothing better to do than provide global corporations with free services; given the disparity of resources, this is unacceptable. Organisations working with disabled people should seriously discourage this kind of arrangement. Neither is it good enough for an organisation to receive a donation for testing and not pass it on to the testers.
It is quite acceptable to pull out at any time during a test, particularly if it is causing stress;
The test is being mounted to see if a product or process is adequate; it is not there to put pressure on the tester; if something does not work properly the tester has helped the producer;
A test cannot start until the tester is sure what the task is and the level of support during the test. If the test is to be totally autonomous this needs to be carefully explained;
It is vital for people to know why they are testing something.

Scenario 10. Exploitation

PR: Thank you for agreeing to test the Whizzo profit pack
Tester: You are welcome. I represent disabled people
PR: Of course we'd like to pay you but the company has a cash flow problem at the moment
Tester: I am sorry; let us hope the profit pack helps
PR: I am sure it will
Tester: So what do you want me to do?
PR: We simply want to see how quickly we can get through the money collecting operation
Tester: Will this help disabled people
PR: Oh, I wouldn't think so; you don't have that kind of money, do you? But we have to go through the motions.

6.4 Be Prepared

Scenario 11. Check the System

Supervisor: As you know Im Whizzo's Head of Testing. I am pleased to see that we are all ready to go?
Tester: There's a funny smell
Supervisor: It must be burning toast in the canteen
Tester: But there isn't a canteen
Supervisor: Well, it must be outside. Never mind. Can you just use your mouse to start the test
Tester: Nothing's happening
Supervisor: Drat! It was all right five minutes ago when I checked everything. (fiddles) I see, somebody else has loaded something that isn't compatible; I'll have to disable it. It won't take a minute
Tester: Isn't that smoke? Yes, smoke! Fire!
Supervisor: Must be a cable or something. Where's the extinguisher?

How often have we been to a lecture or presentation when "the technology" doesn't work. It shows disrespect to our testers and gives us a bad reputation if we are not properly prepared to receive them. Carry out a thorough check of all the systems and carry out all the tests in a 'dummy run' before the testers arrive.

7. On The Job

7.1 It is important to make tests as near to real life as possible. If people are undertaking a simple task like finding a piece of information, they usually perform the task in an unbroken sequence. If people are carrying out a number of related tasks, they might want to stop and say something or have a drink before going on. If the tests are being timed, remember to turn off the timing device while people are relaxing.

7.2 A test should always take place in a relaxed environment as it is the product or process and not the person that is being tested.

7.3 Although the specification may not include it, it is useful to record individual reactions if these are strong. It may well be that a particular way of doing things presents a particular problem to a particular kind of person.

7.4 Step by Step

When observing a test it is best to get the permission of the tester to record the whole test. The performance of the tester can then be analysed in detail. Sometimes it helps to talk to the tester while she is working, particularly if she hits a snag that she wants to talk about; but sometimes patterns emerge from analysing tapes after a test, eg different testers might have found the same problem which they did not think was important when they were working, or which they worked round.

7.5 The tester performance should be matched against the optimal process sequence and variations noted. If the user departs from the sequence, was the signposting clear?

Scenario 12. It's Not Obvious

Supervisor: Hello im another of Whizzo's Test Supervisors
Tester: I've clicked there; and all I get is a whole load of icons; what am I supposed to do now?
Supervisor: Choose one, of course
Tester: But it doesn't say that, does it? Can I choose any icon?
Supervisor: No, of course not. You're supposed to try them each in order from left to right
Tester: But it doesn't say so
Supervisor: It's obvious
Tester: No it's not. Anyway, I've clicked the first one now. Oh, look; how lovely
Supervisor: You are not supposed to be enjoying the picture, you are supposed to be noting its contents and moving on
Tester: Sorry. Where does it say "Note the contents?"
Supervisor: It doesn't; it's obvious.

8. Feedback

Rule 7. Testers are entitled to feedback on three issues at least:

The test results;
Any action being taken as a result;
Is further testing planned?

Scenario 13 And Finally .......

Assessor: Well as a senior Whizzo manager, from our point of view that was pretty successful
Assistant: Why's that then?
Assessor: You see, the objective of Whizzo is to make it just easy enough to use but just difficult enough so we can sell them an upgrade.

Return to Accessibility Criteria