Pilot Usability Test

CS 160

Abe Mirrashidi

Paul Mais

D. Kirk Wylie

Rona Yang

 

1. Introduction

The system being designed is aimed to assist students with online class searching and scheduling. Keyword matching and simultaneous search queries are present to improve usability. Classes that match the search criteria are displayed and the student can add appropriate classes. The key feature of the interface allows automated scheduling. Once all classes have been chosen, the user can click on the "Generate Schedule" icon and a schedule will be presented to him/her. A grid resembling a calendar view is used because most people are familiar with the idea of a calendar used to schedule appointments or classes. Finally, modification of this schedule should be easy and intuitive to the user of any such system.

 

Pilot user testing gives us an opportunity to receive feedback about our design and interface ideas from real people who might actually use our system. Other methods for evaluating user interfaces, such as heuristic evaluation, are good at identifying specific problems with a design. User testing (UT) provides more information than just a list of what’s wrong with an interface. UT is particularly good at finding usability problems, precisely because real people are trying to use our program. UT also lets us know what we did right. Besides soothing bruised egos (designers are not always right), it lets us indemnify why particular interface ideas were effective. We can then apply these effective ideas to parts of our design that did not test so well. Finally, user testing serves as an aid and reinforcement of task analysis. Having users express their thoughts while using our program helps us understand our users better.

 

2. Method

Participants

The participants of the survey were all students attending UC Berkeley. Two participants were EECS/CS students enrolled in CS160 and a third participant was an L&S Undeclared student. Although it is expected that the proficiency of a computer science major at Berkeley to be quite high, the participant in L&S Undeclared only had a general knowledge of computers. This provided some balance to make sure that the interface is suitable for both computer "experts" as well as occasional users or "novices".

 

Apparatus

We conducted our user test in the lab in 330 Soda. One of the NT machines was used to run the demo. Because of crowded lab facilities, only our tester and the facility were sitting in front of the computer. Two other members of our team were standing behind the work area taking notes as our users progressed through the tasks. The final team member taped the user test on a video camera we borrowed for the occasion.

 

Tasks

The three tasks given to the user are intended to simulate three different uses of the application, and all follow the usage of the system from initial contact through successful generation and approval of a schedule.

The first task involves generating a schedule with specific classes that the user already knows they wish to take. This is intended to simulate a student who just wants to see what a schedule with certain classes would look like. We looked to make sure that the users could discover how to search for specific classes and generate a schedule based on those classes.

The second task involves generating a schedule that has certain parameters, and different search criteria. It was intended to model a user who knows that he wants to take classes that fulfill certain parameters, such as Professor or Time of Day, but isn’t sure exactly what classes satisfy those criteria. In this task, we looked to make sure that the users could find the search criteria they were looking for, and make sure that our interface to the search criteria was intuitive to the user.

The final, and most complicated, task was to generate a schedule with four specific classes. This schedule would need to have a new discussion section set for it. Furthermore, there was (intentionally), a lecture timing conflict inherent in the classes that we chose. We looked to ensure that the users were able to discover how to change discussion sections, and whether our chosen interface for resolving lecture conflicts was a reasonable one for the users.

 

Procedure

When the participants of the test arrived, they were greeted by Paul who acted in the role of both greeter and facilitator. He explained the test to them, had them each fill out a consent form and user experience/information questionnaire to make sure that the users were properly selected (based on range of experience), and then continued with the explanation of the program that they were to test. It was explained to the user that he would be taped and people would be taking notes throughout the test, but that he should be sure to relax. A script of the section is in the Appendix.

The user was then shown a brief demonstration that covered the range of interface issues in the application, including searching for classes, assembling a list of classes to schedule, generating a schedule, and changing discussion sections. This demonstration had no overlap with the exact tasks asked of the users.

The three tasks were given to the user in order, one at a time. After each task was shown to the user, Paul explained the task briefly to the user and asked if he had any questions about what he was being asked to perform. After this, the user completed the tasks, interacting with Paul consistently.

After the tasks were accomplished, the user was asked for any suggestions or comments that they might have. During this time, all four team members asked the user questions and also answered questions. The team also supplied some rationale for why we used some of the interface components that we did. This session helped in providing us with useful comments as to how we could improve certain portions of the interface. Finally, the user was asked to fill out a post-testing questionnaire, which is also in the Appendix.

The user was thanked for his or her time, and showed out by Paul, acting once again as the greeter.

 

3. Test Measure

We qualitatively measured how long it took the participants to complete the given tasks and also the time required to perform certain actions (such as deciding to click on a discussion in order to change it) within that task. The goal of this measure was to compare our expectations with what really happened so that we could modify and improve our interface accordingly. We also tried to observe and interpret the participants’ facial and verbal expressions so as to hypothesize whether they liked or disliked some particular part of the interface, and also to determine whether or not they were confused about something. These observations also told us what the users were looking for in our program and whether our program met their expectations. The users’ satisfaction was measured by simply asking if the program met their expectations. Originally, we did not really think that this would be a good measure, but we think that they were honest in their responses- none of the users thought that the program met their expectations. We asked users why this was the case and they were helpful in providing suggestions as to what they would like to see in the interface.

 

4. Results

The user testing helped us find problems based on the actions of the participants. We found most problems by observing what the users did. However, we found a few flaws in the interface by noting what the users did not do. One major example of this was observing how users did not take advantage of the fact that the "Course #" scroll-list was optional. This caused some problems when they would try and select a specific course from a long list. The same problem occurred with all the users when searching for "Landay" from the list of professors.

Two users suggested that the search criteria automatically affect the possible selections. When "Architecture" is chosen, they wanted the list of all architecture courses to be taught. And if "lower division" were selected, the users wanted upper division courses to be removed without having to push "Search". One user made the same observation with relation to the "Course Description" text box.

All the users made comments about the lack of a menu-bar with possible saving or printing options. It seems like users like the idea of a menu-bar even if none of the options in the menu will be implemented.

In the calendar window, two users commented on the courses listed in the top right corner. They tried double-clicking classes to see if anything would happen. One user suggested being able to remove a class or get information on a class by double clicking.

One user was unclear that the pop-up menu displaying conflicting lectures was indeed displaying conflicting lectures, as opposed to discussions or labs. None of the users seemed to mind that they had to remove a class without being able to retrieve more information about the classes.

All of the users had at least minor problems with the calendar grid (much was probably due to limitations in Visual Basic). One user did not understand that when a section is clicked that the blue sections are "possible sections" and that he was supposed to choose one. Another user noticed that if they click on a discussion and then on another class, the discussion remains highlighted and the action is not "cancelled".

While asking users to complete each task, we noticed that a "Clear All" button might make things easier on users trying many different schedulers, particularly if they were still undeclared.

As an ad-hoc experiment we asked two of the three users (who voluntarily agreed) to try and change a physics discussion. The amount of blue on the screen overwhelmed both users.

 

5. Discussion

For the real experiment, we will try to find a quieter place because the lab where we conducted our experiment was very noisy, which may have distracted the user. In addition, we decided that we have given too much information in our demo as to how to perform a search, which we will try to avoid in the real experiment. By watching the videotape, we also found out that when confronted with questions, we argued with the user about why we did what we did instead of listening to the user. We will not repeat that next time.

We have planned to make the following changes to our GUI based on the experiment results:

 

For the search window:

 

For the schedule window:

 

For the conflict dialog:

 

 

6. Appendices

Materials used in the experiments:

 

Summary of "Raw" Data

 

Double click to add class

No "clear all"

Trouble finding "162" or "Landay"

-match word

- don’t realize that you can type in "Landay"

Automatically update. No "search" or "show course description" button

- selecting division does not create instant results

- course description may refer to previous class if not changed

Undo… switch back to calendar without regenerating a schedule

No menu-bar to save or print

Want to click list of classes in top right corner of calendar window

-maybe display course information when one is selected from the calendar

Course description can be modified

Pop up for conflicting classes does not make it clear that the lectures are in conflict

Task is unclear- does not tell user whether to keep Econ 3 or CS 152

Calendar does not resize when window is resized- scroll bar keeps its length

Clicking on discussion, then lecture (or anything else) does not cancel action

Too many choices for classes with many discussions, such as physics

 

Delete on the fly

Confirm on exit

No guidance on how to change discussions (or anything else)