Usability Test Plan

This usability test plan was made for Assassin's Creed IV: Black Flag (AC4) while in graduate school at University of Washington. Our main goal was to evaluate how experienced gamers use the tutorial system in AC4 and deliver design recommendations. I want to give a big thanks to my teammates Muchen Feng and Kenley Gard!

Summary

Our team began by focusing on who the “experienced gamer” is by creating personas to focus our communication. We then played AC4 to get a strong understanding of what exactly the new features are and how we should go about evaluating the game’s tutorial. With this knowledge, we focused on three high level objectives and formed measurable research questions for each objective. After testing our participants, we quantified our observations and presented our findings in order of severity to progressing in gameplay.

Study Plan

Assassin’s Creed IV: Black Flag

Our team will be conducting a usability test on the tutorial system for Assassin’s Creed IV: Black Flag (AC4). We need to evaluate the tutorial system and characterize its effectiveness for experienced users.

Overall Objectives

For experienced gamers new to AC4:

  • Identify helpful tutorial information.
  • Identify superfluous tutorial information.
  • Identify confusing tutorial information.
  • Identify absent tutorial information.
Research Questions

Identify helpful tutorial information:

  1. Is the participant able to perform game functions novel to AC4 on their first attempt?
  2. What familiar action-adventure game functions are reinforced without hindering the user?
  3. Which tutorial prompts does the user find helpful?
  4. How confident is the user in performing game functions?

Identify superfluous tutorial information:

  1. What game controls do users perform before their respective tutorial prompt?
  2. Which tutorial prompts does the user find unnecessary?
  3. Was any part of AC4 tedious for the user?

Identify confusing tutorial information:

  1. Which tutorial prompts does the user find confusing?
  2. Which sequences of AC4 are confusing for the user?

Identify confusing tutorial information:

  1. What information is the user missing to advance the game?
Participant Characteristics

Experienced Gamer – 6 to 8 participants

  • Gender – Male
  • Age – 18 to 35
  • Owns an Xbox or Playstation console.
  • Have played 10 hours in action-adventure genre
  • Does not develop games or work in game industry
  • Have not played AC4 before

Data to be collected and evaluation measures

Summary

This usability study is mainly exploratory in that we will look to generate hypotheses regarding experienced gamers using the AC4 tutorial system. In doing so, we will gather assessment data about the effectiveness of the AC4 tutorial system for experienced gamers. We will collect quantitative data about game function execution, categorization of tutorial prompts, and confidence in learned game functions. We will also collect qualitative data on participants’ experiences using the tutorial system.


Observation study

In our observation study design, all participants will playtest through AC4 from the beginning. We will not be able to counterbalance participants considering the sequential learning involved with initial game functions.

We will conduct 6-8 individual 1.5 hour usability study sessions. The first 15 minutes of each session will be used to explain the session to the participant and confirm basic background information with the participant. During the middle 1 hour, participants will play to progress through AC4. The last 15 minutes will be reserved to administer our post-test and debrief our participant.


Pre-test arrangements (3 minutes)

Have the participant:

  • Review and sign nondisclosure & recording permissions.
  • Confirm background with pre-test survey (same questions as screener).


Introduction to the session (7 minutes)

Discuss:

  • Participant’s experience with usability studies.
  • Importance of their involvement in the study.
  • Moderator’s role.
  • Room configuration, recording systems, observers.
  • The protocol for the rest of the session.
  • Thinking aloud.


Background interview (3 minutes)

Discuss the participant’s:

  • Experiences with various tutorial systems.


Task (1 hour)

Participants will start at the beginning of AC4 with the task of progressing through the game story.


Post-test debriefing (10 minutes)

  • Administer post-test survey.
  • Ask broad questions regarding the effectiveness of the tutorial system to collect general qualitative data.

Task List

Participants will start at the beginning of AC4 with the task of progressing through the game story for 1 hour. While AC4 contains features and game functions that occur past 1 hour of gameplay, the main tutorial system is presented within the 1 hour. Also, an experienced gamer new to AC4 would have to start at the beginning when playing the game outside the context of usability testing.Areas of interest-place tutorial prompts and screenshots.

Test Environment

We will utilize Microsoft Studios research testing facilities to conduct the sessions. Participants will play AC4 on an Xbox 360. The video information on screen will be captured using OVO recording software. We will also bring a recorder to gather a set of backup recordings.

Test Moderators Role

Our role is to carry out our test methodology in an unbiased manner. As the participants progress with the test, we will gather our data through observational notes, participant talk aloud, surveys, and interviews.After our test session, we will analyze and tabulate the data to answer our research questions. We will then present the data to our classmates for feedback; then create a succinct written report.

Data to be collected and evaluation measures

Our measures will answer our research questions. We will collect both performance and preference measures.


Performance:
Collection Method – Remote observation

  • AC4 new game function errors [1]
  • User execution of game function prior to tutorial prompt [5]
  • Task Completion


Preference:
Collection Methods:
Talk Aloud

  • Perceived effectiveness of each tutorial prompt [2, 3, 6, 9]
  • Perceived feeling of game sequences [7, 10]
Surveys
  • User confidence rating for game functions [4]
  • User confidence in tutorial effectiveness.[3]
Semi structured-interviews
  • Perceived effectiveness of each tutorial prompt [2, 3, 6, 9]
  • Perceived feeling of game sequences [7, 10]
  • Missing information to advance [8]
  • Confusing information during gameplay [10]

Report contents and presentation

We will deliver a presentation and written report that:

  • Briefly summarizes the background of the study, including the goals, methodology, logistics, and participant characteristics
  • Presents findings for the original questions to investigate
  • Gives quantitative results and qualitative findings to discuss specifics as appropriate to the question and the data.
  • Discusses the implications of the results
  • Provides recommendations
  • Suggests follow-up research

Figure 1. AC4 Tutorial Usability Study Plan.

Results

Read the whole report here! This was a great learning experience where we got real world experience planning, moderating, and analyzing game usability.

Date September, 2014
Type UX Research, Study Plan
Share

Contact

Are you working on something great? I would love to help make it happen! The best way to reach me is through email. Just do it.