Sea of Thieves - Playtest
Sea of Thieves - Playtest
Solo, Personal Project, Playtesting
Solo, Personal Project, Playtesting
Solo, Personal Project, Playtesting
A personal Formative Research project undertaken to understand the first time experience of a New Player playing the Sea of Thieves tutorial. This was carried out over a duration of 2 weeks in preparation for Sea of Thieves port to PlayStation consoles. It demonstrates the importance of measuring a player's physiological responses such as eye tracking during a playtest and how it can help uncovering causes for certain player behaviors.
A personal Formative Research project undertaken to understand the first time experience of a New Player playing the Sea of Thieves tutorial. This was carried out over a duration of 2 weeks in preparation for Sea of Thieves port to PlayStation consoles. It demonstrates the importance of measuring a player's physiological responses such as eye tracking during a playtest and how it can help uncovering causes for certain player behaviors.
A personal Formative Research project undertaken to understand the first time experience of a New Player playing the Sea of Thieves tutorial. This was carried out over a duration of 2 weeks in preparation for Sea of Thieves port to PlayStation consoles. It demonstrates the importance of measuring a player's physiological responses such as eye tracking during a playtest and how it can help uncovering causes for certain player behaviors.
Client
Client
Personal Project
Personal Project
Services
Services
Primary Research
User Testing
Eye Tracking
Qualitative and Quantitative Analysis
Primary Research
User Testing
Eye Tracking
Qualitative and Quantitative Analysis
Industries
Industries
Video games
Video games
Date
Date
April 2024
April 2024
Overview
Overview
This personal project is a formative UX research study focusing on understanding the first time experience of a New Player playing the Sea of Thieves tutorial by observing their gameplay, measuring their physiological state while playing the game and interviewing them.
This personal project is a formative UX research study focusing on understanding the first time experience of a New Player playing the Sea of Thieves tutorial by observing their gameplay, measuring their physiological state while playing the game and interviewing them.
Research objectives
Research objectives
Understand Player's experience playing the SoT tutorial for the first time
Discover issues that lead to a negative experience.
Learn how the user perceives interface elements such as the inventory , map , prompts etc
Evaluate the Player's emotions after playing the tutorial
Understand Player's experience playing the SoT tutorial for the first time
Discover issues that lead to a negative experience.
Learn how the user perceives interface elements such as the inventory , map , prompts etc
Evaluate the Player's emotions after playing the tutorial
Research Plan
Research Plan
The following research plan was adopted
The following research plan was adopted
Recruitment
Recruitment
The study recruited 4 participants who matched the following criteria
1. Experience level:
Beginner - New to gaming in general ( 1 Participant )
Intermediate - Experienced gamers but new to the adventure/ PvE genre. ( 2 Participants )
Expert - Experienced in adventure and PvE games. ( 1 participants )
2. Gaming Peripherals:
Participants were recruited to ensure equal representation for both, the dualsense controllers and keyboard and mouse as these peripherals are primarily used to game on a Playstation
3. Experience playing Sea of Thieves:
Participants were only recruited if they never saw gameplay and never played the game to ensure that it would be their first experience
4. Language:
One participant was chosen to represent a different language ( Simplified Chinese ) other than English to test the tutorial instructions/prompts of a different language
The study recruited 4 participants who matched the following criteria
1. Experience level:
Beginner - New to gaming in general ( 1 Participant )
Intermediate - Experienced gamers but new to the adventure/ PvE genre. ( 2 Participants )
Expert - Experienced in adventure and PvE games. ( 1 participants )
2. Gaming Peripherals:
Participants were recruited to ensure equal representation for both, the dualsense controllers and keyboard and mouse as these peripherals are primarily used to game on a Playstation
3. Experience playing Sea of Thieves:
Participants were only recruited if they never saw gameplay and never played the game to ensure that it would be their first experience
4. Language:
One participant was chosen to represent a different language ( Simplified Chinese ) other than English to test the tutorial instructions/prompts of a different language
Study Setup
Study Setup
Since the PS5 can be used on both Monitor and Television , two setups were used to carry out User Testing and record physiological responses.
1. Television With Speakers:
This setup mimics the traditional living-room console setup that is commonly used. Eye-Tracking Data was captured using Tobii Glasses 3
2. Monitor with Headphones:
This setup represents the modern way of playing on a PS5 where multiple devices in addition to the PS5 are attached to a monitor and sound is played through the headphones for an immersive gaming experience. Eye tracking data was captured using Tobii Fusion Pro
Since the PS5 can be used on both Monitor and Television , two setups were used to carry out User Testing and record physiological responses.
1. Television With Speakers:
This setup mimics the traditional living-room console setup that is commonly used. Eye-Tracking Data was captured using Tobii Glasses 3
2. Monitor with Headphones:
This setup represents the modern way of playing on a PS5 where multiple devices in addition to the PS5 are attached to a monitor and sound is played through the headphones for an immersive gaming experience. Eye tracking data was captured using Tobii Fusion Pro
Findings
Findings
The Post gameplay questionnaire consisted of Likert scale questions which quantified the different feelings that the participants were feeling after playing the tutorial. This questionnaire was inspired by the Circumplex Model, proposed by James A. Russel.
We see from the chart that The Participants mostly felt Confused , Tired but were also Happy and Excited.
The Post gameplay questionnaire consisted of Likert scale questions which quantified the different feelings that the participants were feeling after playing the tutorial. This questionnaire was inspired by the Circumplex Model, proposed by James A. Russel.
We see from the chart that The Participants mostly felt Confused , Tired but were also Happy and Excited.
User Journey Map
User Journey Map
A user journey map was created to pinpoint issues that would cause these feelings of Confusion and Exhaustion identified previously. It would also help identify which issues lead to negative feelings and which lead to positive feelings by cross-referencing issues during gameplay , post-test interviews and the remarks made by participants while playing.
(It is highly recommended to download the user journey map below in full resolution from here)
A user journey map was created to pinpoint issues that would cause these feelings of Confusion and Exhaustion identified previously. It would also help identify which issues lead to negative feelings and which lead to positive feelings by cross-referencing issues during gameplay , post-test interviews and the remarks made by participants while playing.
(It is highly recommended to download the user journey map below in full resolution from here)
Issues
Issues
The issues highlighted in the user journey map were cross referenced with eye-tracking and gameplay observation to identify the causes and potential solutions
The issues highlighted in the user journey map were cross referenced with eye-tracking and gameplay observation to identify the causes and potential solutions
Prompts
The Tutorial prompts had a couple of issues which was responsible in causing some confusion for the participants
Prompts
The Tutorial prompts had a couple of issues which was responsible in causing some confusion for the participants
Hidden Interaction Buttons
Since the interaction buttons are attached to the object rather than the camera, the tutorial prompts would often cover the interaction buttons causing the participants to press random buttons in order to interact with the chest in this case or think that they did something wrong and retrace their steps
Vague/Confusing Prompts
The instructions contained in some of the prompts caused confusion for all participants ,
The first prompt on the right did not show the picture of the location on the mast where the repairs were required causing participants to go below deck or climb the mast to find the cracks that needed repairing
The second prompt on the right caused participants to click the left stick and hold it instead of "moving the left stick down"
Ignoring Prompt Pictures
3/4 participants ignored the pictures in the tutorial prompt which is essential in locating the correct interaction element, Causing them to spend a lot of time locating the interaction element. The reason for ignoring these prompts can be identified by using a scan path analysis of gaze data as shown below
The participant's gaze path moves in saccades from left to right for each sentence individually. After reading the last line, the eye gaze ends up at the right side , after this the participant looks at the game to make sense of the instruction rather than the picture, as the picture is very far to the left compared to where the participants gaze ends. This is simplified and illustrated below
Level/Tutorial Design
Some issues regarding level design caused significant confusion to players as well
Bucketing Water
The tutorial required players to bucket water from the lower deck of the ship, however,
The prompt was vague and the ship barrel was already filled with water , confusing the participants. This resulted in participants emptying the water barrel and filling it again in hopes of completing the tutorial
Ghost Ship Event
The Ghost Ship event was meant to create feelings of Awe and encourage players to play the game after the tutorial.
However,
3/4 participants missed this event as they were prompted to find cannonballs to fight the ghost ship some time before the kraken swallowed the ghost ship.
This resulted in participants taking a long time to find the cannonball barrel as all 3 of them thought it would be below deck. therefore, missing the event.
Lost in Sea
2/4 players got lost while sailing and kept crashing into invisible barriers. 1 participant completely lost their sense of direction and went back towards the island
This was because the guiding lights were not distinct enough among the other wreckage in the sea forming the invisible barriers.
The players also got lost while getting the cannonballs from the barrel during the ghost ship event , but did not center the wheel. Causing them to go in circles away from their correct path
Barrel Inventory
The Controls for the Barrel inventory is far away from the items in the inventory which would require constant switching of eye gaze between inventory and controls, causing exhaustion.
Hidden Interaction Buttons
Since the interaction buttons are attached to the object rather than the camera, the tutorial prompts would often cover the interaction buttons causing the participants to press random buttons in order to interact with the chest in this case or think that they did something wrong and retrace their steps
Vague/Confusing Prompts
The instructions contained in some of the prompts caused confusion for all participants ,
The first prompt on the right did not show the picture of the location on the mast where the repairs were required causing participants to go below deck or climb the mast to find the cracks that needed repairing
The second prompt on the right caused participants to click the left stick and hold it instead of "moving the left stick down"
Ignoring Prompt Pictures
3/4 participants ignored the pictures in the tutorial prompt which is essential in locating the correct interaction element, Causing them to spend a lot of time locating the interaction element. The reason for ignoring these prompts can be identified by using a scan path analysis of gaze data as shown below
The participant's gaze path moves in saccades from left to right for each sentence individually.
After reading the last line, the eye gaze ends up at the right side. As the picture is very far to the left compared to where the participants gaze ends, the participant looks at the game to make sense of the instruction rather than the picture.
This is simplified and illustrated in the pictures on the left
The participant's gaze path moves in saccades from left to right for each sentence individually.
After reading the last line, the eye gaze ends up at the right side. As the picture is very far to the left compared to where the participants gaze ends, the participant looks at the game to make sense of the instruction rather than the picture.
This is simplified and illustrated in the pictures on the left
Level/Tutorial Design
Some issues regarding level design caused significant confusion to players as well
Bucketing Water
The tutorial required players to bucket water from the lower deck of the ship, however,
The prompt was vague and the ship barrel was already filled with water , confusing the participants. This resulted in participants emptying the water barrel and filling it again in hopes of completing the tutorial
Ghost Ship Event
The Ghost Ship event was meant to create feelings of Awe and encourage players to play the game after the tutorial.
However,
3/4 participants missed this event as they were prompted to find cannonballs to fight the ghost ship some time before the kraken swallowed the ghost ship.
This resulted in participants taking a long time to find the cannonball barrel as all 3 of them thought it would be below deck. therefore, missing the event.
Lost in Sea
2/4 players got lost while sailing and kept crashing into invisible barriers. 1 participant completely lost their sense of direction and went back towards the island
This was because the guiding lights were not distinct enough among the other wreckage in the sea forming the invisible barriers.
The players also got lost while getting the cannonballs from the barrel during the ghost ship event , but did not center the wheel. Causing them to go in circles away from their correct path
Barrel Inventory
The Controls for the Barrel inventory is far away from the items in the inventory which would require constant switching of eye gaze between inventory and controls, causing exhaustion.
Observations - Beginner vs Experienced
Observations - Beginner vs Experienced
Apart from the issues discovered there were also interesting observations made between beginner , intermediate and expert players
Apart from the issues discovered there were also interesting observations made between beginner , intermediate and expert players
Finding the 'X' spot on the treasure map
Finding the 'X' spot on the treasure map
Beginner and Intermediate players took longer to find the X spot on the map. The advance players however, took shorter time to find X , this can be further explored by looking at the average heatmap of eye gaze data when looking at the map
Beginner and Intermediate players took longer to find the X spot on the map. The advance players however, took shorter time to find X , this can be further explored by looking at the average heatmap of eye gaze data when looking at the map
Radial Menu
Beginner and Intermediate players focused more on familiarizing themselves with the radial menu controls whereas the advanced players focused on familiarizing themselves with the items in the radial menu
Suggestions
Suggestions
Since the project was undertaken for a duration of two weeks and due to the lack of Game Assets , prototypes for solutions could not be made. Therefore , here are some suggestion to solve the issues discussed above.
Changing the positions of Barrel Inventory controls
Changing the positions of Barrel Inventory controls
The Position of the barrel inventory could be brough closer to the inventory and placed under or above the item description. thus resulting in lesser eye gaze travel distance and subsequently leading to reducing eye fatigue
The Position of the barrel inventory could be brough closer to the inventory and placed under or above the item description. thus resulting in lesser eye gaze travel distance and subsequently leading to reducing eye fatigue
Changing the positions of Tutorial Prompt Pictures
Changing the positions of Tutorial Prompt Pictures
The tutorial prompt pictures play an important part in helping locate the correct element to interact with. Therefore it could be highlighted and placed on the right side of the text instead of the left side so that it is closer to the player's gaze after reading the tutorial prompt
The tutorial prompt pictures play an important part in helping locate the correct element to interact with. Therefore it could be highlighted and placed on the right side of the text instead of the left side so that it is closer to the player's gaze after reading the tutorial prompt
NPC acts as a guide to find barrels and sail controls
The Pirate lord is an NPC that guides you through the first part of the tutorial , He shows you how to find the sword. and the direction to sail into the Sea of Thieves. This same NPC could be used by players to locate the Sail length controls and barrels on the ship.
The Pirate lord is an NPC that guides you through the first part of the tutorial , He shows you how to find the sword. and the direction to sail into the Sea of Thieves. This same NPC could be used by players to locate the Sail length controls and barrels on the ship.
