For the Usability course final project, I designed and constructed a level in Unreal Tournament 2004 using UnrealEd, the modding tool which ships with the game. I then conducted extensive user testing with iterative level revisions between each test. The testing report is below.
If you want to play the mod and you have UT2004, download this set of files. Check the ReadMe.txt for instructions on how to install.
Test Report
Seven tests of different types were conducted over three versions of the level in the spirit of the RITE testing method discussed during Week 4 of the Usability class (and many times thereafter!)
Many testers commented on both my level and the Unreal Tournament game at large (though they did not necessarily realize the difference), but I am focusing here only on the comments relative to my level, since the UT game itself is not being tested nor is UT something I can "fix" for this project.
Test Composition
Think aloud tests had three parts: brief explanation to orient the tester to UT navigation and mutant gametype, longer for more inexperienced players; a short session of the tester navigating and exploring the level without bots; and a longer session of the tester playing the level with 5 novice or average bots, depending on the experience of the player at playing first-person shooter games.
Multiplayer survey tests only had two parts: brief explanation to orient testers to UT navigation and mutant gametype; and a long multiplayer session (total: 6 human players in level together). All players filled out a survey after the test.
Testers and versions:
I tested three builds of my level with changes from testers incorporated into each new build for retesting.
The test statistics are summarized below:
Analysis and Version Development
The initial think-aloud test was a test run to see if the level would work when transferred to another computer and for me to get a feel for how to run a think-aloud test, since this is my first such project.
I next conducted the multiplayer survey tests to get an overall idea of the problems and potential solutions from a variety of players. Comments on this build specifically described extreme confusion and disorientation due to the transparent nature of the glass level. All rooms were identical in this build which also caused extreme disorientation. Clear space and glass walls/floors were difficult to distinguish, appearing as a variety of problems such as inability to identify doorways and players subsequently falling out of doorways, frequent shooting at walls in an attempt to shoot target players on the other side of the walls, and players getting stuck in the slides in the rooms. Players also commented on the movers breaking, acting unexpectedly, or being inconsistent. Observed problems with movers were: movers would not always work when a player entered platform, movers would move slightly and return to their start position, movers would get stuck/shake/force players off.
With these surveys in mind, I developed the second version of the level making only the changes small enough to build quickly yet significant enough to increase the usability of the playing experience. Each level was themed with a color, red, yellow, green, and blue, by way of all doorways and lighting being one color per level. The floors were changed from glass to a semi-transparent grate texture. The problems with the movers are large scale problems that could not be diagnosed and fixed for this build.
The second build was tested by two think-aloud testers who had not played the level before. Both testers commented on the colors making the level easy to navigate. Notably, the testers did not make the same comments on disorientation as the survey testers had. The male tester did comment on the ceiling of his current room, a surface which is also the floor of the level above him. All of the floors were grates, but looking at the same floor from underneath showed no grating texture, an illogical occurrence in the physical world. Both testers were observed having problems with the movers, with the male tester reacting severely to the problems by avoiding the green mover and the entire green room (by stair or mover), though I've observed equal breakage to movers on all levels. The male tester also noted a headache after playing the level due to the glass texture and some difficulty perceiving depth in the level.
For the third build, I could not diagnose and fix the movers, but I did correct the ceiling textures to match the grates.
For the final think-aloud tests, two testers from the multiplayer Test 2 returned to participate in individual think-aloud tests. Both testers commented on the positive changes to the level, especially on the colors making the level easier to navigate. One tester liked the solid grate flooring and desired that the walls also have a more solid but still "see-through feel" to them, since the glass walls were still difficult for him to see. Both testers showed significantly improved ease of use with the level since their initial experience with the test, with one tester being more familiar with first person shooter style games, yet both testers showing a general improvement from the repeated exposure as well as the changes to the level. Both testers encountered problems with movers and commented on movers being stuck or killing them.
Recommendations
Future builds would need to address the problems with the movers. The problems potentially lie with the change to the mover type made to enable bots to successfully use the movers, or with the spin keyframe animations that the movers cycle through. The mover issues are high profile and incredibly negative and therefore a high priority to fix in a later build.
Developing new textures for the glass that maintain the transparent and aesthetically pleasing nature of the glass but are "more solid" would increase player satisfaction as well as ease of navigation in the level. Tests should be made during this development specifically to see if the changes are adequately addressing the player's needs.
Tests with color-blind users should be done to evaluate the usability of the level with that handicap. If disorientation occurs, which seems very likely, solutions addressing this population need to be developed if a significant portion of the player base is also a member of this special population. (Or, if there is time/money in general to make the level more usable for everyone by making changes.) Potentially, signs with text or some other non-color dependant indicator could be added to the levels to distinguish the levels for players who can not see the color theming. Testers may be able to brainstorm solutions based on their solutions in life experiences.