A UX service provider that tests games, that doesn’t go together at all, you think? You might think so, but UX and gaming methods are not that different. I would like to use a recent example to show you what you need to be particularly careful about when it comes to game testing.
A few months ago, we worked with a major game studio to test a well-known open world game that has been on the market for many years and is played by many millions of gamers around the world. Since the topic of user research has also received more and more attention in the games industry in recent years, user studies on the operation of games are carried out parallel to their further development in order to constantly improve them. This was also the case in our project. The developer was unsure about the user experience of some functions of his game. He suspected that they would hardly be used. While these features were not absolutely necessary to achieve the game’s goal, they were developed for a good reason and should contribute to the “joy of use” and thus to the overall experience of the game. However, this becomes difficult if they are not found by the player.
A well-known example…
An example from The Legend of Zelda: Breath of the Wild: The main character Link’s hearts and energy can be replenished by cooking certain dishes. To do this, the player must find various ingredients throughout the world. The player must find out which ingredients make which dishes and thus how much of an energy boost, primarily through trial and error or by using Google. However, if a player never finds or uses the ability to cook, they would still have the opportunity to successfully complete all of the game’s missions, as they are offered other options to gain energy in the game. Cooking, however, with its “randomness” of the result, brings a special “joy of use” to the players and not using it would be more than a pity. Fortunately, Nintendo’s developers and designers have introduced the cooking feature in a way that is easy to understand, so I too had great fun using it.
This example shows why our client was also interested in obtaining user and usage information on the functions in question.
Exploration vs. moderation
It was important right at the beginning that we gained experience with the game ourselves. Enough time should be planned for this, a difference to the classic usability test. It was essential that we knew the game well (especially with regard to the functions to be tested), because a game test is much more flexible than a usability test. In a usability test, we usually create a test scenario that is run through more or less the same way for all test participants in order to answer all relevant questions. A game involves much more variation, no participant chooses the same path to get to the goal. This could be achieved via very strong guidance from the moderator, but this would significantly limit the authenticity of the game situation. Especially in open world games, the exploration character of the game is crucial and this can only be explored if the players can act as freely as possible. In the Zelda example, this could mean that participants are initially only asked to replenish their energy and observe how they proceed. If the cooking is not used at all, the moderator can give a hint after some time and again observe the first interaction with it. In our test case, while we decided to give a specific starting point in the game, we also kept giving hints about the next steps during the course of the test (so that all the features would be tested). However, we left the interactions themselves completely up to the participants. It was important, however, to help the participants at points relevant to the test, so as not to spend unnecessary time on functions that were not the focus of the question. For example, in Link’s example case, this could be the use of the bow to kill animals, which in turn make meat available for cooking. The use of the bow can be explained because it is not the focus of the question. However, it should be recorded as a marginal finding that the participants had problems with this.
Test participants with special requirements
The prerequisites of the participants of the game test were another important difference to a classical usability test. The test-relevant characteristics of the target group/persona were defined as usual (here, for example, that they should have no or little experience with the specific game, but nevertheless experience with open world games), but in addition, special technical prerequisites were relevant: The game required a certain performance of the computers, for example, and the test participants were also supposed to use a gamepad for the execution rather than a keyboard and mouse. In addition, the participants were supposed to reach a certain game level before the test, since the relevant functions were only usable from a certain level. Link, for example, also only finds certain cooking ingredients once he has discovered certain parts of the world. For our test, we had to rely on the participants independently reaching the game state according to our instructions before the test. As expected, we found that the intrinsic motivation of the participants was very high for such a test and most of them brought the prerequisites into the test. The intrinsic motivation seems to be higher here than in most usability tests, where the direct benefit is not always clear to the participant. The time required and the uncertainty of the test situation is often present in the participants. In a game test, however, the expected fun of playing the game is enough for participants to agree to take part. Accordingly, the recruitment of the 10 participants turned out to be quite uncomplicated.
Due to the complexity of the gameplay, the tests themselves had to be scheduled longer than the normally recommended hour for conducting usability tests. We usually try not to exceed this limit, since attention and concentration of participants suffer in longer tests, especially in a remote context. In addition, especially for tests in a professional context, time resources are often a problem, which is why shorter tests are more attractive. In our case, however, a short test was not practical to try out the relevant features. However, this did not seem to cause problems for the participants due to the short nature of the game situation. The moderator attempted to observe the five main functionalities to be examined in the test in all participants as far as possible, but without pretending too much. It became apparent that, despite the requirements placed on the participants’ technology, not all of them were able to use it. Accordingly, a few participants were not able to test all functions within the two-hour time window due to technical difficulties. Above all, the Internet connection was a problem for some due to the high frame rate required for screen transmission. An internet speed test at the beginning of the test should be performed in the future to detect this problem early.
Where to put all the insights?
The individual tests were recorded for the customer and highlight videos of the most important findings were created afterwards. It became apparent that an additional visualization of the controller would have been interesting. We had mostly asked which buttons the participants used to test certain functions, but on the one hand the participants used different controllers and on the other hand the buttons used could not be queried at every point in time in order not to interrupt the natural course of the game.
We found some interesting insights into the use of the investigated functionalities through the tests. In fact, many players had overlooked or not expected the functionalities. The reason for this was mainly because hints given to the player about their use were overlooked in most cases, and also inconsistency of buttons to use the functionalities was observed. The evaluation took the form of a report that described these and other keyfindings in detail. Compared to the usability test, where we often choose a presentation format, this is recommended due to the complexity and provides a good basis for further development. Here, the additional time required must be taken into account. We also evaluated some additional findings that became apparent on the way to testing the relevant functions. These led to further test scenarios that can be highlighted in the future.
Game Test & Usability Test – same same but different?!
In summary, it can be stated that game testing and usability testing do not differ fundamentally. Preparation, execution and evaluation are very similar. However, game testing requires a more time-intensive preparation on the part of the researcher and the participants than a classic usability test due to the complexity and the variants in the course of the test. However, since the intrinsic motivation for such tests is particularly high, this does not seem to lead to difficulties in recruitment. The development towards more user research in gaming is very desirable and we look forward to many more projects of this kind.
PS: Don’t give up if the cooking result is not quite as expected! 😉
We have aroused your interest? Take a look at our services!