Are Loot Boxes beneficial for Game Engagement? Learn more about a study we conducted about two different progression systems in a controlled environment.
What are Loot Boxes?
Loot boxes are items that can be obtained in video games after completing certain tasks. They provide randomized items and/or other randomized virtual rewards. In recent years, has become widely used in a variety of video games in various ways. In addition to other microtransactions, they have emerged as a critical component of game commercial success.
They can either be obtained using in-game or real-world currency or as a reward for completing in-game tasks.
Sometimes loot box rewards can be exchanged for other in-game items or currency. Many games are offering in-game markets, making it possible to exchange virtual goods bought with real-world money.
It is undoubtedly true that loot boxes come in different visual representations. Although they may look different, the gained rewards are unpredictable before opening a loot box.
Criticism about Loot Boxes
Criticism about loot boxes has been voiced by many researchers, users, and gamers, as well as by governments. Loot boxes are frequently linked to gambling and gaming addiction [1, 2, 3]. The uncertainty of if and what can be won is comparable to classic slot machines.
Furthermore, companies try to hide the real-world price of loot boxes by converting real money into in-game currency. Some companies take this approach a step further, introducing multiple virtual currencies to confuse the process of figuring out how much "real" money users just spent.
There are countless stories on the internet documenting that kids spent their parents' money on loot boxes, became addicted, and so on. I don’t want to dig deeper into this rabbit hole right here, but I encourage you to read more about this topic if you’re interested. There is one paper in particular that stood out to me: "Lifting the Lid on Loot-Boxes" by James Close and Joanne Lloyd [4], an independent paper summarizing the situation to the best of its possibilities.
The developed stimuli
Back to the topic.
We developed a video game. A Brick Breaker video game in two versions. The main goal is to complete all six levels. A level is completed by destroying all blocks/obstacles on the field by guiding a ball into them. If the ball touches the floor, the player takes damage.
Both versions only differ in the progression system that is shown to players after each level. One version provides a skilltree progression system, the other shows a set of loot boxes a user may select from.
Both versions yield the same rewards in the same order, without the users knowing this fact.
Study
So, we conducted a controlled experiment. The skill tree version was employed as a baseline as it has not been suspected of leading to addictive effects in literature. The research question that guided this experiment is:
"Does a loot box-based progression system yield a higher game engagement score than a skill tree-based system?"
Both game versions were tested by independent groups.
After completing the game, users filled out the Game Engagement Questionnaire (GEQ) [5]. It consists of 19 items arranged in difficulty order, with the easiest to agree with at the bottom of the table. Each item corresponds to a psychological construct, as shown below:
The Game Engagement Questionnaire was translated into German.
Results
Participants
61 users participated (45 male, 13 female, 0 diverse). We excluded three participants because they took too long to complete the questionnaire in a reliable manner. The remaining 58 participants (13 females and 45 males) ranged in age from 15 to 39 years.
Reliability tests for all GEQ items in the loot box version resulted in an of 0.73, while those in the skill tree version resulted in an of 0.77. Both groups' values are considered acceptable, with Cronbach's alpha values above 0.7.
Some upstream tests
Also, Shapiro-Wilk normality tests have been performed (Absorption: Lootbox (L) = 0.759, Skilltree (S) = 0.507; Flow: L = 0.493, S = 0.110; Presence L = 0.203, S = 0.532). To further prove or disprove homogeneity of variances, Levene’s f-test has been calculated for each construct (Absorption α = 0.369, Flow α = 0.686, Presence α = 0.545).
Group comparisons
To find out whether there are differences in the perceived level of game engagement, we conducted two sample t-tests for equal variances for the constructs absorption, flow, and presence. We chose the Wilcoxon-test for non-parametric independent samples for the construct Immersion.
As the results in the preceding table show, all measured results are insignificant because they exceed the = 0.05 threshold. This therefore leads to insignificant results for all constructs.
What does this mean?
The participants that played the game using the loot box progression system did not experience a statistically significant difference in game engagement compared to the participants that used the skill tree progression system.
In this study, loot boxes on their own, isolated from monetary practices, did not provide a more engaging experience for players compared to the baseline skill tree.
What is missing in this study? What could be improved?
Real-world currency investments could lead to higher game engagement, as stated by Larche et al. [6]. We didn't or couldn't account for that. Further research is advised to find out if there’s a difference between paid and free loot boxes in terms of player engagement.
Players are more likely to increase their investments in loot boxes if they can see some return on their investment (in rare in-game items) [6]. The concept of value integration of gainable items was not integrated in this study.
In other video games, depending on the type of loot box, duplicate items might drop. Collecting those duplicate parts (and reusing them) might exhibit changes in game engagement.
Many video games detach progression from loot boxes, dropping only cosmetic, not gameplay-altering, items. Thus, future studies might look into the differences between loot boxes yielding cosmetic and essential items. It is also questionable whether the approach of using loot boxes as a progression system is the right way to go.
It might be important to justify the experimental design better than we did.
Some final notes
Please note that this blog entry aims to increase your interest in this topic. A lot of the design decisions made in developing the game and the study itself are questionable.
Don’t take our results at face value.
Thanks for reading 💖 Mathias
A big thank you to |Xion| for developing the game with me and to our supervisor for providing invaluable feedback.
Play the game on itch.io (free).
Check out the sourcecode on a GitLab server (free). Please bare in mind that we changed versioning software at the end of development, so the history does not reflect the development process at all.
Sources. Scientific papers mentioned above. These are not all sources we used, but might peek your interest in the topic. Those are good reads:
[1] Heather Wardle and David Zendle. 2021. Loot Boxes, Gambling, and Problem Gambling Among Young People: Results from a CrossSectional Online Survey. Cyberpsychology, Behavior, and Social Networking 24, 4 (2021), 267–274. https://doi.org/10.1089/cyber.2020.0299
arXiv:https://doi.org/10.1089/cyber.2020.0299 PMID: 33103911.
[2] David Zendle. 2020. Beyond loot boxes: a variety of gambling-like practices in video games are linked to both problem gambling and disordered
gaming. PeerJ 8 (July 2020), e9466. https://doi.org/10.7717/peerj.9466
[3] David Zendle and Paul Cairns. 2018. Video game loot boxes are linked to problem gambling: Results of a large-scale survey. PLOS ONE 13, 11 (11
2018), 1–12. https://doi.org/10.1371/journal.pone.0206767
[4] James Close and Joanne Lloyd. 2021. Lifting the Lid on Loot-Boxes Chance-Based Purchases in Video Games and the Convergence of Gaming and
Gambling. Report. https://www.begambleaware.org/sites/default/files/2021-03/Gaming_and_Gambling_Report_Final.pdf
[5] Jeanne H. Brockmyer, Christine M. Fox, Kathleen A. Curtiss, Evan McBroom, Kimberly M. Burkhart, and Jacquelyn N. Pidruzny. 2009. The
development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology
45, 4 (2009), 624–634. https://doi.org/10.1016/j.jesp.2009.02.016
[6] Chanel J. Larche, Katrina Chini, Christopher Lee, Mike J. Dixon, and Myra Fernandes. 2019. Rare Loot Box Rewards Trigger Larger Arousal and Reward
Responses, and Greater Urge to Open More Loot Boxes. Journal of Gambling Studies 37 (2019), 1573–3602. https://doi.org/10.1007/s10899-019-09913-5