Are Loot Boxes beneficial for Game Engagement? Learn more about a study we conducted about two different progression systems in a controlled environment.
Loot boxes are items that can be obtained in video games after completing certain tasks. They provide randomized items and/or other randomized virtual rewards. Within the last years, the became commonly utilized in a lot of video games in various ways. Besides other microtransactions, they emerged as a key element for the commercial success of games.
They can either be obtained using in-game or real-world currency or as a reward for completing in-game tasks.
Sometimes loot box rewards can be exchanged for other in-game items or currency. Many games are offering in-game markets, making it possible to exchange virtual goods bought with real-world money.
It is undoubtedly true that loot boxes come in different visual representations. Although they may look different, the gained rewards are unpredictable before opening a loot box.
Criticism about loot boxes has been voiced by many researchers, users/gamers and governments. Loot boxes are often associated with gambling and gaming addiction [1, 2, 3]. The uncertainty, if and what can be won, is comparable to classic slot machines.
Furthermore, companies try to hide the real-world price of loot boxes by transitioning the real money to an in-game currency. Some companies take this approach a step further, introducing multiple virtual currencies to confuse the process of figuring out how much “real” money users just spent.
There are countless stories on the internet documenting that kids spent their parents money on loot boxes, became addicted and so on. I don’t want to dig deeper into this rabbit hole right here, but I encourage you to read more about this topic if you’re interested. There is one paper in particular that stood out to me: “Lifting the Lid on Loot-Boxes” by James Close and Joanne Lloyd  - an independent paper summarizing the situation to the best of possibilities.
Back to the topic.
We developed a video game. A Brick Breaker video game in two versions. The main goal is to complete all six levels. A level is completed by destroying all blocks/obstacles on the field by guiding a ball into them. If the ball touches the floor, the player takes damage.
Both versions only differ in the progression system that is shown to players after each level. One version provides a skilltree progression system, the other shows a set of loot boxes a user may select from.
Both versions yield the same rewards in the same order, without the users knowing this fact.
So, we conducted a controlled experiment. The skilltree version was employed as baseline as it has not been suspected of leading to addictive effects in literature. The research question that was guiding this experiment is:
“Does a loot box based progression system yield a higher Game Engagement score than a skilltree based system?”
Both game versions were testes by independent groups.
After completing the game, users filled out the Game Engagement Questionnaire (GEQ) . It consists of 19 items arranged in difficulty order, with the easiest to agree with at the bottom of the table. Each item is corresponding to a psychological construct, as shown below:
The Game Engagement Questionnaire was translated into German.
61 users participated (45 male, 13 female, 0 diverse). We excluded three participants, because they took too long to finish the questionnaire in a reliable manner. 58 participants remained (13 female, 45 male) that ranged from 15 to 39 years.
Reliability tests for all GEQ items of the loot box version resulted in an α = 0.73, while the skilltree version resulted in α = 0.77. Both group’s values are considered acceptable, with Cronbach’s-alpha values above 0.7.
Also, Shapiro-Wilk normality tests have been performed (Absorption: Lootbox (L) = 0.759, Skilltree (S) = 0.507; Flow: L = 0.493, S = 0.110; Presence L = 0.203, S = 0.532). To further prove or disprove homogeneity of variances, Levene’s f-test has been calculated for each construct (Absorption α = 0.369, Flow α = 0.686, Presence α = 0.545).
Two find out whether there are differences in the perceived Game Engagement, we conducted two sample t-tests for equal variances for the constructs Absorption, Flow and Presence. For the construct Immersion, we chose the Wilcoxon-test for non-parametric independent samples.
As the results in the above table show, all measured results are insignificant, due to being above the threshold of α = 0.05. This therefore leads to insignificant results for all constructs.
The participants that played the game using the loot box progression system did not experience a statistically significant difference in Game Engagement compared to the participants that used the skilltree progression system.
In this study, loot boxes on their own, isolated from monetary practices, do not provide a more engaging experience for players, compared to the baseline skilltree.
Real-world currency investments could lead to a higher Game Engagement, as stated by Larche et al. . We didn’t/couldn’t account for that. Further research is advised to find out if there’s a difference between paid and free loot boxes in terms of Game Engagement.
Players are more likely to increase investments into loot boxes, if they can forsee some return of investment (in rare in-game items) . The concept of value integration of gainable items was not integrated in this study.
In other video games, depending on the type of loot box, duplicate items might drop. Collecting those duplicate parts (and re-using them) might exhibit changes in Game Engagement.
Many video games detach progression off of loot boxes, dropping only cosmetic, not gameplay altering, items. Thus, future studies might look into changes between loot boxes yielding cosmetic and essential items. It is also questionable whether the approach of using loot boxes as a progression system is the way to go.
It might be important to justify the experimental design better than we did.
Please note that this blog-entry aims at increasing your interest in this topic. A lot of design decisions made developing the game and the study itself are questionable.
Don’t take our results at face value.
Thanks for reading 💖 Mathias
A big thank you to |Xion| for developing the game with me and to our supervisor for providing invaluable feedback.
Play the game on itch.io (free).
Check out the sourcecode on a GitLab server (free). Please bare in mind that we changed versioning software at the end of development, so the history does not reflect the development process at all.
Sources. Scientific papers mentioned above. These are not all sources we used, but might peek your interest in the topic. Those are good reads:
 Heather Wardle and David Zendle. 2021. Loot Boxes, Gambling, and Problem Gambling Among Young People: Results from a CrossSectional Online Survey. Cyberpsychology, Behavior, and Social Networking 24, 4 (2021), 267–274. https://doi.org/10.1089/cyber.2020.0299 arXiv:https://doi.org/10.1089/cyber.2020.0299 PMID: 33103911.
 David Zendle. 2020. Beyond loot boxes: a variety of gambling-like practices in video games are linked to both problem gambling and disordered gaming. PeerJ 8 (July 2020), e9466. https://doi.org/10.7717/peerj.9466
 David Zendle and Paul Cairns. 2018. Video game loot boxes are linked to problem gambling: Results of a large-scale survey. PLOS ONE 13, 11 (11 2018), 1–12. https://doi.org/10.1371/journal.pone.0206767
 James Close and Joanne Lloyd. 2021. Lifting the Lid on Loot-Boxes Chance-Based Purchases in Video Games and the Convergence of Gaming and Gambling. Report. https://www.begambleaware.org/sites/default/files/2021-03/Gaming_and_Gambling_Report_Final.pdf
 Jeanne H. Brockmyer, Christine M. Fox, Kathleen A. Curtiss, Evan McBroom, Kimberly M. Burkhart, and Jacquelyn N. Pidruzny. 2009. The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing. Journal of Experimental Social Psychology 45, 4 (2009), 624–634. https://doi.org/10.1016/j.jesp.2009.02.016
 Chanel J. Larche, Katrina Chini, Christopher Lee, Mike J. Dixon, and Myra Fernandes. 2019. Rare Loot Box Rewards Trigger Larger Arousal and Reward Responses, and Greater Urge to Open More Loot Boxes. Journal of Gambling Studies 37 (2019), 1573–3602. https://doi.org/10.1007/s10899-019-09913-5