Can Your Game Controller Predict Your Rehab Progress? Scientists Say Yes

Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 2 Related Work In line with our research objectives, we first review previous work in games for rehabilitation, and more precisely the modifications applied to the games to adapt to the rehabilitation specifications. Then we review related work on input gesture modelling in the context of gesture-based interaction. 2.1 Modifying Games for Rehabilitation A large body of research has been dedicated to developing Serious Games (Michael and Chen, 2006; Susi et al., 2007) and design guidelines adapted to different patient’s needs have been established for their creation, such as those targeting stroke patients and upper arm rehabilitation (Burke et al., 2009; Barrett et al., 2016). More recently, repurposing commercial games has been proposed (Walther-Franks et al., 2013) with the intent of leveraging their established gameplay. The distinction between serious games and commercial games can sometimes be ambiguous. Commercial games intentionally designed for physical exercise, such as those present on the Wii Fit platform, or serious games based on well-known commercial games (Burke et al., 2009; Khademi et al., 2014) are conceptually very similar. The gameplay from commercial games is however unparalleled and Barret et al. (Barrett et al., 2016) found in their review that most appreciated serious games were those closely inspired by commercial ones. While likely more engaging than serious games, commercial games are usually less flexible in their potential for adaptation. One commonality in the literature is the design of a new input control to adapt to different motor rehabilitation needs (Plaisant et al., 2000; Ketcheson et al., 2016; Khademi et al., 2014; Monedero et al., 2014; Chatta et al., 2015; Mandryk et al., 2013). Traditional hand-held controllers have been replaced by or augmented with other systems. These sense parts of the user’s body other than the hand, to solicit the motions required for physical rehabilitation while acting as input devices for control. Many studies have relied on the flexibility of optical technologies such as Microsoft Kinect (Bao et al., 2013; Chatta et al., 2015; Walther-franks et al., 2013), Leap Motion controller (Khademi et al., 2014) or motions capture systems (Munoz et al., 2018). Accelerometer based sensors from the Wii system were ˜ also employed (Walther-franks et al., 2013; Monedero et al., 2014) while instrumented versions of exercise equipment, such as recumbent bicycles (Ketcheson et al., 2016), have also been used as input devices. Likewise, our work relies on a custom gamepad to recruit arm gestures in an interaction with a commercial game originally designed for finger-based control. Then, the actual gameplay can be changed to fit the rehabilitation requirements. Walther-Franks et al. (WaltherFranks et al., 2013) proposed applying graphical overlays to unmodified games to fed back how well users performed rehabilitation-effective motions. Several systems have elaborated on this idea. Ketcheson et al. (Ketcheson et al., 2016) displayed the heart rate of the player and showed good results to support anti-sedentary levels of exertion, while Chatta et al. (Chatta et al., 2015) measured increased level of physical activity by employing graphical overlays to motivate players engaged with a commercial racing game. However, Chatta et al. also point out that this decouples exercise and in-game routine and, while it cleverly rely on “inconvenient interactions” (Rekimoto and Tsujita, 2014), such methods can go against common recommendations for serious game design (Sinclair et al., 2009). These are examples of fixed modifications. Strategies of automatic adaptation to the players’ skill level and difficulties have also be investigated, notably in the field of serious gaming (Hendrix et al., 2019; Burke et al., 2009; Sinclair et al., 2009). These works highlight several challenges. Hendrix et al. (Hendrix et al., 2019) rightfully point out that games should not rely on presets, which require by nature a subjective player skill assessment, but adapt to an ideal challenge, Game knwoledge can be useful for that matter (Sinclair et al., 2009). For instance, Burke et al. (Burke et al., 2009) mention that game difficulty can be tuned with the pace of the game, with a difficulty increasing along with the speed at which game elements move in the game. A common approach to tune these parameters is to use heuristic rules. Munoz et al. (Mu ˜ noz et al., 2018) ˜ used heuristic rules to adapt both game difficulty and rehabilitation goals for a serious game they designed, inspired by Pong, targeting an optimal pulse rate. A set of rules linked to rehabilitation goals was also used by Pirovano et al. (Pirovano et al., 2016) to adapt the difficulty of custom games to specific users and a numerical objective was set to maintain an 80% success rate in the game. Similar game design elements are used in our system, and the method we propose is aiming at removing the needs for using presets to adapt for game difficulty. Instead the baseline gameplay model sets the default challenge. 2.2 Input control performance measure and modelling Modifying a control input or swapping one for another purposefully designed is likely to have an impact on performance. A common measure of performance in games is the score, for example linked to the number of hits in First Person Shooter (FPS) games (Gerling et al., 2011). Isokoski and Martin (Isokoski and Martin, 2007) compared a keyboard with a standard mouse, a wheel mouse, a trackmouse and a Xbox360 gamepad. The gamepad was outperformed by all other devices in the task of FPS target acquisition. Another approach is to compute the information transmitted by the control inputs. Compared to a standard mouse, the Wii remote has been shown to perform poorly in terms of throughput, speed and error rate (Natapov et al., 2009), while an Xbox gamepad performed equally well in tracking a target’s velocity (Klochek and MacKenzie, 2006). These differences can be explained by the differences in the limbs and muscles they recruit and is the topic of seminal research works. Card et al. (Card et al., 1991) compiled several Fitts’ law experiments involving different limbs and showed a difference in available information throughput between neck, arm, wrist and finger – with four-fold decrease between finger and arm from 38bit/s to 9.5bit/s. Targeting more complex motions than pointing, Oulasvirta et al. (Oulasvirta et al., 2013) measured the information throughput in full body motions to provide researchers with a quantitative measure of control input capacity. Finally, to help designer in understanding the impact of new input methods on the user body, Bachynskyi et al. (Bachynskyi et al., 2015) used a clustering approach to summarise results about 3d pointing. In summary, newly designed input controls are likely sub-optimal when compared to reference devices and methods to predict performance beyond pointing, in particular (Oulasvirta et al., 2013; Bachynskyi et al., 2015), exist but are heavy to deploy in workshop settings. Models for gameplay interaction have also been developed. For example, Smith et al. (Smith and Nayar, 2016) defined it as the player’s raw input in contrast to methods that were only considering in-game states or event logs. In their work, they showed that models of gameplay can reliably identify player’s identity based on their playstyle measured as “input words”, i.e. strings of actions performed by the users. Patterns linking user identity to input behaviour was also found in the study from Dhakal et al. (Dhakal et al., 2018) in which they observed 136 millions keystroke of text input. In their analysis, several clusters identified users’ typing behaviours. One of their performance measure included interkey interval, which is also a feature we rely on in this work. Models of players have been used to replace humans in games by emulating their behaviour (Pfau et al., 2020). They have also been used for game design generation to optimise game worlds with specific outcome (Karavolos et al., 2018). When one has a direct access to player satisfaction, methods such as those proposed by Yannakakis et al. (Yannakakis and Hallam, 2009) can be employed, in which they trained a model to control the parameters of a custom game in real-time using recorded player satisfaction in pairwise A/B tests and sampled possible game designs through genetic optimisation. In contrast, the present work employs gameplay modelling for the purpose of identifying input configurations compatible with a baseline model. 2.3 Summary The present work is, to the best of our knowledge, the first to develop baseline gameplay models of commercial games for adaptation to physical rehabilitation. In doing so, we tackle two challenges. The first challenge is finding parameters for the interaction loop that allows for rehabilitation and gameplay adaptation; this is addressed through co-design with occupational therapists. The second challenge is building a model of baseline gameplay from recorded game sessions; we solve this by proposing a statistical model, based on data captured in an experimental study, with four parameters that leverages player typing behaviour and recorded effects on game state. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Authors: Authors: (1) Antoine Loriette, IRCAM, CNRS, Sorbonne Universite, Paris, France (antoine.loriette@gmail.com); (2) Baptiste Caramiaux, Sorbonne Universite, CNRS, ISIR, Paris, France (caramiaux@isir.upmc.fr); (3) Sebastian Stein, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (sebastian.stein@glasgow.ac.uk); (4) John H. Williamson, School of Computing Science, University of Glasgow, Glasgow, Scotland, United Kingdom (johnh.williamson@glasgow.ac.uk). Table of Links Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2 Related Work and 2.1 Modifying Games for Rehabilitation 2.2 Input control performance measure and modelling and 2.3 Summary 2.2 Input control performance measure and modelling and 2.3 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3 Motivation from a rehabilitation context and 3.1 Observations and Brainstorming workshops 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 3.2 System description 3.3 Interaction parameterisation 3.4 Summary 4 Experimental Study 4 Experimental Study 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5 Modelling User Behaviour and 5.1 Behavioural Features 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 5.2 Gameplay Reference Model 5.3 Comparing NLL to NSCORE 5.4 Effect of design parameters on NLL 5.5 Sampling Period 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6 Discussion and 6.1 Design workshop and tools for modelling 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 6.2 Interaction parameters adaptation 6.3 Modelling Gameplay 6.4 Limitations and Future work 7 Conclusion 8 Acknowledgements and References 7 Conclusion 8 Acknowledgements and References 8 Acknowledgements and References 2 Related Work In line with our research objectives, we first review previous work in games for rehabilitation, and more precisely the modifications applied to the games to adapt to the rehabilitation specifications. Then we review related work on input gesture modelling in the context of gesture-based interaction. 2.1 Modifying Games for Rehabilitation A large body of research has been dedicated to developing Serious Games (Michael and Chen, 2006; Susi et al., 2007) and design guidelines adapted to different patient’s needs have been established for their creation, such as those targeting stroke patients and upper arm rehabilitation (Burke et al., 2009; Barrett et al., 2016). More recently, repurposing commercial games has been proposed (Walther-Franks et al., 2013) with the intent of leveraging their established gameplay. The distinction between serious games and commercial games can sometimes be ambiguous. Commercial games intentionally designed for physical exercise, such as those present on the Wii Fit platform, or serious games based on well-known commercial games (Burke et al., 2009; Khademi et al., 2014) are conceptually very similar. The gameplay from commercial games is however unparalleled and Barret et al. (Barrett et al., 2016) found in their review that most appreciated serious games were those closely inspired by commercial ones. While likely more engaging than serious games, commercial games are usually less flexible in their potential for adaptation. One commonality in the literature is the design of a new input control to adapt to different motor rehabilitation needs (Plaisant et al., 2000; Ketcheson et al., 2016; Khademi et al., 2014; Monedero et al., 2014; Chatta et al., 2015; Mandryk et al., 2013). Traditional hand-held controllers have been replaced by or augmented with other systems. These sense parts of the user’s body other than the hand, to solicit the motions required for physical rehabilitation while acting as input devices for control. Many studies have relied on the flexibility of optical technologies such as Microsoft Kinect (Bao et al., 2013; Chatta et al., 2015; Walther-franks et al., 2013), Leap Motion controller (Khademi et al., 2014) or motions capture systems (Munoz et al., 2018). Accelerometer based sensors from the Wii system were ˜ also employed (Walther-franks et al., 2013; Monedero et al., 2014) while instrumented versions of exercise equipment, such as recumbent bicycles (Ketcheson et al., 2016), have also been used as input devices. Likewise, our work relies on a custom gamepad to recruit arm gestures in an interaction with a commercial game originally designed for finger-based control. Then, the actual gameplay can be changed to fit the rehabilitation requirements. Walther-Franks et al. (WaltherFranks et al., 2013) proposed applying graphical overlays to unmodified games to fed back how well users performed rehabilitation-effective motions. Several systems have elaborated on this idea. Ketcheson et al. (Ketcheson et al., 2016) displayed the heart rate of the player and showed good results to support anti-sedentary levels of exertion, while Chatta et al. (Chatta et al., 2015) measured increased level of physical activity by employing graphical overlays to motivate players engaged with a commercial racing game. However, Chatta et al. also point out that this decouples exercise and in-game routine and, while it cleverly rely on “inconvenient interactions” (Rekimoto and Tsujita, 2014), such methods can go against common recommendations for serious game design (Sinclair et al., 2009). These are examples of fixed modifications. Strategies of automatic adaptation to the players’ skill level and difficulties have also be investigated, notably in the field of serious gaming (Hendrix et al., 2019; Burke et al., 2009; Sinclair et al., 2009). These works highlight several challenges. Hendrix et al. (Hendrix et al., 2019) rightfully point out that games should not rely on presets, which require by nature a subjective player skill assessment, but adapt to an ideal challenge, Game knwoledge can be useful for that matter (Sinclair et al., 2009). For instance, Burke et al. (Burke et al., 2009) mention that game difficulty can be tuned with the pace of the game, with a difficulty increasing along with the speed at which game elements move in the game. A common approach to tune these parameters is to use heuristic rules. Munoz et al. (Mu ˜ noz et al., 2018) ˜ used heuristic rules to adapt both game difficulty and rehabilitation goals for a serious game they designed, inspired by Pong, targeting an optimal pulse rate. A set of rules linked to rehabilitation goals was also used by Pirovano et al. (Pirovano et al., 2016) to adapt the difficulty of custom games to specific users and a numerical objective was set to maintain an 80% success rate in the game. Similar game design elements are used in our system, and the method we propose is aiming at removing the needs for using presets to adapt for game difficulty. Instead the baseline gameplay model sets the default challenge. 2.2 Input control performance measure and modelling Modifying a control input or swapping one for another purposefully designed is likely to have an impact on performance. A common measure of performance in games is the score, for example linked to the number of hits in First Person Shooter (FPS) games (Gerling et al., 2011). Isokoski and Martin (Isokoski and Martin, 2007) compared a keyboard with a standard mouse, a wheel mouse, a trackmouse and a Xbox360 gamepad. The gamepad was outperformed by all other devices in the task of FPS target acquisition. Another approach is to compute the information transmitted by the control inputs. Compared to a standard mouse, the Wii remote has been shown to perform poorly in terms of throughput, speed and error rate (Natapov et al., 2009), while an Xbox gamepad performed equally well in tracking a target’s velocity (Klochek and MacKenzie, 2006). These differences can be explained by the differences in the limbs and muscles they recruit and is the topic of seminal research works. Card et al. (Card et al., 1991) compiled several Fitts’ law experiments involving different limbs and showed a difference in available information throughput between neck, arm, wrist and finger – with four-fold decrease between finger and arm from 38bit/s to 9.5bit/s. Targeting more complex motions than pointing, Oulasvirta et al. (Oulasvirta et al., 2013) measured the information throughput in full body motions to provide researchers with a quantitative measure of control input capacity. Finally, to help designer in understanding the impact of new input methods on the user body, Bachynskyi et al. (Bachynskyi et al., 2015) used a clustering approach to summarise results about 3d pointing. In summary, newly designed input controls are likely sub-optimal when compared to reference devices and methods to predict performance beyond pointing, in particular (Oulasvirta et al., 2013; Bachynskyi et al., 2015), exist but are heavy to deploy in workshop settings. Models for gameplay interaction have also been developed. For example, Smith et al. (Smith and Nayar, 2016) defined it as the player’s raw input in contrast to methods that were only considering in-game states or event logs. In their work, they showed that models of gameplay can reliably identify player’s identity based on their playstyle measured as “input words”, i.e. strings of actions performed by the users. Patterns linking user identity to input behaviour was also found in the study from Dhakal et al. (Dhakal et al., 2018) in which they observed 136 millions keystroke of text input. In their analysis, several clusters identified users’ typing behaviours. One of their performance measure included interkey interval, which is also a feature we rely on in this work. Models of players have been used to replace humans in games by emulating their behaviour (Pfau et al., 2020). They have also been used for game design generation to optimise game worlds with specific outcome (Karavolos et al., 2018). When one has a direct access to player satisfaction, methods such as those proposed by Yannakakis et al. (Yannakakis and Hallam, 2009) can be employed, in which they trained a model to control the parameters of a custom game in real-time using recorded player satisfaction in pairwise A/B tests and sampled possible game designs through genetic optimisation. In contrast, the present work employs gameplay modelling for the purpose of identifying input configurations compatible with a baseline model. 2.3 Summary The present work is, to the best of our knowledge, the first to develop baseline gameplay models of commercial games for adaptation to physical rehabilitation. In doing so, we tackle two challenges. The first challenge is finding parameters for the interaction loop that allows for rehabilitation and gameplay adaptation; this is addressed through co-design with occupational therapists. The second challenge is building a model of baseline gameplay from recorded game sessions; we solve this by proposing a statistical model, based on data captured in an experimental study, with four parameters that leverages player typing behaviour and recorded effects on game state. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv