Learning Temporal Graph Representations for Intelligent Control in 3D Endless Runner Games

Oddy Virgantara Putra; Daffa Sesa Rabbani; Alvin Fredericco; Arsyapradana Fadlanabil Bahri

doi:10.34148/teknika.v15i1.1416

Authors

Oddy Virgantara Putra Department of Informatics, Faculty of Science and Technology, Universitas Darussalam Gontor, Ponorogo, East Java, Indonesia
Daffa Sesa Rabbani Department of Informatics, Faculty of Science and Technology, Universitas Darussalam Gontor, Ponorogo, East Java, Indonesia
Alvin Fredericco Department of Informatics, Faculty of Science and Technology, Universitas Darussalam Gontor, Ponorogo, East Java, Indonesia
Arsyapradana Fadlanabil Bahri Department of Informatics, Faculty of Science and Technology, Universitas Darussalam Gontor, Ponorogo, East Java, Indonesia

DOI:

https://doi.org/10.34148/teknika.v15i1.1416

Keywords:

Gesture Recognition, Graph Neural Network, Long Short-Term Memory, Media Pipe, Endless Runner

Abstract

Recent advancements in deep learning have significantly enhanced body gesture recognition, enabling real-time interaction between humans and machines through the modeling of spatial–temporal features. However, many existing approaches primarily rely on frame-based or visual feature representations and are often evaluated in offline settings, which limits their stability and responsiveness when applied to real-time 3D game environments that require continuous and dynamic player movement. In this paper, we develop a gesture-controlled endless runner game using a skeleton-based Graph Neural Network–Long Short-Term Memory (GNN–LSTM) model. The proposed system enables real-time interaction without the need for conventional input devices and is directly integrated into a Unity-based game environment. A dataset of 1,000 gesture videos across five classes (Jump In Place, Jump Left, Jump Right, Looking Down, and Still Pose) is processed using MediaPipe Pose to extract 33 body keypoints per frame, which are then normalized and represented as graph structures to capture spatial and temporal motion patterns. Experimental results show that the GNN–LSTM model achieves a validation accuracy of up to 97.5% and a test accuracy of 96%. Although CNN–LSTM attains slightly higher test accuracy, the GNN–LSTM model demonstrates more stable validation performance and robustness by leveraging skeleton-based representations, making it more suitable for real-time gesture control in interactive gameplay. Integrated with Unity, the proposed system allows intuitive and responsive control of character movements during gameplay. These findings highlight the effectiveness of temporal graph-based representations for stable and natural gesture-based human–computer interaction in real-time 3D games.

Downloads

Download data is not yet available.

References

[1] M. Wu, “Gesture Recognition Based on Deep Learning: A Review,” EAI Endorsed Transactions on e-Learning, vol. 10, pp. 1–8, Mar. 2024, doi: 10.4108/eetel.5191.

[2] S. M. Saeed, H. Akbar, T. Nawaz, H. Elahi, and U. S. Khan, “Body-Pose-Guided Action Recognition with Convolutional Long Short-Term Memory (LSTM) in Aerial Videos,” Applied Sciences, vol. 13, no. 16, p. 9384, Aug. 2023, doi: 10.3390/app13169384.

[3] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, in AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018.

[4] H. Lee et al., “Stretchable array electromyography sensor with graph neural network for static and dynamic gestures recognition system,” npj Flexible Electronics, vol. 7, no. 1, p. 20, Apr. 2023, doi: 10.1038/s41528-023-00246-3.

[5] M. Nan, M. Trăscău, and A.-M. Florea, “Spatio-temporal neural network with handcrafted features for skeleton-based action recognition,” Neural Comput. Appl., vol. 36, no. 16, pp. 9221–9243, Jun. 2024, doi: 10.1007/s00521-024-09559-4.

[6] G. Abbate, A. Giusti, V. Schmuck, O. Celiktutan, and A. Paolillo, “Self-supervised prediction of the intention to interact with a service robot,” Rob. Auton. Syst., vol. 171, no. October 2023, p. 104568, 2024, doi: 10.1016/j.robot.2023.104568.

[7] H. Xia and X. Gao, “Multi-scale Mixed Dense Graph Convolution Network for Skeleton-based Action Recognition,” vol. 4, pp. 1–10, 2020, doi: 10.1109/ACCESS.2020.3049029.

[8] Y. Wang, M. Yu, N. Li, E. Gao, G. Wang, and L. Zhu, “Difference-attention graph convolutional network for skeleton-based gesture recognition,” vol. 123, no. 70, 2025.

[9] J. Sanchez, C. Neff, and H. Tabkhi, “Real-World Graph Convolution Networks (RW-GCNs) for Action Recognition in Smart Video Surveillance,” in 2021 IEEE/ACM Symposium on Edge Computing (SEC), 2021, pp. 121–134. doi: 10.1145/3453142.3491293.

[10] X. Han, Y. Cui, X. Chen, Y. Lu, and W. Hu, “Spatio-Temporal Dynamic Attention Graph Convolutional Network Based on Skeleton Gesture Recognition,” Electronics (Basel)., vol. 13, no. 18, p. 3733, Sep. 2024, doi: 10.3390/electronics13183733.

[11] D. D. N. Cahyo and S. Liyan, “Perancangan Desain Antarmuka Pengguna Game Endless Runner Matematika Dasar Berbasis Android Menggunakan Metode System Usability Scale,” Abstrak, vol. 6, pp. 54–64, 2024.

[12] M. Pertiwi and Riwinoto, “Effect Of Playability On User Experience In Game ‘Joko Run,’” Journal of Applied Multimedia and Networking, vol. 3, no. 1, pp. 1–14, 2018.

[13] T. Yu, I. Journal, T. Y. Jia, A. Hussain, Y. Yusof, and M. Motion, “Mobile Game using Hand Gesture,” International Journal of Emerging Trends in Engineering Research, vol. 8, no. 10, pp. 6842–6848, 2020, doi: 10.30534/ijeter/2020/348102020.

[14] M. Schmidt and C. Pöpel, “HETZI-jump and run: Development and evaluation of a gesture controlled game,” CEUR Workshop Proc., vol. 1734, pp. 91–99, 2016.

[15] M. Nan and A. M. Florea, “Fast Temporal Graph Convolutional Model for Skeleton-Based Action Recognition,” Sensors, vol. 22, no. 19, p. 7117, Sep. 2022, doi: 10.3390/s22197117.

[16] L. Kristiana and D. Miyanto, “Penambahan Parameter PM2 . 5 dalam Prediksi Kualitas Udara : Long Short Term Memory,” vol. 8, no. 2, pp. 188–202, 2023.

[17] J.-W. Kim, J.-Y. Choi, E.-J. Ha, and J.-H. Choi, “Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid Model,” Applied Sciences, vol. 13, no. 4, p. 2700, Feb. 2023, doi: 10.3390/app13042700.

[18] S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices,” Remote Sens. (Basel)., vol. 16, no. 3, 2024, doi: 10.3390/rs16030533.