Classification of human motor activity based on multisensory data analysis

Obukhov A.D. Classification of human motor activity based on multisensory data analysis. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2025, vol. 25, no. 5, pp. 833–843 (in Russian). doi: 10.17586/2226-1494-2025-25-5-833-843

An analysis of multisensor data obtained from an electromyograph, inertial measurement devices, a computer-vision system, and virtual-reality trackers was performed in order to solve the problem of classifying human motor activity. The relevance of solving this problem is determined by the necessity of analyzing and recognizing human motor activity when using various hardware and software complexes, for example, rehabilitation and training systems. For the optimal solution of the task of recognizing the type of hand movements with the highest accuracy, the contribution of each signal source is evaluated, and a comparison of various machine-learning models is performed. The approach to processing multisensor data includes: synchronized acquisition of streams from different sources; labeling of the initial data; signal filtering; dual alignment of time series by frequency and duration with approximation to a common constant; formation of a common dataset; training and selection of a machine-learning model for recognizing motor activity of the hands. Nine machine-learning models are considered: logistic regression, k-nearest neighbors, naïve Bayes classifier, decision tree, and ensembles based on them (Random Forest, AdaBoost, Extreme Gradient Boosting, Voting, and Stacking Classifier). The developed approach of synchronization, filtering, and dual alignment of data streams makes it possible to form a unified dataset of multisensor data for model training. An experiment was carried out on the classification of nine categories of hand movements based on the analysis of multisensor data (629 recordings collected from 15 participants). Training was performed on 80 % of the collected data with five-fold cross-validation. The AdaBoost ensemble provides a classification accuracy of 98.8 % on the dataset composed of the combined information from four different sources. In the course of ablation analysis for comparing the data sources, the greatest influence on the final classification accuracy is exerted by information from virtual-reality trackers (up to 98.73 % ± 1.78 % accuracy on the AdaBoost model), while data on muscle activity from the electromyograph turned out to be the least informative. It was determined that high classification accuracy of motor activity can be obtained using inertial measurement devices. The considered study formalizes a reproducible approach to processing multisensor data and makes it possible to objectively compare the contribution of different sources of information and machine-learning models in solving the problem of classifying the motor activity of the user’s hands within rehabilitation and virtual training systems. It is shown that under resource limitations it is possible to refuse part of the data sources without significant loss of classification accuracy, simplifying the hardware configuration of tracking systems and making it possible to move from closed commercial systems (virtual-reality trackers) to more accessible and compact inertial measurement devices.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License