The Keraal Dataset
KinesiothErapy and Rehabilitation for Assisted Ambient Living

The Low Back Pain Physical Rehabilitation Dataset


The Keraal dataset is a medical database of clinical patients carrying out low back-pain rehabilitation exercises and is distributed under and is distributed under a CreativeCommons Attribution/Non-Commercial/Share-Alike license (CC-BY-NC-SA).
This dataset is designed for human body movement analysis in the context of low-back pain physical rehabilitation. This dataset has been acquired during a clinical study where patients were performing 3 low-back pain rehabilitation exercises while being coached by a robotic system, as described in (Blanchard et al, Biomed Research International, 2022). The dataset was recorded from 9 healthy subjects and 12 patients suffering from low-back pain, and were annotated by a rehabilitation doctor. It includes 3D skeleton sequences captured by a Kinect, color video sequences and 2D skeleton data estimated from videos. Moreover, medical expert annotations are associated to each patient's performance for assessment of correctness, recognition of errors, spatio-temporal localization of errors.
Rendering of the skeleton data for the 3 exercises :

hiding face (CTK) Keraal
flank stretch (ELK) Keraal
torso rotation (RTK) Keraal

We believe it can serve the research community of various fields from computer vision, machine learning, robotics, physical medicine and bio-mechanics.

The documentation, code for visualising the published data and processing the raw data to skeleton data is available at https://github.com/nguyensmai/KeraalDataset

Download


Download a sample of the database as a zip file
For the whole database, click to download for each group : Download the pre-trained openpose model

Dataset content


The dataset includes RGB video and skeleton files : from the kinect, the vicon, open-pose and blazepose. It also includes annotation files from a medical annotator. The files constituting the dataset are described in readme_files_description.txt. Our dataset is composed of :
  • Control command files of the robot Poppy to demonstrate the three exercises. They are json files that use the syntax commonly used by the library pypot as described in its documentation and can be used with the web-interface developed by the project Keraal to execute on Poppy. They can be used with the physical robot Poppy or its simulation.
  • the therapists' annotations in a xml anvil format. They indicate whether the execution is correct, the label of the error, the body-part causing the error and the temporal description of the beginning and ending timestamps of the error.
  • anonymised RGB videos. The videos are of avi and resolution 960x544 for Group3. They are of mp4 and resolution 480x360 for groups 1a, 1b, 2a and 2b. The resolution was kept low during the coaching sessions with the robot to allow for real-time coaching.
  • the positions and orientations of each joint of the Microsoft Kinect skeleton. The txt files display in a table, a line per timestamp. The data are presented in ASCII txt format, with space delimiter used for separating the values of positions and orientations of each joint in the order of the skeleton numbering.
  • the 2D positions of each joint of the OpenPose skeleton in the COCO pose output format. The txt files give the x and y positions on each video frame in the format of a dictionary of video frame numbers and joint names.
  • the positions and orientations of each joint of the Vicon skeleton. The txt files display in a table, a line per timestamp.

    • Available recording modalities per group and the format of the data :

    Gr. Annotation RGB videos Kinect OpenPose/BlazePose Vicon Nb rec
    1a xml anvil : err label, bodypart, timespan mp4, 480x360 tabular dictionary NA 249
    1b NA mp4, 480x360 tabular dictionary NA 1631
    2a xml anvil : err label, bodypart, timespan mp4, 480x360 tabular dictionary NA 51
    2b NA mp4, 480x360 tabular dictionary NA 151
    3 error label avi, 960x544 tabular dictionary tabular 540

    The recordings are organized into subfolders corresponding to each recording modality. In the case of groups 1a, 1b , 2a and 2b, each recording is given a unique number, which is the same for all recording modalities. The nomenclature of the files in the dataset includes the following information: modality/GId_Modality_ExerciseName_ParticipantId_RecordingId.extension. In the case of group 3, a 4th level of folders indicate the subject id. The name of the file indicates the label (ex: Error1Bodypart1) of the error and the recording id. The nomenclature of the files in this case is : modality/G3_Modality_ExerciseName_ParticipantIdTrialId_label_RecordingId.extension.

    More details can be read in https://github.com/nguyensmai/KeraalDataset

    Citation


    Please cite this dataset with the following citation :
    Sao Mai Nguyen, Maxime Devanne, Olivier Remy-Neris, Mathieu Lempereur, and Andre Thepaut (2024). A Medical Low-Back Pain Physical Rehabilitation Dataset for Human Body Movement Analysis, International Joint Conference on Neural Networks. [ bib video youtube code ]

    @inproceedings{Nguyen2024IJCNN,
    author = {Sao Mai Nguyen and Maxime Devanne and Olivier Remy-Neris and Mathieu Lempereur and Andre Thepaut},
    booktitle = {International Joint Conference on Neural Networks},
    title = {A Medical Low-Back Pain Physical Rehabilitation Database for Human Body Movement Analysis},
    year = {2024}}

    This dataset was collected during a prospective, controlled, single-blind, two-centre study conducted from October 2017 to May 2019 in two rehabilitation centres in Brittany (France). All participants signed informed consent forms for their participation. The study was written according to the TiDier checklist and descrined in :
    Blanchard, A., Nguyen, S. M., Devanne, M., Simonnet, M., Goff-Pronost, M. L., and Remy-Neris, O. (2022). Technical Feasibility of Supervision of Stretching Exercises by a Humanoid Robot Coach for Chronic Low Back Pain: The R-COOL Randomized Trial. BioMed Research International, 2022(1--10). Hindawi Limited.

    @article{Blanchard2022BRI,
    author = {Agathe Blanchard and Sao Mai Nguyen and Maxime Devanne and Mathieu Simonnet and Myriam Le Goff-Pronost and Olivier Remy-Neris},
    journal = {BioMed Research International},
    month = {mar},
    pages = {1--10},
    title = {Technical Feasibility of Supervision of Stretching Exercises by a Humanoid Robot Coach for Chronic Low Back Pain: The R-COOL Randomized Trial},
    volume = {2022},
    year = 2022}

    Motivation

    For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled?

    The purpose of this dataset is to provide anonymized data from physical rehabilitation patients to train an intelligent tutoring system (ITS) capable of supervising rehabilitation sessions autonomously, by providing instructions for each exercise of the program and real-time feedback how to improve the efficiency of the patient's performance. Thus, the ITS must be capable of
    1. assessing if the rehabilitation exercise is correctly performed,
    2. recognising the error of the patient to give a feedback localising which body part is responsible of the error,
    3. detecting the temporal segment where the error occurred.
    Compared to existing datasets, our Keraal dataset has been recorded within a long-term rehabilitation program, targeting low back pain, and labelled by a doctor. Thus our Keraal dataset is the only benchmarking set with clinical patients and with labels provided by a physician for the four challenges : motion assessment, error recognition, spatial and temporal localization.
    1. Rehabilitation motion assessment. The goal is to assess an observed motion sequence by detecting if the rehabilitation exercise is correctly performed or not. In order to fit real scenarios, only correct demonstrations provided by the dataset are available during training.
    2. Rehabilitation error recognition. In other terms, the goal is not only to detect incorrect sequences but also to classify the observed error among a set of known errors, so as to explain and give feedback. Supervised approaches are considered. To tackle this challenge, we provide both correct and simulated errors with associated labels within the dataset.
    3. Spatial localization of the error. In addition to recognizing the error, the goal is also to identify which body part is responsible of the error.
    4. Temporal localization of the error. The goal is to detect the temporal segment where the detected error occurred along the sequence.

    Who created this dataset and on behalf of which entity ?

    THe Keraal Dataset was created as a collaborative effort between IMT Atlantique and CHRU de Brest as part of the Keraal experiment, led by Sao Mai Nguyen and Andre Thepaut from IMT Atlantique and Olivier Remy Neris from CHRU de Brest. The overseeing of the patients' care was carrried out by Olivier Remy Neris and Mathieu Lempereur of CHRU de Brest. The data collection was carried out by Maxime Devanne at IMT Atlantique. The annotations were carried out by Olivier Remy Neris. The bechnmarking and data curation were carried out by Maxime Devanne and Sao Mai Nguyen, when at respectively UHA and U2IS.

    Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number

    The research work leading to the Keraal dataset is partially supported by the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 601116 as part of Keraal experiment of the ECHORD++ project and by the European Regional Development Fund (ERDF) via the VITAAL Contrat Plan Etat Region.

    Composition

    What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

    The core element of this dataset captures a human carrying a repetition of an exercise. Each exercise is identified and labelled, and is also represented by a Control command file of the humanoid robot Poppy to demonstrate the exercise. Each repetition is recorded as a RGB video and kinect skeleton (articulated figure) sequence or a Vicon motion capture skeleton sequence, and is labelled with which error and when the error was made by the subject.

    What data does each instance consist of?

    Each repetition of exercise can have as raw data :
    • an anonymized RGB video. The videos are avi files and resolution 960x544 for Group3. They are mp4 files and resolution 480x360 for groups 1a, 1b, 2a and 2b. The resolution was kept low during the coaching sessions with the robot to allow for real-time coaching.
    • a skeleton sequence recorded by the kinect. The txt files display in a table, a line per timestamp. The data are presented in ASCII txt format, with space delimiter used for separating the values of positions and orientations of each joint.
    • a skeleton sequence recorded by the vicon. The txt files display in a table, a line per timestamp.
    • annotations of errors and their body part and temporal localization by a doctor in physical rehabilitation and a physiotherapist. They are in a xml anvil format. They indicate whether the execution is correct, the label of the error, the body-part causing the error and the temporal description of the beginning and ending timestamps of the error.
    • Control command files of the robot Poppy to demonstrate the three exercises. They are json files that use the syntax commonly used by the library pypot as described in its documentation and can be used with the web-interface developed by the project Keraal to execute on Poppy. They are used to execute the movement on the physical robot Poppy or its simulation.
    We also added by post-processing of the RGB video:
    • a 2D skeleton sequence obtained by OpenPose : the files record the x and y positions on each video frame in the format of a dictionary of video frame numbers and joint names. OpenPose skeletons use the COCO pose output format.
    • a 3D skeleton sequence obtained by BlazePose : the files record the x, y and z positions on each video frame in the format of a dictionary of video frame numbers and joint names. BlazePose skeletons uses a topology of 33 human body keypoints, which is a superset of COCO, BlazeFace and BlazePalm topologies
    The pose (skeleton) output formats of Kinect, OpenPose, BlazePose and Vicon are :
    skeleton models

    Is there a label or target associated with each instance?

    All recordings are identified with : the exercise name, an identifier of the subject, his group.

    Moreover, part of the data collected have supplementary annotation:

    • Patients in Group1A: 14 recordings per exercises were selected among 6 patients for annotations. Half of these recordings are taken at the beginning of the patient care with the robot, between days 1 and 3, and half of these recordings are taken during the second half of their care program, between days 7 and 12. An initial idea was to analyse if improvement was observed between the beginning and the end of the training. The remaining recordings of the 12 patients are made available without annotation as data of Group1B.
    • healthy subjects in Group2A : 51 recordings per exercises were selected for annotations. The remaining recordings of the 6 healthy adults are made available without annotation as data of Group2B.
    • healthy subjects in Group3: 540 recordings are identified with the error label.

    The video collected from group1A and from group2A were annotated by a medical doctor in physiotherapy, using the Anvil video annotation research tool. The videos are annotated at three levels.

    • On a global evaluation level, an assessment is given as either correct or incorrect (Challenge 1). In the case of an incorrect error, he can indicate if the execution has no errors but finished before the end (label code 4 : incomplete) or the participant did not start the execution of the exercise (label code 5 : motionless).
    • On the error classification level, in the case of an incorrect movement, annotations first indicate whether the error is significant or small as well as the label of the error (Challenge 2). Moreover the body part causing the error is also indicated (Challenge 3). He can also add a comment.
    • On a temporal level, the medical doctor annotating videos can also indicate the time window where the error occurs (Challenge 4), and the same information as previously: whether the error is significant or small, the error label, and the body part causing the error. We note that in addition to the error label identified, other label codes 4 and 5 corresponding to an incomplete and motionless sequence can be used by the medical expert annotating videos.