The REXASI-PRO project uses machine learning (ML) to control wheelchair motors using a deep neural network (DNN-LNA) that interprets sensor data. The goals are to navigate efficiently without collisions, ensure a smooth and safe ride for the user, and maintain the safety of nearby people. Traditional machine learning training methods require extensive and costly data collection, which often does not cover rare scenarios, cannot replicate dangerous situations, and must deal with the privacy of the people involved in the data recordings. To overcome these challenges, the project developed a simulator to create synthetic data that allows the neural network to learn from different scenarios without real-world risks or privacy issues. While the wheelchair in the simulator is driven by an algorithm that is fully aware of its environment, the network learns to navigate solely from the data collected by the virtual sensors.
The data generated by the simulator are stored in ROSBAG files. This means that they can easily be used with pipelines implemented to work with the ROS2 framework, one of the most used in robotics and that used in the REXASI-PRO project. Storing the output of the simulator in ROSBAG files also allows to use the visualisation tools available in the ROS2 framework to inspect it. The video shows an example where one of these tools (RViz) is used to display the data produced during a simulation where a wheelchair is travelling down a corridor with ten people coming from the opposite direction. The image on the top left of the figure shows the RGB data captured by one of the cameras installed on the wheelchair, while the image on the bottom right shows the depth information for the same scene (darker objects are closer). The grid shown in the main window represents the ground, while the reference systems with red, green and blue axes represent the position and orientation of each simulated agent and sensor. The reference systems of the wheelchair and its sensors are easy to identify as they are four, move consistently and in the opposite direction to all the others. The white dots show the raw data collected by the LIDAR installed on the wheelchair. The two straight lines are the wall of the corridor, while the two half moons near each person are generated by the LIDAR laser bouncing in front of their legs.