Research Center Machine Learning

Programming Artificial Intelligence Yourself via “Drag and Drop”

Practical and intuitive – the “Smart Roberta” project of the CCIT Research Center Machine Learning is designed to expand the “Open Roberta” platform of the Roberta initiative of Fraunhofer IAIS to include artificial neural networks (ANN). The goal is to make the functioning of ANN understandable to students by allowing them to work with the networks in a practical way.

For this purpose, the processing of ANN is to be integrated into the Open Roberta platform. The Open Roberta platform has already offered a graphical programming interface for many embedded systems and robots since 2014. Currently, 14 different systems are programmable. In 2020 alone, more than three million people from 120 countries worldwide have programmed robots as well as simulated hardware on the open-source platform free of charge.

In principle, it should be possible to use an ANN directly in a robot system without access to powerful external hardware. This will reduce technical and professional barriers for users. For the integration of the ANN, the library “AIfES” (Artificial Intelligence for Embedded Systems) of the Fraunhofer IMS will be used. In a first step, the framework will be provided as a precompiled software library for a widely used system, the microcontroller “Arduino Nano 33 BLE Sense”, and will make it possible to execute an ANN directly on the microcontroller and also to train it.

Gesture recognition with artificial neural networks

As a use case for “Smart Roberta”, a gesture recognition system was developed that can learn and subsequently recognize individual gestures directly on the embedded system. The integrated acceleration sensor of the microcontroller is used for this purpose.

The training of new gestures directly in the system is to be emphasized, since this is not yet possible with comparable systems. For this purpose, the Fraunhofer IMS scientists have developed a feature extraction that extracts only the most necessary features from the raw signal of the acceleration sensor. The features of a gesture are passed as input to an ANN, which then either learns a new gesture or classifies the gesture.

The AIfES framework dynamically creates a suitable ANN from the self-generated training data. The network size depends on the number of gestures to be recognized. Each gesture to be recognized is assigned to an output neuron of the ANN.

The learned gestures can be stored permanently in order to retrieve them later. For the visual implementation, the framework AIfES was integrated into Open Roberta and the block-based programming language “NEPO” was extended by new blocks for working with ANN. The user can develop a complete ANN application via the Open Roberta Lab with a microcontroller that both trains and then uses the ANN.


About the Roberta Initiative of Fraunhofer IAIS

Since 2002, Fraunhofer IAIS has been supporting STEM education among girls and boys from elementary school to secondary school, as well as in education and training, with its initiative “Roberta – Learning with Robots”. The Roberta coaches from Fraunhofer IAIS have already trained more than 3100 teachers who offer robotics and programming courses at schools throughout Germany and internationally. With Open Roberta, Fraunhofer IAIS, with the support of Google.org, has developed a platform on which millions of children and young people from more than 120 countries worldwide now playfully create programs for various robots and microcontrollers using “drag and drop”. The programming environment is developed on a Fraunhofer server at the Sankt Augustin site with special attention to data protection. For example, no cookies are used on the educational platform.

The user can develop a complete ANN application via the Open Roberta Lab with a microcontroller that both trains and then uses the ANN. © Fraunhofer IAIS