Research Center Machine Learning“Sort with me!” – An interactive, self-learning handling robot
The demonstrator is able to pick up known as well as unknown objects placed at random in front of it by the user without any problems.
“Sort with me!” is an interactive sorting game with a self-learning and interactive handling robot. Via its sensors, the robot recognizes objects placed in front of it and sorts them according to different criteria unknown to the human player. In the next step, the human player has to guess what these criteria are and use them to sort other objects correctly. The winner is the first player to identify the secret sorting rule and communicate it to the robot. The aim of this game is to demonstrate various everyday challenges that intelligent robots must be able to overcome both today and in the future – including visual object recognition, handling objects of all kinds, and voice interaction with users.
When it comes to visual object recognition, robots must be capable of “learning” new objects in real time and recognizing certain object properties such as color, surface texture, or shape. The demonstrator evaluates these properties and uses them to determine sorting rules in the game. To achieve this, the Fraunhofer IPA researchers apply machine-learning methods such as Siamese networks and integrate different sensors.
Gripping technology functions without data basis
Unlike humans, robots find it very difficult to grasp unknown objects. Therefore, the experts have developed a model-free gripping technology for the handling robot, i.e., one that functions without the need for a data basis. Instead, a Convolutional Neural Network (CNN) is used to evaluate the gripping quality of various access points. As a result, the demonstrator is able to pick up known as well as unknown objects placed at random in front of it by the user without any problems.
Such a game becomes particularly entertaining when you can interact directly with your opponent. Thanks to neural networks and appropriate speech output, the handling robot automatically recognizes the language spoken, enabling it to ask questions, give instructions, and reply freely to questions posed by the user.
The hands-on and interactive exhibit uses a simple sorting game to demonstrate that tasks which are easy for humans to perform are in fact extremely difficult for robots.