A Knowledge-Augmented Concept for Programming by Demonstration based on Hand-Object Actions

Zusammenfassung

In this paper, we introduce work towards a knowledge-augmented concept for Programming by Demonstration (PbD) for industrial assembly processes. The hand-object actions in various assembly tasks are abstracted with the semantic model of grasp types based on the Web Ontology Language (OWL) containing task context from the grasp taxonomy stored in a Knowledge Base (KB). A Long Short-Term Memory (LSTM) network is used for recognizing the grasp types from the hand skeleton in human demonstrations. The semantic process description enables the generation of human assembly processes with a sequence of hand-object actions and their conversion to a robot-suitable process variant, which can be executed based on the skill description of a workcell. We showcase the concept with different assembly steps for a ball bearing as an example of matching between human grasp types and robotic skills.