Teach your Robot new Tricks with NLP and Deep Learning

Introduction

Execution of two tasks in a sequential manner. First, the language command is analyzed in the given environment to identify the target object, desired action, and described quantity; then, this information is used to generate a suitable task controller that actuates the robot.

Introducing the Model

System overview of our framework (center). Detailed explanations of the semantic module (right) and the policy translation module (left) can be found below.
Model Overview: Based on the environment perception and the voice command, the system generates a suitable motor primitive specific to the specified task in the given environment. The left half shows the semantic module, whereas the right side shows the policy translation module.

Evaluation

Training

Tutorial: Using the Model

Setting up the Code

Overall, our model achieves a 95% success rate for picking, 85% success for pouring, and 84% success for executing both tasks sequentially.

Custom Code: Model Creation and Inference

Summary and Conclusion

--

--

--

PhD student in Computer Science | Artificial Intelligence, Natural Language Processing, Human-Robot-Collaboration

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Data generators for Machine Learning

The No Free Lunch Theorem

New BigQuery DETECT_ANOMALIES Function

Exploring Machine Learning Solutions for Credit Scoring Farmers in Kenya

Classification of sounds using android mobile phone and the YAMNet ML model

How to design a high-performance neural network on a GPU

Synthesizing Deep Learning Training Data

Using Machine Learning to Auto Detect Column Types in Customer Files

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Simon Stepputtis

Simon Stepputtis

PhD student in Computer Science | Artificial Intelligence, Natural Language Processing, Human-Robot-Collaboration

More from Medium

What is AI Image Recognition and How Does it Work?

How Does Google Use Artificial Intelligence (AI)?

The computer vision bias trilogy: Shortcut learning.

Cows walking on a beach. This image is used to show how AI (artificial intelligence) can misunderstand what it is looking at by looking at the wrong things. You might normally see cows in the field, farms, hills, Swiss alps etc.

Machine Learning vs Deep Learning vs Artificial Intelligence