App creates augmented-reality tutorials from normal videos

Published by: 0

Want to fix your car or indulge in some molecular gastronomy? A system that generates the kind of AR tutorials that trains fighter jet engineers could help

Enter now to win the chance to speak to an astronaut on the ISS

Augmented-reality headsets have long promised to turn unskilled tinkerers into instant experts. Whether you want to change the oil in your car or cut an onion so it doesn’t make you cry, a headset or tablet will paint your surroundings with instructions guiding you like an experienced pro, patiently correcting any missteps along the way.

Sounds great, so what’s the hold-up? Producing the content has always been tricky and expensive. That’s why AR tutorials are generally used only by the kinds of companies that have a fortune to spend, for example on fighter jet maintenance tutorials. But a start-up company is hoping to change that. IOXP in Mannheim, Germany, can take a single video of someone correctly performing a task – and then with the help of artificial intelligence automatically convert it into an AR tutorial.

“The goal is that anyone who wants to fix something at home can download a tutorial to their AR device that shows them how to do so,” says Didier Stricker at the German Research Centre for Artificial Intelligence, and one of the co-founders of IOXP.

There are already how-to videos on YouTube for every conceivable thing you might want to learn, but they won’t tell you when you’re doing it wrong.

To create an AR tutorial using the IOXP system, first you use a standard camera to film someone completing a task. This could be changing the oil in a car, installing a boiler or doing a cross-stitch. Then a host of computer-vision algorithms are set loose on the video to separate it into comprehensible chunks: detecting a person’s hands and what they are doing, recognising different objects, and so on.

From this, the system generates a step-by-step electronic manual detailing how to do the task. Finally, that’s converted into an AR version.

Best of all, the AR instruction manual starts itself as soon as it recognises the task in front of you. For example, if an engineer goes to perform some basic maintenance on a machine, the camera on their AR headset will automatically identify the machine in front of it.

“You then see the hands of the expert in front of you showing you what to do,” says Stricker. If you grasp the wrong dial or object, your hands turn red and the system replays the action that you are meant to be performing, highlighting your hands, relevant knobs, dials, parts, all within an accuracy of half a centimetre.

Bosch has already trialled the system with some of its engineers in Germany, and is currently in talks about a larger roll-out.

“This solves one of the big barriers of adoption: how can you extract the information? Automating this is massive,” says Rab Scott at the Advanced Manufacturing Research Centre at the University of Sheffield, UK.

For the average person, it’s only a matter of time before flat-packed furniture comes with augmented-reality instructions too. “Ikea has all of its instructions available on YouTube. Turning that into an augmented reality app would be easy with IOXP’s technology,” says Scott.

This wouldn’t have to be via a headset, but could be just as effective on a mobile phone or tablet. Imagine trying to replace a part on your boiler. You hold up your phone and it highlights the right bit for you and shows you how to detach the current one. You then hold up the app again and it either proceeds to the next step or tells you you’ve done it wrong.

Technology like this is also going to be crucial in the next few years. “Sixty per cent of engineers in the UK are over 50. In the next 15 years we are going to fall off a cliff unless we can capture their knowledge and deliver it to the next generation of engineers effectively,” says Scott.

More on these topics: