New MIT tech lets Pokemon magically interact with the real world

But get ready for it, because researchers from MIT have kicked the augmented reality game up a notch by inventing a program that allows virtual objects like Pokémon to interact in real time with real world environments. Yep, this means one day you could be snaring a Ponyta as it prances through a field or wasting 20 Pokéballs on a Zubat that keeps messing up your curtains.

The technology, dubbed Interactive Dynamic Video (IDV) not only allows animated characters move about the world but it also allows them to realistically - and this is the cool bit, affect objects in the environment, like Pikachu rustling the leaves of a bush or pushing a swing.

"IDV technique lets us capture the physical behaviour of objects, which gives us a way to play with them in augmented reality," said Abe Davis from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

"By making videos interactive, we can predict how objects and characters will respond to unknown forces and explore new ways to engage with videos."

How it works 

Show me the magic

The technology uses traditional cameras to analyse the tiny, almost invisible vibrations given off by the objects in the frame. These vibration patterns are then fed through a number of algorithms to predict how that object can move. Based on data captured from just 5 seconds of footage, the program can build realistic simulations of an object’s motions right there on the screen in front of you.

"If you want to model how an object behaves and responds to different forces, we show that you can observe the object respond to existing forces and assume that it will respond in a consistent way to new ones," says Davis, adding that they’ve even made it work on some existing YouTube videos.

Not only does the technique allow programmers to drop Pokémon or other animated characters into an environment and have them realistically interact with it, it also works in reverse, allowing users to interact with a virtual world, and poke and prod at objects 'inside' video footage. Think plucking at the strings of a guitar in a video, or telekinetically controlling the leaves of a bush.

"When you look at VR companies like Oculus, they are often simulating virtual objects in real spaces," says Davis. "This sort of work turns that on its head, allowing us to see how far we can go in terms of capturing and manipulating real objects in virtual space."

Their research has yet to appear in a peer reviewed journal, but you can read their recent report online here, and check out more of its capabilities in the video above. We're so ready to have this in our life and we know you are too.