New military virtual reality system lets operators control swarms of drones

Ever since drones became a thing the standard practice for piloting one has been one operator for each unmanned vehicle. That is of course unless they’re autonomous or controlled by other drones or jet fighters, like the drone F/A-18 or the F-35 drone and hunter killer drones that the US Marines want built… But, and let’s pretend humans are still needed in the future of war, what if a single human operator could control tens or hundreds of drones? That’s the question the US military’s bleeding edge research arm DARPA is working on with its “Offensive Swarm-Enabled Tactics,” or OFFSET for short, program.

As leader of one of the two OFFSET integrator teams selected in October 2017, Raytheon has announced that they’ve created a Virtual Reality interface that allows a single user to control large groups of inexpensive unmanned vehicles like these Locust drone swarms that Raytheon and pals showed off a year or so ago. And Northrop Grumman, the other lead integrator, has also announced they’re “designing, developing and deploying an open architecture” for so called drone swarm technologies that use “game based architectures to enable design and integration of swarm tactics,” according to DARPA.

So far Raytheon has tested the technology to control swarms with as many as 50 drones and they plan on growing that number significantly over the coming months, according to Shane Clark, a scientist at Raytheon and principal investigator for the company's OFFSET efforts.

“The goal is to allow a single user to actually control, in real time, up to hundreds of air and ground vehicles that have different capabilities, or are different models,” said Clark.

To manage the swarm Raytheon had to develop an entirely new VR interface and during tests the drones communicated with the “Swarm Tactician” using a laptop over Wi-Fi link, although the eventual plan is to make the technology communications platform agnostic. The tactician then interacted with drones and the simulated environment with a HTC Vive VR headset and a pair of controllers, which one day could be replaced by HTC’s brain controlled VR headset that I showed off a while ago.

“Right now, the data [the platform produces] is just used for real time decision-making, so there isn’t a storage component, though that could be added in the future,” Clark said. “And because interacting with hundreds of individual drones would be complicated, they're managed in groups.”

Grouped drones can show their target area, given task, battery life and status of the communications link all seamlessly, and operators can drill down to see information on the individual drones too, added Clark.

The drone swarms themselves are designed to act as a mobile ad-hoc mesh network with each drone acting as a link connecting the entire swarm together.

At the moment Raytheon’s interface is currently capable of simple commands like selecting a subset of the swarm and tasking it to move to a particular area, or asking drones to spin in place to get a sustained view of the surrounding environment. Many of the drones will be outfitted with electro-optical cameras capable of image recognition, and others will be equipped with LiDAR to allow for 3D high resolution mapping and reconnaissance.

Raytheon is also working on a capability that would allow the operator to use the VR environment to draw around an area to be mapped, select the drones to complete the task and then issue a voice command to map the area, Clark said.

OFFSET is being conducted in a series of “sprints,” and groups of “sprinters” will be selected through the solicitation process to develop applications in several technology areas. The first sprint, released last year, addressed advancements in swarm tactics for a mixed swarm of 50 air and ground robots in an urban environment over a period of 15 to 30 minutes, and relevant capabilities included mapping abilities, locating entry and exit points, deploying sensor networks and maintaining connectivity for war fighters, according to the broad agency announcement.

There have been three contracts awarded this year as part of the OFFSET program to Soar Technology, Charles River Analytics and Lockheed Martin according to updates on the original BAA.

Raytheon’s VR environment will be leveraged by different “sprinters” who will be using it along with the AirSim open source flight simulator developed by Microsoft  to test the different tactics they’ve been chosen to develop.

“In simulation it's easy to postulate a new sensor that gives you a particular capability and see how that might inform what sorts or tactics you could accomplish,” Clark said. “One of the examples they gave in the BAA was what if you had a camera that could see through walls, how would that change things?"

And speaking of cameras that see through walls, what about if there was one that could see round corners too… like this one.

"We’re really interested in understanding some of that more speculative technology," he said, "so that we’re not just building a pile of parts that will become obsolete, but be forward looking.”