Pointing the Way: 3D Computer Cursors Could Navigate Virtual Worlds

Researchers at the University of Montreal have developed techniques to enable computer cursors to navigate around 3D space.

Forget everything you thought you knew about computer cursors. Researchers have come up with a way to turn cursors into a tool that can navigate around 3D space.

Conventional pointers that are controlled with a trackpad and show up as a tiny arrow on a screen will soon be outdated, according to scientists at the University of Montreal in Canada. They have created a way to turn smartphones, tablets or anything with an interactive surface, into a translucent so-called "controlling plane" to select and manipulate objects in a 3D world.

This futuristic technology could play an integral role in how virtual reality software responds to how users move in real life. The 3D cursor was unveiled earlier this month at the SIGGRAPH 2015 Conference, held from Aug. 9 to 13 in Los Angeles.

Traditionally, a mouse and a cursor are confined to a screen "like a jail," said study lead researcher Tomás Dorta, a professor at the University of Montreal's School of Design. "It's the kind of interaction which has to evolve," he told Live Science.

The high-tech cursor developed by Dorta and his colleagues can select objects in the 3D virtual world. Instead of clicking on icons to select things with a trackpad or mouse, the screen of a smartphone or tablet becomes the trackpad itself and produces a translucent plane on the screen that responds to all kinds of movements.

"If I have this cup," Dorta said, picking up a coffee mug. "When it's selected, it's like I have it in my hand."

The controlling plane appears on the screen, which can enlarge or decrease an object when the user pinches or expands it using their fingers. It twists and tilts when the device does and users can also copy and paste with it. In tests so far, the researchers were able to select chairs and tables in a building and organs inside a large, to-scale skeleton image on the screen.

At the moment, the cursor technology can be demonstrated using Hyve3D technology, which is an immersive design system that visualizes 3D sketches on a screen in front of the user. The screen is also collaborative, so people can link their devices to the same software and work together on a project. Contributors can look at the same space from different angles using their various devices, each accessing and manipulating it separately. 

"You can navigate together … working together in the same computer," Dorta said. "Everything 3D, everything collaborative, because the 3D cursor becomes our avatar."

Dorta said potential uses for a collaborative, 3D technology range from interior and architectural design to the development of virtual reality computer games. If phones or a tablets can become 3D cursors, then the ultimate goal is for users to access the same program or desktop as their colleagues, wherever they are, he said.

Eventually, this type of cursor technology could be available for operating systems like Windows and Mac OS, Dorta said. This could enable people to access each other’s desktops and see the files and applications on there in 3D, rather than through a window. Dorta thinks people are currently restricted by the window format on computers, and a 3D version of a desktop would make people’s computer interactions easier. Sending a file also won't require a USB or an online folder — you would just need to swoop at it with your phone to "grab" it and it'll be saved to your device, Dorta said.

Smartphones and tablets can become translucent "controlling plane" to select and manipulate objects in a 3D world.

The traditional computer mouse was invented in 1964, Dorta said, and it's time for something new. The researchers were inspired by the way people interact with the world, and how a computer can seem limited with its 2D restrictions.

"Let's do something in 3D, because we are in 3D," Dorta said.

He added that 3D cursors could open up new possibilities in the world of computing. For one, application windows won't need to "stack" or hide on top of each other on a screen because the cursor could move around in 3D space, Dorta said.

While people have become accustomed to desktops and laptops that present information in a 2D landscape, Dorta said, next-generation users will likely experience a different way of interacting with computers. The researchers have noticed that younger users already have more of a knack for using the 3D technology than adults who are "already contaminated with the cursor."

"When we see kids using the 3D cursor, they don't take time to learn," Dorta said. "They do it quickly because it is like mastering the movement of a hand."

Dorta said innovative cursor technologies will continue to evolve to keep up with ever more virtual lives. "It's not only a little arrow to click," he said. "We are 46 years later. We can do better, I think."

The research was published online July 31 by the Association for Computing Machinery.