Introduction

Spatial controllers are amazing for authoring 3D content, so what if we just didn’t put on the headset? I set out to explore this idea by building a prototype and looking at prior art.

Please note that this is a bit of a digression. Convey's priority is making a great desktop and mobile experience. By default it will not require any peripherals, just the out of box devices. This exploration, like many, may not show up in the final product.

Prio Art

There's already proof in the Dreams creator community that 3D controllers without a headset, work great for 3D editing. Here is Martin Nebelong (a seasoned Dreams artist) sculpting a character using the Playstation Move controllers. Note how naturally he grabs and rotates the object while also making precise sculpts.

This video is from this tweet where he responded to the original Headless VR tweet

Headless VR for Desktop Editors

There are no product solutions like this for desktop Windows/macOS though, or I would have tested them out. Here in the first prototype I built, here I'm using my Mac laptop with the Oculus Quest controllers. Notice how responsive and precise moving and rotating the cube is.

Here I have added support for two controllers and am mirroring the controllers with realistic 3D models.

The prototype works by putting my Quest headset on my desk looking towards the space where I want to use the controllers. A custom VR app runs on the headset that streams the controller information over the network to my desktop demo app. To prevent the headset from powering off I cover the sensor that detects if your face is in the headset, I easily did this by just pushing the headset strap down into the face plate in front of the sensor. There is a simple calibration step required too, but I'll get into that another time.

More Dreams Examples

Martin shares a ton of his Dreams creation on Twitter and Youtube (see his timelapses playlist).

The one that amuses me the most is this one, making Bob Ross style paintings but in full 3D. It really captures how natural 3D controllers are for art forms like painting and sculpting, both being very physical/tactile mediums.

Here's a 2 minute highlight reel to get a sense for just how much art he's created with 3D controllers.

A 3D Painting Simulator

Since posting the first Headless VR video on twitter, I built this prototype that turns your monitor into a paint canvas and uses your touch controllers handle as the paint brush. Using the handle was an attempt to see if tactile feedback made it more immersive. It did but at the cost of being less comfortable than holding the controller the way it was intended. I think there might still be legs to that grip if used on the flat surface of your desk, like a portable Wacom tablet.

Ignore the big delay between movement and the paint showing up, the tracking system is very low latency. The issue was present in the paint brush simulator I forked, try it out for yourself using your mice/touchpads.

A Cautionary Tale

I'd caution people getting too excited about this direction, as there have been many expensive attempts at introducing new types of controller hardware that have been abysmal business failures. An example was Microsoft and the Kinect, which Microsoft pushed very hard and at great cost, and it never really took off except as a fun depth sensor to hack on.

It's rational that new control paradigms are rare as experiences really need to be built from the ground up around them. The difference in market scale between an emerging UX paradigm and the built in controls of billions of devices is enormous. Most software creators, obviously opt to target the default built in inputs. Trying to support multiple control schemes ends up costing way more and risks complexifying development for most often low returns.

I'd like to still fight the fight and spend some time trying to increase the adoption of alternate control schemes and hardware. There is no way the current paradigms are the optimal ones. It should at least be more common to have specialized tools for the job you do day in and day out interacting with computers.

Conclusions

These prototypes have definitely been compelling. I'd hazard that a good sized niche of people would be interested in a more polished solution like this.

In addition to creative tool use cases, these types of controls would be very fun to use when playing video games that support them.

Next up I'm building a richer Headless VR example. A sculpting experience and I'll share it soon. If you have any thoughts please do comment on the original twitter thread where this concept was first shared.

Follow @seflless and @convey_it on Twitter, to see Convey's progress and more experiments.