Breezy can now turn a 3DOF (degree of freedom) device into a 6DOF device by augmenting the missing positional data from a webcam. Spoiler! It is not the cam strapped to my face – this is just for the demo you can watch on my blog, on PeerTube or YouTube.
The cam, that I used for this task, is sitting on my monitor. How this works? Well not with magic! This requires a somewhat decent webcam – really anything from the last decade should suffice – and OpenTrack, of course.
OpenTrack is a head-tracking application with multiple tracker plugins. One of it’s plugins is the Neuralnet Tracker, an AI powered extension that comes with a bunch of different head pose models to choose from. With a webcam connected this can now locally run the detection model with very low latency – so it’s usually blazing fast on most systems!
This alone is already 6DOF and is used a lot for gaming already – so what does Breezy do with this? Simple! It reads the forwarded data via an UDP listener, a very quick way to transmit data on a local network or system [and complements it’s own rotational data with the missing positional data].
With this a Breezy user still gets the rotational data from the XR’s very sensitive IMU, that is short for Inertial Measurement Unit btw, and the not so important positional data sent from OpenTrack.
This works of course only while the webcam can still see the user. So sadly no walking around while using this.
And the best thing? It can also send the data back! This means that the very same combined values can be forwarded – e.g. to a computer game – benefiting from the best available data sources for rotation and position.
That’s not the main use case, of course, and only of importance for some nerds like myself. This is mostly relevant for the productivity features of Breezy, because sometimes a text may be too small to read with the glasses on. We do no longer have to increase the font size – we can now simply lean in! That is a feature that is usually only available with glasses, that come with little cameras of their own, so they can have native 6DOF support. And when I say native I mean that such glasses usually also outsource exactly this calculation to the connected computer. It’s my understanding that this seems to require a lot of computation power, which is something many XR users with the more modern devices complain about.
Well not so much with OpenTrack and the Neuralnet tracker, that utilizes the ONNX runtime under the hood. That’s a high-performance, cross-platform engine to power exactly such models locally. The runtime automatically makes use of the best available hardware acceleration, if there is any.
Overall I’m rather hyped about this feature – especially because I’m using the OpenTrack output option of Breezy for quite some time now, to get a VR like experience with stereoscopic 3D rendering in Side-By-Side mode. I can now keep using my older XR glasses and still enjoy this more modern 6DOF feature. This is rather expensive hardware after all.
And all that on Linux PC!
Breezy xr_driver: https://github.com/wheaney/breezy-desktop by https://www.youtube.com/@WayneHeaney
Official Announcement XR desktop with 6DoF + multiple displays: https://www.youtube.com/watch?v=eFLmjpjF-rA
Close. degree of freedom in that context.
Are we talking 3x acceleration and you're adding 3x position? I've heard the terms a lot but haven't really thought about which DoFs they're referring to.. Can't these devices also measure absolute rotation, so wouldn't that make them 9DoF?
The three degrees of freedom in 3dof are rotational: pitch, yaw and roll. The 3 that get added in 6dof are positional, forward/back, up/down, left/right
Not sure I can follow. There are various IMU systems, of course. The ones I have expose it's data as Quaternions and I used Euler's formula to translate this to rotation (orientation). Been a while but that's what the Breezy xr-driver does now for me under the hood too. With this I have roll, pitch and yaw - the more important ones for games.
Now in combination with a webcam and OpenTrack I also get x, y, z as position (location). Yes. that also has roll, pitch and yaw, but that is not as precise and will lag a tiny bit by design resulting in motion sickness - so that is not used and simply discarded. This can work fine on it's own, of course. It really depends on the fine tuning of the settings and mapping and I did use only this before I accessed the data provided by the IMU.
Does this make a 9DoF device? I don't think so. There's nothing like a magnetometer involved. And what would the use-case be? I'm not a plane xD
I was basically wondering if position and acceleration were considered different degrees of freedom, but I had it a bit mixed up. I've seen cheap 6 DoF boards with accelerometers plus something else to measure rotation IIRC, but these don't have absolute position and I was thinking if you add that maybe it would increase the degrees of freedom