xr
The tracked devices of an XR session and the hand controllers: where each part of the person is, what the controllers' sticks, triggers and buttons read, the hands and their fingers (held controllers or tracked bare hands), what the hands point at, vibration, and the comfort sett…
xr
The tracked devices of an XR session and the hand controllers: where each part of the person is, what the controllers' sticks, triggers and buttons read, the hands and their fingers (held controllers or tracked bare hands), what the hands point at, vibration, and the comfort settings of the person wearing the headset.
local right = xr.controller("right_hand")
if right and right.trigger > 0.8 then
xr.pulse("right_hand", 0.6, 0.1)
end
local head = xr.pose("head")
local ray = xr.pointerRay()
if ray then
print(ray.role, ray.origin.x, ray.direction.z)
end
Poses are read by role
A tracked pose is read by the part of the person it follows: "head",
"left_hand" and "right_hand". xr.roles() lists the roles whose device the
runtime locates now, xr.tracked(role) asks about one, and xr.pose(role, kind?)
returns the pose in world space, or nil while the device is untracked. For a
hand the pose is where it holds the controller, and kind "aim" is where the
controller points. A device that follows another part of the body takes a role of
its own and is read the same way.
Hands
A hand is driven by one source at a time: the controller held in it
(xr.handSource(hand) reads "controller"), or, while it holds none and the
runtime tracks it, hand tracking ("hand_tracking"). Otherwise the source is
"none". A controller always wins over hand tracking.
local pose = xr.hand("right_hand") -- palm, finger directions, pinch
local joints = xr.joints("right_hand") -- the 26 joints by name
if joints then print(joints.joints.index_tip.position.y) end
print(xr.handTracking().available, xr.handTracking().reason)
xr.joints(hand) returns the 26 joints in world space: palm, wrist, then
thumb_metacarpal, thumb_proximal, thumb_distal, thumb_tip, and
<finger>_metacarpal, _proximal, _intermediate, _distal and _tip for
index, middle, ring and little. Each joint states its position,
rotation, radius and whether the runtime locates it (tracked); a joint's -Z
points along its bone toward the next joint and its +Y out of the back of the hand.
Where the runtime does not locate a hand that holds a controller, the joints are the
model's around the controller's grip, with the fingers curled by the trigger and grip
travel (modelled is true).
xr.hand(hand) is the same hand reduced to what moves an avatar's hand: the
palm (the frame whose -Z points along the fingers and +Y out of the back of
the hand), the wrist position and directions, 45 numbers giving the unit
direction each finger bone points along in the palm's frame (finger by finger,
thumb first, three bones each). pinch is the strength of the pinch of thumb
and index finger, 0 apart to 1 touching, and pinching whether it is held: it
begins when the gap between the fingertips' surfaces is 12 mm or less and ends at
30 mm, so it holds through the tremor of the fingers.
A hand tracked without a controller drives the game like a controller does. It
is active, its pinch is its trigger (xr.controller(hand).trigger is 1 while the
pinch is held), so a laser press on a panel button, a select in the editor and
the pad's trigger bindings all answer it, and it points the laser along its aim
pose. The aim comes from the runtime where it states one (XR_EXT_hand_interaction)
and is derived from the hand's joints otherwise, a ray from an estimated shoulder
through the base of the index finger; xr.hand(hand).aimSource says which, and
xr.handTracking().aim says which the runtime offers. Where the runtime offers
no hand tracking, xr.handTracking().reason says why and the controllers are the
only hands.
Without a body the hands are drawn over each eye as a jointed model (a block for
the palm, one along each bone and one at each joint), so the person always sees
their hands. When a body draws a hand with its own arm (see xr.perspective) the
body's hand is that hand.
A game needs nothing from this module
While a session holds input focus the controllers drive the game's input on their own:
| Controller | Pad |
|---|---|
| left stick | left stick (move) |
| right stick, horizontal | right stick (look), as snap or smooth turning |
| right stick, vertical | right stick, which look ignores in a headset |
| A / B (right) | south / east (jump / crouch) |
| X / Y (left) | west / north (interact) |
| left menu button | start |
| grips | left and right shoulders |
| triggers | left and right triggers |
| stick clicks | left and right stick clicks (sprint on the left) |
Every input map's gamepad bindings answer them unchanged, Zin.surface
reports the device class it reported before the session began, and the touch
overlay does not appear. The pad is connected while a controller is held and
disconnected when the session loses input focus.
Turning
The look action reads the right stick's horizontal axis as discrete steps of
xr.snapTurnDegrees() (45 degrees) by default: one flick of the stick is one
step, and the stick has to return to rest before the next flick. Smooth
turning is the opt-in comfort setting xr.setTurnMode("smooth"). The game turns
its own camera through the look action and the headset follows the camera's
yaw; the head tracks fully free, and the stick never tilts the view.
A step is 45 degrees at the look sensitivity every shipped camera rig
defaults to (0.005 radians per look unit); a rig authored with another
sensitivity turns in proportion.
Pointing
The engine's pointer hit (getPointerHitTable(), the hover and selection a mouse
produces on desktop) is cast along the dominant hand's aim pose while the
session holds input focus. xr.pointerRay() returns the same ray. The laser ends where that cast hits; a script that
carries something to a point the cast does not know of calls xr.setPointerReach(distance)
each frame, and xr.setPointerReach(nil) hands the beam back.
The laser is the game's mouse. While the session holds input focus,
input.snapshot().mouse_position is the point the laser rests on (where its cast
hit, else far along the ray) projected into the view the game renders from, in the
game's viewport pixels, every frame and with no press: whatever reacts to the
mouse hovering reacts to the laser resting on it, and mouse-move events follow it.
A ray the game builds from that position (screenToRay(x, y) at the mouse,
physics.raycastFromScreen) is the laser's own ray, so a hit on a mesh gives the
entity, the point and the UV the laser visibly touches. A laser on a panel, or
pointing out of the game's view, leaves the mouse just outside the viewport
(mouse_in_viewport reads false). The press is the controller's trigger, read as
before through the pad.
Panels
A panel is a surface standing in the world that shows UI and takes the
controller's laser as its pointer. It shows screens of the UI, drawn into a
picture of its own that the engine puts over both eyes after the world has been
tonemapped, so the interface stays sharp and keeps exactly the colours its
author gave it. A panel is flat or bent around the person (curve is the
radius in metres of the cylinder), and its pose is measured in the world, in the
room, or in the person's view. With the "tracking" anchor a panel stays where
the person put it and follows the rig when the game moves or turns the player.
With the "view" anchor the pose is stated in the frame at the head, turned by
the head's yaw and pitch and not its roll ({ position = {x = 0, y = -0.05, z = -2.4} }
is 2.4 metres ahead of where the person looks and 5 centimetres below), and the
panel follows the head: it eases towards that place with a time constant of 0.12
seconds, so a small tremor of the head does not shake it, and it stands there at
once when the person recenters, when the session begins and when a script states its
pose or anchor. The laser points at a view-anchored panel where it is drawn,
and a view-anchored panel held by its handle keeps the place it is put in the view.
xr.panels.create({
id = "stats",
screens = { "stats_hud" },
width = 1.2, height = 0.7, curve = 1.5,
pose = xr.panels.placement({ distance = 1.2, yaw = 25 }),
anchor = "tracking",
handle = { 0, 0, 0.6, 0.08 },
})
A panel's corners are rounded by cornerRadius, in metres along the surface
(0.03 unless the panel states another; 0 leaves them square), and the picture,
the frame around it and the laser's hits all stop at that outline.
The world's HUD, the shown screens its game authored (space = "content": every
screen of either the "window" or the "camera" space that is not the
editor's chrome, so a menu placed on the window shows in both eyes like a
camera-space screen), appears with no script on the curved panel "hud", with the "view" anchor,
2.4 metres in front of where the person looks; xr.panels.autoHud(false) turns
that off. The game lays its screens out for that panel: while a session shows them
with no window beside it, ui.screenSize() is the HUD panel's logical size, 1280
points across, and a screen sized from it fills the panel. The HUD panel passes the laser through wherever its UI has nothing to
act on.
While the laser points at a panel the world receives no pointer: the engine's
pointer hit is empty and the pointing hand's trigger and vertical stick belong to
the panel. The trigger presses what the laser is on, and the stick scrolls it.
A press that began in the world stays the world's when the laser crosses a panel.
xr.panels.hover() names the panel under the laser and the uv on it, and
xr.panels.state(id) reads one panel's hover, grab and drop.
A press on a panel always reaches its UI, the handle included, and a press is a click.
A panel's handle is a rectangle of its picture ({ left, top, right, bottom },
0,0 the top left) that moves the panel by a deliberate drag: with the trigger held
(or a hand's pinch), the panel is held once the laser has turned more than 6 degrees
from where the press began, which is more than a controller's trigger pull turns it.
At that moment the UI's press ends outside the panel, so no widget takes it for a
click, and the panel follows the laser at the distance it was held, turns to face the
person, and is pulled and pushed by the stick, until the trigger is released. It stays
where it is put. A press on the handle that is released with the laser turned less
than that clicks the widget under the point where it began, wherever the laser swayed
to, and counts in xr.panels.state(id).taps, which is how a small movable panel (a
point) is also a button. xr.panels.state(id) reads grabAngle, the turn in degrees
that makes a press a grab, and pressTurn, how many degrees the laser has turned
since the press held on the panel began (nil while none is held). A press off the
handle follows the laser for the UI however far it turns. A panel created with
acceptsDrops is a drop target: xr.panels.state(id).droppedOn names it when the grab
ended with the laser on it, and dropTarget is true on it, and its frame is drawn
green, while another panel is held over it. A panel that accepts no drops is never
named.
xr.panels.grab(id, { distance?, offset? }) holds a panel with the pointing hand while
its trigger is held, as if the trigger had been pressed on its handle: a list entry that
was pressed and dragged past the grab angle makes the panel it stands for and grabs it,
and the panel is under the laser from the next frame to the release. The press on the
entry ends there without a click.
A panel created with resize = { corner?, minWidth?, minHeight?, maxWidth?, maxHeight?, corners? }
(flat panels, corners "bottom" or "all") is resized by pulling the trigger on a
corner's zone: the dragged corner follows the laser on the panel's plane, the opposite
corner stays, and the size stops at the limits. The panel keeps its points per metre,
so a larger panel lays out more content rather than larger content.
xr.panels.state(id) reads onCorner, resizing and resizes; the engine marks each
resize corner with three dots that light while the laser is on the corner.
A widget with a dragPayload on a panel is dragged with the trigger as it is with the
mouse. xr.panels.drag() reads the drag in flight (panel, payload, label, overPanel
and the pointing hand) for as long as the trigger is held, wherever the laser points,
so a script that carries the payload into the world (an asset placed under the laser)
follows overPanel and the release, when the reading turns nil.
A dockArea drawn on a panel hands a tab dragged out of it and released over
empty space to its owner (ui.takeDockTearOffs(dockId)) instead of opening a
window inside itself.
The view
The camera the game moves relative to is the rig: the game camera's position and
yaw, written as the camera entity's own transform. While a session draws frames
the camera entity that owns the viewport keeps the game's own pose:
entity(id).position, .rotation, .eulerAngles, .worldMatrix() and the
local reads are the values the game wrote, so a script that reads the camera's
pose, changes it and writes it back moves the camera exactly as it does on a
desktop. Its facing is the person's: .transform.forward, .right and .up are
the head's directions (camera.facing(id, axis) is the same read), and the
Camera component's forward, right and up are the same values. A game that
aims, picks up or prompts along its camera's forward acts where the person
looks, with no code for a headset. Everything parented under the camera (a
carried prop, a viewmodel) is drawn and read in the world under the head.
Movement written relative to the camera follows where the person looks;
xr.pose("head") is the head's full pose.
The client setting xr.aim (xr.aim(), xr.setAim(aim)) chooses what the
camera faces. "head" (the default) is the head's orientation. With
"controller" the camera's facing (transform.forward, right and up, and
the Camera component's) is the dominant hand's aim direction with the head's
up, so a game that aims, picks up or prompts along its camera's forward acts
where the controller points; the eyes still draw from the head, and the setting
applies on the next frame.
What the game writes stays the base. Its pose, local or world, is the game's own
and the rig is built from it, so a first-person look, a follow camera or a script
turning or moving the camera turns and moves the base, and the head composes on
top of it, with its yaw counted once. The game's view for worldToScreen,
screenToRay and the laser's mouse is the game camera's lens from the head's
pose, so they agree with what the person sees. When the session stops, the
camera entity is drawn at the game's own pose the same frame.
getLookHitTable() casts from the eye point along the camera's facing.
The body and the perspective
In play mode a headset sits in the local player's body whenever the game has one.
xr.body() returns { entity, reason }: the body is the avatar the spawn system
bound to the player (avatar), else the entity the game's camera follows
(cameraFollow), else the entity the player's character controller moves
(controller). A candidate is the body only when the game uses it: the game's
camera follows it, or a character controller the local player drives moves it. An
avatar neither followed nor driven (a game that plays from its own fixed camera) is
not a body, and xr.perspectiveStatus().why says so. A game with no such entity (a
strategy camera over many units) has no body and the game's camera is the view. The
search repeats, so a character the game starts following or driving later becomes the
body then.
xr.perspective() is "embodied" or "game":
"embodied"puts the eyes at the body's eye height straight over its root (never over an animated bone, so a walk cycle does not bob the view), with the real head free on top. The yaw is the game camera's, so turning works as it does without a headset, and the game's camera entity keeps running untouched. The body's head and what is attached to it draw for every view but the person's own eyes (other players, the desktop window undereditorand a capture from another camera still see it), and its arms follow the hands."game"is the view from the game's own camera with the head free.
A world whose camera is a first-person camera is embodied and offers no choice.
Any other world with a body is embodied until the person chooses: a click of the
right thumbstick switches, and so does the See from setting in the client
settings (clientSettings.open("vr")). The choice is the person's client setting
xr.perspective, the same in every world: a world reads it
(clientSettings.get("xr.perspective"), xr.perspective()) and does not change
it, so xr.setPerspective is refused from world code. xr.perspectiveStatus() says which
perspective is in force and why, the body found, where the eyes stand, and how much
of the body is hidden from the eyes or reaching for the hands.
The body's head is hidden by drawing the body's skinned meshes a second time for the
eyes alone, with the head bone and everything below it collapsed
(SkinnedModel.collapsedBones), while the meshes themselves draw on the layer
xrSelf, which the eyes do not draw (renderer.setLocalLayers). Parts attached
below the head bone (a hat, hair) and the player's name label draw on xrSelf as well. A camera that names the
render layers it draws sees the body only if it names xrSelf.
Reading the controllers
xr.controller(hand) returns a controller's inputs as of the frame last drawn:
its sticks, trigger, grip and buttons. Poses come from xr.pose; they pass
through the same rig as the eyes, so they move and turn with the game's camera.
Vibration
xr.pulse(hand, amplitude, seconds, frequency?) vibrates a controller. It
returns false and plays nothing while no session holds input focus.
Taking the headset off
A session stays standing while the headset is off: the person can test in VR,
take the headset off and work at the desktop, and put it on again to be back in
VR, with no step in between. While the runtime does not show the session and
reports that nobody wears the headset, the engine begins no XR frame: a runtime
holds each frame of a session it does not show for its own throttle (SteamVR
holds every frame's end for about 100 ms), so the frame loop pauses and the
engine's own frames run at the desktop's pace. The session keeps listening to
the runtime, and when the headset is worn again the frame loop runs, the
runtime shows the session, and the eyes are drawn again. Leaving VR from the
menu (or xr.exit()) still ends the session.
The xr observation (/zero/runtime/observations/xr) states it:
reading.userPresent is what the runtime reports about the person wearing the
headset (null for a runtime that reports nothing, whose session never pauses),
and reading.paused is true while the frame loop is paused. xr.focused() is
false while the headset is off.
Testing without hands
input.simulateXrController(hand, control, ...) and the inputSim toolbox's
xr tool inject controller state where the runtime's action set enters the
engine, so the pad, the turning, the pointer ray and xr.controller follow it
exactly as they follow held controllers. The injected state takes effect while
the session holds input focus. input.simulateXrHead(...) and the xrHead
tool inject the head pose in the tracking space; the eyes move with it.
input.simulateXrSession(true) (or the engine's --xr-sim boot flag, which a
headless engine takes too) starts a simulated session: the engine runs its XR
frame loop over a session that needs no headset. It holds input focus from its
first frame, locates a head standing at the tracking space's origin 1.6 m up
and facing -Z (input.simulateXrHead moves it) with two eyes 64 mm apart
through symmetric 90 degree frusta, and reports the controllers and hands
input.simulateXrController states. The eyes are drawn at 1024 by 1024 pixels
each and shown to no compositor. xr.focused(), xr.pose, xr.pointerRay and
the panels read it as they read a headset. input.simulateXrSession(false)
stops it and every read answers as it does with no session. The session starts
and stops on an engine frame, so wait for xr.focused() after starting it. It
runs on native engines; a headset's session that already stands is left as it
is.
input.simulateXrWorn(false) takes the simulated session's headset off: the
session reports it not worn and stands in the synchronized state, its runtime
holds each begin and end of a frame, and the engine begins none.
input.simulateXrWorn(true) puts it on again; the frame loop runs and the
session takes focus once a frame has ended.
Scoped to this part · feeds back into the world's score.