xr

The tracked devices of an XR session and the hand controllers: where each part of the person is, what the controllers' sticks, triggers and buttons read, the hands and their fingers (held controllers or tracked bare hands), what the hands point at, vibration, and the comfort settings of the person wearing the headset.

local right = xr.controller("right_hand")
if right and right.trigger > 0.8 then
    xr.pulse("right_hand", 0.6, 0.1)
end

local head = xr.pose("head")
local ray = xr.pointerRay()
if ray then
    print(ray.role, ray.origin.x, ray.direction.z)
end

Poses are read by role

A tracked pose is read by the part of the person it follows: "head", "left_hand" and "right_hand". xr.roles() lists the roles whose device the runtime locates now, xr.tracked(role) asks about one, and xr.pose(role, kind?) returns the pose in world space, or nil while the device is untracked. For a hand the pose is where it holds the controller, and kind "aim" is where the controller points. A device that follows another part of the body takes a role of its own and is read the same way.

Hands

A hand is driven by one source at a time: the controller held in it (xr.handSource(hand) reads "controller"), or, while it holds none and the runtime tracks it, hand tracking ("hand_tracking"). Otherwise the source is "none". A controller always wins over hand tracking.

local pose = xr.hand("right_hand")        -- palm, finger directions, pinch
local joints = xr.joints("right_hand")    -- the 26 joints by name
if joints then print(joints.joints.index_tip.position.y) end
print(xr.handTracking().available, xr.handTracking().reason)

xr.joints(hand) returns the 26 joints in world space: palm, wrist, then thumb_metacarpal, thumb_proximal, thumb_distal, thumb_tip, and <finger>_metacarpal, _proximal, _intermediate, _distal and _tip for index, middle, ring and little. Each joint states its position, rotation, radius and whether the runtime locates it (tracked); a joint's -Z points along its bone toward the next joint and its +Y out of the back of the hand. Where the runtime does not locate a hand that holds a controller, the joints are the model's around the controller's grip, with the fingers curled by the trigger and grip travel (modelled is true).

xr.hand(hand) is the same hand reduced to what moves an avatar's hand: the palm (the frame whose -Z points along the fingers and +Y out of the back of the hand), the wrist position and directions, 45 numbers giving the unit direction each finger bone points along in the palm's frame (finger by finger, thumb first, three bones each). pinch is the strength of the pinch of thumb and index finger, 0 apart to 1 touching, and pinching whether it is held: it begins when the gap between the fingertips' surfaces is 12 mm or less and ends at 30 mm, so it holds through the tremor of the fingers.

A hand tracked without a controller drives the game like a controller does. It is active, its pinch is its trigger (xr.controller(hand).trigger is 1 while the pinch is held), so a laser press on a panel button, a select in the editor and the pad's trigger bindings all answer it, and it points the laser along its aim pose. The aim comes from the runtime where it states one (XR_EXT_hand_interaction) and is derived from the hand's joints otherwise, a ray from an estimated shoulder through the base of the index finger; xr.hand(hand).aimSource says which, and xr.handTracking().aim says which the runtime offers. Where the runtime offers no hand tracking, xr.handTracking().reason says why and the controllers are the only hands.

Without a body the hands are drawn over each eye as a jointed model (a block for the palm, one along each bone and one at each joint), so the person always sees their hands. When a body draws a hand with its own arm (see xr.perspective) the body's hand is that hand.

A game needs nothing from this module

While a session holds input focus the controllers drive the game's input on their own:

Controller	Pad
left stick	left stick (move)
right stick, horizontal	right stick (look), as snap or smooth turning
right stick, vertical	right stick, which look ignores in a headset
A / B (right)	south / east (jump / crouch)
X / Y (left)	west / north (interact)
left menu button	start
grips	left and right shoulders
triggers	left and right triggers
stick clicks	left and right stick clicks (sprint on the left)

Every input map's gamepad bindings answer them unchanged, Zin.surface reports the device class it reported before the session began, and the touch overlay does not appear. The pad is connected while a controller is held and disconnected when the session loses input focus.

Turning

The look action reads the right stick's horizontal axis as discrete steps of xr.snapTurnDegrees() (45 degrees) by default: one flick of the stick is one step, and the stick has to return to rest before the next flick. Smooth turning is the opt-in comfort setting xr.setTurnMode("smooth"). The game turns its own camera through the look action and the headset follows the camera's yaw; the head tracks fully free, and the stick never tilts the view.

A step is 45 degrees at the look sensitivity every shipped camera rig defaults to (0.005 radians per look unit); a rig authored with another sensitivity turns in proportion.

Pointing

The engine's pointer hit (getPointerHitTable(), the hover and selection a mouse produces on desktop) is cast along the dominant hand's aim pose while the session holds input focus. xr.pointerRay() returns the same ray. The laser ends where that cast hits; a script that carries something to a point the cast does not know of calls xr.setPointerReach(distance) each frame, and xr.setPointerReach(nil) hands the beam back.

The laser is the game's mouse. While the session holds input focus, input.snapshot().mouse_position is the point the laser rests on (where its cast hit, else far along the ray) projected into the view the game renders from, in the game's viewport pixels, every frame and with no press: whatever reacts to the mouse hovering reacts to the laser resting on it, and mouse-move events follow it. A ray the game builds from that position (screenToRay(x, y) at the mouse, physics.raycastFromScreen) is the laser's own ray, so a hit on a mesh gives the entity, the point and the UV the laser visibly touches. A laser on a panel, or pointing out of the game's view, leaves the mouse just outside the viewport (mouse_in_viewport reads false). The press is the controller's trigger, read as before through the pad.

Panels

A panel is a surface standing in the world that shows UI and takes the controller's laser as its pointer. It shows screens of the UI, drawn into a picture of its own that the engine puts over both eyes after the world has been tonemapped, so the interface stays sharp and keeps exactly the colours its author gave it. A panel is flat or bent around the person (curve is the radius in metres of the cylinder), and its pose is measured in the world, in the room, or in the person's view. With the "tracking" anchor a panel stays where the person put it and follows the rig when the game moves or turns the player. With the "view" anchor the pose is stated in the frame at the head, turned by the head's yaw and pitch and not its roll ({ position = {x = 0, y = -0.05, z = -2.4} } is 2.4 metres ahead of where the person looks and 5 centimetres below), and the panel follows the head: it eases towards that place with a time constant of 0.12 seconds, so a small tremor of the head does not shake it, and it stands there at once when the person recenters, when the session begins and when a script states its pose or anchor. The laser points at a view-anchored panel where it is drawn, and a view-anchored panel held by its handle keeps the place it is put in the view.

xr.panels.create({
    id = "stats",
    screens = { "stats_hud" },
    width = 1.2, height = 0.7, curve = 1.5,
    pose = xr.panels.placement({ distance = 1.2, yaw = 25 }),
    anchor = "tracking",
    handle = { 0, 0, 0.6, 0.08 },
})

A panel's corners are rounded by cornerRadius, in metres along the surface (0.03 unless the panel states another; 0 leaves them square), and the picture, the frame around it and the laser's hits all stop at that outline.

The world's HUD, the shown screens its game authored (space = "content": every screen of either the "window" or the "camera" space that is not the editor's chrome, so a menu placed on the window shows in both eyes like a camera-space screen), appears with no script on the curved panel "hud", with the "view" anchor, 2.4 metres in front of where the person looks; xr.panels.autoHud(false) turns that off. The game lays its screens out for that panel: while a session shows them with no window beside it, ui.screenSize() is the HUD panel's logical size, 1280 points across, and a screen sized from it fills the panel. The HUD panel passes the laser through wherever its UI has nothing to act on.

While the laser points at a panel the world receives no pointer: the engine's pointer hit is empty and the pointing hand's trigger and vertical stick belong to the panel. The trigger presses what the laser is on, and the stick scrolls it. A press that began in the world stays the world's when the laser crosses a panel. xr.panels.hover() names the panel under the laser and the uv on it, and xr.panels.state(id) reads one panel's hover, grab and drop.

A press on a panel always reaches its UI, the handle included, and a press is a click. A panel's handle is a rectangle of its picture ({ left, top, right, bottom }, 0,0 the top left) that moves the panel by a deliberate drag: with the trigger held (or a hand's pinch), the panel is held once the laser has turned more than 6 degrees from where the press began, which is more than a controller's trigger pull turns it. At that moment the UI's press ends outside the panel, so no widget takes it for a click, and the panel follows the laser at the distance it was held, turns to face the person, and is pulled and pushed by the stick, until the trigger is released. It stays where it is put. A press on the handle that is released with the laser turned less than that clicks the widget under the point where it began, wherever the laser swayed to, and counts in xr.panels.state(id).taps, which is how a small movable panel (a point) is also a button. xr.panels.state(id) reads grabAngle, the turn in degrees that makes a press a grab, and pressTurn, how many degrees the laser has turned since the press held on the panel began (nil while none is held). A press off the handle follows the laser for the UI however far it turns. A panel created with acceptsDrops is a drop target: xr.panels.state(id).droppedOn names it when the grab ended with the laser on it, and dropTarget is true on it, and its frame is drawn green, while another panel is held over it. A panel that accepts no drops is never named.

xr.panels.grab(id, { distance?, offset? }) holds a panel with the pointing hand while its trigger is held, as if the trigger had been pressed on its handle: a list entry that was pressed and dragged past the grab angle makes the panel it stands for and grabs it, and the panel is under the laser from the next frame to the release. The press on the entry ends there without a click.

A panel created with resize = { corner?, minWidth?, minHeight?, maxWidth?, maxHeight?, corners? } (flat panels, corners "bottom" or "all") is resized by pulling the trigger on a corner's zone: the dragged corner follows the laser on the panel's plane, the opposite corner stays, and the size stops at the limits. The panel keeps its points per metre, so a larger panel lays out more content rather than larger content. xr.panels.state(id) reads onCorner, resizing and resizes; the engine marks each resize corner with three dots that light while the laser is on the corner.

A widget with a dragPayload on a panel is dragged with the trigger as it is with the mouse. xr.panels.drag() reads the drag in flight (panel, payload, label, overPanel and the pointing hand) for as long as the trigger is held, wherever the laser points, so a script that carries the payload into the world (an asset placed under the laser) follows overPanel and the release, when the reading turns nil.

A dockArea drawn on a panel hands a tab dragged out of it and released over empty space to its owner (ui.takeDockTearOffs(dockId)) instead of opening a window inside itself.

The view

The camera the game moves relative to is the rig: the game camera's position and yaw, written as the camera entity's own transform. While a session draws frames the camera entity that owns the viewport keeps the game's own pose: entity(id).position, .rotation, .eulerAngles, .worldMatrix() and the local reads are the values the game wrote, so a script that reads the camera's pose, changes it and writes it back moves the camera exactly as it does on a desktop. Its facing is the person's: .transform.forward, .right and .up are the head's directions (camera.facing(id, axis) is the same read), and the Camera component's forward, right and up are the same values. A game that aims, picks up or prompts along its camera's forward acts where the person looks, with no code for a headset. Everything parented under the camera (a carried prop, a viewmodel) is drawn and read in the world under the head. Movement written relative to the camera follows where the person looks; xr.pose("head") is the head's full pose.

The client setting xr.aim (xr.aim(), xr.setAim(aim)) chooses what the camera faces. "head" (the default) is the head's orientation. With "controller" the camera's facing (transform.forward, right and up, and the Camera component's) is the dominant hand's aim direction with the head's up, so a game that aims, picks up or prompts along its camera's forward acts where the controller points; the eyes still draw from the head, and the setting applies on the next frame.

What the game writes stays the base. Its pose, local or world, is the game's own and the rig is built from it, so a first-person look, a follow camera or a script turning or moving the camera turns and moves the base, and the head composes on top of it, with its yaw counted once. The game's view for worldToScreen, screenToRay and the laser's mouse is the game camera's lens from the head's pose, so they agree with what the person sees. When the session stops, the camera entity is drawn at the game's own pose the same frame. getLookHitTable() casts from the eye point along the camera's facing.

The body and the perspective

In play mode a headset sits in the local player's body whenever the game has one. xr.body() returns { entity, reason }: the body is the avatar the spawn system bound to the player (avatar), else the entity the game's camera follows (cameraFollow), else the entity the player's character controller moves (controller). A candidate is the body only when the game uses it: the game's camera follows it, or a character controller the local player drives moves it. An avatar neither followed nor driven (a game that plays from its own fixed camera) is not a body, and xr.perspectiveStatus().why says so. A game with no such entity (a strategy camera over many units) has no body and the game's camera is the view. The search repeats, so a character the game starts following or driving later becomes the body then.

xr.perspective() is "embodied" or "game":

• "embodied" puts the eyes at the body's eye height straight over its root (never over an animated bone, so a walk cycle does not bob the view), with the real head free on top. The yaw is the game camera's, so turning works as it does without a headset, and the game's camera entity keeps running untouched. The body's head and what is attached to it draw for every view but the person's own eyes (other players, the desktop window under editor and a capture from another camera still see it), and its arms follow the hands.
• "game" is the view from the game's own camera with the head free.

A world whose camera is a first-person camera is embodied and offers no choice. Any other world with a body is embodied until the person chooses: a click of the right thumbstick switches, and so does the See from setting in the client settings (clientSettings.open("vr")). The choice is the person's client setting xr.perspective, the same in every world: a world reads it (clientSettings.get("xr.perspective"), xr.perspective()) and does not change it, so xr.setPerspective is refused from world code. xr.perspectiveStatus() says which perspective is in force and why, the body found, where the eyes stand, and how much of the body is hidden from the eyes or reaching for the hands.

The body's head is hidden by drawing the body's skinned meshes a second time for the eyes alone, with the head bone and everything below it collapsed (SkinnedModel.collapsedBones), while the meshes themselves draw on the layer xrSelf, which the eyes do not draw (renderer.setLocalLayers). Parts attached below the head bone (a hat, hair) and the player's name label draw on xrSelf as well. A camera that names the render layers it draws sees the body only if it names xrSelf.

Reading the controllers

xr.controller(hand) returns a controller's inputs as of the frame last drawn: its sticks, trigger, grip and buttons. Poses come from xr.pose; they pass through the same rig as the eyes, so they move and turn with the game's camera.

Vibration

xr.pulse(hand, amplitude, seconds, frequency?) vibrates a controller. It returns false and plays nothing while no session holds input focus.

Taking the headset off

A session stays standing while the headset is off: the person can test in VR, take the headset off and work at the desktop, and put it on again to be back in VR, with no step in between. While the runtime does not show the session and reports that nobody wears the headset, the engine begins no XR frame: a runtime holds each frame of a session it does not show for its own throttle (SteamVR holds every frame's end for about 100 ms), so the frame loop pauses and the engine's own frames run at the desktop's pace. The session keeps listening to the runtime, and when the headset is worn again the frame loop runs, the runtime shows the session, and the eyes are drawn again. Leaving VR from the menu (or xr.exit()) still ends the session.

The xr observation (/zero/runtime/observations/xr) states it: reading.userPresent is what the runtime reports about the person wearing the headset (null for a runtime that reports nothing, whose session never pauses), and reading.paused is true while the frame loop is paused. xr.focused() is false while the headset is off.

Testing without hands

input.simulateXrController(hand, control, ...) and the inputSim toolbox's xr tool inject controller state where the runtime's action set enters the engine, so the pad, the turning, the pointer ray and xr.controller follow it exactly as they follow held controllers. The injected state takes effect while the session holds input focus. input.simulateXrHead(...) and the xrHead tool inject the head pose in the tracking space; the eyes move with it.

input.simulateXrSession(true) (or the engine's --xr-sim boot flag, which a headless engine takes too) starts a simulated session: the engine runs its XR frame loop over a session that needs no headset. It holds input focus from its first frame, locates a head standing at the tracking space's origin 1.6 m up and facing -Z (input.simulateXrHead moves it) with two eyes 64 mm apart through symmetric 90 degree frusta, and reports the controllers and hands input.simulateXrController states. The eyes are drawn at 1024 by 1024 pixels each and shown to no compositor. xr.focused(), xr.pose, xr.pointerRay and the panels read it as they read a headset. input.simulateXrSession(false) stops it and every read answers as it does with no session. The session starts and stops on an engine frame, so wait for xr.focused() after starting it. It runs on native engines; a headset's session that already stands is left as it is.

input.simulateXrWorn(false) takes the simulated session's headset off: the session reports it not worn and stands in the synchronized state, its runtime holds each begin and end of a frame, and the engine begins none. input.simulateXrWorn(true) puts it on again; the frame loop runs and the session takes focus once a frame has ended.