Camera input: a webcam or phone camera in the world

• a live texture that shows the camera's current frame, usable anywhere a texture is: on a surface (an in-world TV, a mirror, a picture frame), in a UI image, and in an av.record recording, because it is scene content like any other;
• the pixels of the current frame, for code that looks at the picture: colour picking, motion, QR codes, face or hand tracking.

The browser, Linux and Android builds capture: in the browser and on Android the feed runs once the person allows the camera, and on Linux once the device opens. A native Windows build lists no devices and opens every feed as failed, with a reason naming its platform; code written against cameraInput there runs unchanged in the builds that capture, where the picture appears. engine.platform says which build is running ("browser" or "native").

Show the webcam on a screen in the world

local feed, why = cameraInput.open({ width = 1280, height = 720 })
assert(feed, why)

-- A flat box for the screen, wearing an unlit session material whose picture
-- is the feed. A session material lives as long as this session, so a live
-- device's picture writes nothing into the world's files.
local screen = renderer.material.create(
    { shader = "unlit", textures = { base_color_texture = feed.texture } }, "tv_screen")
local tv = entity.spawn("TV", { position = { 0, 1.5, -3 }, scale = { 1.6, 0.9, 0.02 } })
tv.component.add("Model", { model = "cube" })
tv.component.get("Model"):applySessionMaterial(screen)

-- The browser asks the person at the machine first; wait for the answer.
local ok, reason = feed:awaitRunning(20)
if not ok then
    print("no camera picture:", reason)   -- e.g. "denied: camera access was refused ..."
end

feed.texture is a texture handle string. It names the feed's texture from the moment the feed opens, so it can go on a material before the first frame arrives; the surface shows the picture as soon as there is one. On a material that already exists, renderer.material.setTexture("tv_screen", "base_color_texture", feed.texture) puts it in the slot.

The same handle goes in UI, as an image in a game screen:

ui.registerScreen("selfie", {
    type = "vertical",
    style = { padding = 12 },
    children = {
        { type = "image", props = { src = feed.texture }, style = { width = 320, height = 180 } },
    },
}, 10)
ui.showScreen("selfie")

A feed is a state, not a call

cameraInput.open returns a feed at once. Opening a camera means asking the person at the machine for access and waiting for an answer that may never come, so where the feed stands is something you read:

feed:status().state	Meaning
permissionPending	The platform is asking for camera access.
starting	Access was granted and the device is opening.
running	Frames are arriving.
denied	Access was refused. reason carries the platform's message.
failed	The camera did not open or stopped. reason names why: no camera, a camera another program holds, a size or rate it cannot deliver, a platform with no capture backend.
idle	The feed was stopped.

feed:awaitRunning(timeout) waits out permissionPending and starting, is always bounded, and answers true, or false plus the state and reason.

A request that cannot be made at all answers nil plus the reason instead of a feed: a size or frame rate outside its range, a facing other than "front" or "back", a device the platform does not offer.

What the camera delivers

local s = feed:status()
print(s.width, s.height, s.fps)     -- what the device delivers, not what was asked
print(s.frames)                     -- advances once per new frame
print(s.rotation, s.mirrored)       -- how to show it upright and as the person expects

width, height and fps passed to open are hints: the camera answers with the nearest it can deliver, and status() says what that is.

frames counts every frame delivered. Two reads with the same count saw the same picture, so a loop that analyses frames skips the ones it has seen:

local seen = 0
task.spawn(function()
    while feed:status().state == "running" do
        local now = feed:status().frames
        if now ~= seen then
            seen = now
            -- a new frame
        end
        task.wait()
    end
end)

mirrored is true for a front camera: people expect to see themselves as in a mirror, so show the picture flipped left to right to the person in front of it. The unlit shader flips it with its UV transform:

local screen = renderer.material.create({
    shader = "unlit",
    properties = { uv_scale = { -1, 1 }, uv_offset = { 1, 0 } },
    textures = { base_color_texture = feed.texture },
}, "tv_screen_mirrored")

rotation is the degrees clockwise the picture turns to stand upright on the display, turned first and mirrored after. On a phone it follows the display: a phone camera's sensor is mounted at an angle to the phone, and when the screen rotates the status reports the new turn within a quarter of a second. After a quarter turn (90 or 270) the upright picture is height wide and width tall.

Reading the pixels

local cpu, why = feed:readPixels()                          -- the whole frame
local patch = feed:readPixels({ x = 0, y = 0, w = 8, h = 8 }) -- a rectangle
if cpu then
    local r, g, b, a = cpu:readPixel(10, 20)
    local png = cpu:encodePng()
    cpu:unload()
end

The answer is a TextureCpuHandle of RGBA8 pixels, the same kind renderer.texture.loadCpu gives, so :readPixel, :rgba, :histogram and :encodePng all work on it. Capture keeps running while you read. nil plus the reason comes back while no frame has arrived yet, after the feed stopped, and for a rectangle outside the frame. :unload() a handle once you are done with it.

Choosing a camera

for _, d in ipairs(cameraInput.devices()) do
    print(d.id, d.name, d.facing, d.default)
end

cameraInput.open({ facing = "back" })          -- a phone's main camera
cameraInput.open({ device = "…an id from devices()…" })

Until camera access has been granted, devices() lists each camera with an empty id and name: the list is part of what the permission protects. Open by facing, or with no device to take the default, and the list names the cameras once access is granted. cameraInput.onDeviceChange(function(devices) ... end) hears cameras being plugged in and removed, and the list being named after a grant; it returns a function that removes the listener, and a listener a component registered ends when that component is removed.

Two feeds on the same camera share it: the second open reads the same frames and the same texture, and the camera is opened once.

Ending a feed

feed:stop() stops the feed, and the camera is let go once no feed is reading it, which is what turns the platform's camera light off. A feed also ends when the content that opened it ends: the component that opened it is removed, the module that opened it reloads, the world unloads. cameraInput.feeds() lists every open feed with the context that holds it (the same tag scopes.list() shows), and cameraInput.stopAll() ends them all.

Platforms

The browser build captures from cameras: it asks the browser for the camera, so the browser's own permission prompt and camera indicator are what the person sees.

The Linux build captures from USB (UVC) webcams through their /dev/video* device nodes, and from the cameras PipeWire offers that no device node does, which is where a laptop's MIPI camera appears. A device node the user may not open is listed by name and opens as denied, saying so. Inside a Flatpak sandbox it asks the desktop's camera portal instead: the feed reads permissionPending while the desktop asks the person, and denied with the portal's answer when access is refused. It opens each camera at the size and rate asked for where the camera offers them, else at the nearest size it offers and the nearest rate at that size, and the feed's status reports what it delivers. Cameras that are plugged in or removed show up in devices() and fire onDeviceChange within about a second. The engine's --camera-route option names the route (v4l2, pipewire or portal) when the default is not the one wanted, and --fake-camera pattern presents the test harnesses' fake camera in place of the machine's own.

The Android build captures from the phone's front and back cameras and from an external USB camera, each listed with its facing, through Android's camera service. The first open asks the person for the Camera permission: the feed reads permissionPending while the system's prompt is up, then running, or denied when they refuse; a refusal the system will not ask about again ("Don't allow" chosen twice, or "don't ask again") reads denied naming where in Settings the permission is granted, and a feed opened after granting it there runs. A camera switched off by the Camera access privacy toggle (Android 12 and later) reads denied naming the toggle. rotation combines the camera's sensor mounting with the display's rotation and follows the screen as it turns, and the front camera reads mirrored. When the app goes to the background the system takes the camera from it: every feed lets its camera go and reads starting, and opens it again when the app returns. The build runs on Android 8.0 (API 26) and later.

A build with no capture backend for its platform (a native Windows build) lists no devices and opens every feed as failed, with a reason naming the platform, so content reads why there is no picture.

Finding the rest

lsp.methods("cameraInput") lists the calls. topics/render-textures covers where a texture can go; topics/frame-streaming carries a texture's frames off the machine; topics/microphone is the same shape for sound coming in.