Streaming frames out of the engine

Where capture answers a request with one image, a frame-stream session is a continuous video-out: raw pixels, frame after frame, pushed for as long as the session is attached.

The shape of it

Three things, in order: a texture to render into, somewhere for the bytes to go, and the session that binds them.

-- 1. A render target, and a camera that draws into it (the render-textures guide).
local rt = renderer.texture.create({ width = 128, height = 32, name = "panel" })
entity.spawn({
    name = "panelCam",
    position = { 0, 0, 6 },
    components = { Camera = { textureHandle = rt.guid, postProcessing = false } },
})

-- 2. Somewhere to send the pixels — any stream URL (the byte-streams guide).
local out = task.await(stream.open("tcp://127.0.0.1:9000", {
    outboundCapacity = 128 * 32 * 3 * 4,   -- room for a few frames
}))

-- 3. Carry the camera's frames out at 30 per second.
local session, why = frameStream.attach(rt.guid, out, { fps = 30, format = "rgb24" })
if not session then error(why) end

-- Later:
local s = frameStream.status(session)
print(s.frames, s.dropped, s.achievedFps)

frameStream.detach(session)
stream.close(out)

Any camera works — a second camera pointed at a corner of the world, or the same view the player sees. The camera renders into a texture; the session reads that texture back and writes it out.

What arrives at the far end

One frame's pixels, then the next frame's, with nothing between them. A frame is width * height * bytesPerPixel bytes of tight rows, top row first, written to the stream in a single call — so a consumer that has counted its bytes reads a whole frame or none of one, never half.

• format is "rgb24" (3 bytes per pixel, the default) or "rgba8" (4). Any other value raises, naming both.
• flipY = true writes the last texture row first, for a display that is scanned bottom-up.
• fps caps how often a frame is taken; omit it to take one per rendered frame.

A consumer that reconnects usually wants more than bare pixels — a magic number to resynchronise on, a length, a sequence counter. That protocol belongs to whoever speaks it, so write it yourself: attach the session to a loopback stream, drain whole frames out of it, prepend your header and write the result on to the real stream. Attaching straight to the destination skips that copy and is the right choice when the consumer wants the pixels bare.

Rate, and the three ways a frame goes missing

fps is independent of the rate the engine renders at: a session asking for 30 in an engine running at 144 takes roughly every fifth frame; one asking for 30 in an engine running at 20 takes every frame it can. status().achievedFps is what it actually reached — compare it against requestedFps to tell a display running slow from one running as asked. It reads 0 until two frames have been accepted.

When frames do not arrive, frameStream.status() separates the causes, because each calls for a different fix:

Reading	What it means	What to change
droppedBackpressure	The consumer is reading slower than the session produces.	Speed up the consumer, or open the stream with a bigger outboundCapacity.
stalledReadbacks	A frame came due while every staging buffer still held a copy on its way from the GPU.	Ask for fewer frames per second.
lastOutcome == "tooLarge"	A whole frame is bigger than the stream's entire outbound capacity, so every frame of that size is refused too.	Reopen the stream with an outboundCapacity that holds one frame.

The readback runs off the render thread and the write refuses rather than waits, so a slow consumer falls behind by frames instead of holding the renderer back. frameStream.list() names every live session, which is what a hot-reloading module uses to detach the one it left behind.

Where it runs

Frame egress runs on native and in the browser alike — the readback is submitted and polled without blocking on either, and packing pixels is plain byte work. What differs is which transports a stream can be opened over: stream.transports() is the probe, and it names tcp's absence in a browser tab there.

Finding the rest

lsp.methods("frameStream") lists this API and lsp.describe("frameStream.attach") gives its exact signature. The render-textures guide covers making the texture and pointing a camera at it, and the other two destinations for a rendered view (a UI viewport widget, a material on a mesh). The byte-streams guide covers stream.open, the URL schemes, and what the write outcomes mean. To let something on the network ask for a frame instead of being pushed one, httpServer.route serves a path from this engine.

The model to hold: a camera fills a texture on the GPU, a stream carries bytes somewhere, and a frame-stream session is the pump that reads the one back and writes it into the other.