inputSim

Every simulated input, across every device a player might hold. Drive the world as a keyboard-and-mouse player, as a phone, or as someone with a controller — and read back what actually reached the game.

Why all three, always

A world checked only from a keyboard is a world whose keyboard experience is the only one anybody verified. That is how a game ships playable on a desktop and dead on a phone: not because the touch path was written badly, but because nothing ever drove it.

So this toolbox covers the three classes symmetrically, and device makes a desktop session present as any of them.

Driving the game: actions

A game reads its input through the controls its input maps declare — move, jump, sprint. action drives one by name with the value the game should read, which is how a test or an agent plays a game: the same call works whatever device the player would hold, and nothing about a device — which key a map binds, where the pointer is — has to be worked out.

tools.use("inputSim", "bindings")                      -- the live controls and their kinds
tools.use("inputSim", "action", "jump")                -- tap a button
tools.use("inputSim", "action", "move", { 0, 1 }, 2)   -- walk forward for 2 s
tools.use("inputSim", "action", "sprint", true, nil, true)  -- hold until `release`
tools.use("inputSim", "fired")                         -- reported under class "driven"

A button takes true, a 1-D axis a number, a 2-D axis { x, y }. The value joins the control like another device: its shaping applies, its press and release fire, and a context or a suppression that stands the control down stands it down too. release lets go of every driven control.

The device verbs below prove the devices themselves: that a key, a pad or the touch overlay reaches the control a player would press it for.

The loop

tools.use("inputSim", "device", "touch")     -- present as a phone
tools.use("inputSim", "bindings")            -- what controls are live?
tools.use("inputSim", "layout")              -- what is on screen?
tools.use("inputSim", "tapButton", "jump")   -- tap it where a player would
tools.use("inputSim", "fired")               -- did it reach the game?

bindings first, always. Nothing is live until a component activates an input map, so an empty list means no awake component asked for input — which is a different problem from a control that fires and does nothing.

fired is what turns a simulated input into an assertion. A capture can look identical whether or not the input landed; this cannot. Its count is the number of engine frames the control was active, so a tap that holds a button across two frames reads 2 for one press. To count presses, subscribe with control:onPressed.

One argument shape

Every verb here takes its arguments POSITIONALLY, in one order across the whole toolbox: what first — the button, key, stick, trigger, label or contact id — then where as flat x, y (or dx, dy), then how long in seconds or how far spread in frames, then which pad slot. Every optional argument trails. No verb wraps its arguments in an options table, so a verb's shape is known from having called its neighbour.

tools.use("inputSim", "click",     0, 400, 300)   -- what, where
tools.use("inputSim", "mouseDown", 0)             -- the same what
tools.use("inputSim", "touch",     400, 300, 0.5) -- where, how long
tools.use("inputSim", "padStick",  "left", 0, -1) -- what, where

Every surface that names arguments resolves those names against the same positional list, so one call is written four ways and reaches the same slots: click(0, 400, 300) in Luau, { button = 0, x = 400, y = 300 } through tools.use, { button: 0, x: 400, y: 300 } through tools.use in a tool_exec script, and --button 0 --x 400 --y 300 in the shell. A verb whose every argument is optional takes {} for "no arguments at all", and a name the verb does not declare is refused with the ones it does.

Who the input is for

What this toolbox simulates is the game's input: a key, click, motion, wheel or touch reaches the scene alone. It reaches the scene whatever a person at the window has focused — a panel they clicked, a terminal they are typing a message into — and it reaches no UI widget, so a run driving the game never types into the person's panel and the person's typing never stands the run down. Every key, pointer and touch verb takes a trailing target: "game", the default, or "ui", which reaches the UI the way a person's device does. Driving a UI by its elements — clicking a button, typing into a field — is the gui toolbox's work.

tools.use("inputSim", "key", "KeyW", 1)            -- walks the player
tools.use("inputSim", "key", "KeyA", nil, "ui")    -- types into the focused field

A mouse button is written either way, index or name: 0 / "left", 1 / "right", 2 / "middle" — the same pair Zin.state.mouseButtonDown reads. A standard pad's button is a canonical position name ("south", "dpad_up", …), because a standard pad has no index a player could recognise; a wheel's, a stick's or a button board's own elements are moved by their index with padDeviceAxis / padDeviceButton.

Where a point is measured from

mouseMove, click, touch, touchDown and touchMove take their point in physical pixels, measured in the space the input's receiver reads positions in, so target decides it.

With target "game" (the default) the point is in the space capture reports: the scene's own pixels, measured from the top-left of the picture a capture returns. That is the space Zin.state.get().mouse_position reads a pointer back in, Zin.touch.slots() a contact, Zin.events every pointer and touch event, and the space worldToScreen answers in, so a point read off any of them is a point the others take.

tools.use("inputSim", "mouseMove", 300, 200)
Zin.state.get().mouse_position          -- 300, 200
local id = tools.use("inputSim", "touchDown", 300, 200)
Zin.touch.slots()[1]                    -- x = 300, y = 200

The window can hold the scene inside a rect with editor chrome above and to the left of it; the verbs carry that offset themselves.

With target "ui" the point is on the window: the space every UI widget is recorded in, wherever its screen draws (the window itself, or the picture a Scene panel presents). A widget's gui.bounds rect is in logical points, and times ui.pixelRatio() it is the point to aim at:

local b = tools.use("gui", "bounds", "play-button")
local r = ui.pixelRatio()
tools.use("inputSim", "click", 0, b.centerX * r, b.centerY * r, nil, "ui")
tools.use("inputSim", "touch", b.centerX * r, b.centerY * r, nil, "ui")

gui.clickElement does the same by id and refuses a widget the last frame did not draw where its rect claims, so it is the call for pressing a named button. The gui pointer verbs speak logical points and convert by the ratio themselves: gui.clickAt takes one, gui.clickElement and gui.hoverElement answer the one they aimed at, and gui.clickAt(b.centerX, b.centerY) presses the same place as the click above.

A macro's ops are those same verbs, so an absolute point written for one is the point the other takes.

A mouse event the primary touch contact drives carries that contact's id as touch in Zin.events, so a reader tells a finger's press from a mouse's.

Keyboard and mouse

Tool	What it does
key	Tap a key — down, optional hold, up
keyDown / keyUp	Hold and release
click	A full click, optionally at a point
mouseDown / mouseUp	Hold and release a button
mouseMove	Move the cursor to a point; the delta is what look bindings read
mouseMoveBy	Move the pointer by a motion, over steps frames — repeat it to hold a look axis
scroll	Turn the wheel
lockPointer	Lock or release the pointer — mouse-look is usually gated on it

keyDown and mouseDown hold until something releases them, and a hold outlives the call that made it — including a driver task that was cancelled mid-press. held says what is down; release puts all of it down at once.

Touch

Tool	What it does
touch	A whole tap at a point
touchDown / touchMove / touchUp	One finger, by contact id — several at once
stick	Push the on-screen movement stick
drag	Drag across the look zone, spread over frames
pinch	Pinch or spread two fingers
tapButton	Tap an on-screen button by its id, its actionName or its label

tapButton takes any of the three names layout lists per button. The id and the actionName read the same in every language, so a test that names a button by either keeps working for a player in any language; the label is the words on the button as they read now. It aims at the button's real centre and raises when no live button answers to the name, listing the ones that do. That is deliberate: a tap that silently lands on nothing looks exactly like a control that works.

Gamepad

Tool	What it does
pad / padOff	Connect or disconnect a simulated pad
padPress	Press a button — down, optional hold, up
padDown / padUp	Hold and release
padStick	Push a thumbstick
padTrigger	Squeeze a trigger to an analog travel
padDeviceAxis	Move one of the pad's own axes by index: a wheel, a pedal, a throttle
padDeviceButton	Push or release one of the pad's own buttons by index

Buttons are named by POSITION, never by the letter printed on them: south, east, west, north, left_shoulder, right_shoulder, left_trigger, right_trigger, left_stick, right_stick, select, start, guide, dpad_up, dpad_down, dpad_left, dpad_right. The same call works whatever pad is connected.

The name passed to pad decides which legends a prompt draws, so connecting "Sony DualSense Wireless Controller" is how a scheme's PlayStation prompts get checked.

A controller that is not a standard pad, a steering wheel with pedals, a flight stick, a button board, is connected with its element counts and driven by index:

local slot = tools.use("inputSim", "pad", "Racing Wheel", { axes = 6, buttons = 25 })
tools.use("inputSim", "padDeviceAxis", 2, 1)       -- floor the third axis
tools.use("inputSim", "padDeviceButton", 19, true) -- hold the twentieth button

A device's own axes start at 0 unless rest says where they rest: tools.use("inputSim", "pad", "Pedals", { axes = 3, rest = -1 }) connects a pedal box with every pedal released, the way a real one reports on connect.

A simulated pad offers rumble unless connected with rumble = false, and force-feedback effects when connected with force = true; Zin.gamepad.get(slot).rumble and .force read back what a game played on it.

Reading back

Tool	What it answers
device	Which class the session presents as — and forces one
bindings	Which controls are live, and which map contributed each
layout	What the on-screen controls are actually drawing
fired	Which controls were active since last asked, for how many engine frames, and from which device
held	What the session is holding down right now, across every device

layout is how a phantom control is caught: a label on screen that no entry in bindings explains is a button nothing consumes.

held is how a leftover hold is caught: keys, mouse buttons, fingers, the stick and connected pads in one answer, with atRest true when the session holds nothing. A control that keeps firing after a take ended reads here as the key that is still down, so recovering from it is a release rather than a guess at what was pressed.

Replaying a sequence

A run of events worth repeating becomes a .inputMacro asset, so the same sequence drives the same world on every later check.

Tool	What it does
recordMacro	Capture real input for duration seconds
saveMacro	Write an event list to a .inputMacro asset
macro	Play an event list back through the input surface
listMacros	Every .inputMacro asset in the project

An event is { t, op, arg }. op names one of the verbs above that drives input and arg carries that verb's arguments — the value itself when the verb takes one, a table keyed by the verb's parameter names when it takes several — so every device class a verb reaches is a device class a macro reaches. A touchDown event's arg.id is the name its contact goes under, and the touchMove and touchUp events carrying that id drive it.

tools.use("inputSim", "macro", {
    { t = 0.0, op = "pad" },
    { t = 0.4, op = "padDown", arg = "south" },
    { t = 0.6, op = "padUp", arg = "south" },
    { t = 0.8, op = "touchDown", arg = { x = 714, y = 984, id = 1 } },
    { t = 1.2, op = "touchUp", arg = 1 },
})

recordMacro reads what every device is HOLDING each frame — keys, mouse buttons, pad buttons, the pad itself, fingers on the glass — and what each one is MEASURING every sampleInterval seconds — mouse motion, the wheel, thumbsticks, triggers, a sliding contact.

release is the reset between takes — it lifts every contact, drops the on-screen stick, releases every held key and mouse button, and disconnects every connected pad, so one take never leaks a held control into the next. It reports how many of each it let go, and held reads the same set before or after.