RoombaExperiment

# fruitFly A whole fruit fly nervous system running as a spiking network on the GPU, a fly body it drives, and a bench for experiments on it. Install this package and a world holds the complete adult male *Drosophila* central nervous system (MaleCNS v1.0: 164,587 neurons, 25.6 million connections, CC-BY 4.0) as a leaky integrate-and-fire network you can stimulate, silence, read out and watch, plus a fly whose taste, looming, walking, turning, feeding, flight and escape run through that wiring. ## Contents The network: - `maleCNS.connectome/` is MaleCNS v1.0 packed for the GPU (109 MB, the edge table in two parts). It is an instance of the `connectome` asset type, which lives at the world root (`/zero/source/connectome.assetType/`) and travels with the package as a dependency: any nervous system's wiring as one asset (`manifest.json`, `neurons.bin`, `offsets.bin`, `edges.bin` or its parts). `guides { path = "types/connectome" }` is the reference. - `lif_integrate`, `lif_propagate`, `lif_clear`, `lif_reset` (`.computeShader`) are the four kernels of the network: membrane step, event-driven spike propagation over the CSR edge table, step close, reset. - `brain.module/` is `Brain.load(connectomeRef, params)`, which binds a connectome to the kernels and returns the network: `step`, `update`, `select`, `stimulate`, `silence`, `watch`, `rates`, `activity`, `classRates`, `groupActivity`, `drives` and the cloud. - `FruitFlyBrain.component/` puts a network on an entity: it loads the connectome the `connectome` field names, steps it every frame, draws the cloud, and exposes `stimulate` / `silence` / `watch` / `rates` / `reset` (the first three are synced, so every peer's copy receives the same inputs). - `neuron.shader/`, `neuron.material/` and `cloud_gather.computeShader/` are the instanced neuron cloud, each neuron lit by its own activity. - `crackle.soundClip/` is the network's own sound: a sparse crackle that is silent when the network is silent and a hiss when it runs away. The fly: - `flyBody.module/` is the fly's body as an entity tree with a tripod gait, wings, proboscis and a hop. - `FruitFly.component/` is the embodied fly: it reads the world into the brain's sensory neurons and reads motor commands back out of named cell types. - `FoodPatch.component/` is a disc of food on the ground the fly can taste. - `wing.material/`, `eye.material/` and the `wingbeat`, `feed` and `jump` sound clips are what the fly looks and sounds like. The instrument: - `FruitFlyLab.component/` is the bench window: network state and knobs, live readouts with traces and rasters, hot classes, live drives, a neuron panel, a legend, and a selection line to stimulate, silence, restore, count or watch any set of neurons. - `selection.module/` is the selection grammar the bench and everything else parse with. - `protocol.module/` is the guide's experiments as records anything can run, and the verdict a run computes from its own numbers. - `FruitFlyExperiments.component/` is the experiments panel: those protocols with one press each, a run list, a compare overlay and a verdict line. - `FirstRun.component/` is the four-step card that walks a newcomer from a reset network to a proboscis coming out. - `lab.inputMap/` and `experiments.inputMap/` are the bench's and the panel's controls, each binding keyboard, gamepad and touch. - `experiments.guide/` is the bench manual and the five experiments. The lab around it: - `labFloor.material/` and `FruitFlyLook.component/` are the floor, the fog, the vignette and the lighting intent, held as data rather than as code. - `arenaRing.mesh/` is the ring on the floor the fly stays inside. ## Usage ```luau -- The brain: one entity, one network. The default connectome is maleCNS. local brainEnt = entity.spawn("Brain") brainEnt.position = { 0, 6, -8 } brainEnt.component.add("FruitFlyBrain", { gain = 0.6, timeScale = 1 }) -- An embodied fly driven by it, and something for it to taste. local fly = entity.spawn("Fly") fly.component.add("FruitFly", { brain = brainEnt, bodyLength = 0.6 }) local food = entity.spawn("Food") food.position = { 3, 0, 0 } food.component.add("FoodPatch", { radius = 0.6, sugar = 1 }) -- The bench, the experiments panel and the card. entity.spawn("Lab").component.add("FruitFlyLab", { brain = brainEnt, fly = fly }) entity.spawn("Experiments").component.add("FruitFlyExperiments", { brain = brainEnt, fly = fly }) entity.spawn("FirstRun").component.add("FirstRun", { brain = brainEnt, fly = fly }) -- Or from code: local comp = brainEnt.component.get("FruitFlyBrain") comp:watch("MN9", { types = { "MN9" } }) -- proboscis motor neurons comp:stimulate({ types = { "claw_tpGRN" } }, 100) -- tarsal taste pegs at 100 Hz print(comp:rates().MN9) -- Hz per neuron, updated every frame ``` A selection names neurons by the dataset's own vocabulary: `types` (exact cell-type names), `typePrefix`, `pattern` (a Lua pattern over the type name), `classes`, `superclasses`, `sides` (`L`/`R`/`M`), `excitatory`, `indices`, `limit`. `brain:info(i)` answers what any neuron is. ## The model Every neuron is a leaky integrate-and-fire unit with the constants Shiu et al. 2024 (Nature) fitted to the FlyWire brain: rest and reset at -52 mV, threshold -45 mV, membrane time constant 20 ms, synaptic time constant 5 ms, refractory period 2.2 ms, transmission delay 1.8 ms, 0.275 mV per synapse. A presynaptic neuron whose consensus transmitter is GABA, glutamate or histamine inhibits; every other transmitter excites. A connection carries its synapse count; connections under `minSynapses` (default 5) carry nothing. `gain` scales every synapse. MaleCNS traces about 2.5 times the synapses per connection FlyWire does, so at gain 1 the network runs away under any drive; at 0.6 (the default) driving the 50 tarsal taste-peg neurons at 100 Hz holds the proboscis motor neurons MN9 at about 115 Hz while the Kenyon cells stay under 0.1 Hz, and driving the 595 Johnston's organ neurons instead leaves MN9 at zero. At 0.5 the taste drive reaches nothing. The bench's gain slider is the experiment, and the experiments panel's "Gain recovery" runs all three in one press. Everything runs on the GPU: the state, the input ring, the spike list and the edge table. The CPU sees the per-group spike counters each frame and, on request, the per-neuron activity lane. Integer atomics make a step order-independent, so two peers stepping the same inputs read the same network. ## The fly `FruitFly` reads named cell types out of the brain and writes the world into others; every name is in `FruitFly.component/init.luau`: | command | cell types | |---|---| | walk | the leg motor neurons (`vnc_motor` types ending in ` MN`), per side | | turn | `DNa01`, `DNa02` left minus right | | proboscis | `MN9` | | flight | the wing power-muscle motor neurons (`DLMn`, `DVMn`) | | jump | a `DNp01` (giant fibre) burst | | sense | cell types | driven by | |---|---|---| | tarsal taste | `claw_tpGRN`, `dorsal_tpGRN`, per side | that side's front foot on a `FoodPatch` | | labellar taste | `LB*` labellar bristle GRNs | the extended proboscis on a `FoodPatch` | | looming, LPLC2 path | `LPLC2` | a player's avatar growing fast in front of the fly | | looming, LC4 path | `LC4` | the same, driven as a separate path so one can be silenced | The two looming paths are driven separately and weighted by how head-on the approach is, so an avatar walking up behind the fly does not loom and silencing one detector class is an experiment with an answer. The body moves kinematically at `walkSpeed` body lengths per second; the fly faces its entity's -Z. On a shared world the entity's owner writes the senses into the brain (a synced call) and every peer poses its own copy from its own network. ## Controls | Action | Keyboard | Gamepad | Touch | |---|---|---|---| | Bench: show or hide | L | Start | The "Lab" button | | Bench: move focus | Down and Up, Tab | D-pad down and up | Tap the control | | Bench: activate the focused control | Enter | North (Y) | Tap | | Bench: nudge the focused slider | Left and Right | D-pad left and right | Drag it | | Bench: identify a neuron | Click it in the cloud | West (X) | Tap it | | Experiments: show or hide | P | Select | The "Experiments" button | | Experiments: next protocol | N | Right stick press | Tap the protocol | | Experiments: previous protocol | B | None: the list wraps | Tap the protocol | | Experiments: run | R | Right trigger | RUN | | Experiments: compare | C | Left trigger | COMPARE | Both maps name their controls apart (`lab*` and `exp*`) and take disjoint keys and buttons, so the bench and the panel can be open together and neither answers for the other. ## Data and license MaleCNS v1.0: HHMI Janelia FlyEM, Google Research and the Cambridge connectomics group, released 8 June 2026 (Berg et al. 2026, Cell, doi:10.1016/j.cell.2026.08.015), CC-BY 4.0. The pack was written by a script that reads the public flat-connectome export (`connectome-weights ... traced-only`, body annotations, consensus neurotransmitters), assigns dense indices, sorts edges by presynaptic neuron and writes the four files `types/connectome` describes; the manifest carries each source file's SHA-256. ## Notes - Loading the network reads the 102 MB edge table into a GPU buffer once per `FruitFlyBrain`; the VFS copy is released afterwards. - The neuron cloud is one instanced draw of every neuron with a soma (139,741). Its transforms are static; the activity lanes are written by the integrate kernel. - `maxStepsPerFrame` bounds GPU time per frame; at 32 steps of 1 ms a 60 fps engine keeps up with real time. - A neuron reads out under at most three groups at once. The fly holds seven of its own and the protocols watch five more under the guide's names, so a bench watch of the same cell types under the same name shares the group rather than spending another slot.

# flyBody A fruit fly's body as an entity tree, and a pose for it. `build` stands one under a root entity; `update` moves the root and poses every part from a motor command. The body is a thorax, a head with two compound eyes and a pair of antennae reaching forward and down, a banded abdomen, a proboscis that extends to feed, six legs of two segments each walking a tripod gait, and two wings that lie back over the abdomen at rest and swing out to beat. ```luau local FlyBody = require("~.flyBody") local body = FlyBody.build(public.entity, { length = 0.6 }) body.groundY = 0 body.maxRadius = 12 -- the body cannot be moved past this radius body:update(dt, { speed = 0.3, turn = 0.2, proboscis = 0, flight = 0 }) ``` The motor command is `{ speed, turn, proboscis, flight, jump, land }`: speed in world units per second along the fly's facing, turn in radians per second (positive turns left), proboscis and flight 0 to 1, `jump` to hop, `land` to put the body down. Flight lifts the body only while it is already off the ground, so a beating wing on the floor is a beating wing and the legs keep walking under it. Readings: `eye()` returns the head's world position and the facing direction; `mouthPosition()` the proboscis tip; `rightFrontFoot()`, `leftFrontFoot()` and `frontFeet()` where the front feet touch; `heading()` the yaw. `destroy()` takes down only the entities this body built, so anything else parented to the same root is left standing. The wing and eye materials are `fruitFly.wing` (a thin, mostly clear membrane) and `fruitFly.eye` (smooth enough to hold a highlight).

# maleCNS The complete central nervous system of an adult male *Drosophila melanogaster*: brain, both optic lobes and the ventral nerve cord, 164,587 traced neurons and 25,563,197 neuron-to-neuron connections carrying 125 million synapses. Packed from the MaleCNS v1.0 flat-connectome export (HHMI Janelia FlyEM, Google Research and the Cambridge connectomics group; released 8 June 2026, published as Berg et al. 2026, *Sexual dimorphism in the complete Drosophila male central nervous system connectome*, Cell, doi:10.1016/j.cell.2026.08.015). The export used is the `traced-only`, minimum-confidence 0.5 connectivity table, with the curated body annotations (cell type, class, superclass, side, sexual dimorphism, fru/dsx expression) and the per-body consensus neurotransmitter prediction. Soma positions come from the annotation table's soma location in 8 nm voxels, converted to micrometres and centred; 139,741 neurons carry one, the rest sit at the origin with the has-soma flag clear. Data license: CC-BY 4.0, https://male-cns.janelia.org/. The manifest carries the SHA-256 of each source file. Sign convention: a neuron whose consensus transmitter is GABA, glutamate or histamine inhibits every target; every other transmitter excites (Shiu et al. 2024, Nature). Connection strengths are synapse counts; the wiring says which neurons reach which and how many synapses they make, and nothing about the efficacy of each synapse in a living fly.

# brain A whole connectome running as a leaky integrate-and-fire network on the GPU. `Brain.load` binds any `connectome` asset to the package's kernels and returns a network; the `FruitFlyBrain` component is the usual owner, and anything holding the network can step, drive, cut, read out and draw it. ## Exports - `M.load(connectomeRef, params?) -> Brain`: allocate the network at rest. `params` overrides any key of `Params`: `dtMs`, `vRest`, `vThresh`, `vReset`, `tauM`, `tauSyn`, `refracMs`, `delayMs`, `weightMv`, `gain`, `minSynapses`, `traceMs`, `maxSpikes`, `seed`, `timeScale`, `maxStepsPerFrame`. A key that is not one of those raises, as `setParams` does, and so does a value that is not a number. - `brain:update(dt)`: once a frame: queue the steps `dt` covers, issue and drain the counter readback, and fold the activity snapshot when something has asked for one. - `brain:step(n)`: queue `n` steps outright. - `brain:select(sel) -> { index }`: the neurons a selection names, ascending. The array is the network's own and memoized on the selection, so read it rather than writing into it. - `brain:stimulate(sel, hz, source?)`: Poisson drive. The rate a neuron sees is the largest any source asked for, so the fly's senses and the bench's experiments can drive the same neurons without taking each other's drive away. `brain:drives()` lists `{ source, count, hz }` per source and `brain:clearStimulation(source?)` clears one source or all of them. - `brain:silence(sel, on)`: ablate or restore. - `brain:watch(name, sel)` / `brain:unwatch(name)` / `brain:rates()`: read groups out as Hz per neuron. A neuron carries at most three groups at once and a watch that would need a fourth on any neuron is refused, naming the neuron and the groups it holds. Watching a name again with the same selection is a no-op; `unwatch` frees the slots and the group's id. - `brain:classRates(limit?)`: the cell classes with the most activity right now, from the 60 ms spike trace. `brain:groupActivity(name)`: that trace per neuron for a watched group of at most 32, which is what a raster draws a row from. Both arm a per-neuron readback that is taken at most every twelve frames and only while something is asking, because it reads the whole activity lane. - `brain:activity()`: network mean rate, total spikes, dropped spikes, simulated time. - `brain:info(i)`: what one neuron is. - `brain:buildCloud(opts, onReady?)` / `brain:setCloudDrawn(on)` / `brain:cloudDrawn()` / `brain:cloudBounds()` / `brain:setCloudOrigin(xyz)` / `brain:showCloud(opts)` / `brain:hideCloud()` / `brain:worldPosition(i)` / `brain:neuronAt(screenX, screenY, radiusPx?)`: the instanced neuron cloud. - `brain:setParams(patch)`, `brain:setRunning(on)`, `brain:setTimeScale(x)`, `brain:setMaxStepsPerFrame(n)`, `brain:reset()`, `brain:ready()`, `brain:destroy()`, `brain:describe()`. ## Usage ```luau local Brain = require("~.brain") -- from inside the package local brain = Brain.load(asset.resolve("fruitFly.maleCNS", "connectome"), { gain = 0.6 }) brain:watch("MN9", { types = { "MN9" } }) brain:stimulate({ types = { "claw_tpGRN" } }, 100, "bench") -- every frame: brain:update(dt) print(brain:rates().MN9) ``` A selection is `{ types?, typePrefix?, pattern?, classes?, superclasses?, sides?, excitatory?, indices?, limit? }`; every given key must hold. ## The cloud, and why it is built in two halves A registered instanced population has its whole transform run recomposed on the CPU every frame, whether or not the cull draws any of it, so 139,741 neurons cost about 4.7 ms of frame whether or not anyone is looking at the brain. `buildCloud` fills the buffers and `setCloudDrawn` registers or drops the draw without touching them, which is what lets `FruitFlyBrain` put the registration down while the camera is pointed somewhere else and put it back the frame the cloud enters view. Taking it down leaves `worldPosition`, `neuronAt` and `cloudBounds` answering as before. Only neurons that have a soma get a slot, which is 139,741 of 164,587. Slot and neuron index are therefore different numbers, so the `cloud_gather` kernel copies each slot's activity out of the neuron it draws once a frame, and `slotToNeuron` maps back for picking. The fill is spread over frames from a task, about forty scan slices and eighteen upload slices, so building the cloud costs about a millisecond a frame for a second rather than one frame of a third of a second. ## Notes - The network is silent until something is driven; a spike always traces back to a stimulus. - Every GPU buffer is named `brain.<id>.<name>`; `compute.observe()` lists them. `destroy` releases them all. - The kernels' `params` are per shader, so two networks on one engine share the model constants the last `applyParams` wrote. - Editing this file hot-reloads its methods onto networks that are already running. A network records the `SHAPE` it was made with and `FruitFlyBrain` loads a fresh one when the module's shape has moved past it, so raise `SHAPE` whenever a field is added to or taken off a network or its cloud.

# connectome A whole nervous system's wiring diagram as one asset: every neuron with its soma position, cell type, class and neurotransmitter, and every neuron-to-neuron connection with its synapse count, packed into the binary form a GPU spiking simulation binds directly. The `brain` module of the `fruitFly` package runs any instance of this type; the type itself carries no animal. ``` maleCNS.connectome/ manifest.json counts, provenance, byte layouts, vocabularies neurons.bin 32 bytes per neuron offsets.bin u32[neurons + 1], CSR row starts into edges.bin edges.bin u32 per connection: target index | synapse count << 18 README.md the animal, the dataset, the release, the license .metadata description + tags ``` ## The manifest ```json { "dataset": "MaleCNS v1.0 ...", "license": "CC-BY 4.0", "citation": "...", "neurons": 164587, "edges": 25563197, "edge_format": "...", "neuron_record": "...", "flags": "...", "sign_rule": "...", "soma_centre_um": [x, y, z], "vocab": { "types": [...], "superclasses": [...], "classes": [...], "neurotransmitters": [...], "sides": [...], "dimorphism": [...], "fruDsx": [...] } } ``` ## The neuron record (32 bytes, little-endian) | offset | type | field | |---|---|---| | 0 | f32 | soma x, micrometres, centred on `soma_centre_um` | | 4 | f32 | soma y | | 8 | f32 | soma z | | 12 | u32 | index into `vocab.types` | | 16 | u32 | flags (below) | | 20 | u32 | source body id, low 32 bits | | 24 | u32 | source body id, high 32 bits | | 28 | u32 | out-degree (number of edges in this neuron's row) | Flags: bits 0-5 superclass, 6-13 class, 14-17 neurotransmitter, 18-19 side, bit 20 has a soma position, 21-23 dimorphism, 24-26 fruDsx, bit 27 set when the neuron excites its targets (clear when it inhibits). Every bit field indexes the vocabulary of the same name, 0 being `unknown`. ## Parts A payload larger than one blob may carry is stored as numbered parts, `edges.0.bin`, `edges.1.bin`, ..., and the manifest's `parts` says how many (`"parts": { "edges": 2 }`). `ref:parts("edges")` lists them with the byte offset each starts at, so a GPU upload writes each part in place; `ref:bytes("edges")` concatenates them when a caller wants the whole table. A payload without an entry in `parts` is the single file `<name>.bin`. ## The edge (4 bytes) Bits 0-17 hold the postsynaptic neuron's index, bits 18-31 the synapse count clamped at 16383. Row `i` of the table is `edges[offsets[i] .. offsets[i+1])`, so a neuron's outgoing connections are one contiguous run, which is what an event-driven spike propagation reads. ## Reading one from Luau ```luau local ref = asset.resolve("maleCNS", "connectome") local m = ref:manifest() -- the decoded manifest local neurons = ref:bytes("neurons") -- a Luau `buffer` of neurons.bin local edges = ref:bytes("edges") -- the whole edge table, for a GPU upload ref:counts() -- { neurons = ..., edges = ... } ``` `ref:bytes(name)` reads the whole file into a Luau `buffer`, so a call for `edges` on a large connectome holds a hundred megabytes until the caller drops it; hand it to a GPU buffer's `writeBytes` and let it go. ## Making one A connectome is written by a packer outside the engine (the `fruitFly` package's README names the script for MaleCNS). It reads the dataset's own export, assigns dense indices, sorts edges by presynaptic neuron, and writes the four files above. `asset.create("connectome", "<name>")` scaffolds the folder with an empty manifest for a packer to fill.