Picking · Meep

PickingSystem answers one question: what is drawn under this pixel. An entity opts in by carrying a Pickable marker, the application hands the system a batch of view points as PickingQuery objects, and one PickingResult comes back per point. It is the path from a pointer position to an entity id, and it lives at @woosh/meep-engine/src/engine/graphics3/PickingSystem.js with the rest of the ECS rendering systems.

Three properties change how the calling code is written, and all three are deliberate.

It is asynchronous and batched, on purpose

pick(queries) takes an array and returns a promise. There is no synchronous entry point, and there could not be one: the answer is read back off the GPU. A compute pass gathers the queried texels from the frame that draws them, and the results are read when the device has finished with it, so an answer is a frame or two late and arrives for the whole batch at once.

Write the calling code as if the answer arrives later, because it does:

The world can change while the question is out. Check dataset.entityExists(entity) before acting on a result.
Batch what you can. Asking for four points in one call is one round trip, not four.
Nothing in the current tick can come to depend on the result.

It picks what was actually drawn

The rasterizer already writes, for every pixel, the meshes row it drew there and the depth it drew it at. A pick reads those two texels with textureLoad, gathers them into a storage buffer and resolves the row back to an entity after readback. The precision is therefore the frame’s own: triangle-exact, under skinning, through whatever alpha the material tested - with nothing kept on the CPU for it. No bounds, no tree, no per-frame refresh, and no second representation that could disagree with what the player is looking at.

Two consequences worth designing around:

Only what is drawn can be picked. Something culled, hidden, or standing behind something else is not under that pixel. There is no “everything along the ray” mode, and no way to ask about what is not on screen.
The answer is the nearest surface, and it is whatever is in front. A prop with no gameplay meaning standing in front of a chest means “the prop”, not “the chest”.

There is no index

Nothing pays for picking on a frame where nobody picks, and nothing pays for it per entity either. The system keeps a set of the entities that opted in and nothing else: no spatial structure, no bounds, no per-frame update work. A frame with queries waiting adds a compute gather with one thread per query and one readback, carrying eight bytes of results per query; a frame with none adds nothing at all. Both textures are read as bindings, so picking needs neither COPY_SRC on the visibility buffer nor an illegal sub-region copy of a depth texture.

That also removes a class of bug outright. There is no index to fall out of step with the scene, so a pick can never land on where something used to be.

Setting it up

PickingSystem takes one collaborator: the GraphicsEngine, whose frames draw the view the picks read.

import { EngineHarness }  from "@woosh/meep-engine/src/engine/EngineHarness.js";
import { PickingSystem }  from "@woosh/meep-engine/src/engine/graphics3/PickingSystem.js";

const engine = await EngineHarness.bootstrap({
    configuration(config, engine) {
        config.addSystem(new PickingSystem(engine.graphics));
    },
});

Something still has to get the scene drawn - meshes & materials covers that side. Three things follow from reading the answer off the frame:

Anything the renderer draws can be picked. A pixel names a meshes row, and every drawn primitive has one however it got there: a model placed through SGMesh and MeshSystem, or an entity built directly from a MeshletGeometry and a ShadedGeometry. There is no separate index to be absent from.
The opt-in may sit on an ancestor. A model placed through SGMesh does not draw at the entity carrying the component - it draws at the child entities the prefab was spawned into. The system resolves the drawn entity and then walks up ParentEntity to the nearest ancestor carrying Pickable, so the marker goes on the entity you want back and the primitives beneath it need nothing.
PickingSystem.dependencies is [Pickable]. No transform is required: where a thing is, is where it was drawn.

PickingSystem3 is a deprecated re-export of the same class. Import PickingSystem.

The `Pickable` marker

import Entity        from "@woosh/meep-engine/src/engine/ecs/Entity.js";
import { Pickable }  from "@woosh/meep-engine/src/engine/graphics3/Pickable.js";
import { SGMesh }    from "@woosh/meep-engine/src/engine/graphics/ecs/mesh-v2/aggregate/SGMesh.js";
import { Transform64 } from "@woosh/meep-engine/src/engine/ecs/transform/Transform64.js";

const model = new SGMesh();
model.url = "/models/crate.glb";

const transform = new Transform64();
transform.setTranslation(3, 0, -2);

new Entity()
    .add(model)
    .add(transform)
    .add(new Pickable())
    .build(engine.entityManager.dataset);

Pickable carries no data and has nothing to configure - the picking shape comes from what the entity actually looks like, so there is no second way for it to disagree with the model. It is paid for per entity that asks, which is the point: a level is mostly scenery nobody will ever click on, and scenery costs nothing here.

It serializes through PickableSerializationAdapter (already in populateEngineSerializationRegistry). The adapter writes no bytes: what has to survive a save is that the entity opted in, and the presence of the component is that.

Queries

import { PickingQuery } from "@woosh/meep-engine/src/engine/graphics3/PickingQuery.js";

const query = PickingQuery.from_viewport(0.5, 0.5);   // the centre of the view

Field	Type	Meaning
`u`	`number`	Across the view. `0` at the left edge, `1` at the right.
`v`	`number`	Down the view. `0` at the top edge, `1` at the bottom.

A query names a point of the view, not a ray, because the answer comes off the visibility buffer - the entity the rasterizer drew there, at the depth it drew it - and a ray that does not start at the camera has no pixel to read.

It is a fraction rather than a pixel so that a caller holding a pointer position need know nothing about the resolution the frame was drawn at; the frame that answers turns the fraction into the texel it drew. A point outside [0, 1) on either axis is off the view: it still produces a result, in order, and that result is a miss.

Results

import { PickingResult } from "@woosh/meep-engine/src/engine/graphics3/PickingResult.js";

Member	Type	Meaning
`entity`	`number`	What was hit, or `PickingResult.NOTHING` (`-1`).
`position`	`Vector3`	Where the ray met it, in world space. Untouched on a miss.
`distance`	`number`	How far along the ray that was. `Infinity` on a miss.
`found`	`boolean` (getter)	`entity !== PickingResult.NOTHING`.

position is the world point the pixel’s depth stands for, unprojected through what the frame actually projected with rather than through wherever the camera has moved to since; distance is how far that point is from the eye that drew it.

The contract is positional. A batch always answers with one result per query, in the order the queries were given, and a point that found nothing still produces a row - so the two arrays line up by index and a caller never has to search for the answer to its own question. distance being Infinity on a miss means an unfiltered batch still sorts and compares sensibly.

Each query yields whatever was drawn nearest the camera at that pixel. There is no “everything behind it too” mode, no filter callback, and no layer mask - if you need to exclude something, exclude it by not giving it Pickable, or filter the entity id after the fact.

A click, to an entity

GraphicsEngine.normalizeViewportPoint(input, result) turns viewport pixels (origin top-left) into a clip-space point (-1 to 1, +Y up), writing into a Vector2 you own. A PickingQuery wants that same point as a fraction with v measured downwards, which is one line either way: u = (x + 1) / 2, v = (1 - y) / 2.

viewportProjectionRay(x, y, source, direction) is still there and still builds a world-space ray through the one camera the graphics engine owns - it is what a terrain pick or a physics raycast starts from. Picking no longer needs it.

import Vector2 from "@woosh/meep-engine/src/core/geom/Vector2.js";
import { PickingQuery }  from "@woosh/meep-engine/src/engine/graphics3/PickingQuery.js";
import { PickingSystem } from "@woosh/meep-engine/src/engine/graphics3/PickingSystem.js";

const picking = engine.entityManager.getSystem(PickingSystem);

const clip = new Vector2();

engine.devices.pointer.on.tap.add(async (position) => {
    engine.graphics.normalizeViewportPoint(position, clip);

    const [result] = await picking.pick([
        PickingQuery.from_viewport((clip.x + 1) / 2, (1 - clip.y) / 2),
    ]);

    if (!result.found) {
        return;
    }

    const dataset = engine.entityManager.dataset;

    if (!dataset.entityExists(result.entity)) {
        // the level went away, or that entity did, while the question was out
        return;
    }

    console.log("picked", result.entity, "at", result.distance, result.position);
});

on.tap sends (position, event), where position is the pointer device’s own live Vector2 - safe to read synchronously, which is all the first two lines do.

To ignore clicks that landed on UI rather than on the game field, compare engine.devices.pointer.getTargetElement() against engine.graphics.domElement before building the query. TooltipComponentSystem (src/engine/graphics3/TooltipComponentSystem.js) does exactly that and is the reference consumer of this API in the engine: it converts the pointer the same way, awaits a one-query batch every tick, and acts on whatever comes back without checking whether the pointer moved in the meantime - the answer describes where the pointer was, the next tick asks about where it is now.

Entities whose model has not arrived

An entity can be Pickable and still not be findable, because its model has not loaded and nothing has yet been drawn for it. Nothing has to notice the moment that changes: the opt-in is a set membership, and the first frame that draws the model is the first frame that can answer for it. The reverse costs nothing either - a model that goes away stops being drawn, and the entity keeps its opt-in.

Two counters are exposed as diagnostics. picking.pickable_count is how many entities have opted in; picking.waiting_count is how many batches have been asked and not yet answered by a frame. Neither is a count of what is pickable on screen right now - only a frame knows that.

What picking does not cover

Picking is one narrow tool: a point of the view in, an entity id out, at the precision of whatever was drawn there. Three neighbouring questions go somewhere else entirely.

The question	The tool
What did the player click on?	`PickingSystem.pick`
Where on the ground did the player click?	`Terrain.raycastFirstSync`, or the `pick()` grid helper
Does this ray hit something solid, and at what angle?	`physics.raycast` - see spatial queries
Would a body fit here? Sweep a shape?	`physics.overlap` / `physics.shapeCast`

Terrain picking does not come through here. Terrain carries no Pickable, and the answer wanted from it is a place rather than an entity. A terrain hit is an exact raycast against the terrain’s own height representation: Terrain.raycastFirstSync(surfacePoint, ox, oy, oz, dx, dy, dz) fills a SurfacePoint3 with a position and a normal, synchronously. pick(x, y, graphics, terrain, callback) in @woosh/meep-engine/src/engine/ecs/grid/pick.js wraps the whole thing - it does the viewport-to-ray conversion itself and hands the callback the hit in grid coordinates as well as world coordinates. Reach for it whenever the answer you want is a place rather than an entity; it is more precise than picking, not less.

Physics raycasts are a different world. Spatial queries run against colliders in the physics world, and they are synchronous, exact against each shape, accept a filter(entity, collider), and report surface normals. Reach for physics.raycast when the question is about the simulated world - line of sight, a projectile, a ground probe, “what would this bullet hit”. Reach for PickingSystem when the question is about what the player is pointing at on screen and the answer you want is an entity id. An entity can have a collider and no Pickable, or the reverse; the two indexes are unrelated.

Finally, picking is not a general CPU raycast against a mesh. For triangle-accurate intersection against one specific primitive there is ShadedGeometry.query_raycast_nearest(contact, ray, transform_matrix4) - contact is a SurfacePoint3, ray a 6-tuple [ox, oy, oz, dx, dy, dz], and transform_matrix4 a 16-element matrix array. It requires a decoded Geometry: a MeshletGeometry keeps its triangles quantized inside meshlet pages, so a ShadedGeometry placed from a loaded model cannot be asked this without decoding it first via geometry_build_from_meshlet_geometry. It is a tool for tooling, not for gameplay.