Skip to content
Aman SinghWorkshop
AmanSinghNp / palmlab-gesture-playground

PalmLab Clay Lab

Shape a 3D clay sphere with two hands. The webcam feeds MediaPipe's hand landmarker in the browser, pinches become stretch and squash, and React Three Fiber deforms 1,617 vertices every frame. It opens in a demo mode that needs no camera, loads the vision model only when you ask for it, and ships with tests, lint and a Pages deploy.

Try the app ↗View source ↗
768Lines of TS/TSX
10 / 10Tests passing
3Clay states
300 kBMain JS, gzip
7.8 MBModel, on demand
1,617Vertices per frame
Demo

The Clay Lab in demo mode

Recorded in headless Chromium with software WebGL. A scripted pair of hands cycles idle → one pinch → two pinches every 160 ms. The telemetry column tracks clay state, pinch count, span, scale and wrist roll.
Clay Lab showing the sphere squashed into a tilted ellipse, clay state COMPRESS, two pinches
Compress. Two pinches squash the sphere along the axis between them. The squash here is tilted about 40° from the pinch axis, which is bug 1 below.
Clay Lab in camera mode with a small live preview and the status LIVE, no hands detected
Camera mode. Reached LIVE with Chromium's fake webcam. The test pattern has no hands, so the lab shows IDLE and "Place both hands in frame".
The Clay Lab stacked into a single column on a phone-sized screen
Phone width. The three columns stack below 720 px.
Interaction

Two hands, three states

Hand signalEffect on the clay
Midpoint of both hand centresPosition (x, y, depth)
Distance between hands ÷ 0.34Scale, clamped to 0.45 – 2.8
Circular mean of both wrist anglesRoll
One pinch (thumb tip to index tip < 0.085)Stretch toward the pinch
Two pinchesCompress along the axis between them
No pinchIdle: relax back to a sphere

An earlier version had three visual modes, called Sculpt, DJ and Physics. They were cut so the app does one interaction well. The gesture classifier still recognises open palm, fist and point, but only pinch changes anything.

Privacy is built in. There is no server. The demo runs with no permission prompt, and the 153 kB vision bundle, the WASM runtime and the 7.8 MB model download only after you press Use camera.

Architecture

Pure domain, one adapter, one renderer

Adapters · side effects Webcam 1280×720getUserMedia, on clickmirrored preview useHandTrackingHandLandmarker, GPU → CPU2 hands × 21 points, per rAF Demo tickerevery 160 ms, deterministic createDemoFrame(step)synthetic hand pair GestureFramesame shape from both Domain · pure · unit tested deriveClayControls position · midpointscale · span ÷ 0.34roll · circular meanpinchPointsidle | stretch | compress memoised per frame Render Stage · R3F useFrame1,617 target vertices, lerp at 9/s, recompute normalsmeshPhysicalMaterial, clay #bd7359 App telemetrylandmark map, meters,aria-live cue usePrefersReducedMotionfreezes demo, no wobble, faster blend
Frame pipeline. Camera and demo both emit the same GestureFrame, so everything downstream does not care which one is running. The domain modules import neither React nor the DOM, which is what makes them testable without a camera.
How it works

From pinch to deformation

// gestures.ts:39-50
export function classifyGesture(landmarks, handedness = 'Right') {
  if (landmarks.length < 21) return 'neutral'
  const pinchDistance = distance(landmarks[4], landmarks[8])
  if (pinchDistance < 0.085) return 'pinch'
  const extended = countExtendedFingers(landmarks, handedness)
  if (extended >= 4) return 'open-palm'
  if (extended === 0) return 'fist'
  ...
}

One threshold, one frame. No hysteresis, no smoothing, not scaled to hand size.

// Stage.tsx:26-37 · stretch toward the pinch
const direction = normalizeDirection(
  (point.x - controls.position.x) * 1.7,
  (controls.position.y - point.y) * 1.7)
const projection = Math.max(0, x*direction.x + y*direction.y)
const influence = Math.min(1, projection / 1.12)
return { x: x + direction.x * influence * 0.58,
         y: y + direction.y * influence * 0.58,
         z: z * (1 + influence * 0.12) }

// Stage.tsx:88 · the only smoothing in the pipeline
const blend = Math.min(1, delta * (reducedMotion ? 24 : 9))

The y term is mirrored, and the result is applied inside a group that is already rotated by the roll.

pinch clay centre Scratch check with the real deriveClayControls intended stretch: 166° (toward the left pinch) rendered stretch: −76° (straight down) 1. y difference is mirrored (hand space is y-up) 2. direction is computed in screen space but applied in the group's local frame, already rotated by roll ≈ 90° for upright hands In the scripted demo the error is 34° to 69°.
The stretch points the wrong way. Fixing it means taking point.y − position.y and rotating the direction by −roll before use, or deforming a child mesh that isn't rotated.
Measurements

Build and bundle

What the browser downloads, gzip where it applies
The model and the vision bundle load only in camera mode. The model is not compressed further.
Lines per file
Grey is test code.

Verified locally on Node 24: npm ci, typecheck, lint and build are clean; 10 of 10 tests pass in 123 ms. Vite warns that the 1,091 kB main chunk is over 500 kB. CI runs typecheck, tests, lint and build on every push, and a Dependabot PR for a dev-only vitest advisory is open.