ASL Interpreter
A fingerspelling prototype: a desktop app that tries to read American Sign Language letters from a webcam and type them out. MediaPipe finds 21 points on the hand, those points become 42 numbers, and a 17,400-parameter network picks one of 24 letters. The pipeline runs end to end, from data capture to a 24.5 KB TensorFlow Lite model.
Tested on synthetic input only. Every number here comes from a model trained on synthetic data, and the screenshots use a blank frame or invented values. Recognition on real signing hands hasn't been validated: the repo ships no trained model or hand data.
The app's own window
asl_app.main() ran headless with a blank 1280 × 720 frame in place of the webcam. It loaded the TFLite model and 24 classes, and rendered the "No hand detected" state.
draw_ui_overlay called with invented values: B at 97%, word "HELLO", confirmation flash on. As it happens, the app can't type the double L in HELLO without lowering the hand.A genuine recognition screenshot needs a model trained on real hand images and a signing hand in front of the camera. The repo ships neither: models and data are gitignored, and an earlier committed model was missing its graph file, so it could never load.
From hand shape to 42 numbers
Why landmarks instead of pixels. MediaPipe has already done the hard vision work, so the classifier only sees geometry. Background and skin tone drop out, and a small dense network is enough. Training and the live app call the same normalisation function, which keeps the two from drifting apart.
Invariance, measured. Scaling the hand 2× changed no feature. Rotating it 20° changed features by up to 0.75, and mirroring it (a left hand) by up to 1.48. A commit titled "Fix rotation-invariant normalization" made the scale Euclidean; it did not add rotation invariance.
# utils.py:84-113, condensed wrist = landmark_list[0] normalized = [{'x': lm['x'] - wrist['x'], 'y': lm['y'] - wrist['y']} for lm in landmark_list] mcp = landmark_list[9] dist = np.sqrt((mcp['x'] - wrist['x'])**2 + (mcp['y'] - wrist['y'])**2) scale = max(float(dist), 0.01) features = [lm['x'] / scale for lm in normalized] + \ [lm['y'] / scale for lm in normalized]
Capture, train, export, run
A small network is enough for geometry
model.summary(); 17,016 trainable plus 384 batch-norm statistics.# scripts/train_model.py:78-95 model = keras.Sequential([ layers.Input(shape=(42,)), layers.Dense(128, activation='relu'), layers.BatchNormalization(), layers.Dropout(0.3), layers.Dense(64, activation='relu'), layers.BatchNormalization(), layers.Dropout(0.3), layers.Dense(32, activation='relu'), layers.Dropout(0.2), layers.Dense(num_classes, activation='softmax')]) model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
Sizes and call times come from a model trained on synthetic data during this review, on an Apple Silicon Mac. They measure the classifier call only, not MediaPipe or the camera, and are indicative.