Work / Independent

LightTable: Portable Interactive Projection

An affordable, portable interactive projection system that turns ordinary surfaces into responsive sensory environments, using computer vision for hand and motion tracking and a real-time visual engine.

Role: Designer and developer
Context: illusionTechnique (independent)
Technologies: Python, OpenCV, MediaPipe, Godot 4, OSC, Projection
Technical approach: A Python and OpenCV computer vision layer offers several interchangeable tracking modes: hand tracking with MediaPipe, plus blob, colour, edge, motion and object tracking, with smoothing and calibration tools for aligning to different surfaces.
A Godot 4 visual layer renders a library of about fifteen interactive scenes, including water ripples, balloons, a piano, a galaxy, bouncing balls, colour trails, freehand drawing, an alphabet puzzle and a talk-back scene, chosen from an on-screen menu.
The two layers communicate over OSC, so either side can be changed independently.
A web-based remote lets a phone or tablet configure and control the system, and the whole thing is tuned to run on mini-PC hardware.
Outcome / state: In development
Links: Source on GitHub

System overview diagram: a mini projector and webcam above a table, a mini PC for processing, and a six-step flow from capture to interaction — System overview: a webcam captures the surface, the mini PC detects touches and gestures and runs the application, and a short-throw projector displays the result.

Concept illustration of a parent and child touching a projected water scene on a round coffee table — Concept illustration: a water scene projected onto a coffee table.

Concept illustration of a parent and child playing with a projected underwater scene on a wooden coffee table — Concept illustration: a ceiling-mounted unit projecting down onto a home table.

Code

Kalman smoothing for hand and object tracking · cv_layer/smoothing.py

Raw camera detections jitter, and detection adds latency. Each tracked hand or object gets its own constant-velocity Kalman filter, which smooths the position and predicts slightly ahead so the visuals feel immediate.

import cv2
import numpy as np


class KalmanSmoother:
    """
    Per-ID constant-velocity Kalman filter.
    State vector: [x, y, vx, vy]   Measurement: [x, y]

    Very effective for smoothing noisy position data while predicting ahead to
    compensate for detection latency.  measurementNoiseCov controls how much
    the filter trusts the raw measurement vs. its own prediction.
    """

    def __init__(self):
        self._filters: dict = {}

    @staticmethod
    def _make_kf() -> cv2.KalmanFilter:
        kf = cv2.KalmanFilter(4, 2)
        # x, y observed; vx, vy inferred
        kf.measurementMatrix = np.array(
            [[1, 0, 0, 0],
             [0, 1, 0, 0]], dtype=np.float32
        )
        # Constant-velocity transition
        kf.transitionMatrix = np.array(
            [[1, 0, 1, 0],
             [0, 1, 0, 1],
             [0, 0, 1, 0],
             [0, 0, 0, 1]], dtype=np.float32
        )
        kf.processNoiseCov      = np.eye(4, dtype=np.float32) * 1e-3
        kf.measurementNoiseCov  = np.eye(2, dtype=np.float32) * 5e-2
        return kf

    def smooth(self, track_id: int, x: float, y: float):
        """Returns (smoothed_x, smoothed_y) using the per-ID Kalman filter."""
        if track_id not in self._filters:
            kf = self._make_kf()
            kf.statePre = np.array([[x], [y], [0.0], [0.0]], dtype=np.float32)
            self._filters[track_id] = kf

        kf = self._filters[track_id]
        kf.predict()
        corrected = kf.correct(np.array([[x], [y]], dtype=np.float32))
        return float(corrected[0]), float(corrected[1])

    def clear(self, track_id=None):
        if track_id is None:
            self._filters.clear()
        else:
            self._filters.pop(track_id, None)

A common interface for swappable trackers · cv_layer/trackers/base_tracker.py

Hand, blob, colour, edge, motion and object tracking all implement this small contract, so the system can switch modes at runtime and each tracker exposes its own tuning controls.

from abc import ABC, abstractmethod


class BaseTracker(ABC):
    """Common interface for all LightTable CV trackers."""

    name = "Base"

    @abstractmethod
    def process(self, frame) -> list:
        """
        Process a BGR camera frame.
        Returns a list of detection dicts, each containing at least:
            id (int), x (float), y (float)
        """

    def draw_debug(self, frame, detections: list):
        """Draw debug visuals onto frame in-place and return it."""
        return frame

    def get_trackbar_settings(self) -> list:
        """
        Return tracker-specific trackbar descriptors.
        Each dict: {name: str, max: int, default: int}
        """
        return []

    def on_trackbar(self, name: str, value: int):
        """Called whenever one of this tracker's trackbars changes."""

    def reset(self):
        """Reset internal state (call on camera switch or re-init)."""