Engineering a Touchless AR Hand Gesture Canvas with MediaPipe & React

Vision-Based Interaction Without Touch

In AR_HandGesture, I set out to turn any standard web camera into an Augmented Reality air-drawing canvas running in pure client-side JavaScript.

21-Landmark Hand Kinematics

MediaPipe Hands outputs normalized 3D coordinates (x, y, z) for 21 joints. To create an intuitive control interface, we map fingertip Euclidean distances to distinct operational states:

1. Drawing State (Index Tip L₈ down & Thumb L₄ near):

Dist(L₈, L₄) = √((x₈ - x₄)^2 + (y₈ - y₄)^2) < 0.045

1. Color Palette Selector (Index & Middle finger extended parallel).
2. Eraser / Clear (Open Palm gesture).

// Landmark distance calculation for pinch trigger
function checkPinchGesture(landmarks) {
  const thumb = landmarks[4];
  const index = landmarks[8];
  const distance = Math.hypot(thumb.x - index.x, thumb.y - index.y);
  return distance < 0.05; // Pinch active
}

Smoothing Jitter with Moving Exponential Averages

Hand tracking data over 60 FPS video streams inherently contains micromovement noise. I implemented an exponential smoothing algorithm so air strokes look fluid and calligraphic rather than jagged:

Live Demonstration

Available at arhandgesture.vercel.app.