Engineering a Touchless AR Hand Gesture Canvas with MediaPipe & React

Vision-Based Interaction Without Touch

In AR_HandGesture, I set out to turn any standard web camera into an Augmented Reality air-drawing canvas running in pure client-side JavaScript.

21-Landmark Hand Kinematics

MediaPipe Hands outputs normalized 3D coordinates $(x, y, z)$ for 21 joints. To create an intuitive control interface, we map fingertip Euclidean distances to distinct operational states:

1. Drawing State (Index Tip $L_8$ down & Thumb $L_4$ near):

$$\text{Dist}(L_8, L_4) = \sqrt{(x_8 - x_4)^2 + (y_8 - y_4)^2} < 0.045$$

1. Color Palette Selector (Index & Middle finger extended parallel).
2. Eraser / Clear (Open Palm gesture).

// Landmark distance calculation for pinch trigger
function checkPinchGesture(landmarks) {
  const thumb = landmarks[4];
  const index = landmarks[8];
  const distance = Math.hypot(thumb.x - index.x, thumb.y - index.y);
  return distance < 0.05; // Pinch active
}

Smoothing Jitter with Moving Exponential Averages

Hand tracking data over 60 FPS video streams inherently contains micromovement noise. I implemented an exponential smoothing algorithm so air strokes look fluid and calligraphic rather than jagged:

$$P_{\text{smoothed}} = \alpha \cdot P_{\text{current}} + (1 - \alpha) \cdot P_{\text{previous}}$$

Live Demonstration

Available at arhandgesture.vercel.app.