Building an Intuitive Gesture Recognition System

One of Notidian’s standout features is its ability to understand what you’re trying to draw. Draw a rough circle, and it can perfect it. Sketch a wobbly line, and it can straighten it. This isn’t magic - it’s the result of a carefully crafted gesture recognition system.

The Challenge of Intent

Understanding user intent from raw input data is complex. A series of points could represent:

• A deliberate freehand stroke
• An attempt at a geometric shape
• A gesture command (like undo or delete)
• The beginning of text input

Our system needs to distinguish between these intents in real-time, without interrupting the creative flow.

Architecture Overview

The gesture recognition system consists of three main components:

1. Input Processing Pipeline

Raw input events are processed and normalized:

interface StrokePoint {
	x: number;
	y: number;
	pressure: number;
	timestamp: number;
	velocity: Vector2;
}

class InputProcessor {
	processPoint(event: PointerEvent): StrokePoint {
		return {
			x: event.clientX,
			y: event.clientY,
			pressure: event.pressure || 0.5,
			timestamp: performance.now(),
			velocity: this.calculateVelocity(event),
		};
	}
}

2. Feature Extraction

We extract meaningful features from the stroke:

interface StrokeFeatures {
	length: number;
	curvature: number;
	closedness: number;
	cornerCount: number;
	aspectRatio: number;
	velocity: {
		mean: number;
		variance: number;
	};
}

3. Shape Recognition

Our shape detector uses multiple algorithms in parallel:

Geometric Analysis

For basic shapes, we use geometric properties:

function detectCircle(points: StrokePoint[]): CircleResult {
	const center = calculateCentroid(points);
	const radii = points.map((p) => distance(p, center));
	const meanRadius = mean(radii);
	const variance = standardDeviation(radii);

	const confidence = 1 - variance / meanRadius;
	return { isCircle: confidence > 0.85, confidence, center, radius: meanRadius };
}

Template Matching

For complex shapes, we use template matching with the $1 recognizer algorithm:

class TemplateRecognizer {
	recognize(stroke: StrokePoint[]): RecognitionResult {
		const normalized = this.normalize(stroke);
		let bestMatch = null;
		let bestScore = 0;

		for (const template of this.templates) {
			const score = this.compareStrokes(normalized, template);
			if (score > bestScore) {
				bestScore = score;
				bestMatch = template;
			}
		}

		return { shape: bestMatch?.name, confidence: bestScore };
	}
}

Smart Shape Assistance

When a shape is recognized, we offer intelligent assistance:

Progressive Disclosure

• Subtle hint: Shape preview appears transparently
• User confirmation: Press space or pause to accept
• Auto-reject: Continue drawing to keep freehand

Contextual Perfection

Different contexts require different levels of “perfection”:

function perfectShape(shape: RecognizedShape, context: DrawingContext): Shape {
  if (context.mode === 'technical') {
    // Snap to grid, perfect angles
    return snapToGrid(perfectGeometry(shape));
  } else if (context.mode === 'artistic') {
    // Maintain hand-drawn character
    return smoothShape(shape, preserveCharacter: true);
  }
  return shape;
}

Gesture Commands

Beyond shape recognition, we detect command gestures:

Scribble to Erase

Rapid back-and-forth motion triggers erase:

function detectScribble(points: StrokePoint[]): boolean {
	const directions = calculateDirectionChanges(points);
	const speed = calculateAverageSpeed(points);

	return directions > 6 && speed > SCRIBBLE_SPEED_THRESHOLD;
}

Tap Gestures

Quick taps for tool switching and commands:

• Single tap: Select
• Double tap: Edit mode
• Triple tap: Delete
• Long press: Context menu

Machine Learning Enhancement

We use a lightweight neural network for ambiguous cases:

Training Data

• 50,000+ hand-drawn samples
• Multiple drawing styles and speeds
• Various input devices (mouse, stylus, touch)

Model Architecture

model = Sequential([
  Dense(128, activation='relu', input_shape=(feature_count,)),
  Dropout(0.2),
  Dense(64, activation='relu'),
  Dense(num_shapes, activation='softmax')
])

Real-time Inference

The model runs in WebAssembly for consistent performance:

async function classifyStroke(features: Float32Array): Promise<Classification> {
	const output = await wasmModel.predict(features);
	return {
		shape: shapes[argmax(output)],
		confidence: max(output),
	};
}

Performance Considerations

Gesture recognition must be fast to feel responsive:

Optimization Strategies

1. Early rejection: Quick checks eliminate unlikely matches
2. Progressive refinement: Start with fast approximations
3. Parallel processing: Use Web Workers for complex calculations
4. Caching: Store recent recognition results

Benchmarks

• Average recognition time: < 5ms
• 99th percentile: < 15ms
• Memory usage: < 2MB

User Experience Design

Predictable Behavior

Users learn the system quickly because:

• Consistent recognition thresholds
• Visual feedback during drawing
• Undo always available

Customization

Users can adjust:

• Recognition sensitivity
• Auto-correction aggressiveness
• Enabled shape types
• Gesture shortcuts

Future Directions

We’re exploring several enhancements:

Contextual Intelligence

• Learn from user’s drawing style
• Adapt to current artwork context
• Predict next likely shape

Advanced Gestures

• Multi-touch gestures for tablets
• Pen tilt and rotation support
• Air gestures for AR/VR

Collaborative Recognition

• Share recognition models between users
• Crowd-sourced gesture templates
• Team-specific shortcuts

Conclusion

Building an intuitive gesture recognition system requires balancing technical sophistication with user experience simplicity. By combining geometric analysis, template matching, and machine learning, Notidian creates a natural drawing experience that understands your intent without getting in your way.

The key is not perfect recognition, but appropriate assistance - helping when wanted, staying invisible when not.