air_pointer

Merge pull request #32 from Ranveer-Singh-Gour/perf/gate-debug-info-o…

Bug fixes

• Release-build latency logging removed — the per-60-frame latency debugPrint in the web GestureInputSource now only runs in debug builds (kDebugMode); it previously logged to the console in release builds too.
• Wasm compilation selects the web implementation — the conditional export for GestureInputSource now keys on dart.library.js_interop instead of the legacy dart.library.html, so flutter build web --wasm gets the MediaPipe implementation rather than the native stub.
• Corrected the hand_detection LandmarkProvider doc example — the sketch in landmark_provider.dart called detector.detect(image) with a raw CameraImage (that method takes decoded image bytes; camera streams need detectFromCameraImage) and treated Hand.landmarks as already normalised, when hand_detection returns them in the source image's pixel space. The example now uses the correct camera-streaming call, normalises landmarks by the frame's width/height, and aliases hand_detection's Handedness/HandLandmarkType to avoid colliding with air_pointer's own.
• Worker no longer killed before it can release MediaPipe resources — GestureInputSource.dispose() (web) posted a 'dispose' message and called Worker.terminate() in the same tick; since postMessage delivery is asynchronous, the worker was always hard-terminated before it processed the message, so landmarker.close() (which releases the WASM/WebGL delegate) never ran. dispose() now gives the worker a short window to shut itself down gracefully before terminating it as a backstop.

Performance

• Skip debug-only work when nobody's listening (web) — GestureInputSource parsed worldLandmarks, computed per-hand bounding boxes, and built a full GestureDebugInfo snapshot on every processed frame regardless of whether anything was subscribed to debugInfo. That work now only runs when debugInfo has an active listener, avoiding needless per-frame allocation for consumers who don't use the debug stream.
• Extend the debug-only skip to the worker itself (web) — the previous fix only gated main-thread parsing; the inference worker still built worldHands/handedness and transferred them across the thread boundary on every frame regardless of listener state. GestureInputSource now tells the worker (via a new setDebugEnabled message, sent from debugInfo's onListen/onCancel) whether to compute that data at all, and the main thread only dartifies the JS payload fields it actually needs per frame instead of eagerly converting the whole message.

New

• GestureInputSource.workerUrl (web, default 'hand_tracker_worker.js') — the inference worker script location is now configurable, for apps that serve the worker from a subdirectory. Must be same-origin.

Bug fixes

• Drag-to-scroll interference — CanvasScrollEvent could fire immediately
  after releasing a drag when the hand briefly adopted a pointing-like shape
  during the transition. Fixed by adding a scrollConfirmFrames gate (default
  2, ~67 ms at 30 fps) that requires N consecutive pointing frames before
  scroll activates, consistent with the existing pinchConfirmFrames guard for
  drag. Set scrollConfirmFrames: 1 to restore the old single-frame-baseline
  behaviour.

New

• handLandmarkConnections — exported constant listing the 21 skeleton
  bone pairs for the MediaPipe hand topology. Iterate over it to draw a
  skeleton overlay without hand-coding the connections yourself.

• CanvasGestureEvent.confidence (double, default 1.0) — ML
  confidence of the gesture classification. Always 1.0 on web (rule-based
  heuristic); populated from the backend's score on native when using a
  model-based LandmarkProvider such as hand_detection.

• HandDetectionFrame.gestureConfidence / secondHandGestureConfidence
  (double, default 1.0) — per-gesture confidence fields for native backends
  to expose ML scores that flow through to CanvasGestureEvent.confidence.

• GestureInputSource confidence thresholds (web) — three new constructor
  params minHandDetectionConfidence, minHandPresenceConfidence, and
  minTrackingConfidence (each default 0.5, matching MediaPipe defaults).
  Tune these to trade sensitivity against false positives without modifying the
  JS worker.

• HandDetectionFrame.worldLandmarks / secondWorldLandmarks —
  optional world-space landmark lists (metric scale, hand-centre origin) for
  native backends to expose orientation-invariant landmark data. On web,
  populated from MediaPipe's worldLandmarks result automatically.

• GestureDebugInfo.worldLandmarks / secondWorldLandmarks — world
  landmarks forwarded through the debug snapshot so overlays and classifiers
  can consume them.

• HandDetectionFrame.boundingBox / secondHandBoundingBox (Rect?)
  — axis-aligned bounding box of each detected hand in normalised image
  coordinates. Native backends set this from their ML model's output. On web,
  computed automatically as the min/max envelope of the 21 image-space
  landmarks.

• GestureDebugInfo.boundingBox / secondHandBoundingBox (Rect?)
  — bounding box forwarded through the debug snapshot for overlay rendering.

• StylusInputSource — input source for Apple Pencil, Samsung S-Pen, and
  any PointerDeviceKind.stylus / PointerDeviceKind.invertedStylus device.
  Maps pen contact to element-drag canvas events (Down/Move/Up, tap, double-tap,
  hover, cancel). Filters out mouse/touch/trackpad events so it can be combined
  with MouseInputSource or TouchInputSource without duplicates. Exposes an
  eraserModeStream that emits true / false on eraser-vs-tip mode changes.

• CanvasDownEvent.pressure / CanvasMoveEvent.pressure (double,
  default 1.0) — hardware pen pressure (0.0–1.0) carried by stylus down and
  move events. Backwards-compatible: non-stylus sources emit 1.0; existing
  construction and pattern-matching compiles unchanged.

• defaultPointerSource() — factory function that returns the best-fit
  CanvasInputSource for the current platform: TouchInputSource on Android /
  iOS, MouseInputSource everywhere else (including web). Uses kIsWeb and
  defaultTargetPlatform from package:flutter/foundation.dart, so it is safe
  to call on web without conditional imports. GestureInputSource is not
  included — it requires platform-specific configuration that cannot be inferred
  automatically.

• TouchInputSource — mobile-native input source for Android and iOS.
  Single-finger drag → CanvasScrollEvent (direct-manipulation pan); two-finger
  pinch/spread → CanvasScaleEvent; fling → CanvasScrollEvent with non-zero
  velocity field for consumer momentum animations. Tap detection uses raw
  Listener events rather than GestureDetector so taps are never dropped by
  ScaleGestureRecognizer's kPanSlop movement threshold.

• CanvasScrollEvent.velocity (Offset, default Offset.zero) — fling
  velocity field added to CanvasScrollEvent. Non-zero only on fling events
  emitted by TouchInputSource. Backwards-compatible: existing code that
  constructs or pattern-matches CanvasScrollEvent compiles unchanged.

Documentation

• README — Documented three previously undocumented public API surfaces:
  CanvasInputController muting (muteWhenActive/activeStream),
  GestureInputSource.statusStream lifecycle states, and
  GestureInputSource.setFilterParams().
• README — Added self-hosting section documenting mediaPipeBaseUrl and
  modelAssetUrl constructor parameters (Flutter Web only); corrected the
  "no self-hosted model" limitation which was inaccurate — self-hosting has
  been wired since 0.1.0.
• KNOWN_LIMITATIONS — Corrected "No rotation gesture" bullet: rotation IS
  emitted by GestureInputSource via CanvasScaleEvent.rotation; only
  MouseInputSource omits it (Flutter's ScaleGestureRecognizer limitation).
• CONTRIBUTING — Fixed wrong directory paths (lib/src/filters/ →
  lib/src/filter/, removed non-existent lib/src/calibration/); added
  development prerequisites table.
• SECURITY — Replaced GitHub default template with accurate policy for a
  0.2.x package (correct version table, advisory link, SLA, scope table).
• CLAUDE.md — Added architecture guide for AI-assisted development,
  covering the js_interop boundary invariant, quality gates, testing philosophy,
  and breaking-change rules.

Fixes

• pubspec.yaml version bumped from 0.2.0 to 0.2.1 (was incorrectly
  behind CHANGELOG.md); added homepage and issue_tracker fields.
• GestureInputSource (native) — dispose() now wraps _frameSub?.cancel()
  in unawaited(), consistent with the three StreamController.close() calls
  below it.

Example

• 3D Room — Camera now starts inside the room (dist 900 → 250) for an
  immersive interior view instead of a dollhouse perspective from outside.
• 3D Room — Furniture can now be rotated on its Y-axis: horizontal scroll
  while a piece is selected rotates it in-place; two-finger twist
  (CanvasScaleEvent.rotation) also rotates the selected piece instead of the
  camera.

Bug fixes

• Drag-to-scroll interference — CanvasScrollEvent could fire immediately after releasing a drag when the hand briefly adopted a pointing-like shape during the transition. Fixed by adding a scrollConfirmFrames gate (default 2, ~67 ms at 30 fps) that requires N consecutive pointing frames before scroll activates, consistent with the existing pinchConfirmFrames guard for drag. Set scrollConfirmFrames: 1 to restore the old single-frame-baseline behaviour.

New

• handLandmarkConnections — exported constant listing the 21 skeleton bone pairs for the MediaPipe hand topology. Iterate over it to draw a skeleton overlay without hand-coding the connections yourself.

• CanvasGestureEvent.confidence (double, default 1.0) — ML confidence of the gesture classification. Always 1.0 on web (rule-based heuristic); populated from the backend's score on native when using a model-based LandmarkProvider such as hand_detection.

• HandDetectionFrame.gestureConfidence / secondHandGestureConfidence (double, default 1.0) — per-gesture confidence fields for native backends to expose ML scores that flow through to CanvasGestureEvent.confidence.

• GestureInputSource confidence thresholds (web) — three new constructor params minHandDetectionConfidence, minHandPresenceConfidence, and minTrackingConfidence (each default 0.5, matching MediaPipe defaults). Tune these to trade sensitivity against false positives without modifying the JS worker.

• HandDetectionFrame.worldLandmarks / secondWorldLandmarks — optional world-space landmark lists (metric scale, hand-centre origin) for native backends to expose orientation-invariant landmark data. On web, populated from MediaPipe's worldLandmarks result automatically.

• GestureDebugInfo.worldLandmarks / secondWorldLandmarks — world landmarks forwarded through the debug snapshot so overlays and classifiers can consume them.

• HandDetectionFrame.boundingBox / secondHandBoundingBox (Rect?) — axis-aligned bounding box of each detected hand in normalised image coordinates. Native backends set this from their ML model's output. On web, computed automatically as the min/max envelope of the 21 image-space landmarks.

• GestureDebugInfo.boundingBox / secondHandBoundingBox (Rect?) — bounding box forwarded through the debug snapshot for overlay rendering.

• StylusInputSource — input source for Apple Pencil, Samsung S-Pen, and any PointerDeviceKind.stylus / PointerDeviceKind.invertedStylus device. Maps pen contact to element-drag canvas events (Down/Move/Up, tap, double-tap, hover, cancel). Filters out mouse/touch/trackpad events so it can be combined with MouseInputSource or TouchInputSource without duplicates. Exposes an eraserModeStream that emits true / false on eraser-vs-tip mode changes.

• CanvasDownEvent.pressure / CanvasMoveEvent.pressure (double, default 1.0) — hardware pen pressure (0.0–1.0) carried by stylus down and move events. Backwards-compatible: non-stylus sources emit 1.0; existing construction and pattern-matching compiles unchanged.

• defaultPointerSource() — factory function that returns the best-fit CanvasInputSource for the current platform: TouchInputSource on Android / iOS, MouseInputSource everywhere else (including web). Uses kIsWeb and defaultTargetPlatform from package:flutter/foundation.dart, so it is safe to call on web without conditional imports. GestureInputSource is not included — it requires platform-specific configuration that cannot be inferred automatically.

• TouchInputSource — mobile-native input source for Android and iOS. Single-finger drag → CanvasScrollEvent (direct-manipulation pan); two-finger pinch/spread → CanvasScaleEvent; fling → CanvasScrollEvent with non-zero velocity field for consumer momentum animations. Tap detection uses raw Listener events rather than GestureDetector so taps are never dropped by ScaleGestureRecognizer's kPanSlop movement threshold.

• CanvasScrollEvent.velocity (Offset, default Offset.zero) — fling velocity field added to CanvasScrollEvent. Non-zero only on fling events emitted by TouchInputSource. Backwards-compatible: existing code that constructs or pattern-matches CanvasScrollEvent compiles unchanged.

Documentation

• README — Documented three previously undocumented public API surfaces: CanvasInputController muting (muteWhenActive/activeStream), GestureInputSource.statusStream lifecycle states, and GestureInputSource.setFilterParams().
• README — Added self-hosting section documenting mediaPipeBaseUrl and modelAssetUrl constructor parameters (Flutter Web only); corrected the "no self-hosted model" limitation which was inaccurate — self-hosting has been wired since 0.1.0.
• KNOWN_LIMITATIONS — Corrected "No rotation gesture" bullet: rotation IS emitted by GestureInputSource via CanvasScaleEvent.rotation; only MouseInputSource omits it (Flutter's ScaleGestureRecognizer limitation).
• CONTRIBUTING — Fixed wrong directory paths (lib/src/filters/ → lib/src/filter/, removed non-existent lib/src/calibration/); added development prerequisites table.
• SECURITY — Replaced GitHub default template with accurate policy for a 0.2.x package (correct version table, advisory link, SLA, scope table).
• CLAUDE.md — Added architecture guide for AI-assisted development, covering the js_interop boundary invariant, quality gates, testing philosophy, and breaking-change rules.

Fixes

• pubspec.yaml version bumped from 0.2.0 to 0.2.1 (was incorrectly behind CHANGELOG.md); added homepage and issue_tracker fields.
• GestureInputSource (native) — dispose() now wraps _frameSub?.cancel() in unawaited(), consistent with the three StreamController.close() calls below it.

Example

• 3D Room — Camera now starts inside the room (dist 900 → 250) for an immersive interior view instead of a dollhouse perspective from outside.
• 3D Room — Furniture can now be rotated on its Y-axis: horizontal scroll while a piece is selected rotates it in-place; two-finger twist (CanvasScaleEvent.rotation) also rotates the selected piece instead of the camera.

New events

• CanvasDoubleTapEvent — emitted alongside the second CanvasTapEvent when two taps occur within doubleTapWindow (default 300 ms). MouseInputSource detects it via a DateTime gap between releases; HandGestureRecognizer via elapsed time between consecutive dwell taps.
• CanvasLongPressEvent — emitted after the pointer (or cursor) holds still for longPressDuration. MouseInputSource uses GestureDetector.onLongPressStart; HandGestureRecognizer uses a configurable longPressDuration threshold shared with the dwell timer (shorter threshold always wins).
• CanvasGestureEvent(gesture) — edge-triggered on each RecognizedGesture change (non-none) for both the primary and secondary hand. Fires once when a gesture starts; resets when the hand is lost so the same gesture can fire again on re-entry.

HandGestureRecognizer

• longPressDuration — new constructor param; Duration.zero = disabled (default).
• doubleTapWindow — new constructor param controlling the inter-tap interval that qualifies as a double tap (default 300 ms).
• longPressDurationS / doubleTapWindowS — public getters for the above.
• Pointing-finger scroll now emits both horizontal and vertical deltas (Offset(scrollDx, scrollDy) instead of Offset(0, scrollDy)).
• _checkDwell refactored to _checkDwellEvents returning List<PointerInputEvent> to support multiple simultaneous events (tap + double-tap).

MouseInputSource

• CanvasCancelEvent emitted from Listener.onPointerCancel when the OS interrupts an active drag (context menu, window switch, etc.).
• CanvasLongPressEvent via GestureDetector.onLongPressStart. If a drag was in progress, CanvasCancelEvent is emitted first to close it cleanly.
• Double-tap detected via a DateTime gap between successive tap releases.

GestureInputSource

• longPressDuration and doubleTapWindow forwarded to HandGestureRecognizer on both web and native variants.
• maxHands param (default 2) forwarded to the MediaPipe web worker's numHands option; set to 1 to improve performance in single-hand apps.
• CanvasGestureEvent emitted for secondary hand gestures in addition to the primary hand.
• _lastGesture resets to none when tracking is lost, so the same gesture fires again when the hand re-enters the frame.

Breaking changes

• OneEuroFilter removed from the public barrel export (air_pointer.dart). It was an internal smoothing detail; import directly from package:air_pointer/src/filter/one_euro_filter.dart if needed.

Initial release.

Events

• PointerInputEvent sealed hierarchy — CanvasTapEvent, CanvasDownEvent, CanvasMoveEvent, CanvasUpEvent, CanvasCancelEvent, CanvasHoverEvent, CanvasScrollEvent (with isTrackpad), CanvasScaleEvent (with rotation), CanvasScaleEndEvent, CanvasSwipeEvent (with SwipeDirection and velocity). All events use the Canvas prefix to avoid collisions with Flutter's own pointer events.

Core abstractions

• CanvasInputSource — abstract boundary; all input origins implement this contract.
• CanvasInputController — merges events from multiple CanvasInputSources into a single broadcast stream; folds buildSurface wrappers in order.

MouseInputSource

• Maps Flutter gesture-arena callbacks to canvas events: tap → CanvasTapEvent, one-finger drag → Down/Move/Up, two-finger pinch → CanvasScaleEvent, scroll wheel and native trackpad pinch (PointerScaleEvent) → CanvasScrollEvent / CanvasScaleEvent, mouse hover → CanvasHoverEvent.
• scrollMultiplier — scales scroll deltas before emission.
• tapSlop — configurable tap-slop threshold (default 10 px).
• isTrackpad on CanvasScrollEvent — true for macOS/iOS two-finger pan; consumers can skip ticker-based inertia since the OS already provides momentum.

GestureInputSource (Flutter Web)

MediaPipe HandLandmarker running in a dedicated web worker (off the main thread). Zero-copy ImageBitmap transfer. Camera permission / hardware / context errors all produce typed HandTrackingStatus states.

• HandGestureRecognizer — pure-Dart state machine; fully testable without a camera.
  • Acquisition gate: N consecutive frames (default 3) to confirm hand presence.
  • Hysteresis: separate close (default 0.05) and open (default 0.08) thresholds prevent chatter near the boundary.
  • Grace window: N frames (default 5) before declaring the hand lost; cursor freezes and dwell progress is preserved through brief occlusions.
  • Clutch / Midas-touch guard: pinch is blocked until the hand opens after confirmation, preventing accidental drags when the hand enters the frame already pinched.
  • CanvasCancelEvent (not CanvasUpEvent) when the hand exits during an active drag.
  • Two-hand spread → CanvasScaleEvent with rotation delta; CanvasScaleEndEvent when the second hand leaves.
  • Dwell-click: cursor must hold still within dwellRadius for dwellDuration to emit CanvasTapEvent. Progress is reported via GestureDebugInfo.dwellProgress (0–1). Dwell is preserved through the grace window so brief occlusions don't reset progress.
  • Pointing-finger scroll: index extended + middle curled → CanvasScrollEvent driven by vertical fingertip movement. Enabled via scrollEnabled: true; scaled by scrollScale.
  • Swipe gesture: fast directional movement (velocity > swipeThreshold px/s) in an open hand emits CanvasSwipeEvent. 60/40 dominance ratio prevents diagonal false-positives; 400 ms cooldown suppresses repeated firing from one gesture. Disabled by default (swipeThreshold: 0).
• OneEuroFilter — adaptive low-pass filter (Casiez et al., CHI 2012) for landmark coordinates. Exposes velocity for prediction and swipe detection.
• GestureCalibrator — accumulates open/closed pose samples from GestureDebugInfo.pinchDistance and computes per-user CalibrationResult.
• LandmarkProvider — platform interface for native landmark sources (e.g. TFLite on iOS/Android); web implementation uses MediaPipe via web worker.
• Debug support — GestureInputSource.debugInfo stream of GestureDebugInfo (phase, pinch distance, dwell progress, pointing flag, landmarks, worker latency, round-trip latency). buildCameraPreview returns a live camera widget.

Example

example/ ships two demos driven entirely through CanvasInputController:

• Sandbox canvas — draggable boxes with dot-grid background, inertia scrolling, two-finger pinch-to-zoom, dwell-click, pointing-finger scroll, debug overlay, camera preview, calibration dialog, and zoom badge.
• Netflix-style demo — scrollable content grid with hero section, card rows with horizontal scrolling, detail overlay, and full air-pointer interaction (pinch-drag with inertia, dwell-click with progress ring, pointing scroll, swipe navigation).