Daily Note: Opportunistic Control Interfaces and Augmented Reality

These notes are a summary of concepts presented in “Opportunistic Tangible User Interfaces for Augmented Reality.”

S. Henderson and S. Feiner, “Opportunistic Tangible User Interfaces for Augmented Reality,” in IEEE Transactions on Visualization and Computer Graphics, vol. 16, no. 1, pp. 4-16, Jan.-Feb. 2010, doi: 10.1109/TVCG.2009.91.

  1. Opportunistic Control Interfaces
    • Opportunistic Control
      • Tangible user interface leveraging naturally occurring, tactilely interesting, unused affordances in an environment
    • Affordances
      • Properties of objects that determine usability; serve as tactile landmarks providing passive haptic feedback
    • Core concept
      • Augmenting these tactile landmarks with overlaid 3D widgets for visual feedback
  2. Key Properties of Opportunistic Control Interfaces
    • Utilization of existing objects
      • Leveraging unused objects within the task domain as interface components
    • Passive haptics and gesture recognition
      • Exploiting object features for tactile feedback and hand gesture recognition
    • Minimized external artifacts
      • Reducing reliance on additional interface elements
  3. Haptic Feedback
    • Active haptics
      • Motor-driven force and torque as part of the interface
    • Passive haptics
      • Environmental elements naturally responding to user interaction
  4. Interaction and Design
    • Integration with task environment
      • Interact with AR applications using naturally occurring surfaces
    • Tangible representation
      • Existing environment features adapted for control; leaves no artifacts after use
    • Fallback mechanisms
      • Smooth transition to conventional passive haptics for feature-deficient environments
  5. Components of Opportunistic Controls
    • Physical region
      • Bounds the affordances serving as tactile landmarks
    • 3D widget
      • Virtual geometry mapped to the affordance
    • Gesture grammar
      • Recognized hand gestures linked to the opportunistic control
    • Functional mapping
      • Links gestures to widget functions
    • Output Proximity
      • Application outputs occur near the opportunistic control’s affordance region
  6. Types of Opportunistic Controls
    • Button-based
      • Tangible resemblance to buttons; may provide locational or elastic feedback
    • Valuator-based
      • Interact with linear/curved surfaces (e.g., pipes, cords)
    • Movable
      • Objects with physical movement (bend, slide, rotate)
  7. Design Guidelines
    • Avoid desensitizing or overloading objects with strong predefined functions
    • Prevent endangering users or damaging objects during interaction
    • Consider transparency of widget models to reveal underlying affordances
  8. Gesture Recognition and Parsing
    • Hardware
      • Single overhead camera with optical tracking (e.g., ARTag)
    • Algorithm
      • Segment and analyze bounding boxes of opportunistic controls
      • Identify hand/finger positions for gesture matching
      • Parse gestures using a finite-state machine linked to the 3D widget
    • Key distinctions between hovering versus selection
  9. Interaction Types
    • 3D object selection and manipulation
      • Widgets for selecting, rotating, and translating objects
    • Scene control
      • Manipulate camera positions/orientations in 3D scenes
    • 2D document visualization
      • Pan and zoom
    • Valuator widgets
      • Discrete (e.g., buttons) and continuous (e.g., sliders)
    • Menu selection
      • Choosing items from lists
  10. Heuristics for Opportunistic Control Implementation
    • Position affordances at eye level and within arm’s reach
    • Minimize physical exertion for interaction
    • Allow user-defined mapping of physical features to widgets
    • Enable system recognition of suitable features for opportunistic controls
  11. Further Investigation
    • Perception of unconventional affordances as opportunistic controls
    • Materiality and comfort of physical interaction
    • Support for two-handed techniques and field-of-view constraints