Daily Note: Accessible Multimodal 3D Printed Models with Conversational and Haptic Features

These notes are a summary of concepts presented in “3D Printed Maps and Icons for Inclusion: Testing in the Wild by People who are Blind or have Low Vision.”

Samuel Reinders, Swamy Ananthanarayan, Matthew Butler, and Kim Marriott. 2023. Designing Conversational Multimodal 3D Printed Models with People who are Blind. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (DIS ’23). Association for Computing Machinery, New York, NY, USA, 2172–2188. https://doi.org/10.1145/3563657.3595989

  1. Overview of the System
    • Combines conversational interface, haptic vibration, and touch for accessible interactions
    • Fully functional 3D model integrating touch interaction, haptic feedback, and audio responses
  2. Elements
    • Touch interaction
      • Button or touch-triggered responses with deliberate touch gestures (Single, Double, Long Press)
    • Haptic feedback
      • Vibration responses for confirmation and directional cues, customizable intensity and duration
    • Conversational interface
      • Supports speech input/output with customization options for voice, speed, and verbosity
    • Audio labels
      • Real-time text-to-speech or pre-recorded audio
      • Useful for blind users not familiar with Braille
      • Updateable information for adaptability
  3. Accessibility and Usability Enhancements
    • Natural multimodal interactions
      • Supports gestures, speech, and buttons
      • Users can switch interaction modes based on context or preference
      • Demonstrative pronouns like “this” and “that” for intuitive interaction
    • Customization and personalization
      • Adjustable touch sensitivity, gesture timing, and feedback intensity
      • Customizable voice types, speeds, and verbosity for auditory responses
    • Seamless tactile exploration
      • Easily identifiable touch points (protruding/recessed features)
      • Activate press gestures or on/off options to reduce unintended interactions
  4. Haptic and Auditory Augmentation
    • Localized feedback
      • Haptic vibrations tied to specific components for better spatial understanding
      • Vibrations used for locating objects or confirming touch gestures
    • Coupled audio-haptic output
      • Combined feedback enhances context during tactile exploration
      • Supports comparisons and multi-object interactions
  5. Conversational Interface Improvements
    • Natural dialogue support
      • Handles follow-up queries and demonstrative pronouns
      • Simplified command phrases (e.g., “Yes, search for it” simplified to “Yes”)
      • Offers search and additional information when needed
    • User control
      • Customizable verbosity and output length
      • Wake words and Long Press gestures for activation
      • Feedback mechanisms for secure, confident interactions
    • Comfort and privacy:
      • Avoids public conversational interactions; headphones as an option
      • Reference sources to enhance trustworthiness
  6. Design Recommendations
    • Uninterrupted tactile exploration
      • Physical design of touch points to ease discovery
      • Activation mechanisms (e.g., single press) to ensure deliberate interactions
    • Leverage familiar interactions
      • Build on smartphone and smart speaker interaction experiences
      • Incorporate similar gestures and commands for accessibility
    • Customization for confidence
      • Tailor input/output preferences for user comfort
      • Adjust feedback and interaction settings for inclusivity
    • Enhanced interaction support
      • Enable intuitive dialogue with pronouns and simplified commands
      • Align responses to conversational and touch interaction needs
    • Rich haptic integration
      • Tightly couple haptic feedback with auditory cues for intuitive understanding
      • Use haptic signals for spatial orientation and object identification