Daily Note: Motion-Based Augmentative and Alternative Communication (AAC) Systems

These notes are a summary of concepts presented in “Nonverbal Communication through Expressive Objects.”

Stephanie Valencia, Mark Steidl, Michael L. Rivera, Cynthia L. Bennett, Jeffrey P. Bigham, and Henny Admoni. 2023. Nonverbal Communication through Expressive Objects. Commun. ACM 67, 1 (January 2024), 123–131. https://doi.org/10.1145/3610939

Current Challenges in AAC Systems
- Limited customization to word selection and speech production levels
- Lack of support for augmentations to enhance nonverbal communication
- Asynchronous text-based communication challenges synchronous interactions
Nonverbal communication aids
- Regulates turn-taking in conversations
- Convey personality
- Increase social agency
Synchronous timing challenges in asynchronous AAC mediums
- Composing messages in text-to-speech
- Responding to synchronous, speech-based communication partners
Motion-Based AAC Systems
- Design process (five stages)
  - Discovery: Identifying opportunities for nonverbal expressive sidekicks
    - Definition: Discussing the functionality and future potential of sidekicks
      - Development: Ideating and prototyping
        
        Delivery: Pilot testing and refining prototypes
        
        Longitudinal Evaluation: Diary studies and real-world usage feedback
- Group-Based Communication Examples
  - Prepare discussion questions in advance for AAC users
  - Use one-word answers with professor expansion
  - Switch-activated LED light strips for communication aids
Key Design Considerations for Devices
- Functional, social, and aesthetic factors
  - Conversational goals supported
    - Control methods
      - Introduction to conversations
        
        Behavior when inactive
        
        Physical appearance
        
        Device presence
        
        Always accessible but ambient when inactive
        
        Avoid requiring user retrieval or reattachment
- Inputs and controls
  - Gesture-based inputs may cause fatigue
  - Infrared vs. Bluetooth remote control communication
Real-World Application and Feedback
- Contexts
  - In-person and virtual communication
  - Varying communication partners
- Observations
  - Effects of technology setups and channel variations
  - Long-term use inspired refinements and improvement ideas
Motion-Based AAC Through Expressive Objects
- Potential for new communication modalities
- Insights from physical sidekick support for augmented communication
- Open development platform for feature integration
- Long-term planning ensures sustainability and maintenance
Co-Design Approach
- Iterative process includes identifying “what doesn’t work”
- Valuable lessons from exploring non-feasible options
- Collaboration fosters innovation and avoids replicating ineffective designs