Daily Note: Situation-Specific Augmented and Alternative Communication (AAC) Device

These notes are a summary of concepts presented in “AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings.”

Fontana de Vargas, Mauricio & Dai, Jiamin & Moffatt, Karyn. (2022). AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings. 1-18. 10.1145/3517428.3544805.

  1. Application Characteristics
    • Automatically generates communication boards using words/phrases from photographs
    • Combines descriptive, narrative, and semantic methods for vocabulary generation
    • Aims to reduce workload, stimulate AAC usage, and facilitate symbolic understanding
  2. Traditional AAC Applications
    • Hierarchical symbol organization imposes cognitive and memory demands
    • Small, pre-programmed vocabularies limit scalability to unplanned situations
    • Requires significant effort from conversation partners for pre-programming
  3. Visual Scene Displays (VSDs) with JIT Programming
    • Associates language concepts with photographs or natural scenes
    • Enables “on-the-fly” programming during interactions
    • Increases teachable moments and symbolic communication turns
    • Still requires manual selection and programming of vocabularies
  4. Automated Vocabulary Generation
    • Uses contextual information like photos, speech, location, or sensor data
    • Methods explored include image captioning, OCR, and semantic expansion
    • Limitations
      • Limited to visible items in the photo
      • Biases in datasets (e.g., gender or language style)
  5. Application Users
    • Context-dependent communicators
      • Use symbolic communication but need more vocabulary and syntax tools
      • Goal: Increase literacy and interaction in diverse contexts
    • Emergent Communicators
      • Rely on gestures or body language, focused on current contexts
      • Goal: Establish symbolic expression through VSDs or static boards
  6. Features of the Prototype Application
    • Generates vocabulary spanning main parts of speech: pronouns, nouns, verbs, adjectives
    • Generation methods
      • Descriptive: Simple scene descriptions
      • Related: Semantically connected words (e.g., using SWOW model)
      • Narrative: Storytelling phrases linked to similar photo captions
      • Editable vocabulary: Add or correct terms, personalize interface settings
  7. Benefits and Outcomes
    • For AAC users
      • Expands expressive language
      • Supports independent use of AAC systems
    • For conversation partners
      • Reduces workload while enhancing language stimulation
      • Enables modeling language during meaningful interactions
  8. Application Customization and Personalization
    • Core vocabulary available across all pages for consistency
    • Adjustable layout: Symbol buttons, photo size, font size, and interface colors
    • Feature to add familiar people as pronoun symbols
  9. Challenges and Insights
    • Errors in vocabulary quality tied to photo content or misidentification
    • Users valued automation despite occasional biases and inaccuracies
    • Human-AI collaboration proved essential for effective AAC use
  10. Future Improvements
    • Enhance automatic identification for diverse real-world scenarios
    • Reduce biases in machine learning models
    • Explore more intuitive interfaces to increase symbolic understanding and accessibility