These notes are a summary of concepts presented in “AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings.”
Fontana de Vargas, Mauricio & Dai, Jiamin & Moffatt, Karyn. (2022). AAC with Automated Vocabulary from Photographs: Insights from School and Speech-Language Therapy Settings. 1-18. 10.1145/3517428.3544805.
- Application Characteristics
- Automatically generates communication boards using words/phrases from photographs
- Combines descriptive, narrative, and semantic methods for vocabulary generation
- Aims to reduce workload, stimulate AAC usage, and facilitate symbolic understanding
- Traditional AAC Applications
- Hierarchical symbol organization imposes cognitive and memory demands
- Small, pre-programmed vocabularies limit scalability to unplanned situations
- Requires significant effort from conversation partners for pre-programming
- Visual Scene Displays (VSDs) with JIT Programming
- Associates language concepts with photographs or natural scenes
- Enables “on-the-fly” programming during interactions
- Increases teachable moments and symbolic communication turns
- Still requires manual selection and programming of vocabularies
- Automated Vocabulary Generation
- Uses contextual information like photos, speech, location, or sensor data
- Methods explored include image captioning, OCR, and semantic expansion
- Limitations
- Limited to visible items in the photo
- Biases in datasets (e.g., gender or language style)
- Application Users
- Context-dependent communicators
- Use symbolic communication but need more vocabulary and syntax tools
- Goal: Increase literacy and interaction in diverse contexts
- Emergent Communicators
- Rely on gestures or body language, focused on current contexts
- Goal: Establish symbolic expression through VSDs or static boards
- Context-dependent communicators
- Features of the Prototype Application
- Generates vocabulary spanning main parts of speech: pronouns, nouns, verbs, adjectives
- Generation methods
- Descriptive: Simple scene descriptions
- Related: Semantically connected words (e.g., using SWOW model)
- Narrative: Storytelling phrases linked to similar photo captions
- Editable vocabulary: Add or correct terms, personalize interface settings
- Benefits and Outcomes
- For AAC users
- Expands expressive language
- Supports independent use of AAC systems
- For conversation partners
- Reduces workload while enhancing language stimulation
- Enables modeling language during meaningful interactions
- For AAC users
- Application Customization and Personalization
- Core vocabulary available across all pages for consistency
- Adjustable layout: Symbol buttons, photo size, font size, and interface colors
- Feature to add familiar people as pronoun symbols
- Challenges and Insights
- Errors in vocabulary quality tied to photo content or misidentification
- Users valued automation despite occasional biases and inaccuracies
- Human-AI collaboration proved essential for effective AAC use
- Future Improvements
- Enhance automatic identification for diverse real-world scenarios
- Reduce biases in machine learning models
- Explore more intuitive interfaces to increase symbolic understanding and accessibility