Daily Note: Interactive Tangible Objects and Capacitive Touchscreens

These notes are a summary of concepts presented in “Itsy-Bits: Fabrication and Recognition of 3D-Printed Tangibles with Small Footprints on Capacitive Touchscreens.”

Martin Schmitz, Florian Müller, Max Mühlhäuser, Jan Riemann, and Huy Viet Viet Le. 2021. Itsy-Bits: Fabrication and Recognition of 3D-Printed Tangibles with Small Footprints on Capacitive Touchscreens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 419, 1–12. https://doi.org/10.1145/3411764.3445502

  1. Tangible Interaction
    • Interactive tangible objects enable haptic control of on-screen content
    • Identification of objects includes their location and orientation
  2. Interaction Techniques
    • Touch and deformation
    • Use of physical controls and construction
  3. Challenges in Tangible Design
    • Small tangibles often require costly hardware for active sensing
    • Cheaper tangibles tend to be bulky due to spatially separated touchpoint patterns
  4. Machine Learning for Tangible Recognition
    • Classifies up to 30 conductive shapes using a capacitive data set
    • Process involves mapping raw capacitive data to ground-truth postures
  5. Approaches to Embedding Interactive Capabilities
    • Attaching sensors like capacitive or acoustic sensors, cameras, or accelerometers
    • Emerging methods include
      • Embedding light pipes, sound pipes, or media-filled structures in 3D-printed objects
      • Using conductive spray or plastic to create interactive features
  6. Localization of Tangibles
    • Techniques include
      • Optical markers
      • Magnetic sensor grids
      • Localized NFC
  7. Capacitive Touchscreen Interactions
    • Detection via spatial touchpoint patterns and passive-resistive components
    • Unique touchpoint patterns using conductive and insulating materials
    • Conductive shapes create capacitive images detectable by the touchscreen
  8. Machine Learning Workflow
    • Detect blobs of lit pixels in capacitive images
    • Classify shapes, sizes, and orientations using a machine learning model
    • Track and identify tangibles based on capacitive signals
  9. 3D-Printed Composite Structures
    • Components
      • Grip structure: forms a path to the touchscreen
      • Shape structure: interacts with the screen
      • Wiring structure: connects grip and shape structures
    • All structures made from conductive polymer with customizable sizes and shapes
  10. Designing Distinct Shapes
    • Criteria for distinctiveness
      • Rotation-variant modifications to shapes
      • Maximizing differences at low resolutions
    • Examples of shapes: arrow, circle, heart, hexagon, star, etc
    • Geometric features vary in curvature, corners, and edges to ensure classification accuracy
  11. Applications of Tangibles
    • Enable highly individualized experiences like custom game characters (e.g., 3D-scanned player miniatures)
    • Expand tangible user interactions beyond mass production
  12. Heuristics
    • Shapes must remain distinct at varying resolutions
    • Low-resolution challenges: overlapping features of shapes (e.g., circle vs. hexagon)
    • Shape size impacts classification accuracy