Recently, OpenAI announced a research preview of Operator. OpenAI describes it as an agent that can access the web to perform tasks for the user, including using its own browser to interact with web pages. A more interesting question revolves around which interface elements or data structures are necessary for an operator to use. The term “operator” extends to encompass an intelligent system that completes tasks on the user’s behalf (OpenAI refers to this as a Computer-Using Agent). Researchers from Apple and additional affiliates have also conducted research on this topic with Ferrit-UI 2, focusing on the development of a generalist model for user interface understanding across platforms.
An environment without interfaces seems improbable, but if communication occurs between two operators to complete a task, what remains of the interface? Task alerts, ranging from status updates to completion, may be one interface element. Task description and tagging systems seem like additional interface elements. Images or other visual elements needed for user confirmation are additional elements. With these elements in mind, it is unclear whether contemporary interface design patterns are appropriate.