beta
/Autonomous Graphical User Interface Help Desk Automation With Verified State Transitions And Provenance Logging
Abstract

A technique automates help desk task completion for applications when source code is unavailable. Input data representing a computer issue is received at a server platform. An artificial intelligence system autonomously explores a graphical user interface by simulating user actions and capturing screenshots to derive task parameters. A workflow representation is constructed that includes operating-system-level input events and associated state transitions. The workflow representation is executed by generating the operating-system-level input events and verifying state transitions through a perception layer that captures pre-execution and post-execution states, including screenshots or accessibility tree snapshots. When a failure in a state transition is detected, the workflow representation is adapted to include alternative operating-system-level input events. Executed input events and corresponding verified state transitions are logged in a provenance module with audit metadata, and in some embodiments cryptographic metadata including timestamps, unique identifiers, and hashes of pre-execution and post-execution states.

Full Text

What is claimed is:

A technique automates help desk task completion for applications when source code is unavailable. Input data representing a computer issue is received at a server platform. An artificial intelligence system autonomously explores a graphical user interface by simulating user actions and capturing screenshots to derive task parameters. A workflow representation is constructed that includes operating-system-level input events and associated state transitions. The workflow representation is executed by generating the operating-system-level input events and verifying state transitions through a perception layer that captures pre-execution and post-execution states, including screenshots or accessibility tree snapshots. When a failure in a state transition is detected, the workflow representation is adapted to include alternative operating-system-level input events. Executed input events and corresponding verified state transitions are logged in a provenance module with audit metadata, and in some embodiments cryptographic metadata including timestamps, unique identifiers, and hashes of pre-execution and post-execution states.
Timeline
Filed
03/12/2026
Published
07/16/2026
Granted
Not Available
IPC Codes(1)
G06F 11/3604:Analysis of software for verifying properties of programs (testing of software G06F 11/3668)