This survey paper explores the burgeoning field of Large Language Model (LLM)-powered Graphical User Interface (GUI) agents. It examines the evolution of GUI automation from rule-based systems to intelligent agents leveraging LLMs, computer vision, and natural language processing. The paper details the architecture and workflow of these agents, including components like memory and planning mechanisms. Furthermore, it analyzes various datasets used for training and optimizing these agents, different evaluation metrics and benchmarks used to assess their performance, and finally discusses the challenges and future directions of the field, such as safety, reliability, and ethical considerations.

https://arxiv.org/pdf/2411.18279