Introduction
The main problem plaguing traditional desktop automation has been one of persistent structural weakness: the fragility of pixel-based coordinates or inflexible API wrappers that fall apart the instant there is an update to the GUI of the application. The GUI agents for foundations often find themselves struggling because of prompt ambiguity, oddities of local setups, and procedural subtleties, which cannot be captured in plain language.
The combination of environment-based reinforcement learning with visual demonstration learning solves this problem. With the introduction of screen pixels as the sole point of truth in real-time, the agent gains the ability to interpret the state of the interface and adapt to the drift of UI and even unusual interface layouts. This is precisely what UI-Mate does by allowing users to record a simple video of the task once and transform the video into coordinate-free subgoals, while constantly analyzing screenshots and providing an open-weight foundation model for executing multi-application computer tasks without using fragile scripts on Windows, macOS, and Linux.
What is UI-Mate?
UI-Mate is a GUI agent framework which provides automation for long-term and multiple turn tasks in a native manner on any operating system. This project is maintained by the HY Frontier Team of Tencent. Being an agent which operates on raw pixel level, it converts language input and visual instructions into Platform agnostic mouse and keyboard actions on local software.
Model Variants
It comes in three different open-weight checkpoints:
- UI-Mate-9B: Developed from Qwen3.5-9B, used as a parameter-efficient GUI agent for generic computer manipulation.
- UI-Mate-27B: Developed from Qwen3.6-27B, used for long-horizon, application-specific software manipulation.
- UI-Mate-democua-27B: Fine-tuned using a special combination of generic computer usage data and demonstrations augmented trajectories. The model is capable of completing the task by following the demonstrations workflows while still maintaining the capability to follow instructions in the absence of any demonstrations.
Key Features of UI-Mate
- In-Context Demonstration Guided Execution (DemoCUA): Extracts a screen recording into a sequence of subtasks which are coordinate-free procedural checklists. It retains visual authority in real-time to dynamically omit repeated steps, add low-level actions (like scrolling and popups closure) and re-plan on the spot in case the screen deviates from the recorded one.
- Sharp Visual Reference and Cross-Platform Actuation: Processes the visual input in a normalized visual coordinate space of size 1000 × 1000 units, outputs tool calls for click actions, drag-and-drop, scroll, type text and key combination actions that can be resolved to physical monitor resolution.
- Long-Horizon Stability in Information: Retains very high stability of intent and coreference throughout a sequence of 100 interaction moves or more without any problems with context loss or drift of goal.
- Coordinate-Free Procedural Transfer: Evaluates spatial elements of the UI based on live screen view instead of copy-pasting of absolute coordinate values from recordings; hence, allows transferring of workflows across varying sizes and resolutions and visual themes of windows.
Use Cases of UI-Mate
- Zero-Shot Procedural Migration for Legacy Software Applications: Automates native, offline desktop applications (legacy ERPs, local database clients, or proprietary engineering tools) with inaccessible HTML DOM tree or API. Engineer records the workflow on an older OS, and UI-Mate locates the visual elements to execute the very same process in a new design layout of applications without any code refactoring.
- Visual Verification of Closed-Loop OS Sandbox Training: Facilitates RL agent training in the context of native OS sandboxing (Ubuntu, Windows, macOS). Data engine automatically performs randomization of desktop wallpapers, positioning of windows, and theme settings to avoid spatial shortcut learning. Step-by-step visual result verification through screenshotting validates milestones through Process Credit Model (PCM).
- Locally Hosted Automation of Processes in Secure Air-Gapped Environment: Distributes quantized FP8/INT4 models directly to locally hosted enterprise workstations either through vLLM or SGLang. Finance or government entities can automate multi-application processes such as querying of SQL GUI locally and processing of spreadsheets without going to cloud services.
- Environment Realism Benchmarking & Latency Auditing: Measures friction empirically on real-world desktop environment. Through testing, we found that processing instructions with noisy real-world spreadsheet data leads to 51.7% increase in trajectory length from 38.5 to 58.4 steps than when working with synthetic data. Profiling of latency shows that model prefill (1,597 ms) and decoding (314 ms) take up 91% of overall latency per step, setting benchmarks for hardware optimization of VLM serving.
- Screen-Observation-Only Security Control & Shell Defense: Secures enterprise network by limiting agents' interactions only to visual screen analysis and simulation of input actions, skipping direct execution of bash command-line or system shell API. It allows security specialists to constantly monitor trajectories recorded by the agent visually while human-in-the-loop control lets them to stop trajectory recording, analyze reasonings or even add their text instructions midway.
How does UI-Mate work?
UI-Mate has been engineered with an automated training data flywheel (see below img#1), an asynchronous reinforcement learning stack (illustrated in img#2), and runtime context injection. Its data engine is able to construct executable sandboxed executions on Ubuntu, Windows, and macOS, while randomizing file structures and environment visuals to evade synthetic bias. Coverage on the three-tiered capability tree (application domain, coarse capability, and fine-grained operation) is tracked to overcome training deficiencies, filtering trajectories through a multimodal judge and visual milestone check.
https://arxiv.org/pdf/2608.15930
Training is done using the Asynchronous Group-Relative Policy Optimization (GRPO) to reduce GPU downtime from OS rollout of variable duration. Rollout updates are made asynchronously as trajectory buffer becomes filled. To prevent policy staleness due to asynchronous rollout updates, IcePop discards isolated tokens having extreme ratio values, whereas SeqClip ensures that geometric means of token likelihoods do not drift. Decision-Turn Centering has been used to assign credits to actions to get rid of length biases due to failed trial-and-error approach, alongside with Token-Level Normalization which is meant to eliminate length bias in chain-of-thought output. There is an option to use Process Credit Model (PCM) for aligning active rollouts to milestone trees.
Source - https://arxiv.org/pdf/2608.15930
During inference, DemoCUA processes recordings to create subtask JSON files. These get injected as context blocks by the runtime engine. Intermediate low-level actions are intentionally excluded from training rollouts so that the model has to make those inferences visually using live screen shots. On macOS, the model uses an Accessibility (AX) API bridge for passing coordinates and resorts to PyAutoGUI click simulation if UI elements are not exposed directly.
Performance Evaluation with Other Models
In the core OSWorld-Verified testbed, the larger dense version of UI-Mate achieves a new open-weight state-of-the-art success rate average, whereas the parameter-efficient version performs exceptionally well. In this performance comparison, it has been established that UI-Mate performs way better than the dense foundation model, specialist open-weight agents, and large parameter scale Mixture-of-Experts architectures, and performs competitively against proprietary frontier flagships. The main implication of this finding is that dense and environment-informed reinforcement learning with pixel-wise actuation can do better computer control than parameter scaling and sparse mixture of experts routing.
source - https://ui-mate.github.io/
In the realistic OSWorkerBench dataset, the larger and denser model holds a prominent strict success rate and progress score relative to its smaller competitor. UI-Mate is designed to evaluate more complicated information flow and long-horizon problems involving many normalized applications running on the desktop, and UI-Mate performs significantly better than its own dense model and large general models in Multi-App and Long-Memory subsets. This result indicates the importance of late information and decision credit allocation of UI-Mate since keeping the visual state of applications over long time periods avoids goal degradation in complicated multi-application enterprise tasks.
source - https://ui-mate.github.io/
When considering Other evaluations, in which the larger dense model performs better than both the base model and large Mixture-of-Experts baselines and paired demonstration evaluation on the dedicated demonstration-guided checkpoint, the model demonstrates remarkable adaptability. Same task demonstration guidance helps to solve previously unsolvable tasks in public benchmarks and achieves higher strict success in office workflow subtasks. In special application scenarios such as Godot and QGIS, demonstration guidance helps to increase completion scores and reduce trajectory lengths. The importance of all of these benchmarks together is that the scaling of parameters of the model is the major factor for the complexity of multi-application sub-domains like Office, Daily, and Workflow tasks, while visual demonstrations help in eliminating exploratory paths.
How to Access and Use UI-Mate?
All model weights in Bfloat16 and FP8 quantized format are available on Hugging Face under the license. Local execution scripts, service recipes and a Mac OS client for Apple Silicon computers are provided through GitHub and project website. This model works locally via vLLM or SGLang using standard OpenAI endpoints and parameters.
Limitations
Technical limitations include the current context window format which places the workflow injection at the beginning of the prompt, thus any update to the subtasks invalidates the shared KV-cache. Future work plans to place the workflow blocks at the end of the context in order to be able to utilize an append-only KV-cache. Furthermore, a limitation lies in the gap between specialized professional software compared to flagship products (75.5% vs 81.6% for Kimi-K2.6), as well as in late stage dropouts in multi-step chains. Improving on policy entropy collapse in RL training when using reasoning tracebacks and generalization of non-identical variant tasks are still works in progress.
Conclusion
UI-Mate represents the departure point for more flexible solutions than brittle coordinate-based scripting or prompt engineering, showing that the native control of computers visually is most effectively done via pixel-based reinforcement learning. By divorcing instruction of tasks from fixed positions on the screen, it gives open weight foundation models the flexibility necessary to perform complex multi-application workflows.
Sources:
https://ui-mate.github.io/
https://huggingface.co/papers/2608.15930
https://arxiv.org/pdf/2608.15930
https://huggingface.co/collections/tencent/ui-mate
https://github.com/Tencent/UI-Mate
https://huggingface.co/tencent/UI-Mate-27B
https://huggingface.co/tencent/UI-Mate-9B
https://huggingface.co/tencent/UI-Mate-democua-27B
Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.
No comments:
Post a Comment