Why MiMo-V2.6 Pro Defeats Claude Fable 5 In DeepSWE v1.1

Introduction

Open-weights artificial intelligence is now at a crossroads where getting more intelligence for money determines the feasibility of production. The development of native omnimodal architectures together with scalable reinforcement learning is a recipe for establishing a transparent environment for model improvement. Making a foundation model work as an engine of both creation and analysis makes it possible for modern systems to merge textual reasoning with spatial, musical, visual, and code execution. What is essential about such models is that using groupwise agentic grading gives a way to get sophisticated reward signals, which help models to tune their complicated tool-using strategies without getting stuck in binary reward schemes. Alongside with using special optimization algorithms to guarantee training stability, open-weights models deliver frontier execution for much lower costs than those of proprietary APIs.

MiMo-V2.6 is one of the milestones in this direction – it occupies the top of world leaderboards among open-source projects and also has affordable API prices. This technology presents a single unified baseline that combines post-training reinforcement learning, open environment verifiers, and native multimodal perception.

What is MiMo-V2.6?

MiMo-V2.6 refers to an omni-native sparse mixture-of-experts (MoE) foundation model family that is purposefully designed to advance the frontier of intelligence based on inference and training costs. The model can process text, high-res images, videos, and raw audio inside a single context window of 1M tokens. MiMo-V2.6 uses reinforcement learning with large compute on multiple domains in agent-based environments to perform end-to-end tasks.

Model Variants

• MiMo-V2.6-Pro / Pro-RL: The premier MoE model with 1.02T parameter count in total with 42B parameters per token with routing to 384 experts (8 experts per token). Structured with 70 layers (one dense layer initially and 69 MoE blocks) and hidden dimension of 6,144, it has been designed for deep reasoning and long-horizon software engineering. It can be downloaded in safetensors form from a 524GB file after spending $2.62M on post-training RL compute.
• MiMo-V2.6-Flash / Flash-RL: The efficiency-balanced MoE variant featuring 309B/310B total parameters with 15 Billion active parameters per token across 256 routed experts (8 active per token). Built with 48 Transformer layers (1 dense + 47 MoE blocks) and a hidden dimension of 4,096, it delivers near-flagship agentic performance at a significantly reduced compute footprint ($0.85M RL post-training cost) and is available as a 159GB safetensors file.
• MiMo-V2.6-Distill-Qwen-9B: A compact 9B parameter dense image-text-to-text model fine-tuned from Qwen3.5-9B via Supervised Fine-Tuning (SFT) on 77.4 Billion synthetic tokens (27.2B loss-bearing tokens) generated directly by MiMo-V2.6. Balanced across coding , general agents, visual coding, and cybersecurity, it brings high-efficiency agentic capabilities to resource-constrained edge deployments.

Use Cases of MiMo-V2.6

• Making Animated 3D Scenes & Working with Robots: It can convert any type of written text, photo, or video into a 3D object, something functional in Blender. This is achieved by simplifying complex modeling and programming processes associated with robotics and gives one a real-time feedback on things happening to robots in 3D.
• Music Composition, UI Design & Multimedia Editing: It performs music composition and produces MIDI files people can work with on DAWs, produces UI design screens via Figma, and voice recordings. All of this make the creative work easier and gives independent creators a chance to compete with major companies by taking control of their audio-visual content.
• Materials development and theoretical proofs validation: Reads complex patents and literature and configures its computer programs in order to search applications of materials in the green industry while acting with active agents and generating codes for formal proofs of mathematical theorems—this enables minimizing costs of laboratory testing and patent-approving processes for scientific researchers and obtaining a pulpit for formal proofs generator.
• High Throughput Production Hosting Using Block Diffusion Speculation: Makes use of the block diffusion speculative decode along with the optimization of the inference engine in order to considerably increase the speed of the production of the output without compromising on accuracy.
• Agentic Backbone Distillation for Scalable Enterprise Swarm Deployment: Derives the lightweight agentic backbone from multimodal agentic trajectories in software engineering, general agents, visual design, and cybersecurity and enhances it with domain-specific reinforcement learning, enabling enterprises to deploy very accurate autonomous agent swarms at extremely low cost.

How does MiMo-V2.6 Work?

MiMo-V2.6 uses a hybrid sparse MoE backbone, unique omnimodal encoders, and a multi-stage training pipeline specifically for reinforcement learning scale that does not destabilize model representations. The backbone consists of alternating Local Sliding Window Attention (SWA) with Global Attention (GA). To ensure stable early representation learning, the first block of Transformers is designed with global attention using a dense Feed-Forward Network (FFN), while all other blocks use sparse MoE FFN without sharing any experts. Local SWA shrinks Key-Value (KV) cache memory overhead by 6× to 7×, while attention sink biases are learnable to ensure long-context coherence up to 1M tokens.

source- https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_technical_report.pdf

Visual and audio signals are analyzed using specialized encoding streams. Vision utilizes MiMo-ViT, which is a 681M parameters' Vision Transformer with alternating row-major and column-major SWA tokenization along with spatial 2x2 merge. Audio analysis consists of two stages: AudioTokenizer (308M parameters), which includes 20 RVQ codebooks at 25Hz, and Audio Patch Encoder (127M parameters), which includes grouping every four frames to decrease token frequency to 6.25Hz. Mid-training context length has been increased from 32K → 256K → 1M. For weight optimization, AdamW method is being replaced by Muown (variant of Muon method with row-norm optimization) to ensure high data efficiency and avoid spectral norm drift and loss spikes during large batch training.

source- https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_technical_report.pdf

The post-training procedure utilizes 'You Only RL Once' methodology by employing an asynchronous Group Relative Policy Optimization (GRPO) process on a batch of various tasks (coding, general agents, visual design, cybersecurity, context following) of size G=16 (25,000 rollouts per step). It is important to note that MoE routers remain frozen during the process of RL training. This ensures that there is no issue with router drift or expert-load collapse, which used to increase coefficient of variation from 0.78 to 2.0 and maximum expert load from 6x to 16x. The efficiency of post-RL training is improved via Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD^2) using autonomous student rollouts along with prefix-conditioned single-turn rollouts. The Sample Mixer architecture manages extreme execution variances (up to 90× token length and 66× duration variance) through Adaptive Scheduling and Predictive Dispatch.

Performance Evaluation with Other Models

On the Artificial Analysis Intelligence Index v4.3, the performance score of MiMo-V2.6-Pro comes out to be 46.32, marking it as the best-performing open-source foundation model in the world. This model beats open-weights contenders like Kimi K3 and Qwen3.8 Max, and keeps up well with the top-performing proprietary frontier models like Claude Opus 5 and GPT-5.6 Sol. The main importance of the performance is in the intelligence-to-cost ratio because the model performs at the top level and offers the same cheap API cost as V2.5 generation.

source - https://mimo.xiaomi.com/mimo-v2-6

On the DeepSWE v1.1 benchmark for long-horizon software engineering agents shown in Table below, MiMo-V2.6-Pro scores 71.9, whereas MiMo-V2.6-Flash scores 67.9. This shows a massive rise of 52.9 points from the previous generation MiMo-V2.5-Pro. In this case, MiMo-V2.6-Pro beats Claude Fable 5 and stands on par with top-performing proprietary models like Claude Opus 5, GPT-5.6 Sol, and DeepSeek V4.1 Flash. The main importance of the performance is that scaling agentic reinforcement learning compute directly results in better repository-level debugging and autonomous multi-file editing.

source- https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_technical_report.pdf

In terms of general agential workflow processes, MiMo-V2.6 excels in SaaS API orchestration through AutomationBench v1.0.6 and performs best in command line problem-solving through Terminal Bench 2.1, Model Context Protocol integration on Toolathlon-Verified, and economic wealth generation through GDPVal 2.1 among the open-weights benchmarks. In specific task fields, the model family achieves outstanding results in cybersecurity vulnerability replication through CyberGym and front-end visual coding through MiMo VisualCoding. Although there is a notable lag in performance in the field of competitive programming through ProgramBench and offensive exploit creation through ExploitGym, MiMo-V2.6 demonstrates the effectiveness of multi-task reinforcement learning in transferring tool-use skills into different agent environments.

How to access and use MiMo-V2.6?

Model weights, training logs and codebase of MiMo-V2.6 are completely open-sourced with MIT license. Developers and researchers can have access to the model checkpoints through the Hugging Face and ModelScope repositories . Models can be served using popular inference engines like vLLM or SGLang in an offline environment. For local run, hardware specifications depend on variant. Live telemetry, technical documentations and web demos can be found at the official website from Xiaomi.     

Limitations

The technical report documents several real-world infrastructure failure modes encountered during 1,000+ GPU RL training runs. These include hardware-level GPU memory double-bit errors (DBE), grader network unreachability, partial-rollout memory pool exhaustion, and expert-parallel activation Out-Of-Memory (OOM) spikes caused by up to 30x load imbalances across ranks prior to freezing routers. Host CPU OOM bottlenecks during trajectory packing also presented challenges.

Potential Architectural Advancements & Future Directions

In the pursuit of further advancing on the concept of omnimodal reinforcement learning, could future developments in this model include an entropy-aware, adaptively-controlled process for unfreezing routers post-freezing in the learning process? Through the combination of the localization of gradient values and dynamic expert load balancing, it is possible to incorporate this strategy, which would allow for the late adaptation of routers without the problems associated with experts load collapsing and activation memory spiking in large rollout clusters.

Also, in order to tackle host memory constraints in the trajectory packing process, could the adoption of a zero-copy, GPU-based memory pool in the sample dispatching pipeline be achieved, thus enabling continuous streamlining of multimodal rollouts? Lastly, could it be possible to unify visual and audio encoding processes in one cross-modal latent space?

Conclusion

From MiMo-V2.6, one can see that increasing reinforcement learning compute post-training across different verifier backed environments helps to get model capability improvements faster compared to increasing data pre-training. MiMo-V2.6 shows that it is capable of achieving frontier-level execution of tasks in software engineering, 3D creations, and tool orchestration. Xiaomi is an auditable foundation for autonomous agents.

Sources
https://mimo.xiaomi.com/mimo-v2-6
https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_technical_report.pdf
https://huggingface.co/collections/XiaomiMiMo/mimo-v26
https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL
https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL
https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.

Jev: System One Decision Engine Delivering 193.6x Faster Execution

 

Introduction

In modern software engineering, there is a need to reconcile two very different paradigms – the deterministic predictability of compiled programs and the probabilistic nature of machine intelligence. Traditionally, implementing machine learning into a product entailed compromising heavily – wrapping the nondeterministic text streams into validation routines, dealing with the schema hallucinations, and suffering significant latency penalties. Reliable and deterministic output capable of being used by software directly thus became one of the key architectural challenges. In addition, it is necessary to have an architecture providing reliable software integration and full control of the logic, which means being able to avoid parsing errors at runtime and not break the program flow due to an invalid payload.

Implementing the decision-making models into the enterprise-scale infrastructure requires an architecture providing massive scalability in the production setting, while also becoming much faster and cheaper than multi-billion parameter autoregressive models traditionally used as baselines. Jev, an implementation of a special class of System One models by TypeSafe AI, solves this systemic issue by exposing the structured probabilistic judgment to the application code. It achieves this without relying on token-by-token generation and the need to evaluate the state in just one pass.

What is Jev?

Jev is an advanced, non-autoregressive System One decision engine developed by TypeSafe AI, the company co-founded by the former OpenAI employee Diogo Almeida. The engine was specifically designed for machine-to-machine reasoning, not for human-machine communication, and is capable of taking application state in unstructured form, including plain text, execution traces, or even JSON objects along with typed queries.

Key Features of Jev

• Non-Autoregressive Forward-Pass Evaluation: Evaluates input state and structured queries in a non-autoregressive forward pass, without any token-by-token autoregressive generation to avoid latency and execution overhead.
• Parallel Hardware-Aware Sampler: Scores multiple independent questions in parallel to a single shared context window without any linear accumulation of latency.
• Absolute 100% Type Safety: Avoids string generation and attains a zero type error rate along with eliminating the need for any schema validation code, regex parsing, and run-time retries.
• Three AI Building Blocks: Intelligence is expressed in terms of only three structured query types: Choice (picking one option from a defined list of 255 options), Score (rating state on a particular numerical or ordinal rubric) and Noul (testing boolean true/false condition with a 0 to 1 probability distribution).
• Probability Output Trained Through RLCD: Trained using RLCD instead of RLHF-based preference alignment that ensures statistically calibrated probability outputs.
• Unilateral Metering Scheme: Charged in terms of input tokens rate of $0.042 per million tokens ($42/B tokens) and with output tokens being offered absolutely free of cost ($0.00).

Use Cases of Jev

• Speculative Fan-Out State Assessment: Running multiple-variable policy assessment, routing policies, and compliance policies all at once on a single payload, so the software can use all of the decision outcomes in a single atomic operation.
• Low-Latency Command Safety & Security Firewall: Acting as a synchronous, pre-execution guard that filters raw API calls and shell commands for prompt injection attacks or policy breaches before executing them downstream.
• Reflexive Physical Control in Real-Time in Physical and Simulated Worlds: Acting as a real-time decision engine for simulated worlds, game engines, and robots who need to perform reflexive actions in response to observational data.
• Verification Layer for Generative LLMs: Acting as an additional validation layer placed after traditional conversational models that check if the responses are factually correct, comply with policies and have a proper form before sending responses to the end-user.
• Confidence-Controlled Choice of Agentic Tools: Being the decision brain of agentic tools, which will allow you to make a choice using the Choice primitive and immediately escalate a low confidence decision to the fall back procedure.
• High Throughput Routing & Scoring of Tickets and Emails: Sorting, scoring the level of urgency, and routing a high volume of tickets and emails directly into the database queue with no human involvement.

How Does Jev Work?

Structurally at its core, Jev changes the interaction between foundation models and application code, substituting probabilistic evaluation for the token generation process. In order to ask a question, application passes an unstructured input in the form of a JSON object, an execution log or any other text, together with a series of questions, described in Choice, Score, or Noul primitives in the form of an array of atomic questions. Instead of passing this information to autoregressive decoders generating text letter by letter, Jev transfers the state and the list of questions to the hardware-aware parallel sampler.

source - https://docs.typesafe.ai/introduction

The way the model is executed is closely associated with its post-training alignment technique called Reinforcement Learning for Calibrated Decisions (RLCD). As opposed to standard language models trained via Reinforcement Learning from Human Feedback (RLHF), which are optimized for maximizing human preference and hence suffer from mode dropping, overconfidence and deceptive certainty around decision boundaries, RLCD aims at optimizing the probabilities of the output of the model directly against its empirical classification outcomes. Hence, Jev produces mathematically calibrated confidence scores along with every typed value.

Potential Architectural Enhancements

Could it be possible to compile Jev’s non-autoregressive, parallel sampling engine on edge-level neural processing units (NPUs) or on specialized FPGA acceleration boards? Moving this engine from the hosted environment of cloud nodes to hardware-based environments can open up the opportunity to have less than 10 milliseconds reflex loops for embedded robots, navigation fleets, and industrial IoT. Moreover, adding state differencing along with the activation cache will enable the monitoring system to analyze live data streams without re-tokenizing static backgrounds.

On the other hand, concerning post-training and research side, extension of the RLCD framework by cross-modal joint embeddings will enable direct probability-based evaluation of video streams, audio data, spatial sensors without any text transduction at all. To overcome one-pass cardinality limitations, it is possible to implement dynamic and speculative tree cascading right into the forward pass. Such modification will enable hierarchical routing of decisions across thousands of available options in one-pass manner and still guarantee type safety.

Performance Evaluation with Other Models

For measuring the efficiency of decision-making in real-world scenarios, TypeSafe AI performed the performance evaluation of other models with the help of structured compute graphs of the real-life production business logic and not through any static benchmark datasets. The baseline reference distributions have been derived based on the average results of the leading frontier models such as GPT-6 Astra and Claude Fable 5.1. In order to perform comparative tests, all the standard LLMs were evaluated using TypeSafe AI’s System One, which forced them to output only structured decisions and probabilities.

source - https://typesafe.ai/blog/introducing-system-one-models-and-jev

Jev showed an extremely radical difference when tested with the speed and latency benchmarks. Where the standard frontier LLMs needed 3 to 329 seconds to execute structured decision workflows, Jev completed the task within 70ms to 500ms. On representative production workflow benchmarks, the time required to perform decision cycles was 0.114 seconds for Jev against 8.566 seconds for the standard LLMs, which gave a 193.6x improvement. Moreover, per-workflow execution costs were reduced from $0.013880 for standard frontier models to $0.000081 for Jev, giving a cost reduction of 444.6x.

The synopsis of the benchmarks highlights a paradigm shift in production economics and reliability of systems. The fact that Jev achieves the feat of having a type error rate of 0% means it removes validation delay, regex parsing, and retries that are a constant feature of conventional LLMs. Input pricing comes down to $0.042 million tokens, which is 238x cheaper than the Claude Fable 5.1. Output tokens come free of charge, making Jev bring back the Pareto frontier for workflow intelligence to cost ratio.

How to Access and Use Jev?

The proprietary and hosted Jev API is available as early access on the TypeSafe AI platform through an onboarding waitlist. The model cannot be executed locally as there are no open-weights or self-hosted versions of the model available yet. The integration of Jev into production-ready systems can be done using the official Client SDKs that have been provided in Python and JavaScript/TypeScript or directly via REST API.

Limitations and Future Work

Despite the efficiency that Jev is able to offer in terms of speed and reduced costs, the unique architecture used in Jev has limitations of function as well. First, it does not support open-ended string creation at all making it unusable in conversational chatbots, copilots and generative code creation. Built solely for single pass System 1 classification, it cannot perform multi-step inference or analysis of multi-faceted tradeoffs in one prompt and requires developers to break down problems into smaller atomic questions within application logic. The Choice primitive supports a maximum cardinality of 255 choices – thus, requiring a slower process of scoring in two stages when dealing with greater number of choices – as well as limited input states which include only text and data.

In order to overcome these limitations, the technical road map of TypeSafe AI includes development in areas of expanding input types to cover visual and multimodal data, System One model pipeline extension, developer workshops and hackathons and onboarding from the waitlist. Future work will focus on showing computational and economic viability of its parallel sampling architecture and RLCD training.

Conclusion

Jev demonstrates that specialization of foundation models for non-autoregressive decision execution provides outstanding benefits. It even demonstrates that separation of probabilistic reasoning and text generation allows for creating a fast, affordable, and type-safe primitive that becomes a part of the deterministic codebase. For infrastructure professionals looking for the ways to automate the logic without significant latency and validation costs, Jev sets a benchmark for the future of machine-native AI infrastructure.

Sources
https://typesafe.ai/blog/introducing-system-one-models-and-jev
https://typesafe.ai/
https://docs.typesafe.ai/introduction

Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.

Nex-N2.5: Open-Weight AI That Controls Web Browsers and Desktop OS

Introduction

There is currently an enormous paradigm shift taking place in modern sparse open-weight architectures in how they process multimodal streams. Rather than treating graphical streams as passive data input or using text-based command chains only, today’s architectures such as Nex-N2.5 take advantage of optical perception streams for ongoing runtime error correction. This architectural advance provides a way of overcoming the frailties of traditional automation pipeline design, which cannot cope with minor interface rendering errors or unexpected terminal outputs during multi-turn processes.

It is important to strike a balance between selective parameter passing and efficiency when running autonomous extended processes in the enterprise environment. Being able to adapt the computing infrastructure for advanced computation to work for workflow is key to achieving efficient execution in software engineering, web navigation, and interface control. It is through making such hardware optimization work in combination with closed-loop image validation to provide self-rectification in an autonomous manner that modern architectures such as Nex-N2.5 achieve desktop agency and multi-step reasoning without dense latency cost.

What is Nex-N2.5?

Nex-N2.5 is a family of Mixture-of-Experts (MoE) models developed by Nex-AGI, explicitly purpose-built for visually grounded agency, native desktop and web GUI navigation, autonomous software development, and closed-loop execution. Transitioning from traditional text-heavy agent frameworks, the model utilizes continuous visual perception as a real-time verification interface to execute code, manage operating systems, navigate complex web applications, and self-correct system actions based on observed environmental states.

Model Variants

• Nex-N2.5-mini: Runs on 35 Billion total parameters (MoE) with 3 Billion active parameters per token (A3B). Built on top of the Qwen3.5-35B-A3B-Base multimodal foundation, this model is designed to achieve fast instruction-following, low-latency API serving, and on-the-fly tool execution. Efficiently running on a single node featuring 2×H100 GPUs with Tensor Parallelism (TP=2).
• Nex-N2.5-Pro: Equipped with 397 Billion total parameters (MoE) with 17 Billion active parameters per token (A17B) and built on top of the Qwen3.5-397B-A17B multimodal base, this is the main workhorse model for complex reasoning, multi-agent orchestration, and all-in-one developer stack. Designed for efficient running on a single node with 8×H100 GPUs (TP=8) using the dedicated NexRT inference engine.
• Nex-N2.5-Max: This is the flagship variant equipped with 1.6 Trillion total parameters (MoE) with 49 Billion active parameters per token (A49B). In contrast to other variants, this one uses the DeepSeek-V4-Pro-Base text-only foundation model for deep reasoning, scientific research modeling, and high-level architectural code generation. Efficiently running on multi-node clusters.

Key Features of Nex-N2.5

• Visually Grounded Agency & Closed Loop Self Correction: Controls web browsers and desktop operating systems through a visually grounded feedback loop. This model performs visual action execution, visual UI update evaluation, detects rendering bugs/page anomalies, and dynamically re-plans execution in real time, without any human involvement.
• Native GUI Navigation & Normalized Grounding: Can perform exact cursor-mouse actions in both desktop and web applications. For achieving spatial precision in different resolutions (OSWorld, WebArena, and WebTest benchmarks), normalized grounding is used in a standardized 0–1000 spatial grid.
• Coherent Logic & Multi-Step State Traversal: Maintains coherent reasoning capabilities across the entire process of task decomposition, strategic changes, visual state traversal, and self assessment within complex context switches.
• Trillion Parameter Scale Post Training Pipeline: Represents the first time Nex-AGI is attempting a post training process at a 1.6 trillion parameter scale. The pipeline receives live terminal output, web DOM tree, and screenshot streams directly into the training loop.
• Heterogeneous Base Foundation Models: Uses a combination approach by utilizing multimodal Qwen3.5 models with visual spatial interactions (Mini and Pro) as well as a 1.6T text only DeepSeek-V4-Pro base (Max) for logical synthesis.

Use Cases of Nex-N2.5

• Dual Model Autonomous Code Refactoring & Visual Playtesting: Facilitates automation of CI/CD pipelines where the state-of-the-art reasoning model rewrites complex codebase, and a multimodal model serves as the workhorse that builds up the software, opens the GUI, validates the UI rendering against design specifications and sends the screenshot differences to the code refactoring model to perform corrections before merging.
• Closed Loop Visual GUI Navigation with Autonomous Corrections: Navigates through browser and desktop applications checking the visual state after each click, and autonomously changes the path execution if unexpected popups appear or sites are broken.
• Multi-Backbone Models Inference under a Single Gateway: Simplifies enterprise inference management through serving of lightweight, medium, and ultra-heavy models under one standardized API gateway with custom server-side reasoning and tool-parsing routers.
• Zero-Script Legacy ERP and Desktop Workflow Automation: Automates multi-step workflows within enterprise applications including legacy ERP and desktop applications as well as mainframes emulators, using screen vision and normal coordinates control, without the use of unreliable APIs and RPA scripts.
• Engineering Task Criticality-Based SLA-Allocation of Compute Resources: Minimizes company compute cost by allocating developer traffic depending on the criticality of their tasks, sending code completion to high performance 2× H100 machines, feature requests to mid-performance 8× H100 machines and architecture-related tasks to high end multi-node machines.
• Cost Margin and Latency Management via Reasoning Control: Controls product margins and latencies by configuring API requests to skip the thinking trace, use adaptive thinking trace or deep reasoning trace depending on the criticality of the request.

How Does Nex-N2.5 Work?

Nex-N2.5 integrates an execution-feedback loop into its post-training flow, end-to-end. Instead of making predictions for the next tokens based only on pairs of instruction and response that don’t change during training, the training procedure makes the model experience the live execution environment by receiving terminal output streams, web DOM tree structures, and desktop screenshot streams in real-time. Consequently, the model learns to map raw vision observations directly to useful tool invocations and normalized keyboard/mouse coordinates on a 0-1000 spatial grid.

To enable efficient decoding in real-time on the 8xH100/H200 GPU cluster of Nex-N2.5-Pro, Nex-AGI has developed the dedicated inference engine NexRT. The engine supports Standard Decoding with Multi-Token Prediction (MTP) and DFlash block-diffusion. The engine features custom CUDA kernels, full decode CUDA Graphs, GPU Direct between nodes communication, and Context-Parallel Attention (CP) that allows avoiding duplicate KV cache reads with extended context windows.

Performance Comparison with other Models

In the context of web browsing and long context information synthesis, BrowseComp draws attention to Nex-N2.5-Max which has achieved the #1 rank worldwide and is directly better than some of the best proprietary frontier models like Claude Opus 5 and GPT-5.6 Sol. In addition to that, under the same evaluation paradigm, Pro and mini versions have shown significant generational improvements compared to their predecessors. While showing superiority in web browsing, Nex-N2.5-Pro has achieved the #1 rank worldwide in OSWorld-G spatial grounding benchmarks, beating other best visual and multimodal models like Qwen3.8-Max, GLM-5.3-Flash, GPT-5.6 Sol, and Claude Opus 5.

source - https://nex-agi.com/

When it comes to software engineering for repositories and debugging multiple files, the evaluations on SWE-Bench Pro and DeepSWE v1.1 show how well the model can generate code and solve problems. The model, compared to leading open-weight models such as DeepSeek-V4-Pro and GLM-5.3, shows better results on most of the important agentic, web, and programming benchmarks, while on other benchmarks, such as SWE-Bench Pro and GUI navigation in space, it shows competitive performance with some targeted leads. This good performance showcases the efficiency of the model suite in dealing with large full-stack codebases and autonomous engineering problems.

source - https://nex-agi.com/

In all types of agentic workflows, desktop operating system controls, and multi-turn execution of tools, Nex-N2.5 remains consistently powerful in different evaluation sets. In all evaluation sets for workflow and knowledge performance such as GDPval-AA v2, AutomationBench, and Toolathlon Verified, the Max version is superior to top frontier agents such as GPT-5.6 Sol, GLM-5.3, Kimi-K3, and DeepSeek-V4-Pro. Also, the excellent performance on Terminal-Bench, OSWorld-Verified, and OSWorld-2 shows the cross-domain versatility of the family of models, proving that the feedback loop for perception and execution in reality is successful.

How to Access and Use Nex-N2.5?

Model Weights for Nex-N2.5 are available under the Apache-2.0 License through Hugging Face and ModelScope, while the codebase is available at GitHub. Hosted API endpoints can be accessed through OpenRouter.  Reasoning efforts during API calls can be dynamically controlled through 'reasoning_effort'. During SGLang deployment, launching scripts require providing '--tool-call-parser qwen3_coder', as well as corresponding reasoning parsers: '--reasoning-parser qwen3' for Mini/Pro or '--reasoning-parser deepseek-r1' for Max.

Limitations and Future Work

Using the Nex-N2.5 framework involves significant overhead in terms of hardware infrastructure as the Max model needs at least 16× H200 GPUs in 2 nodes with DeepEP/DeepGEMM networking, whereas the Pro model needs an 8× H100 node with SGLang patches for ideal token decoding. Going forward, work will focus on using trillion-scale post-training knowledge for larger foundation backbones and complete open-source availability of the NexCUA evaluation framework.

Conclusion

Nex-N2.5 presents an implementation strategy for open weight agentic architectures in transforming computer vision into a proactive visual execution cycle instead of a passive description method. It illustrates the way in which enterprise automation systems can move away from fragile brittle scripting to visual self-correcting autonomy through the use of precise spatial grounding along with customized inference engines such as NexRT.

Sources:
https://nex-agi.com/
https://github.com/nex-agi/Nex-N2.5
https://huggingface.co/collections/nex-agi/nex-n25
https://huggingface.co/nex-agi/Nex-N2.5-Max
https://huggingface.co/nex-agi/Nex-N2.5-Pro
https://huggingface.co/nex-agi/Nex-N2.5-mini


Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.

How Claude Fable 5.1 Cuts Heavy Agent Costs by 45 Percent

Introduction

Generative models of great capacity come with a fundamental trade-off of being unshackled from reasoning power versus safety. In mission-critical engineering, telemetry, and software workflow automation, continuous reasoning in large contexts calls for a balance of great model autonomy and dynamic hazard mitigation. Instead of using static refusal of execution that leads to a stoppage of multi-turn reasoning in complex processes, modern frontier systems use classifier shielded designs. This design separates dual-purpose threats while still keeping great analysis capabilities, thus leading to fast growth in bio-computational and planetary mapping domains. Using secure access routes for accredited domain experts and run-time safety zones for deployment of the model, a framework is set in place for stable sovereign enterprise operations. Evaluation of Claude Fable 5.1 offers great insight into this balance of agential autonomy, safety, and long-horizon execution cost in modern system design.

What is Fable 5.1?

Fable 5.1 Claude is the latest state-of-the-art Claude generation model from Anthropic. It is built for advanced horizon reasoning and software engineering with a very large window of context. It is one of the Claude models that functions as a classifier-shielded system and represents the topmost point of the entire Claude generation model family. The model was designed to perform agentic operations within multi-hour long loops.

Key Features of Fable 5.1

• The prompt caching optimization: It minimizes read costs to $0.25 for each million tokens, which is a decline of 75% from the traditional costs of token usage. This results in the reduction of operational expenditures up to 25% for normal workloads, and up to 45% for contextually heavy tasks involving agents.
• Protection of Context State & Intellectual Property: Anti-distillation mechanisms are included in the system to stop any new API accounts from modifying earlier context states while using multi-turn services. Thanks to this design choice, internal processes such as thinking blocks and logical sequence of thought are safe from extraction.
• Shielding of classifiers using precise calibrations: A newly installed and redesigned system of real-time safety probes allowing for a decrease by 60% of the number of cases of cyber-guardrails in each session and a reduction of 85% of their application on harmless questions related to basic biology or medicine.
• Granular Code Analysis Guardrail Thresholds: These custom-tuned safety parameters are set up with a view to provide a method for performing automated static code analysis and source code vulnerability detection irrespective of the level of access. They help to differentiate ordinary code inspection from penetration testing or exploitation. 
• Cryptographic Output Provenance: It is essentially a statistical watermarking measure that is incorporated into a circuit design. It gives a mathematical means to establish the authorship of the work done by the models and comply with the AI Act of the EU authorities. 
• Sovereign Cloud Data Isolation (EFS): Fable is built on an infrastructure that permits the storage of interaction logs and the use of CMEK within the private cloud of Amazon S3, Google Cloud Storage, or Azure Blob. This function eliminates the need for using third-party logging services while keeping platforms cost-free. Its operation adheres to strict ZDR requirements.

Use Cases of Fable 5.1

• Zero-Trust Continuous Codebase Auditing and Vulnerability Discovery: DevSecOps and software developers are able to deploy autonomous agents overnight scanning through millions of lines of codes. The model is capable of conducting deep static  analysis and analyzing complex vendor library dependencies, finding memory leaks and zero-day vulnerabilities without causing repetitive false-positive security denials. 
• Legally Verifiable Content Generation and Regulatory Compliance: Enterprises’ compliance specialists and legal technology teams are able to generate complex regulatory submissions, corporate policies, and intellectual property disclosures. Cryptographic watermark will ensure compliance with transparency requirements in the European Union whereas EFS will make sure that the private information will be stored exclusively in sovereign clouds.
• Fail-Safe High Acuity Scientific Research and Spatial Modeling: Academic and research institutions will be able to conduct high throughput computational modeling including multi-decadal planetary radar data for topographical mapping of planets or biocomputational simulation without risks of being halted due to dual-use query classification.
• Unattended Multi-Hour Agentic Workflows & System Migrations: The infrastructure and process automation experts can run more than 30 hours unattended migrations and diagnostics. The autonomous agents from Fable 5.1 correct runtime mistakes, control the parallel execution pipelines of experiments, log the internal activity, and rebuild the old applications without losing any context and logic consistency on multiple steps.
• Parametric CAD Engineering and Multimodal Technical Operations: Using parametric CAD engineering and multi-mode technical processes, hardware engineers and CAD engineers can upload large Spatial Plans and geometric figures that can then be used for real-time optimization of tolerances, validation of parametric designs, and the resolution of various assembly problems through the one million input parameters.

How does Fable 5.1 Work?

Fable 5.1 by Claude uses a dense transformer-based reasoning model capable of processing large volumes of data within a real-time and multistage classification shielding. Upon processing any input prompt through the 1,000,000-token capacity context window, specific probes will analyze the input tokens and generated tokens on trajectory. While other models may simply shut down the request once they recognize dual-use signals in restricted areas, such as biological sequences and cyber exploitation, Claude uses Active Fallback Routing. This feature reroutes any queries with risks to specific fallback models, such as Claude Opus 4.8 for cybersecurity vectors and Claude Opus 5 for biological computational vectors.

Further system security and alignment are enabled by the use of the anti-distillation defense mechanism and Enterprise Frontier Safeguards (EFS). The anti-distillation mechanism keeps track of multi-turn API conversations and prevents accounts from changing context blocks in history in order to distill reasoning traces without changing the natural thoughts outputs. On the other hand, the EFS system makes sure that data durability is separated from model hosting and creates API streams that transfer prompt history and key management information directly to customer-owned storage buckets.

Performance Evaluation with Other Models

The benchmarking tests to evaluate the performance of Fable 5.1 in terms of long-horizon software engineering and agentic execution have demonstrated its excellent capability. In the initial evaluations of the capabilities of the system highlighted in the table below, Fable 5.1 has been very successful, scoring 81.2% in SWE-bench Pro. This is better than the previous model, Fable 5 (80.0%), as well as other models such as Claude Opus 5 (79.2%) and GPT-5.6 Sol (64.6%). The performance shows that the model has the capability to not take the shortcuts leading to lower quality work and solve the fundamental issues of software. Moreover, the early access partners reported that Fable 5.1 has executed agentic runs for 38 hours without any problem.

source - https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf

In the case of task completion and reasoning for particular science-related terminal tasks, Fable 5.1 made great progress. As can be seen from the table above, the model obtained 52.6% on Terminal-Bench-Science 0.1 benchmark, which is more than two times better compared to Fable 5 (24.7%) and beats Opus 5 (29.0%). In case of Humanity's Last Exam (HLE) Fable 5.1 managed to get 60.9% without tools and 65.0% with tools, beating Fable 5 (57.8%/63.8%) and Opus 5 (56.6%/63.6%). Furthermore, the model achieved impressive result of 73.4% accuracy on CursorBench 3.2.0 at max effort.

source - https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf

Further benchmark synthesis demonstrates the flexibility of the model for multimodal, legal, and agentic tasks. Fable 5.1 did great on benchmarks for legal agent frameworks (90.81% mean criterion-pass rate on Legal Agent Benchmark - LAB) and vision-based data synthesis (GDP.pdf at 85.4% without tools). What is especially important to highlight is the safety and security profile of the model: the attack success rate of the model was only 0.1% at k=1 on the external Indirect Prompt Injection (IPI) benchmark.

How to Access and Use Fable 5.1?

The Fable 5.1 variant of Claude is available as a proprietary API endpoint hosted in the cloud with the ID claude-fable-5-1 . It has native integrations with Claude Code, Claude Cowork, AWS Bedrock, Google Cloud Vertex AI, and Microsoft Azure. Due to its large scale of parameters and proprietary classifier shielded architecture; only API calls can be used to interact locally.

Limitations

Nevertheless, there are certain operational limitations to Fable 5.1 that systems engineers have to consider. First of all, when classifier probes lead to the engagement of Active Fallback Routing in automated benchmarking tests, the execution path will be redirected to fallback models, resulting in a zero-score failure in the given testing phase even though the security hazard has been dealt with successfully. Furthermore, red-team evaluations reveal that although the model demonstrates good resistance to single-turn attacks, it is vulnerable to multi-turn framing attacks, such as deep academic role-playing.

Conclusion

Claude Fable 5.1 illustrates how state-of-the-art model reasoning can coexist alongside thorough safety at a corporation without having to compromise either one. With the substitution of the coarse-grained rejection systems for the flexible classification protection, proactive fallback routing, and aggressive lowering of caching costs for prompts, Anthropic was able to develop a system that allows for reliable operation of agentic loops lasting several hours.

Sources:
https://www.anthropic.com/claude-fable-and-mythos-5-1
https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf
https://www.anthropic.com/news/enterprise-frontier-safeguards
https://www.anthropic.com/news/improving-fable-5-s-biology-safeguards


Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.

GLM-5.3-Flash: 320B Open-Source Native Multimodal Agentic Model

Introduction

There is an evolution from text-based processing into the natively-perceiving and interacting with the graphics interface. Enterprise teams working on scaling automation face significant challenges through the use of disjointed, textual-only pipelines. Building an agentic infrastructure at scale requires a totally different engine; one which incorporates screen-based programming and iterative graphical rendering tests directly in the core loop of its logic.

GLM-5.3-Flash sets a new benchmark in this regard. The use of sparse and linear structural design shows that running tasks in the autonomous manner over long horizons is not necessarily accompanied by excessive computational cost. Companies should use it since it provides cutting-edge agentic capabilities along with document processing in extremely economical terms.

What is GLM-5.3-Flash?

GLM-5.3-Flash is a 320B parameter (18B active) native multimodal foundation model created by Z.ai, designed to be an autonomous partner in production as opposed to a conversational interface. Training on 30 trillion token multimodal dataset, it can process text, code, images, as well as any file format at once, allowing it to perform multi-step professional workloads - from automated UI development to auditable financial research - using agentic frameworks.

Key Features of GLM-5.3-Flash

• Native Graphical User Interface and Interface Automation: Vision is natively integrated within the running processes and can directly interact with desktop and web applications. It uses frameworks such as vLLM, SGLang, or TokenSpeed and applies Computer Use protocols in order to convert screen recordings and designs into Next.js applications, automatically verifying their states of interaction and components.
• Spatial and Parametric Engineering Design Generation: Besides creating websites, it understands 3D spatial relations. It can create parametric CAD scripts using the build123d framework and complete 3D scenes in Blender while continuously adjusting their layouts based on the feedback about the structure.
• End-to-End Office Document Audit: The tool builds complex PDF, PPTX, DOCX, and XLSX documents and manages both information architecture and formatting at the same time. It creates the renderings in order to detect and correct possible layout issues such as text overflowing, wrong alignment of charts, etc.
• Long-Context Execution Support: It was designed for running multi-step operations and has a huge 1 million token context window. Using the efficient IndexPool algorithm, it compresses key vector caches in order to minimize memory usage and keep large projects and financial documentation open and ready for long-horizon analysis.

Use Cases of GLM-5.3-Flash

• Automatic Visual Audit for Massive Numbers of Enterprise Documents: In generating and processing massive numbers of documents on a daily basis like PDFs, PPTXs, and DOCXs, there is bound to be some issues related to poor formatting in terms of things like overlapping tables or misplaced text. This application uses an automatic quality assurance pipeline based on screenshot rendering of the documents.
• Automated Generation of Executable 3D Parametric CAD from 2D Physical Blueprint: Design engineers get an opportunity to transform detailed and technically advanced multiview physical blueprint images into executable 3D CAD script automatically. While doing so in headless environment, the process visually checks the physical boundary and mathematical tolerance against the original 2D blueprint image to accelerate the entire process manually.
• Massive Refactoring & Security Patching of Legacy Codebase: In the context of huge and monolithic code repositories which include almost a million tokens, semantic searches, security patches, or legacy API contract updates turn into cost-prohibitive operations. Using this tool, you can run huge agentic sweeps through dozens or hundreds of code repositories at off-peak hours like weekends to patch deep-rooted structure-related bugs.
• Running Hundreds of Real-Time Desktop Agents Optimized for Sovereign AI Accelerators: Enterprises required to use only domestically produced AI accelerators need extremely efficient performance for operating GUI-operating systems. It allows launching hundreds of concurrent real-time agents which perform simultaneous clicking, typing and reading of live screenshots.

How Does GLM-5.3-Flash Work?

In terms of architecture, GLM-5.3-Flash comes up with the innovative Sparse-Linear Hybrid Attention architecture, becoming the first openly available model of such scale to be built around this specific combination of structures. It operates with 320B total parameters, but activates 18B parameters per token using the advanced MoE sparsity. To focus on extremely high inference speed and low latency as its priority, the model cuts down its total depth in half, working with 45 layers instead of 92 layers in the GLM-4.5 family. Linear attention structures solely take care of the representation of local dependencies, while the sparse attention layers extract relevant global context using the highly optimized, lightweight indexer.

source - https://z.ai/blog/glm-5.3-flash

In order to tackle the critical problems of memory scaling related to large context windows, the model employs IndexPool Key Compression technique. This approach succeeds in compressing four different indexer key vectors into one by using the specific weighted pooling, which reduces KV-cache size 4.44 times and cuts attention computations 3.01 times compared to the flagship GLM-5.3. It is pre-trained on an absolutely unprecedented 30-trillion-token multimodal dataset (much larger than both GLM-5 and DeepSeek-V3) and uses Manifold-Constrained Hyper-Connections (mHC) topology to optimize its scaling behavior. As for physical deployment, it uses Encode-Prefill-Decode (EPD) disaggregated cluster architecture. Thanks to the specific ReplaySSM kernels, hybrid INT8/FP8/BF16 cache quantization, W8A8 weight-activation quantization, and layer-split memory allocation, multimodal encoding and token-by-token decoding are divided into separate worker pools.

Architectural Equivalents & Optimization Paths

Even though GLM-5.3-Flash creates a very high benchmark, an analysis of other comparable hybrid architectures, particularly Kimi K3 and NVIDIA Nemotron 3, shows that these architectures have different operational characteristics. The thing is that Kimi K3 and Nemotron 3 are based on similar design principles: delegation of local dependencies to algorithms and use of dense/sparse attention exclusively for global context retrieval. Still, GLM-5.3-Flash outshines in terms of extreme latency reduction and hardware scalability. Namely, through the use of unique IndexPool Key Compression, this model reduces the cost of supporting its 1M-token context window by pooling four keys in one, which is very helpful during long-horizon codebase sweeps. In addition, through a conscious cutback of the neural network to just 45 layers, it reaches the ultra-low latency necessary for visual interface real-time operations.

However, in turn, other architectures have some structural advantages that point towards clear optimization paths. Nemotron 3 Super uses Mamba-2 State Space Models (SSMs) which by definition have the ability to track local dependencies in a more memory-efficient way compared to linear attention of GLM-5.3-Flash. At the same time, Kimi K3 implements Attention Residuals throughout a far larger 93-layer neural network, allowing selective representation retrieval which does not allow information loss at extremely large sequence lengths. There are many opportunities for improvements in terms of implementation of GLM-5.3-Flash, and it can greatly benefit from both mentioned above techniques. Thus, Attention Residuals may perfectly compensate any logic drops due to the shallow depth of 45 layers in the architecture.

Performance Evaluation with Other Models

Starting from foundational pre-training evaluations, the 18B active parameter GLM-5.3-Flash-Base shows clear supremacy over bigger base models. Tested in table below: Pre-trained Base Model Comparison, its best benchmark score appears to be achieved on LiveCodeBench-Base, where it scores 37.6 points. This clearly outperforms the old flagship GLM-4.5-Base and gains a decisive win against the big GLM-5-Base. Significance of this evaluation proves that despite utilizing very optimized active parameter count – less than half of the parameters used in GLM-4.5-Base and GLM-5-Base – the model manages to achieve better baseline logical and coding reasoning, thus proving structural efficiency of its 30T multimodal pre-training and sparse-linear hybrid architecture.

source - https://z.ai/blog/glm-5.3-flash

During evaluation of the Chat/Instruct variant in complex coding and agentic execution environments, GLM-5.3-Flash scores its second top benchmark breakthrough on DeepSWE v1.1 (as shown in table below), scoring 63.4 points. This is a huge advance for its predecessor GLM-5.2, outperforming such closed-source flagships as Claude Opus 4.8. Moreover, it scores 1773 points on GDPval-AA v2, outperforming not only Claude Opus 4.8 but Gemini 3.7 Flash as well. This evaluation proves that the model has the ability to perform end-to-end, multi-step software engineering resolutions independently, thus proving that operational cost optimizations do not mean poor quality or instability of reasoning and execution.

source - https://z.ai/blog/glm-5.3-flash

Within the wider array of benchmarks, GLM-5.3-Flash continues to demonstrate its supremacy compared to older flagships and smaller rivals. The scores of GLM-5.3-Flash include 48.8 on AutomationBench v1.0.6, 78.4 on Toolathlon Verified, and 84.3 on Terminal Bench 2.1, which always outperforms Claude Opus 4.8 in complicated agents. With respect to multimodal vision benchmarks, GLM-5.3-Flash scores 89.4 on CharXiv Reasoning w/ Tools, 80.5 on MMVU, and 77.8 on MVbench, easily beating smaller MoE models such as DeepSeek-V4-Vision-Exp. All of the above clearly show that the model provides cutting-edge vision-language understanding and tool operation.

How to Access and Use GLM-5.3-Flash?

The engineers and system integrators will have access to its core resources straight from the official Hugging Face repository of the model. This open-source and commercial-friendly licensed model comes with weights that can either be hosted locally or scaled out using stacks like vLLM and SGLang. For execution of workflows, it runs inside agent harnesses like Claude Code and connects to the ZCode desktop client in order to control the GUI visually. Teams that prefer using APIs may make use of GLM Coding Plan with its point system for quotas.

Limitations and Future Work

Apart from being revolutionary in terms of parameter efficiency, there are some limitations to GLM-5.3-Flash’s architecture. In particular, the size of its KV-cache, although heavily optimized by means of the IndexPool Key Compression pipeline, is somewhat bigger than specialized and very compact models such as Kimi-K3 and DeepSeek-V4-Flash. There is room left for improvements here as well as in terms of decreasing memory consumption further. Future architectural iterations will undoubtedly continue in that direction to ensure the most efficient processing of 1M-token context windows domestically on AI accelerator clusters with the highest possible performance in terms of throughput.

Conclusion

The ability to incorporate visual understanding into the process of coding through using a unique sparse-linear framework allows eliminating the expenses involved in applying complex logic and, hence, the financial barriers of the implementation of AI infrastructure. No matter whether it is managing a sovereign fleet, generating accurate physical CAD, or conducting visual audit of enterprise-wide systems – it offers a completely new paradigm of engineering implementation of AI.

Sources:
blog: https://z.ai/blog/glm-5.3-flash
Guide Document: https://docs.z.ai/guides/vlm/glm-5.3-flash
Model Weights: https://huggingface.co/zai-org/GLM-5.3-Flash


Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.

UI-Mate: Converting Screen Views into Platform-Agnostic Actions

Introduction

The main problem plaguing traditional desktop automation has been one of persistent structural weakness: the fragility of pixel-based coordinates or inflexible API wrappers that fall apart the instant there is an update to the GUI of the application. The GUI agents for foundations often find themselves struggling because of prompt ambiguity, oddities of local setups, and procedural subtleties, which cannot be captured in plain language.

The combination of environment-based reinforcement learning with visual demonstration learning solves this problem. With the introduction of screen pixels as the sole point of truth in real-time, the agent gains the ability to interpret the state of the interface and adapt to the drift of UI and even unusual interface layouts. This is precisely what UI-Mate does by allowing users to record a simple video of the task once and transform the video into coordinate-free subgoals, while constantly analyzing screenshots and providing an open-weight foundation model for executing multi-application computer tasks without using fragile scripts on Windows, macOS, and Linux.

What is UI-Mate?

UI-Mate is a  GUI agent framework which provides automation for long-term and multiple turn tasks in a native manner on any operating system. This project is maintained by the HY Frontier Team of Tencent. Being an agent which operates on raw pixel level, it converts language input and visual instructions into Platform agnostic mouse and keyboard actions on local software.

Model Variants

It comes in three different open-weight checkpoints:

• UI-Mate-9B: Developed from Qwen3.5-9B, used as a parameter-efficient GUI agent for generic computer manipulation.
• UI-Mate-27B: Developed from Qwen3.6-27B, used for long-horizon, application-specific software manipulation.
• UI-Mate-democua-27B: Fine-tuned using a special combination of generic computer usage data and demonstrations augmented trajectories. The model is capable of completing the task by following the demonstrations workflows while still maintaining the capability to follow instructions in the absence of any demonstrations.

Key Features of UI-Mate

• In-Context Demonstration Guided Execution (DemoCUA): Extracts a screen recording into a sequence of subtasks which are coordinate-free procedural checklists. It retains visual authority in real-time to dynamically omit repeated steps, add low-level actions (like scrolling and popups closure) and re-plan on the spot in case the screen deviates from the recorded one.
• Sharp Visual Reference and Cross-Platform Actuation: Processes the visual input in a normalized visual coordinate space of size 1000 × 1000 units, outputs tool calls for click actions, drag-and-drop, scroll, type text and key combination actions that can be resolved to physical monitor resolution.
• Long-Horizon Stability in Information: Retains very high stability of intent and coreference throughout a sequence of 100 interaction moves or more without any problems with context loss or drift of goal.
• Coordinate-Free Procedural Transfer: Evaluates spatial elements of the UI based on live screen view instead of copy-pasting of absolute coordinate values from recordings; hence, allows transferring of workflows across varying sizes and resolutions and visual themes of windows.

Use Cases of UI-Mate

• Zero-Shot Procedural Migration for Legacy Software Applications: Automates native, offline desktop applications (legacy ERPs, local database clients, or proprietary engineering tools) with inaccessible HTML DOM tree or API. Engineer records the workflow on an older OS, and UI-Mate locates the visual elements to execute the very same process in a new design layout of applications without any code refactoring.
• Visual Verification of Closed-Loop OS Sandbox Training: Facilitates RL agent training in the context of native OS sandboxing (Ubuntu, Windows, macOS). Data engine automatically performs randomization of desktop wallpapers, positioning of windows, and theme settings to avoid spatial shortcut learning. Step-by-step visual result verification through screenshotting validates milestones through Process Credit Model (PCM).
• Locally Hosted Automation of Processes in Secure Air-Gapped Environment: Distributes quantized FP8/INT4 models directly to locally hosted enterprise workstations either through vLLM or SGLang. Finance or government entities can automate multi-application processes such as querying of SQL GUI locally and processing of spreadsheets without going to cloud services.
• Environment Realism Benchmarking & Latency Auditing: Measures friction empirically on real-world desktop environment. Through testing, we found that processing instructions with noisy real-world spreadsheet data leads to 51.7% increase in trajectory length from 38.5 to 58.4 steps than when working with synthetic data. Profiling of latency shows that model prefill (1,597 ms) and decoding (314 ms) take up 91% of overall latency per step, setting benchmarks for hardware optimization of VLM serving.
• Screen-Observation-Only Security Control & Shell Defense: Secures enterprise network by limiting agents' interactions only to visual screen analysis and simulation of input actions, skipping direct execution of bash command-line or system shell API. It allows security specialists to constantly monitor trajectories recorded by the agent visually while human-in-the-loop control lets them to stop trajectory recording, analyze reasonings or even add their text instructions midway.

How does UI-Mate work?

UI-Mate has been engineered with an automated training data flywheel (see below img#1), an asynchronous reinforcement learning stack (illustrated in img#2), and runtime context injection. Its data engine is able to construct executable sandboxed executions on Ubuntu, Windows, and macOS, while randomizing file structures and environment visuals to evade synthetic bias. Coverage on the three-tiered capability tree (application domain, coarse capability, and fine-grained operation) is tracked to overcome training deficiencies, filtering trajectories through a multimodal judge and visual milestone check.

https://arxiv.org/pdf/2608.15930

Training is done using the Asynchronous Group-Relative Policy Optimization (GRPO) to reduce GPU downtime from OS rollout of variable duration. Rollout updates are made asynchronously as trajectory buffer becomes filled. To prevent policy staleness due to asynchronous rollout updates, IcePop discards isolated tokens having extreme ratio values, whereas SeqClip ensures that geometric means of token likelihoods do not drift. Decision-Turn Centering has been used to assign credits to actions to get rid of length biases due to failed trial-and-error approach, alongside with Token-Level Normalization which is meant to eliminate length bias in chain-of-thought output. There is an option to use Process Credit Model (PCM) for aligning active rollouts to milestone trees.

Source - https://arxiv.org/pdf/2608.15930

During inference, DemoCUA processes recordings to create subtask JSON files. These get injected as context blocks by the runtime engine. Intermediate low-level actions are intentionally excluded from training rollouts so that the model has to make those inferences visually using live screen shots. On macOS, the model uses an Accessibility (AX) API bridge for passing coordinates and resorts to PyAutoGUI click simulation if UI elements are not exposed directly.

Performance Evaluation with Other Models

In the core OSWorld-Verified testbed, the larger dense version of UI-Mate achieves a new open-weight state-of-the-art success rate average, whereas the parameter-efficient version performs exceptionally well. In this performance comparison, it has been established that UI-Mate performs way better than the dense foundation model, specialist open-weight agents, and large parameter scale Mixture-of-Experts architectures, and performs competitively against proprietary frontier flagships. The main implication of this finding is that dense and environment-informed reinforcement learning with pixel-wise actuation can do better computer control than parameter scaling and sparse mixture of experts routing.

source - https://ui-mate.github.io/

In the realistic OSWorkerBench dataset, the larger and denser model holds a prominent strict success rate and progress score relative to its smaller competitor. UI-Mate is designed to evaluate more complicated information flow and long-horizon problems involving many normalized applications running on the desktop, and UI-Mate performs significantly better than its own dense model and large general models in Multi-App and Long-Memory subsets. This result indicates the importance of late information and decision credit allocation of UI-Mate since keeping the visual state of applications over long time periods avoids goal degradation in complicated multi-application enterprise tasks.

source - https://ui-mate.github.io/

When considering Other evaluations, in which the larger dense model performs better than both the base model and large Mixture-of-Experts baselines and paired demonstration evaluation on the dedicated demonstration-guided checkpoint, the model demonstrates remarkable adaptability. Same task demonstration guidance helps to solve previously unsolvable tasks in public benchmarks and achieves higher strict success in office workflow subtasks. In special application scenarios such as Godot and QGIS, demonstration guidance helps to increase completion scores and reduce trajectory lengths. The importance of all of these benchmarks together is that the scaling of parameters of the model is the major factor for the complexity of multi-application sub-domains like Office, Daily, and Workflow tasks, while visual demonstrations help in eliminating exploratory paths.

How to Access and Use UI-Mate?

All model weights in Bfloat16 and FP8 quantized format are available on Hugging Face under the  license. Local execution scripts, service recipes and a Mac OS client for Apple Silicon computers are provided through GitHub and project website. This model works locally via vLLM or SGLang using standard OpenAI endpoints and parameters.

Limitations

Technical limitations include the current context window format which places the workflow injection at the beginning of the prompt, thus any update to the subtasks invalidates the shared KV-cache. Future work plans to place the workflow blocks at the end of the context in order to be able to utilize an append-only KV-cache. Furthermore, a limitation lies in the gap between specialized professional software compared to flagship products (75.5% vs 81.6% for Kimi-K2.6), as well as in late stage dropouts in multi-step chains. Improving on policy entropy collapse in RL training when using reasoning tracebacks and generalization of non-identical variant tasks are still works in progress.

Conclusion

UI-Mate represents the departure point for more flexible solutions than brittle coordinate-based scripting or prompt engineering, showing that the native control of computers visually is most effectively done via pixel-based reinforcement learning. By divorcing instruction of tasks from fixed positions on the screen, it gives open weight foundation models the flexibility necessary to perform complex multi-application workflows.

Sources:
https://ui-mate.github.io/
https://huggingface.co/papers/2608.15930
https://arxiv.org/pdf/2608.15930
https://huggingface.co/collections/tencent/ui-mate
https://github.com/Tencent/UI-Mate
https://huggingface.co/tencent/UI-Mate-27B
https://huggingface.co/tencent/UI-Mate-9B
https://huggingface.co/tencent/UI-Mate-democua-27B

Disclaimer - This article is intended purely for informational purposes. It is not sponsored or endorsed by any company or organization, nor does it serve as an advertisement or promotion for any product or service. All information presented is based on publicly available resources and is subject to change. Readers are encouraged to conduct their own research and due diligence.