Shenzhen Zhilian Weichuang Technology Co., Ltd.

CPU returns to the C position: intelligent AI rewrites the scale of computing power in data centers

Publication Date: 2026-10-27 Source:Electronic Engineering Album Key words: CPU AI ARM View count:

For a long time, the industry's "ruler" for measuring AI infrastructure was only the set of scales called tokens: tokens per second, tokens per watt, and tokens per dollar. These three numbers supported almost all performance narratives in the era of big models. However, as AI shifts from "dialogue generation" to "agent execution", this "ruler" gradually becomes inaccurate.


According to OpenRouter data cited by US venture capital firm a16z, the token usage of intelligent AI has increased by approximately 14 times since February 2026. But more crucial than quantity is the change in nature - tokens are no longer the endpoint of tasks, but rather intermediate products of intelligent detection of context, calling tools, executing code, and verifying results.


We not only need to evaluate performance from the perspective of tokens, but also from the perspective of agents and the tasks they perform. ”Ravi Malhotra, Director of Architecture for Arm Cloud AI Solutions, said in a recent media interview.


Around this transformation, Arm recently released a CPU performance evaluation white paper for intelligent agent AI, with support from Fujitsu, Meta, and OpenAI. Independent analysis firm Signal65 has also completed third-party validation based on this methodology. The change in evaluation criteria reflects the repositioning of the CPU's role in AI data centers.


Intelligent agents become 'virtual users', CPU returns to C-bit


To understand why the evaluation criteria must be changed, we need to first understand what intelligent AI has done to data centers.


In the white paper, the intelligent agent is described as a "virtual user" with a persistent computing environment, which can maintain state, retrieve context, call tools and services, and continuously evaluate execution results until the task is completed. When millions or even billions of such 'users' operate around the clock, computing, memory, storage, and input/output (I/O) must have sustained support capabilities at high concurrency. This not only puts forward new requirements for CPUs, but also means that the industry needs to establish new evaluation standards to assess the system's ability to handle the actual workload completed by intelligent agents.


In the era of big model, CPU is more of a "supporting role" around the accelerator; In the era of intelligent agents, Arm divides its key tasks into three categories: first, converting generated tokens into actions - building, compiling, testing code, accessing web servers and databases, which consume a large number of CPU cycles; The second is to serve the Token generation itself - as the AI head node responsible for job scheduling, data preparation, and transmission, placing KV Cache and vector database in memory and storage, and feeding them to the accelerator as soon as possible; The third is orchestration and isolation - linking token generation and action execution into an end-to-end process, and providing an isolated execution environment for each agent through sandboxing.


If the CPU is not fast enough, more expensive GPUs will be idle, which is a result that no one wants to see. ”In Malhotra's view, the speed of the CPU determines the utilization rate of expensive accelerators.


1.jpg


Of particular note is that intelligent AI is not a replacement for existing cloud native workloads, but rather an overlay and amplification on top of them. Simply put, every search, service call, code execution, and data request made by an intelligent agent is transformed into a task of the underlying infrastructure. Claude Code、 The emergence of cutting-edge applications such as Alibaba Qoder and Meta Muse confirms that this additive effect is moving from concept to deployment.


At the same time, as intelligent agents gain more tools and data permissions, the requirements for runtime isolation and security are rising. Infrastructure processors such as DPUs are beginning to take on independent control planes for network, storage, policy execution, telemetry, and isolation. The computing landscape of data centers is being redrawn around intelligent agents.


Why is the old ruler inaccurate?


So, why are traditional benchmark tests no longer applicable? Malhotra's view is that its root lies in the form of intelligent agent workloads, which are fundamentally different from traditional cloud workloads.


On the one hand, different intelligent agent use cases have vastly different demands for CPU. Arm categorizes the currently active applications of intelligent agents into five prototypes: software engineering, enterprise automation, scientific research (such as financial analysis), personal assistants, and robotics technology. The compilation and testing of software engineering agents take a long time, and the performance of a single agent is crucial; The single operation of an enterprise customer service agent may only be a web page or database access, with low requirements for a single instance, but the massive number of operations puts extreme demands on concurrency and throughput. No two agents are exactly the same, "Malhotra said, meaning that a single score is difficult to cover all scenarios.


2.jpg


On the other hand, intelligent agent tasks run for a long time and are sensitive to concurrency, making "sustained performance" the focus. If throughput decreases as resource utilization increases, more theoretical computing power does not necessarily mean that more tasks can be completed. Further constraints come from the physical reality of data centers - power consumption, heat dissipation, and cabinet space all have fixed upper limits. The ultimate concern of operators is "how much effective work can be continuously completed within these limits?" This is precisely why the evaluation perspective must shift from single-chip to rack.


The customer deploys and uses the entire rack in the data center, not just a single CPU. ”Malhotra explained that the comparison of "chip vs chip" cannot reflect the differences in energy efficiency, memory utilization efficiency, server cost, and how many servers can be accommodated in a single rack. A complete performance evaluation also needs to find a new balance between throughput and SLA (Service Level Agreement).


For software development workloads such as Terminal Bench and SWE bench, running one to two agents per core may already be the upper limit, and increasing concurrency will not meet SLA. Therefore, the trade-off between concurrency and response quality has become a new variable in evaluation.


3.jpg


Arm's Solution: Parallel Component Benchmarking and End to End Agent Benchmarking


In response to the various new issues mentioned above, Arm has defined three evaluation principles in the white paper: representative intelligent agent application scenarios, repeatable CPU performance testing methods, and end-to-end result verification. Fujitsu, Meta, and OpenAI all support this framework.


Specifically, the methodology is divided into two layers. The first layer is component level benchmark testing. Specifically, testing the costs of components such as web servers, databases, and model services separately to eliminate interference from model inference and network variables, in order to accurately characterize the CPU's own capabilities.


During this process, these component benchmarks and agent benchmarks conducted testing and evaluation of the CPU in different directions. For example, tool calling class components (such as web service nginx, database MySQL) focus on IPC, I/O, and concurrency; Scheduling focuses on CPU concurrency and utilization; In the reasoning and planning process, the CPU serves as the head node, involving vector databases such as Milvus and model service framework vLLM. The focus is on examining single threaded performance, branch prediction, caching, as well as memory latency and bandwidth.


However, component testing can only answer 'how strong each component is', but cannot reflect how these components are connected and collaborate in the agent workflow - this is precisely the value of the second layer benchmark in methodology. It is reported that the second layer is an end-to-end intelligent agent benchmark test, which uses emerging benchmarks such as SWE bench and Terminal bench as references to examine the scalability and concurrency of real intelligent agent workflows, and verify whether improvements in the component layer can translate into better task completion results.


Sandbox expenses are also included in the evaluation scope. Due to data and privacy considerations, intelligent agents generally need to operate in isolated environments, and the time cost of creating sandboxes, even in milliseconds, can be amplified into significant system overhead under scale. Arm's testing covered scenarios such as cold creation, hot creation, and oversubscription where multiple containers run in parallel on the same physical core, and compared mainstream sandbox technologies such as Docker, Firecracker, and gVisor.


4.jpg


Malhotra stated that Arm proposes a methodology rather than informal industry standards or single benchmark testing, and does not recommend assigning weights to the five types of use case prototypes. We suggest that users first determine which type of prototype their use case belongs to, and then choose the corresponding end-to-end intelligent agent benchmark and components for evaluation based on this. In the industry rhythm of "almost every week or every other week, a new one will emerge" for intelligent agent benchmark, leaving room for methodological evolution may be more practical than rushing to calibrate.


Where does 1.99 times rack level performance come from?


The methodology ultimately needs to be validated by actual measurement data and rack level model results.


Independent analysis firm Signal65 has released an independent validation report based on this methodology, comparing the performance of Arm AGI CPU and the fifth generation AMD EPYC Turin on workloads such as tool invocation, orchestration, inference, and planning. The two platforms use the same benchmark test script and build rack level performance models based on measured server level data.


The results showed that among the seven workloads, the Arm AGI CPU achieved an average rack level performance of approximately 1.99 times that of EPYC 9755. In terms of scenario classification, the tool calling scenario is 1.7 times, the intelligent agent workflow and task orchestration scenario is 2 times, and the inference and planning scenario is 2.2 times. Compared to EPYC 9965, Arm AGI CPU can reduce sandbox creation latency by up to 45%.


5.jpg

6.jpg


In the report, Signal65 pointed out that Arm AGI CPUs have an average performance lead of about 20% at the per core and CPU SoC levels, with the remaining incremental gains coming from efficiency and scalability: lower power consumption per core means that more cores can be deployed within the same rack power budget. That is to say, the combination of per core performance, per CPU performance, and the number of cores per rack ultimately results in a rack level multiplier advantage.


This confirms Arm CPU's consistent design philosophy of making energy efficiency the core constraint of design, rather than making trade-offs afterwards. From Neoverse N1 to V3 (the core of Arm AGI CPU), Arm continuously optimizes microarchitectures around real cloud workloads such as SQL databases, caching, and inference by collaborating with ultra large scale cloud service providers; AGI CPUs, on the other hand, further enhance their support for the continuous operation of intelligent agent loads with high core density, per core memory configuration, and high bandwidth I/O.


Sandbox data is equally significant. In high concurrency scenarios such as enterprise operations, multiple agents run on each core, and lower isolation costs mean that more customer support work orders can be processed per unit time, resulting in lower task costs. Performance advantages can only be transformed into cost advantages in specific operational processes, which is the practical significance of rack level evaluation.


From methodology to product, an evolving narrative of evaluation


At the product level, Arm has designed two server reference designs, AGI CPU 1OU dual node and 2U2P. Supermicro, Lenovo, and ASRock provide production platforms and commercial servers of different sizes and configurations; At the rack level, it covers both air-cooled and liquid cooled solutions, including two specifications: 19 inch and Open Rack dual width. At the recent OCP Global Summit, Arm is promoting the contribution of AGI CPU server reference designs to OCP, and Meta's Yosemite YV5 rack level design also includes AGI CPUs.


7.jpg


The evaluation methodology has gained endorsement from heavyweight players in the industry and is also part of the ecological narrative. Toshio Yoshida, Senior Vice President and Head of the Advanced Computing Development Division at Fujitsu, emphasized the importance of a transparent, repeatable, and representative evaluation system in the development of the next-generation Arm architecture CPU FUJITSU-MONAKA.


Colin Jarvis, the global head of cutting-edge deployment engineering at OpenAI, pointed out that as AI systems evolve from generating content to completing complex tasks through reasoning, using tools, and interacting with software environments, industry collaboration around representative benchmark testing is crucial for characterizing emerging workloads and optimizing the overall performance of AI systems.


But it should also be noted that the evaluation criteria for intelligent AI are far from settled. Arm, AMD, and Intel each have their own intelligent agent CPU benchmarks, with different load prototypes, deployment topologies, and aggregation calibers. The intelligent agent benchmarks themselves are still rapidly evolving. As Malhotra said, this work "will never stop here," but for the industry, the emergence of a usable methodology may be more valuable than the arrival of a unified standard - at least from today on, when discussing CPU performance in the era of intelligent agents, the industry has a common language that refers to real workloads.