RESEARCH SERIES

Agentic Inference
& AI4AI.

Inference systems as a testbed for AI-driven optimization.

We study how AI agents can improve the computational systems that support their execution. Inference offers executable changes, measurable feedback, and explicit resource and quality constraints.

INFERENCE FOR AGENTS

Run agents efficiently.

Reuse computation, organize requests, and allocate inference budgets.

AI FOR INFERENCE

Let agents improve systems.

Propose changes, conduct controlled experiments, and validate improvements.

01

SELF-EVOLVING OPTIMIZATION

EvoTailor

EvoTailor: Self-Evolving Agent-Based Configuration Optimization for LLM Inference Serving

Under review

Optimize the system and the search process.

EvoTailor investigates how agents can optimize LLM serving configurations for a given model, hardware environment, and workload. An inner loop evaluates deployment trials and incorporates performance feedback. An outer loop revises the search process through validated changes.

Adaptive search spaces, runtime resource awareness, and experience reuse provide a concrete setting for AI4AI. The current focus is vLLM serving, with end-to-end task time considered alongside resource and service constraints.

02

REUSABLE COMPUTATION

CacheSpec

CacheSpec: Finding the Sweet Spot for Small Models in Large Language Models

EMNLP 2026 Oral

Jingquan Chen^*, Jie Feng^*^†, Jinghua Piao^†, Shaogang Hu, Yong Li^†

Give small models precise, useful roles.

CacheSpec stores reusable, parameterized programs. A small model extracts and binds variables from new requests, allowing validated computation to be reused across related inputs. The same model also provides speculative drafts during target-model generation.

Program reuse and lightweight, verifiable model roles reduce repeated generation. Validation and fallback support task quality.

Paper Code arXiv

CacheSpec — Figure 1. Inference framework from the EMNLP 2026 paper. SourceClick the figure to view full size.

03

SHARED-CONTEXT EXECUTION

OpenCity

Designing A Scalable LLM Agent Framework for Large-scale Urban Segregation Simulation

KDD 2026 AI4S

Qingbin Zeng, Yuwei Yan, Zhiheng Zheng, Jingzhe Yuan, Jun Zhang, Jie Feng, Fengli Xu, James Evans, Yong Li

Organize inference around the workload.

OpenCity groups agents with similar static attributes to share compressed context while retaining individual dynamic states and memories. Concurrent request scheduling reduces communication and waiting overhead.

Population-scale simulation provides a structured workload for studying efficient LLM execution. Explore the social simulation perspective

Paper Code Earlier preprint

OpenCity — Figure 1. Concurrent request scheduling (earlier arXiv preprint, 2024). SourceClick the figure to view full size.

04

ADAPTIVE INFERENCE ALLOCATION

APS

APS: Bias-Controlled Adaptive Prototype Simulation for Population-Scale LLM Agents

NeurIPS 2026 Oral

Quan Zheng, Yan Gao, Shaobin He, Haoxiang Guan, Yuanhe Tian, Jie Feng, Ming Wang, Shuxin Zheng, Zhen Liu

Spend computation where it matters.

APS allocates direct LLM queries to adaptive prototypes, selected heterogeneous agents, and shadow-audit agents. Residual estimates guide correction and subsequent budget allocation.

The research question connects computational savings with explicit control of approximation bias. Explore the population simulation perspective

Paper

APS — Figure 1. Adaptive prototype simulation framework (arXiv version). SourceClick the figure to view full size.

RESEARCH OUTLOOK

Questions we are exploring next.

How can agents choose useful interventions, verify improvements under limited budgets, and transfer optimization strategies across workloads and hardware? We evaluate task quality, end-to-end cost, and the overhead of search and adaptation.

Back to the research program

Agentic Inference& AI4AI.