A Compiler-Inspired Workflow to Run Any Model on Any Chip

Introduction

Go to market teams at AI hardware companies often struggle with new models because efficient model execution is deceptively complex. A model may fail because kernels are missing, the compiler lowers a graph incorrectly, the runtime schedules work incorrectly, memory limits are exceeded, or the deployment cannot scale across chips and nodes.

Many teams start by debugging the full model in the full deployment context. That approach requires expensive accelerator time to reason about kernels, compilers, runtimes, memory, and communication all at once. Working at the scope of the full model requires deep expertise and makes agentic workflows unreliable because the agent is asked to solve too many interacting problems without enough structure.

Kernelize takes a different approach.

Our goal is to support a high-level request like: “Make GLM-5 run fast on this multi-node system.” To make that practical, Kernelize gives the agent a structured workflow, a knowledge base, and specialized sub-agents that break the full task into smaller, verifiable steps.

This post shares what we have learned about structuring that workflow. The recommendations below come from applying our compiler experience to AI-powered model bring-up and optimization. A compiler accepts a full program, but it optimizes through intermediate representations, constrained passes, invariant checks, and many small transformations. Kernelize applies the same discipline to model support: decompose the problem, verify each step, and connect every local fix back to the full deployment.

Build a Common Workflow for all Goals

Model bring-up and performance optimization have different goals, but the core workflow is the same. During bring-up, the system needs to ask what is the smallest model, layer, or operator that reproduces the failure? During optimization, the system needs to check the same areas, but with a focus on performance bottlenecks. In both cases, the full model is the source of truth, but smaller steps are needed to do the work efficiently.

A full multi-node deployment is too large, expensive, and noisy for rapid iteration. If an MoE layer in a model is slow, the issue might be routing, token packing, communication, misaligned kernels, memory layout, or load imbalance. If a model fails to compile, the issue might be a missing ATen op, kernel, unsupported lowering, unexpected dtype, or a runtime constraint.

Regardless of the goal, the Kernelize workflow isolates the issue before trying to fix it:

1. Identify one failure or bottleneck in the full model.

2. Minify the model to find the smallest model that still reproduces the issue.

3. Capture a correctness and performance reference.

4. Assign the problem to the right agent or tool.

5. Generate or apply a focused patch.

6. Verify the patch in isolation.

7. Reconnect the patch to the full model.

8. Repeat until the model meets the deployment goal.

The top-level prompt can still be ambitious because the execution path is deliberately constrained. The entire flow is structured to be common for a variety of high-level tasks.

Decompose each Task into Small Steps on a Reproducer

“Make GLM-5 run fast on this multi-node system” is exactly the kind of prompt we want Kernelize to support. The Kernelize platform treats the prompt and model like a compiler treats a full program. The request is decomposed into smaller optimizations and lowering passes.

For example, one agent inspects the model graph and identifies unsupported operators. Other agents create a minified reproducer, capture CPU input/output pairs, call Triton and other kernel generators, and benchmark the resulting fused kernels. If a bug is encountered, another set of agents could prepare compiler issues for review, with the exact IR, failure mode, and potentially a suggested fix.

The structure around focused agents results in a lower cost, more deterministic flow. Each agent operates on a set of subtasks with a narrow scope and clear artifact inside a verification loop. The system knows what kind of evidence is needed before a patch can be trusted. It also knows how to connect the local fix back to the full model.

Careful structure and the right optimization scope is what makes agentic AI practical for compiler and kernel work. Each agent is asked to solve a bounded problem and produce something that can be checked. Correctness is supported by capturing input and output pairs from a CPU or another working implementation. Performance testing requires an isolated benchmark that compares latency, memory traffic, or throughput before and after a change.

The result is a compiler-like agentic workflow with short passes, explicit artifacts, and programmatic verification. And, just as importantly, an audit trail of the steps and artifacts so experts can see what happened and make any needed corrections. Over time, many AI-powered passes can be hardened back into the compiler or runtime, reducing the need for expensive agents.

Start from the Most Abstract Fix

If a PyTorch or ATen implementation can express the behavior correctly, use it. If a portable Triton kernel is enough, use Triton. If the target requires a lower-level vendor kernel, move lower when the higher abstractions no longer work.

During bring-up, higher abstraction keeps the model moving. A PyTorch fallback, a larger boundary patch, or a simple generated Triton kernel may be the right first step even if it is not the final production solution.

During optimization, higher abstraction keeps the improvement reusable. A performance fix expressed in Triton or in a compiler pass can often help more than one chip, model, or deployment shape. A fix written too early in device-specific code may only help one benchmark, requiring expensive duplicate effort later.

Custom kernels written in a low-level, chip-specific language are often the final answer for production performance. The key is to first isolate the problem, then rapidly explore alternatives in PyTorch, a generated Triton kernel, a compiler lowering, or a runtime change. Exploration uncovers the details needed to write the best hand-optimized vendor kernel.

Kernels should be generated offline, validated and cached with regression tests. Avoid runtime generation, which can lead to nondeterministic agent behavior in the deployment path.

Scale from the Reproducer Back to the Full Model

Kernelize connects every isolated patch back to the larger flow. A minified model can prove kernel semantics. A focused benchmark can prove a performance improvement. But the patch still has to be reconnected to the full model, the real runtime, and the real deployment environment.

New issues can be discovered during scale up. Larger tensor shapes may trigger new compiler paths. Full weights may reveal dtype or layout assumptions. More layers may expose memory pressure. A multi-node deployment may introduce communication and scheduling bottlenecks that were invisible in the isolated test.

The workflow creates a trail of artifacts: minified models, captured references, isolated benchmarks, generated patches, verified kernels, profile data, and compiler recommendations. The reproducer can be scaled up incrementally based on these artifacts, isolating each additional issue in a newer, slightly larger reproducer. Keeping these steps small might be less efficient than the intuitive leaps of an expert, but it is far more predictable to guide AI on this path.

A Small Example: torch.histc and MoE

At the API level, histc computes a histogram of a tensor. If min and max must be inferred from the input, the implementation first needs reductions. Then each value must be classified into a bin. Then many input values may update the same output bin. The many coner cases and possible optimizations makes histc small enough to isolate, but complex enough to stress the Kernelize Platform.

A traditional approach is to debug histc only inside the full model running across multiple nodes. Model analysis requires understanding routing, packing, kernel support, communication, and scheduling. The Kernelize approach is to isolate a minified GLM-5-style model that can expose a histc-like issue without requiring the full model, full memory footprint, or full multi-node deployment. Once the isolated behavior is correct, the team can reconnect it to the full model.

Optimization can be applied incrementally during scale up. A fused MoE dispatch path, communication-aware kernel, or topology-aware expert placement strategy should be developed from isolated evidence before being inserted into the full deployment. We will show examples of this type of scale up optimization in a future post.

Model Support Should Look Like a Compiler

AI agents become far more useful when they operate inside a compiler-like workflow. A compiler accepts a full program, but optimizes it through small, structured passes. Kernelize accepts a full model and high-level prompt, but executes it through small, verifiable tasks.

The Kernelize approach is how a prompt like “Make GLM-5 run fast on this multi-node system” becomes practical. The system knows how to decompose the request, assign the right agents, verify their outputs, and reconnect each fix to the larger model.

For hardware GTM teams, a repeatable flow changes the customer conversation. Each new model becomes a structured workflow instead of a one-off benchmark. Each step does not need to deliver the perfect design. It needs to be correct, explainable, and ready for the next pass.