From Perception to Driving Actions: Alpamayo on Chimera

By Jacques van Rhyn and Gang Li

October 22, 2026

We mapped NVIDIA's Alpamayo 1.5, a 10-billion-parameter driving model that explains its decisions and plans a path, onto four Chimera GPNPU cores in W4A16. On held-out driving clips, it writes the same reasoning as NVIDIA's reference model, token for token. Our initial release will land in Chimera SDK 26.10, and we are already working to make it faster. We expect to get it to about 0.9 seconds per sample in subsequent releases.

A red light seems simple: see the signal, stop the car. But deciding when to brake means weighing traffic, road geometry, and what the people around you might do next. NVIDIA's Alpamayo 1.5 is built to connect that kind of scene reasoning to motion planning.

We mapped Alpamayo 1.5's vision encoder, language prefill, token generation, and action expert onto four Chimera GPNPU cores. Each stage presents a different compute and memory problem, and because Chimera is programmable, we solve each one in software. The work extends our BEVFormer-Tiny perception demo into language reasoning and driving actions.

Our initial release will land in Chimera SDK 26.10. It runs a full sample in under 1.2 seconds on those four cores in our instruction-set simulator (ISS). We've already started optimizing it: engineers and AI agents in our Kernel Arena are working through the model's kernels, and we expect that work to make the model faster in future releases.

Four frames from the demo's held-out synthetic morning scene, sampled at 10 Hz and resized to 256×512. Cosmos-Drive-Dreams by NVIDIA Corporation, CC BY 4.0. Dataset source.

What Alpamayo Does

NVIDIA's Alpamayo 1.5 is a vision-language-action model, or VLA. The name describes three connected capabilities: process visual information, use language to reason about the situation, and generate actions that describe future motion.

The model takes camera images, text, and the vehicle's recent motion as inputs. It produces both reasoning text and a proposed trajectory, a sequence of future positions and orientations. NVIDIA calls the explanation a Chain of Causation: it connects features of the scene to a driving decision. For example, the model might identify a red signal as the reason to stop. That explanation is one output to examine alongside the trajectory when evaluating the model.

Alpamayo 1.5 uses a Cosmos-Reason2 vision-language backbone of 8.8 billion parameters and a separate 2.3-billion-parameter action expert, the part that predicts motion. Its output covers 64 waypoints over 6.4 seconds. The current Quadric evaluation uses one front camera, with frames resized to 256×512.

Bringing the Model to Chimera

Alpamayo moves through four stages: image encoding, context processing, text generation, and action refinement. Each calls for a different balance of computation and memory access. Mapping that changing workload efficiently is where the engineering gets interesting.

Chimera combines matrix, vector, and scalar computation in one programmable architecture. We use ChiPy, the Chimera SDK's Python interface for composing neural-network modules, to connect and check the model's stages. C++ compute kernels, the routines that execute operations on the GPNPU, give us control over how the work runs:

• Vision encoding turns camera frames into numerical features. We distribute images across the four cores so multiple frames can be processed in parallel.
• Prefill processes those features together with text and vehicle-motion history. It saves attention data in a key/value cache for later stages to reuse. We process the input in 128-row chunks so every intermediate activation need not fit on-chip at once.
• Decode generates the explanation one token at a time. A token is a word or part of a word. We divide the matrix outputs and attention work across cores to handle this repeated computation.
• The action expert reads the saved context and refines acceleration and curvature values through ten steps. We distribute its smaller matrix operations across cores; the host CPU then converts the resulting actions into trajectory waypoints.

The useful flexibility is in those choices. We can change the division of work, data layout, and memory schedule in software to suit each stage. Every stage on the GPNPU runs in W4A16: 4-bit weights with 16-bit activations and a 16-bit key/value cache. Weights take a quarter of their 16-bit size, which matters most in decode, where each token streams the language model's weights from external memory. We solve the 4-bit weights with GPTQ rather than rounding them, calibrated on seven Cosmos-Drive-Dreams clips.

The mapped neural stages run on Chimera and share saved attention context. Preprocessing, supporting calculations, action updates, and trajectory conversion retain host CPU work. Simplified from the demo's pipeline diagram.

Where Performance Is Headed

The initial release in 26.10 will give us something concrete to optimize, and that work has already begun. Our blog Enter the Arena: How Our Toolchain Started Improving Itself describes how engineers and AI agents find faster ways to execute Chimera compute kernels, verify that outputs remain identical, and carry useful improvements back into the toolchain.

For Alpamayo, the Arena started with two kernels, one in the vision encoder and one in the action expert. Together they already cut the time per sample by about 13% on the same four-core configuration, with outputs identical byte for byte. Prefill comes next and has the most room: it shares its layer kernel with the action expert the Arena just sped up. Decode is already close to the memory-bandwidth limit, since each token streams the language model's weights from external memory, so gains there will come from moving fewer bytes. We expect to get it to about 0.9 seconds per sample on the same four cores in subsequent releases.

Measurement note: Alpamayo initial release for the upcoming Chimera SDK 26.10, four Chimera QC-Ultra cores at 1.7 GHz, 8 MB of on-chip memory and 32 GB/s of DDR bandwidth per core, measured in the instruction-set simulator (ISS). The default single-camera sample covers 4 frames, a 633-token prompt, 16 decode steps and 10 action refinement steps. The figures for the Arena-optimized kernels are estimates, scaled from per-block and per-layer measurements.

The host CPU still handles calculations around the action expert: preparing its inputs, processing its outputs and updating the candidate actions, then converting the final actions into waypoints. Full-chain correctness checks run through a host reference implementation; separate ISS checks validate the corresponding device kernels. The timing scope above excludes a complete camera-to-trajectory system measurement.

Checking the Predicted Motion

A driving model's output is a path, so validation follows the computation all the way to the predicted waypoints. We compare the port with NVIDIA's reference on the same inputs, checking both the generated reasoning and the trajectory. Intermediate checks help isolate where numerical differences arise, including the saved scene context that connects the language model to the action expert. Separately, ISS checks compare the compiled device stages with the host reference implementation.

We tested the fully quantized model, with every stage in W4A16, on four driving clips it never saw during calibration: morning, sunny, rainy, and night. On all four, it writes exactly the same reasoning as NVIDIA's reference model, token for token, from "Keep distance to the lead vehicle since it is directly ahead in our lane" to "Stop for the red traffic light since the signal is red" at night. Its predicted paths stay within 0.11–1.72 m of NVIDIA's on average across all 64 waypoints. Measured against where the car actually drove, the port's error is within 0.1 m of NVIDIA's own error on average.

Predicted path for the held-out morning clip shown above (front camera, 256×512), from the Chimera SDK Alpamayo example's nightly test run. The dark line is NVIDIA's reference model and the green line is the path the car actually drove. The other three lines are our port with progressively more of the model quantized; the fully quantized W4A16 version stays within 0.85 m of NVIDIA's path on average. All three run through the host reference implementation of the device kernels. NVIDIA's model and our port both predict a drift to the right that the driver did not make.

Different Models on the Same Hardware

BEVFormer learns where to sample surround-camera features. Alpamayo generates language and repeatedly refines driving actions. The striking part is that the same chip runs both. In the instruction-set simulator, both models use the same four-core Chimera QC-Ultra configuration with 8 MB of on-chip memory. Nothing in the hardware changes between them: we change the software to accommodate the workload, then keep improving how it executes.

Automotive silicon takes years to develop. AI models keep changing. Chimera lets us respond in software: implement a new attention mechanism or action predictor, validate it, then push performance further through better data movement, parallel execution, and compiler improvements. BEVFormer and Alpamayo show what that flexibility buys: a growing range of automotive workloads and more performance from the same programmable processor IP.

Contact Quadric to discuss your next automotive AI workload.

Explore Quadric IP:

• GPNPU Processor IP - 32 to 864TOPs
• GPNPU Processor IP - 16 to 108 TOPs
• GPNPU Processor IP - 4 to 28 TOPs