Recode v1.5: Batching Integer Programs in a C11 JIT

Problem. A tiny integer kernel can spend more time passing through the managed runtime than doing arithmetic. Change. I added a batch API, tightened the native emitter and built independent compiler comparisons. Evidence. The v1.5 reports retain cold and warm timings, failure cases, and regressions.

Recode is a C11 execution library for Linux x86-64. Its integer-program API takes a validated, deliberately restricted instruction stream, then interprets it or compiles a native function under a managed runtime. The first versions were useful for studying compilation and executable mapping, but repeatedly evaluating short programs exposed a different cost: the work around each call could dwarf the generated code itself.

For v1.5 I wanted to process multiple independent inputs using one registered program, without handing raw executable pointers to callers or removing the checks that made the existing API usable. The goal was less overhead per record, not a heavier general-purpose optimizer. This article describes the batch implementation, the preceding emitter changes, and the comparisons introduced in PR #26.

Where scalar execution spent its time

Calling rc_runtime_call_integer_view repeatedly means entering the public runtime for every item. The runtime checks the handle, looks up its translation, chooses an interpretation or native tier, maintains lifetime protection, updates counters and runs sampling logic. The actual native function may need only a handful of machine instructions. Each scalar call also traverses synchronization even when its executable mapping is already cached.

The v1.5 installed-record experiment instrumented the warm paths: 10,000 scalar FIXED calls made 20,000 mutex acquisitions, while a batch of 10,000 items made two. Both recorded 1,250 clock reads because timing samples were still taken at item boundaries, and neither allocated in the warm loop. Those are diagnostic operation counts, not latency measurements. They explain why reducing entry overhead is worth trying without pretending all public-call cost comes from locks.

Logical request flow from the v1.5 API contract. The native tier still enters the generated function separately for every record.

One program, multiple independent records

rc_runtime_call_integer_batch takes a runtime handle, an array of rc_integer_batch_item descriptors, a results buffer, its capacity, a mandatory completed output and an optional instruction error. An item has an argument pointer, an argument count and an optional rc_memory_view. The registered program is shared; arguments and read-only view bytes belong to the individual record. Each item runs in input order.

I did not implement batching by calling the existing public scalar API in a loop. That would retain the lookup, acquisition, policy and statistics costs for each item. Instead, the runtime validates every descriptor up front, acquires the translation once, chooses the tier once and processes records outside the cache mutex. A native entry is held under one call-long pin. A concurrent eviction, removal or invalidation cannot release executable code while that batch is using it.

The public interface below is a small, self-contained usage example, based on the v1.5 header. It builds a program that adds its argument to a little-endian 64-bit value from each record's memory view. The portfolio CI extracts this example, compiles it against the published v1.5.0 headers and library on Linux x86-64, and checks that it prints 9 16.

#include "recode_integer.h"
#include <stdint.h>
#include <stdio.h>

int main(void) {
    struct rc_builder *builder = NULL;
    struct rc_integer_program *program = NULL;
    struct rc_runtime *runtime = NULL;
    rc_handle handle;
    struct rc_runtime_config config = rc_runtime_default_config();
    config.policy = RC_POLICY_FIXED;
    const struct rc_integer_insn ops[] = {
        {.op = RC_INT_ARGUMENT, .dst = 0, .immediate = 0},
        {.op = RC_INT_CONSTANT, .dst = 1, .immediate = 0},
        {.op = RC_INT_LOAD_U64, .dst = 1, .left = 1},
        {.op = RC_INT_ADD, .dst = 2, .left = 0, .right = 1},
        {.op = RC_INT_RETURN, .left = 2}
    };

    if (rc_builder_create(3, 1, &builder, NULL) != RC_OK) return 1;
    for (size_t i = 0; i < sizeof ops / sizeof ops[0]; ++i)
        if (rc_builder_append(builder, ops[i], NULL) != RC_OK) return 1;
    if (rc_builder_finalize(builder, &program, NULL) != RC_OK) return 1;
    rc_builder_destroy(builder);
    if (rc_runtime_create(&config, &runtime, NULL) != RC_OK) return 1;
    if (rc_runtime_register_integer(runtime, program, &handle, NULL) != RC_OK) return 1;

    const uint8_t bytes[2][8] = {{7}, {11}};
    const uint64_t args[2] = {2, 5};
    const struct rc_memory_view views[2] = {
        {bytes[0], 8}, {bytes[1], 8}
    };
    const struct rc_integer_batch_item items[2] = {
        {&args[0], 1, &views[0]}, {&args[1], 1, &views[1]}
    };
    uint64_t results[2] = {0, 0};
    size_t completed = 0;
    struct rc_error error = {0};
    enum rc_status status = rc_runtime_call_integer_batch(
        runtime, handle, items, 2, results, 2, &completed, &error);
    if (status != RC_OK || completed != 2) return 1;
    printf("%llu %llu\n", (unsigned long long)results[0],
           (unsigned long long)results[1]); /* 9 16 */

    if (rc_runtime_remove(runtime, handle, NULL) != RC_OK) return 1;
    if (rc_runtime_destroy(&runtime, NULL) != RC_OK) return 1;
    rc_integer_program_destroy(program);
    return 0;
}

Each item's storage is borrowed until the call returns. Recode does not copy or retain these descriptors or views for later execution. Read-only input ranges can overlap one another, but results, completed and error storage must be disjoint from every input and from each other. The API checks range overlap and address overflow, not whether an arbitrary pointer actually points to a valid C object. The caller remains responsible for object lifetimes and concurrent writers.

A partial batch is not a transaction

Validation occurs in two stages. Structural validation checks every item before the first evaluation: handle type, argument arity, views, array lengths, capacity and output aliases. Ordinary structural errors execute nothing and set completed to zero. Detected overlapping output is a special case: the call returns RC_ARGUMENT without writing any output, including completed or the error structure. A zero-item call validates the handle but performs no invocation or compilation.

Execution-time faults are different. A bounded load can discover an invalid offset only when that instruction is reached. A fault at record k leaves successful results [0, k) intact, leaves output slot k and every later slot untouched, and sets completed = k. The error's offset names the failing instruction, not the record; completed identifies the record. Arithmetic wraps modulo 2⁶⁴, and a skipped load does not fault.

This is deliberately fail-fast rather than atomic. It avoids silently overwriting later application data or suggesting a failed request rolled back successful evaluations. It also means callers must initialize output slots if they intend to inspect the untouched suffix. Tests cover first, middle and final read failures, aliases, zero-item batches, guard pages and mapping failures. A long-running batch can hold native code pinned long enough to postpone eviction; it provides no timeout, cancellation or in-process sandbox isolation.

The smaller emitter was a separate change

Before batching I also tightened integer native emission. Virtual registers had authoritative stack slots; accesses to the first 16 slots can use short disp8 encodings instead of wider displacements. Inside straight-line regions, a limited RAX value-forwarding rule can avoid some redundant reloads. Forwarding is invalidated on control transfers and at targets, and every virtual write still reaches its stack slot. This is not register allocation or elimination of stack traffic.

The immutable v1.0-to-v1.5 study records arithmetic native text shrinking from 116 to 81 bytes and branch-heavy text from 1,275 to 850 bytes. A separate kernel study measured arithmetic execution from 6.336 to 5.548 ns and branch-heavy execution from 23.541 to 19.568 ns. These are native kernel measurements, not managed public-call gains. The same report records regressions in several managed calls and in all six final FIXED compilation cells. I have not substituted an illustrative assembly listing for disassembly that was not reproduced here.

Comparing with LLVM ORC and MIR

The comparator lives outside Recode's default build. Its LLVM adapter uses the real ORC LLJIT API, emits equivalent integer operations, verifies and optimizes the module with the O2 pipeline, and materializes native code at lookup. The MIR adapter parses and links MIR modules and generates native code at its optimization level 2. The six shared programs cover arithmetic, bitwise, conditional logic, data-dependent branches, an eight-iteration loop and bounded record loads. Independent C results check correctness. Optional competitor dependencies are not silently replaced with stubs.

Cold and warm phases answer different questions. Cold totals include initialization, program construction, registration/compilation and exactly N calls in the corrected final harness. Warm calls occur after checked warmup. The v1.5 comparison used three alternating process rounds on an Intel N150 running Linux 7.3.0-4-cachyos, GCC 16.2.1 and LLVM 23.1.4, with MIR pinned to dc2c763. Recode's compiled package used -O3 -DNDEBUG; LLVM and MIR used optimization level 2. CPU affinity was controlled, not frequency or background load.

Medians from the v1.5 performance report. Compilation intervals follow each engine's documented adapter boundaries and are not identical compiler-phase measurements. Peak RSS is Linux VmHWM for the process, not private heap or page-rounded executable code. C compilation is ahead of this test.

Recode compiled these supported small functions at a substantially lower measured cost, but LLVM and MIR executed the warm arithmetic kernel through their tested invocation paths much faster. The batch comparison also found the external native paths faster. At 65,536 records the Recode FIXED process used around 6.9 MiB peak RSS versus 5.8 MiB for MIR, because caller-owned record arrays and descriptors count too. No general compiler or memory ranking follows from a few small programs.

What batching actually changed

The installed threshold-excess record rule is a separate fixture from the six common LLVM/MIR programs. Its preformed nine-byte records, independent thresholds and checked result arrays are identical in scalar and batch modes; preparation and reference validation are outside the warm timer. The following are the three-round median FIXED times, in ns per record.

At 128 records, FIXED batching improved from 60.359 to 23.020 ns per record, about 43 million records/s in that fixture. One-item batching regressed; the request setup is not free. A 65,536-record cold batch also lost to scalar calls because the entire first batch remains interpreted, while scalar calls can promote partway through their sequence. At that size, ADAPTIVE warm timing regressed from 62.868 to 69.460 ns per record in the final comparison. Its conservative tier model was not adjusted to manufacture a favorable result.

Two details make these results easy to misread. First, this installed-record measurement is not the same harness as the v1.6 baseline comparison; its 23.020 ns figure must not be juxtaposed with a later 19.040 ns baseline as if they were repeated samples of one test. Second, smaller generated text is not smaller executable allocation: these tiny functions still use 4,096-byte page-rounded mappings. Caches, compiler memory and application arrays contribute to RSS separately.

The limits that led to v1.6

Batching reduced the overhead of public API entry, but it did not fuse the native loop. Every record still enters the generated function independently, and native execution still performs iteration, bounds handling, output writes and periodic sampling. Even the v1.5 integer emitter writes every virtual register to a stack home; its RAX shortcut only saves some reloads. Generated text became smaller, but validation/layout work could make compilation slower. The v1.5 scalar comparison even records arithmetic FIXED moving from 59.519 to 59.755 ns and loop interpretation from 241.357 to 254.963 ns.

These measurements pointed to distinct remaining costs: stack traffic in generated code, per-record native calls in the C batch loop, and repeated preparation during compilation. The next release addressed the first and part of the second and third, while keeping the public contract unchanged. For the code-generation details, see Recode v1.6: Cutting Stack Traffic in an x86-64 JIT.

v1.5 established a usable public batch interface with explicit partial-failure semantics and showed where it pays off on measured small workloads. It did not turn Recode into an arbitrary-binary optimizer, a sandbox, or a faster general alternative to LLVM.