Rebuilding the Borealis runtime

Borealis provides the execution environments behind Cobalt’s coding agents and Enclave’s browser sessions. Both products depend on how quickly those environments become usable. A coding agent cannot run its first command while its computer is still starting. Browser automation cannot begin while its browser environment is being prepared. Every second spent there delays the work the user actually requested, before the agent has had a chance to do anything.

Our naive first implementation was functional: it could prepare an environment, boot it and run a workload. In use, that sequence was too slow. We were doing expensive preparation after the request arrived, then waiting for services the workload might never need. Cold template preparation alone could take about 30 seconds. Even once that work was complete, guest startup added seconds of avoidable delay. Keeping computers running avoided the next startup, but tied up host resources through long periods of inactivity.

As we increased concurrency, the same design also limited how much work the hosts could serve. Allocation counted occupied slots instead of the CPU and memory pressure of the workloads actually running. Independent checkpoints queued behind a shared upload limit, and recovery downloaded data already present locally. More computers meant more waiting and duplicated work. We needed a runtime that could respond quickly to a single request and keep doing so as the number of computers grew.

The first gains came from looking at what those seconds contained. One configuration-initialisation stage fell from 6.792 seconds to 21–32 milliseconds when we stopped it watching for configuration changes from the filesystem root. It still loaded the same configuration. Later, we moved image preparation out of the create request and kept compatible VMs ready in a pool. By the final production benchmark, native activation of a pooled VM took a median of 37.48 milliseconds across ten requests. The pool did its preparation ahead of the request; API and product setup added their own delays, which we measured separately.

As we worked through those delays, the larger problem became clearer. We wanted a computer that could wait without permanently occupying a running machine, then continue when someone sent a command or opened a preview. Its identity should survive a change of physical host. That became the basis of Runtime v2’s serverless logical VM model.

Over ten days, the work reached image preparation, snapshot storage, host allocation, billing and both products consuming the runtime. This account follows the implementation through production acceptance on 14 October 2026, including the delays and failures that remained.

Giving a VM a life beyond one host

An agent might spend a minute working and then wait twenty minutes for a person. Suspending its computer during that interval is useful only if the next interaction can continue from the state it left behind. Our early disk-based recovery brought files back through a cold boot. To preserve running processes as well, we needed to capture memory and machine state alongside the disks.

We chose Firecracker microVMs on KVM-capable Linux hosts. The control plane would keep the logical VM’s identity and decide where it should run. A host agent would communicate with that controller, while a privileged native supervisor performed local VM operations. The application would keep using the same computer, with Borealis handling suspension, wake-up and recovery from a checkpoint.

Recovery made ownership a practical problem. A controller can lose contact with a host and retry a request whose outcome it never received. The old host may still be running. We used ownership epochs and operation fences to reject commands from superseded owners, and checkpoint generations to identify the exact combination of memory, machine state and disks to restore. The hypervisor’s jailer, process confinement and network isolation remained responsible for separating guests.

A permanent VM with a committed checkpoint can recover on another compatible host while retaining its logical identity. An ephemeral VM uses the same managed lifecycle, but its state can be lost with its host. We made that durability choice explicit later in the work, after seeing how much unnecessary checkpointing disposable computers generated.

This is the serverless model we wanted: an application uses a computer without managing the physical server behind it. The logical population can grow beyond the number of computers active at once. Hot suspension can still retain RAM; moving state into cold checkpoint storage lets the runtime reclaim execution capacity. Placement must account for CPU, memory and storage pressure whenever a computer becomes active again.

Getting onto hardware we could measure

Our first tests ran inside a nested ARM64 Linux environment on the development machine. One two-VM run took roughly four minutes from initial boot through workload-readiness checks, including service startup and probing. It exercised the lifecycle, but the nested environment and broad timing window made it unsuitable as a baseline for bare-metal startup. We kept it with the functional test evidence.

The first dedicated host gave us a different problem. During a full smoke test it disappeared and rebooted, interrupting a Cobalt VM before recovery testing had completed. The next boot reported a CPU machine-check error. Disabling a firmware performance setting had not prevented the recurrence.

We moved the main tests to OVH while the original provider investigated and supplied a replacement. From then on, each result needed the host boot, runtime version, image revision and cache state beside it. The hardware failure had shown how easily an interrupted test could be mistaken for a software result if we recorded only the final API response.

Removing work from guest startup

With stable hardware, we could trace the wait for the first authenticated command. The .NET guest service was starting with / as its content root and configuring JSON reload watchers from there. In a controlled, same-binary comparison, disabling reload watching reduced the host-builder stage from 6.792 seconds to 21–32 ms. Initial configuration loading stayed in place.

After the source fix, two fresh 4-vCPU, 8-GiB guests reached authenticated execution in 3.577 and 3.661 seconds, with their images already local. Docker readiness accounted for 1.394–1.566 seconds of that startup. We were waiting for a daemon even when the first operation was an ordinary shell command.

We changed Docker to start when needed. The first agent operation could proceed while a later Docker operation ensured the daemon was available. That required coordination between simultaneous callers, so we tested three concurrent requests, an offline container build and run, and checkpoint/resume with Docker both idle and active.

Fresh readiness then measured 2.114–2.293 seconds with a warmed supervisor. The first request after a supervisor restart took 3.301 seconds, which we recorded separately. Deferring Docker removed roughly the wait we had attributed to it, while the cold supervisor still had startup work of its own.

That investigation led to a much larger delay outside the guest: turning an image into a usable template during allocation.

Continuing work with hot resume

For an existing computer, hot resume is the fastest way back to work. Borealis pauses the VM’s execution while keeping its memory and running processes on the host. Waking it continues that execution directly, without booting the operating system or loading a checkpoint from storage.

In the 8 October native-runtime benchmark, thirteen 4-vCPU, 8-GiB VMs went through pause and resume checks:

That makes short idle periods practical: pause a running computer between interactions, then continue with its applications and in-memory state intact. In these host-local tests, even the command confirming that the computer was usable finished within 48 ms of starting the wake-up. Prompts, commands and previews can trigger that wake automatically, so the user can carry on without a manual resume step.

Hot suspension retains memory on the same host. When that memory needs reclaiming, or work needs to move to another compatible host, Borealis uses checkpoint recovery. Keeping these paths separate lets ordinary wake-ups take the fast path while durable snapshots support longer idle periods and recovery.

Preparing images before allocation

Cold template preparation could occupy about thirty seconds of a create request. Faster copying helped, but it still left the caller waiting for work that later VMs would repeat or reuse. We moved preparation into an explicit image lifecycle: import or build the image, prepare its runtime-compatible baseline, then create VMs from that revision.

The API and CLI needed to expose those as separate operations. A Dockerfile first goes through the build process; its prepared output becomes the input to VM creation. Users can see when the image is ready and reuse it without triggering another build during allocation.

We could not pre-position every image on every host. With thousands of images and eligible hosts, that would multiply storage and transfer costs before anyone used most of those copies. Prepared artifacts therefore live in durable storage, with local caches populated as needed and placement able to use hosts that already have the image. First arrival on a host still includes a transfer; our local startup measurements begin after it has finished.

Once the image is local, Borealis can materialise and activate its prepared baseline on demand, or hand over an instance prepared earlier in a pool. The second path spends background work and host resources to reduce the next request’s wait. Btrfs reflinks make writable disk creation cheap: a nominal 16-GiB disk initially shares unchanged extents with its base and consumes additional blocks as the guest writes.

The 11 October test matrix measured both paths on the shared OVH development host. The images were prepared and local before timing began:

All five pooled requests consumed pool instances. Their native activation ranged from 33 to 60 ms, but the public create-through-execution range was 476 ms to 2.11 seconds. In this small sample, pooling reduced each measured interval, while orchestration and transport still kept the public path above our sub-100-ms target.

Repairing pool refill and CPU compatibility

During deployment, some requests were taking hundreds of milliseconds despite the pool being enabled. We needed to know whether they had consumed a ready VM or fallen back to the prepared baseline. We began recording the physical VM identities in the pool before allocation and matching the assigned VM afterwards.

That exposed refill problems after reboot. Builder state from the previous host boot needed retiring, and template-builder policy had to permit the preparation work used to replenish the pool. The control-plane endpoint could be healthy while the host failed to produce any ready instances.

Moving templates between CPU families exposed a second fault. Both hosts could be x86-64 while differing in the CPU state they could restore. We added compatibility to template and pool selection and rebuilt incompatible artifacts for their destination. Activation metadata also had to agree with the signed template metadata used to authorise the operation.

After repair and a reboot check, ten consecutive production allocations matched VMs observed in the ready pool. Every VM executed the verification command and was cleaned up afterwards.

Those ten requests had a native median of 37.48 ms, ranging from 33.30 to 48.52 ms. The server’s wake interval had a 376.29 ms median, and the operator’s observed API readiness had an 884.51 ms median. These overlapping clocks show where time accumulated around native activation. The sample confirmed the repaired path; measuring fleet latency would require a longer run under representative load.

Cobalt used a separate pool partition from the benchmark account. Its first smoke request took the prepared-template path. The next consumed a VM observed in its pool and reported 46 ms for provisioning, followed by 2.8 seconds of setup and 2.9 seconds total readiness. We had no matched earlier production measurement for that product path. The setup stage remained an unresolved delay to profile.

Reusing snapshot data we already had

Frequent suspension made checkpoint costs impossible to ignore. One early generation took roughly 181 seconds to publish, including about 51 seconds importing sealed content. It uploaded approximately 1.49 GB and downloaded 8.21 GB for repeated verification. We were spending bandwidth reading our own output back several times.

We gave verified immutable objects reusable receipts, including across verifier process restarts. A receipt binds to the expected content identity, allowing later operations to reuse that verification. Imported data still passes integrity checks; an internal handoff of the same immutable data can use the existing result. Local recovery likewise began using verified cache material already on the host.

Compression helped reduce transfer volume, but it could not replace the content identity checks. A compressed-stream checksum can detect corruption without establishing which VM or checkpoint generation the bytes belong to. We retained those bindings while removing repeated full reads.

The remote representation also became simpler. Each snapshot is an immutable package under a unique storage prefix. Payload objects upload in parallel, then the manifest publishes after all its dependencies exist. That final publication makes the generation available for recovery; an interrupted upload leaves an incomplete package that can be cleaned up.

Layers avoid duplicating the shared base of VMs created from one image. Later checkpoints describe changes relative to earlier layers. Restore and deletion therefore follow a dependency graph: a VM’s own snapshot can be removed while a base or parent remains available to other users of that layer.

We used the same manifest information to delete known objects directly. Repeated bucket scans had been rediscovering keys the runtime already knew, with more work as the namespace grew. Cleanup still coordinates with publication so that it cannot remove a snapshot whose manifest is becoming visible.

Checkpoint bytes travel directly between the host and object storage. The cloud controller manages the lifecycle and permissions. We also removed a host-wide eight-upload ceiling that made unrelated VMs queue behind one another. With payload concurrency scoped per VM, a three-VM test recorded 24 simultaneous uploads, eight per VM. The host could now use available transfer capacity across independent checkpoints.

The paired restore cases on 10 October showed the value of retaining data locally. Each case restored through the local and remote paths: local operations took 119, 61 and 61 ms without object-payload transfer; their remote counterparts took 4.979, 2.452 and 1.734 seconds, downloading 81.7, 18.2 and 18.1 MB respectively. The comparison measures the cost of download and verification for those snapshots. It is separate from the earlier publication trace and from product readiness.

We placed regional S3-compatible storage near the hosts, with local NVMe caches in front. A regional shared cache such as Ceph remained a future option. Hosts in different regions would use explicit placement and recovery transfers, keeping cross-region latency out of ordinary local cache access.

Letting completed work finish before upload

The product tests revealed another consequence of checkpointing. A coding turn could complete successfully while Cobalt continued showing it as working, waiting for the runtime to finish uploading its state.

We had treated work completion, local capture and durable publication as one user-visible operation. They happen at different times. In four deployed samples on 12 October, local capture took 0.61–2.92 seconds. Host publication took 30.81–37.43 seconds, and the interval from the end of local capture to checkpoint settlement was 39.25–47.95 seconds. Publication falls within that settlement interval, so the figures must not be added together. These samples exercised preview idleness, a command and an agent prompt.

We separated those lifecycles. Cobalt reports a completed turn when the work finishes; Borealis continues checkpoint publication and exposes its progress to infrastructure operators. Recovery uses the last committed generation. Until a new upload commits, it has not advanced that recovery point.

Disposable computers made the cost more obvious. We had been taking snapshots of environments whose loss the caller was willing to accept. New computers now default to ephemeral: they can suspend and resume locally, while accepting host-failure loss. Permanent computers opt into checkpointing. The task retains its history independently and can release its computer when finished.

Users can also switch an existing computer’s durability policy. That change goes through the runtime’s transition logic so that the recorded policy agrees with the checkpoint work required. When an ephemeral computer is lost, the product needs to explain the loss and offer recreation where possible; the task history still exists.

Storage cleanup follows the same ownership rules. Pending uploads can hold the only copy of new state, while an uploaded cache entry can be fetched again. We kept hot VM disks and pending publication on mirrored primary storage. Hosts with a separate drive could use it for image and published-snapshot caches.

To test pressure handling, we accumulated an upload backlog on a deliberately small 512-MiB filesystem. It admitted ten captures, rejected two when reservations exhausted available capacity, survived a restart with the backlog, then drained and verified the captures. The test checked that the runtime could stop accepting new storage obligations while preserving the ones it already held.

Allocating against load and testing to failure

The allocator still counted free slots while we were designing for computers that spent most of their lives asleep. It could reject a host with many idle VMs even when little work was running, or admit another workload to a host whose few active VMs had saturated it.

We changed admission to use current CPU and memory pressure. Resource observations maintain a host admission barrier, which placement consults before starting or resuming a VM. Resume goes through the same decision because it makes workload demand active again. A host at full CPU utilisation can reject more runnable work even with spare RAM.

The OVH test host had an Intel Xeon Gold 6312U with 24 physical cores, 48 hardware threads, 128 GB of ECC RAM and three 1.92-TB NVMe drives. We added Enclave VMs while each ran a continuous browser loop with page updates, JavaScript and screenshots. Our first attempts stopped at configured limits. To find the hardware failure point, we repeated the ramps until the host ran out of memory.

With 2-vCPU, 2-GiB guests, the last fully validated fleet contained 133 VMs and the run attempted a 134th. With 4-vCPU, 8-GiB guests, the corresponding counts were 98 and 99. Memory overcommit allowed those light loops to fit; they were using less than their configured maximum RAM. A workload touching all assigned memory would reach the host’s limit much sooner.

Near exhaustion, the instrumentation became less reliable too. One cgroup-derived CPU interval appeared to exceed the host’s logical CPU count because counter reads and timestamps were no longer aligned under scheduling pressure. We labelled that artefact in the evidence. The final complete fleet observations also preceded the recorded out-of-memory events, leaving the gaps shown in the caption. The failure counts apply to these loops; production sizing needs measurements of the intended workload and sustained health.

The later Cobalt ramp tested the product path with one-minute commands. Earlier coding-task attempts had exhausted agent-account usage, preventing us from separating model availability from runtime capacity. Deterministic commands removed that dependency. On 13 October, the run completed 100 commands and suspended all 100 computers. End-to-end readiness had a 1.96-second median and 5.32-second p95, directing the next investigation into the application services around the VM.

Following the delay into Cobalt and billing

Wake polling added a wait after the VM had become ready. Repeated preparation of fresh workspaces and separate connection handshakes added more. We replaced polling with events and removed unnecessary setup for newly created computers. We also split the activity display into provisioning, computer setup, workspace preparation and total readiness so the next trace would show which stage was slow.

Billing became a contention point when requests arrived together. Each reservation recalculated an account’s allowance while provisioning waited for permission to proceed. We separated the expensive calculation from the common admission decision by keeping a precomputed allowance view.

For accounts comfortably within their allowance, we accepted roughly a minute of stale calculation while reservations settled. Near exhaustion, checks became stricter. Post-pay accounts bypassed the prepaid-allowance check. VM activity measurement continued independently, preserving the timings needed to settle usage afterwards.

We found a similar duplication in suspension policy. Cobalt had its own auto-suspend settings while Borealis was already managing idle VMs. We removed the Cobalt policy and made prompts, commands and previews trigger the runtime’s wake path. An active preview needed to keep its computer usable, then permit idle suspension once that activity stopped.

This put routine suspension behind the computer interface. Longer recovery or a failed admission still needs an explanation, but opening a preview should initiate the necessary wake itself. Asking the user to resume the computer first would expose the scheduling work we had set out to take over.

Making the deployment survive a reboot

The refactor crossed Borealis, Cobalt, Enclave, billing, the SDK and the CLI. We had to publish updated contracts before their consumers could restore packages and build. Each repository then needed its own rebase, review and release checks before we could exercise the combined product.

On the hosts, several changes worked during interactive setup but failed after restart. Storage mounts, networking and service startup needed a repeatable procedure, including a reboot test. Pool refill was one of the failures that this caught: an API health check alone could not tell us whether the host would produce ready VMs after boot.

Lease expiry exposed a more serious lifecycle fault. If a control-plane interruption prevented renewal, a suspended VM could be treated as expired and destroyed. We changed that behaviour so loss of contact preserves the VM while ownership and recovery logic resolve the interruption.

Accounting for ten days of work

Planning began on 4 October. The first substantial Runtime v2 implementation commit was on 5 October, and production pool repair and acceptance completed on 14 October. The nine days from the first implementation commit included hardware investigation, product changes, reviews, deployments and repeated smoke tests. Borealis and its consuming products already existed; we were refactoring and extending that system.

We used Codex throughout, with Astra on the main thread and Sol 6.1 for much of the parallel work. At 20:00 UTC on 14 October, we collected the final cumulative token counter from each linked local session. There were 34 sessions, of which 33 had usage records. Taking one final counter per session avoided counting earlier cumulative totals again.

Of the 7.47 billion tokens, 98.6% were cached input: context processed repeatedly across the workstream. Fresh input plus output came to 103.37 million tokens. The output total already includes 4.65 million reasoning tokens.

The logs cover related infrastructure and product work as well as the runtime implementation. They omit one session without a usage record and exclude external Cobalt review agents and smoke-test model calls outside these local sessions. Calendar dates measure elapsed project time; we did not collect a human-hours timesheet.

We did the work through a Pro subscription, using four manual usage resets while two global resets also occurred. To estimate what the recorded workload would have cost through the API, we applied the standard rates published for GPT-6 Astra and GPT-6.1 Sol, checked on 14 October. Including cached input and long-context premiums gave $5,477.94.

That estimate covers tokens. Tool charges, infrastructure and human time are additional, and the local records did not identify the service tier. Applying a simple 2× fast-tier multiplier to the same usage gives about $10,956. We used the Pro subscription for this work; these figures estimate API billing for the recorded workload.

Repricing the same recorded requests entirely as Sol 6.1 gives $1,072.40, about 80% less at standard rates. This holds token consumption and cache behaviour fixed. We did not rerun the project on Sol, so the calculation cannot tell us whether it would require more attempts on the difficult parts. We chose Sol as the default for subsequent work, with Astra available when a problem warrants escalation.

Cached input contributed roughly 81% of the estimated standard API cost. The discount on each cache hit was outweighed by how often we carried a large history forward. That gave us a reason to keep subsequent investigations more tightly scoped and put reusable findings in evidence files.

An always on computer

At the end of the refactor, Borealis owned a logical computer’s identity across suspension and recovery. A permanent VM could return from a committed checkpoint on a compatible host; an ephemeral one could accept loss in exchange for avoiding durable snapshot work. Applications see an always-on computer, while Borealis handles suspension, wake-up and host placement behind the scenes.

Image preparation now happens before a computer is requested, so a prepared pool can hand over a VM in tens of milliseconds. When an agent finishes, its work is complete from the user’s perspective while snapshot uploads continue in the background. Subsequent starts reuse verified data already on the host. The application can treat the computer as always available without keeping it running continuously.

Figures open at full size. The hero artwork is illustrative; the activity-panel screenshot is an unaltered production capture.