Antler

In development

Early access

Docs

Delivered segments

A workspace can write its records to storage its owner holds: an S3-compatible bucket the customer names. The records go out in the order of the log (log.md) as segments, so that everything is checked from the bucket's files alone, without the service that wrote them.

Vector: 23-segments.

The files

Under the delivery's prefix:

antler-verify is given the directory that holds head.json and segments/, as copied out of the bucket.

A segment is written once and never rewritten. head.json is rewritten after each segment.

A segment

A JSON-lines file: a header line, one entry line per leaf of the log, and a trailer line. A file that does not end with a line feed, or has no trailer, was cut. Each line is decoded strictly: an unknown member is an error. A verifier refuses a segment file over 256 MiB.

The header

Format antler-segment-v1. Members, in order:

MemberTypeMeaning
formatstringantler-segment-v1
org_idstringThe workspace's id
originstringThe origin of the workspace's log, which every checkpoint must name
indexintegerThe segment's number, from 1
from_leafintegerThe leaf of the segment's first entry: 0 for the first segment, otherwise the to_leaf of the segment before
prev_sha256stringThe SHA-256 of the whole file of the segment before; empty for the first
created_atstringWhen the segment was written
keysobjectThe published keys, as in a bundle's manifest
checkpoint_keystringThe checkpoint key, as a verifier key
workspace, regionstringThe workspace and its region; optional

An entry

One of three forms, each with the entry's leaf:

{"leaf":5,"line":"<the record line>"}
{"leaf":6,"line":"<the stored line>","hash":"<the signed hash>","server_redacted":true}
{"leaf":7,"device_id":"d_…","seq":3,"hash":"<the record's hash>","missing":true}

The entries of a segment are the leaves from_leaf to to_leaf - 1, in order, with none left out.

The trailer

Members, in order:

MemberTypeMeaning
to_leafintegerThe leaf after the segment's last entry
entries_sha256stringThe SHA-256 of the entry lines as written, each with its line feed
device_keysmapFor each chain with a record or redacted entry in the segment, the key it is signed with, base64: the device's key, or the audit key for server
checkpointstringThe signed checkpoint note of size to_leaf, when the log has one: the segment's seal; optional
timestampslistTimestamp tokens (below); only in a sealed segment; optional

A segment holds at most 20,000 entries and ends at the largest checkpoint within that bound. Where no checkpoint falls there it ends unsealed, and a later segment's seal covers it. A segment may hold no entry at all: it then repeats the seal of the segment before, to carry tokens that arrived after it.

A timestamp entry:

{"checkpoint":"<signed note>","token":"<base64 DER>","consistency":["<hex>", …]}

checkpoint is the note the token is over (timestamps.md). Its size is at most the seal's. When it is smaller, consistency is the consistency proof from it to the seal (log.md); when it is the seal's size, its root is the seal's and there is no proof.

head.json

{"format":"antler-segment-head-v1","org_id":"…","segments":3,"to_leaf":25,"last_sha256":"<hex>","updated_at":"…"}

segments is the number of the newest segment, to_leaf its to_leaf, and last_sha256 the SHA-256 of its file.

Checking a delivery

With the keys pinned by fingerprint among the keys and checkpoint_key of the newest segment's header:

  1. All there, in order. The segment files are numbered 1 to n with none missing. Each header's index is its number, its origin and org_id are the first's, its from_leaf is the to_leaf before it (0 for the first), and its prev_sha256 is the hash of the file before it (empty for the first).
  2. Entries. entries_sha256 matches the entry lines; to_leaf is from_leaf plus their number; each entry's leaf is its place.
  3. The tree. Append the record hash of every entry, in order, to a Merkle tree as log.md defines it. Where a trailer has a checkpoint: verify the note with the checkpoint key under origin; its size must be the number of leaves so far and its root the tree's root. A root that differs means a record was altered, removed or added. No proof is taken on trust: the verifier holds every leaf.
  4. Chains. For each chain, in leaf order: a record entry is parsed strictly and checked as a record, and links to the entry before it of its chain as a chain's records do (records.md), declared gaps included. A redacted or missing entry stands in the chain with its seq and hash, unchecked, and the next record must link to it. A chain's first entry is seq 1.
  5. Keys. A device's key, from device_keys of the first segment that holds one of its records, must derive its device_id. It is accepted because the pinned checkpoint key signs a log that holds every record the key signed; a key pinned for the device replaces it. The key named for server must be a pinned audit key. A chain with no usable key is a failure.
  6. Timestamps. For each timestamp entry: verify its note; check it against the seal (the same root, or the consistency proof); check the token against it (timestamps.md).
  7. The end. head.json must be of format antler-segment-head-v1 for the same org_id. If it names more segments than the directory holds, segments were removed from the end: a failure. If it names the newest, its to_leaf and last_sha256 must be that segment's. If it names an earlier one, they must be that one's, and the result is unproven: the copy was taken while a segment was being written, or the head was put back.

The delivery is valid when nothing failed, head.json names the newest segment, and a seal covers every entry. It is unproven when nothing failed and the newest entries are in an unsealed segment, or head.json is absent or older than the newest segment. It is invalid otherwise.

What nothing signs

head.json is signed by nothing. It lets a verifier notice a segment removed from the end of a copy; someone who can write the bucket can put back an older head.json together with the segments it named, and the files alone then show a shorter, valid delivery. Keep the bucket under versioning or object lock, so that an older state cannot replace a newer one unseen, and compare the size a verifier reports with a size you noted earlier.

In a header, created_at, workspace and region are stated by the writer and checked against nothing; they are still part of the file whose hash the next segment and head.json name. head.json's updated_at is covered by nothing.

A device's key in device_keys is trusted only as step 5 says.