Delivered segments
A workspace can write its records to storage its owner holds: an S3-compatible bucket the customer names. The records go out in the order of the log (log.md) as segments, so that everything is checked from the bucket's files alone, without the service that wrote them.
Vector: 23-segments.
The files
Under the delivery's prefix:
segments/seg-000000000001.jsonl,segments/seg-000000000002.jsonl, …: the segments, numbered from 1 with 12 digits;head.json: how far the delivery has written.
antler-verify is given the directory that holds head.json and segments/, as copied out of the bucket.
A segment is written once and never rewritten. head.json is rewritten after each segment.
A segment
A JSON-lines file: a header line, one entry line per leaf of the log, and a trailer line. A file that does not end with a line feed, or has no trailer, was cut. Each line is decoded strictly: an unknown member is an error. A verifier refuses a segment file over 256 MiB.
The header
Format antler-segment-v1. Members, in order:
| Member | Type | Meaning |
|---|---|---|
format | string | antler-segment-v1 |
org_id | string | The workspace's id |
origin | string | The origin of the workspace's log, which every checkpoint must name |
index | integer | The segment's number, from 1 |
from_leaf | integer | The leaf of the segment's first entry: 0 for the first segment, otherwise the to_leaf of the segment before |
prev_sha256 | string | The SHA-256 of the whole file of the segment before; empty for the first |
created_at | string | When the segment was written |
keys | object | The published keys, as in a bundle's manifest |
checkpoint_key | string | The checkpoint key, as a verifier key |
workspace, region | string | The workspace and its region; optional |
An entry
One of three forms, each with the entry's leaf:
{"leaf":5,"line":"<the record line>"}
{"leaf":6,"line":"<the stored line>","hash":"<the signed hash>","server_redacted":true}
{"leaf":7,"device_id":"d_…","seq":3,"hash":"<the record's hash>","missing":true}
- A record entry carries the record's line (records.md) as a string and nothing else. Its record hash is the leaf.
- A redacted entry is a record the server stored redacted: the line as stored, which no longer matches its hash, and the hash the device signed, which is the leaf. The line is shown, not checked.
- A missing entry is a record the server no longer held when the segment was written (it was past retention): its chain,
seqand hash. The hash is the leaf; the content is not in the segments.
The entries of a segment are the leaves from_leaf to to_leaf - 1, in order, with none left out.
The trailer
Members, in order:
| Member | Type | Meaning |
|---|---|---|
to_leaf | integer | The leaf after the segment's last entry |
entries_sha256 | string | The SHA-256 of the entry lines as written, each with its line feed |
device_keys | map | For each chain with a record or redacted entry in the segment, the key it is signed with, base64: the device's key, or the audit key for server |
checkpoint | string | The signed checkpoint note of size to_leaf, when the log has one: the segment's seal; optional |
timestamps | list | Timestamp tokens (below); only in a sealed segment; optional |
A segment holds at most 20,000 entries and ends at the largest checkpoint within that bound. Where no checkpoint falls there it ends unsealed, and a later segment's seal covers it. A segment may hold no entry at all: it then repeats the seal of the segment before, to carry tokens that arrived after it.
A timestamp entry:
{"checkpoint":"<signed note>","token":"<base64 DER>","consistency":["<hex>", …]}
checkpoint is the note the token is over (timestamps.md). Its size is at most the seal's. When it is smaller, consistency is the consistency proof from it to the seal (log.md); when it is the seal's size, its root is the seal's and there is no proof.
head.json
{"format":"antler-segment-head-v1","org_id":"…","segments":3,"to_leaf":25,"last_sha256":"<hex>","updated_at":"…"}
segments is the number of the newest segment, to_leaf its to_leaf, and last_sha256 the SHA-256 of its file.
Checking a delivery
With the keys pinned by fingerprint among the keys and checkpoint_key of the newest segment's header:
- All there, in order. The segment files are numbered 1 to n with none missing. Each header's
indexis its number, itsoriginandorg_idare the first's, itsfrom_leafis theto_leafbefore it (0 for the first), and itsprev_sha256is the hash of the file before it (empty for the first). - Entries.
entries_sha256matches the entry lines;to_leafisfrom_leafplus their number; each entry'sleafis its place. - The tree. Append the record hash of every entry, in order, to a Merkle tree as log.md defines it. Where a trailer has a
checkpoint: verify the note with the checkpoint key underorigin; its size must be the number of leaves so far and its root the tree's root. A root that differs means a record was altered, removed or added. No proof is taken on trust: the verifier holds every leaf. - Chains. For each chain, in leaf order: a record entry is parsed strictly and checked as a record, and links to the entry before it of its chain as a chain's records do (records.md), declared gaps included. A redacted or missing entry stands in the chain with its
seqand hash, unchecked, and the next record must link to it. A chain's first entry isseq1. - Keys. A device's key, from
device_keysof the first segment that holds one of its records, must derive itsdevice_id. It is accepted because the pinned checkpoint key signs a log that holds every record the key signed; a key pinned for the device replaces it. The key named forservermust be a pinned audit key. A chain with no usable key is a failure. - Timestamps. For each timestamp entry: verify its note; check it against the seal (the same root, or the consistency proof); check the token against it (timestamps.md).
- The end.
head.jsonmust be of formatantler-segment-head-v1for the sameorg_id. If it names more segments than the directory holds, segments were removed from the end: a failure. If it names the newest, itsto_leafandlast_sha256must be that segment's. If it names an earlier one, they must be that one's, and the result is unproven: the copy was taken while a segment was being written, or the head was put back.
The delivery is valid when nothing failed, head.json names the newest segment, and a seal covers every entry. It is unproven when nothing failed and the newest entries are in an unsealed segment, or head.json is absent or older than the newest segment. It is invalid otherwise.
What nothing signs
head.json is signed by nothing. It lets a verifier notice a segment removed from the end of a copy; someone who can write the bucket can put back an older head.json together with the segments it named, and the files alone then show a shorter, valid delivery. Keep the bucket under versioning or object lock, so that an older state cannot replace a newer one unseen, and compare the size a verifier reports with a size you noted earlier.
In a header, created_at, workspace and region are stated by the writer and checked against nothing; they are still part of the file whose hash the next segment and head.json name. head.json's updated_at is covered by nothing.
A device's key in device_keys is trusted only as step 5 says.