# Cairn record format

A cairn is JSON lines: one record per line, oldest first. Download a shared cairn as
`https://cairn.oasis-protection.com/c/<paste>.jsonl` (sent with `Content-Encoding: gzip`; `curl --compressed` decodes it).

Times: a field ending in `_ms` is milliseconds and `_ns` nanoseconds; `at` is an ISO instant in UTC.
A duration whose stamps did not both come is absent, never 0. A field named `sum` adds steps that can
run on several threads at once, so it can pass the wall time. A field named `since_last_sample`
covers the window since the last `health` record.

## Reading a cairn

1. A field that is null is left out of the line: the writer drops nulls. Absent means unknown or not happening, never zero. A field that exists in one mode only is absent in the other: `fawe` without FAWE, `storage.engine` on Packs Server, `storage.packs_server` on the embedded store, `seal` and `merge` records on the embedded store only, `os.open_file_descriptors` on Unix only, `pass` while a pass runs only. A field described as "or null" is absent.
2. Counters come in three families. `..._since_last_sample` is a delta over the window: divide by `window_ms` for a rate (a window can cover two minutes). `..._since_jvm_start`, `..._since_open`, and `..._since_start` are totals that restart at each boot (`since_open` also at a store move): subtract two samples of one boot. `..._now` is a gauge.
3. A health sample whose server thread did not answer (`main_thread_answered` false) has no `server` and no `ticks`, holds `main_thread_stack`, and lacks the worlds in `disk`. Its tick counts roll into the next answered sample, so divide tick counts by `ticks.intervals_since_last_sample`, not by `window_ms`.
4. `ticks.intervals_over_55ms_since_last_sample` counts late ticks: intervals above 55 ms, the first histogram bound. Older cairns carry `intervals_over_50ms_since_last_sample` instead; it counted on-time ticks too (about half of all ticks), so do not read it.
5. `health.storage.rollback_refusal_now` is the reason a rollback, restore, or undo started now would be refused (`STORE`, `MOVING`, `REBUILDING`, `BUSY`, or `STARTING` while the store is not open yet), and is absent when one could start. Older cairns carry `refusal_reason_now` instead; it read `STARTING` on a healthy store, so do not read it.
6. While `pass` is present, `server.mspt_avg_last_100_ticks` near `config.rollback.target-tick-ms` is the pass using the tick as set, not a fault.
7. On a `lag` record, `interval_ms - previous_tick_duration_ms - max(previous_tick_time_remaining_ms, 0)` is the time lost after the measured work of the tick: the tick end handlers of plugins (the pass writes there), the OS, or a safepoint.
8. `at` is the time of what the record describes: the start of the pause for `gc_pause`, the time of the store for `incident`, the log time for `error`, the end of the group for `chain`, the ask for `lookup`. Records are not written in `at` order, so sort by `at` and break ties by `seq`. `t_ns` subtracts only inside one boot.
9. A boot with a `start` and no `stop`, followed by another boot, ended in a crash or a kill. Two boots with the same `start.jvm_started_at` are a plugin reload in one JVM, which changes the meaning of every `since_jvm_start` counter.
10. `daily` is cut at local midnight of `start.timezone`. A restart splits a day between `stop.day_so_far` and the next `daily`.
11. A record of a rare kind is written to `days/` and to `year/`: read a cairn by hand with duplicates removed by `boot` and `seq`. A gap in `seq`, or a `dropped_before` field, means records were lost.
12. `health.jvm.ledger_since_last_sample.roles` names threads with their digits replaced by `N`. A role absent in a sample did no measurable work in it. The lines add up to `process_cpu_ms`, and thread names identify the plugin or library behind the work (spark, FAWE, a database driver).
13. To read a problem report: start with the `start` and `config` of the boot, then the findings (`deadlock`, `main_thread_answered` false, `storage.failure`, `stall`, `outage`, `incident`, an `error` of level SEVERE), then `lag` and `gc_pause` near the reported time, then `health` around it.

## How to read operations

1. Null fields are absent. A missing timing field means the moment did not come: draw a gap, not a zero.
2. One rollback, restore, or undo is one `chain` record, written at its end (`at` is the end). Its `passes` are the first pass and its sweeps, or its replays. `empty_sweep` is a final sweep that found nothing. `failed_pass` is a pass that failed before it had a report.
3. `group.started_at` is the time of the command. `group.wall_ms` leaves out the wait of the staff member at a prompt (`context.plan.prompt_wait_ms`), as do the ledger and the GC deltas of the group. So `ended_at - started_at = wall_ms + plan.prompt_wait_ms`. For a group with no command (a recovery, an API call with no display) the clock starts at the chain.
4. A field named `sum` adds work that runs on several threads or overlaps in time, so it can pass the wall time by orders of magnitude. Divide by the count (`slices`, `reads`, loads) for a mean, or by the wall time for the threads busy on average. Never subtract a sum from a wall time.
5. Parts that add up: begin, `write_queue_ms + flush_pump_ms + register_name_ms + answer_wait_ms + start_hop_ms = total_ms`, and `answer_wait_ms = store_seal_ms + store_queued_ms + store_rebuild_ms + store_claim_ms + store_recorded_ms + unaccounted_ms` (`open_stream_ms` lies inside `answer_wait_ms`). Ticks: `ticks - full_ticks` is the sum of `ticks_not_full_by_cause` (`evacuation` is not in `ticks`). A pass: `settle.start_after_request_ms + settle.wall_ms + close.wall_ms` is about `wall_ms`. Group: `outside_passes_ms = wall_ms - pass_wall_sum_ms`; it holds the plan, each `before_sweep` gap, and the `begin.total_ms` and `before_sweep` of `empty_sweep`, which record the final sweep check. A cairn of an older build lacks the `empty_sweep.before_sweep`, so a wait of about 150 ms before the last sweep check is in no field there. In a settle, the three `settle_*` times are far below `wall_ms`; the rest is the ticks that write leaves. The ledger roles add up to `process_cpu_ms`.
6. Outcome, as the API reports it: failed when `group.failure_reason` or `group.failure` is present; cancelled when `group.stopped` or `group.cancelled` is present; stopped by the server when `halted_by_stop` or `lost_to_outage` is true; otherwise done. When `sweeps` is 10 the group ran to the sweep bound and the world may still have been changing.
7. Joins: `group.operation` and `group.group` to `confirm.operation`, `saved.last_chain_operation`, `stall.pass.operation`, and `plan.forces_group`; `requester` to `prompt`, `refusal`, and `lookup`; `boot` and `t_ns` for order inside one start.
8. A slow rollback: compare the time before the first write (the plan and the first begin) with the writing time. In the writing, read the share of full ticks, `no_slice_answered`, `credit_stall_wall_ms`, the read-ahead wait, and the capacity waits; then the page rows per edit and `store_after.merge_backlog` of the operation before. For a wrong result, read the accounting fields and `problems`. For a lag spike, read `overrun_max_ms`, `task_max_ms`, `late_ticks`, and the `gc_pause` and `lag` records in the window.
9. Values: pass kinds `pass`, `sweep`, `replay`. The `prompt` record names a prompt `CONFIRM`, `SPEED`, `EDIT_RUNNING`, `FORCE`, or `READ_CONFIG`; `context.plan.prompts[].kind` uses the command's words `CONFIRM_LARGE`, `CONFIRM_WIDE`, `CONFIRM_UNNAMED`, `SPEED`, `EDIT_RUNNING`. `plan.choice` is `normal`, `fast`, `turbo`, or `force`.
10. Units: `lookahead.loads` counts chunks; `lookahead.answer_credit_slices` counts slices of 2 MiB; `allocated_mb` is MiB; tick buckets are upper bounds in ms with an open last bucket. `apply.capacity_gate` holds the time each bound held during the run of its pass, start to end, and is absent for a pass that failed before it opened. In cairns of older builds it ran to the end of the group, so the first pass held the time of its sweeps too. A selection bound `from` of 0 or `to` of 9223372036854775807 means no bound (at or above 2^53, read it as open); `rooted_before` 0 means the selection is not a sweep's.

## Fields of every record

| Field | Unit | Meaning |
| --- | --- | --- |
| `boot` | 8 hex | The id of one start of the plugin. Two records of one boot share one monotonic clock. |
| `seq` | integer | The number of the record in its boot, from 1. Gaps mean dropped records. |
| `at` | ISO instant | The wall time of what the record describes (for a chain, its end; for a lookup, the ask). |
| `t_ns` | ns | System.nanoTime() when the record was made. Subtract only within one boot. |
| `kind` | text | The kind of record; one of the kinds below. |
| `dropped_before` | count | Present when the write queue (8,192 records) was full: the records lost just before this one. |

## Kinds

### `start`

The start of the plugin: versions, machine, JVM, plugins, and worlds. Rare: also kept 12 months in `year/`. Written by `CairnStart.start`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `oasis` | text | The Oasis version. |
| `release` | text | The store release Oasis writes. |
| `paper` | text | Bukkit.getVersion(). |
| `minecraft` | text | The Minecraft version. |
| `data_version` | integer | The world data version. |
| `java` | text | The Java runtime version. |
| `java_vendor` | text | The Java vendor. |
| `jvm` | text | The VM name and version. |
| `processors` | count | Processors the JVM sees. |
| `heap_max_mb` | MB | The most heap the JVM may use. |
| `container_memory_bytes` | bytes | The memory of the machine, or of its container. |
| `engine` | text | embedded or server. |
| `fawe` | bool | Whether FastAsyncWorldEdit is enabled. |
| `online_mode` | bool | The online mode of the server. |
| `plugins` | array | name, version, enabled_at_oasis_start. |
| `worlds` | array | name, environment, view_distance, simulation_distance, auto_save. |
| `jvm_arguments` | text[] | The JVM arguments, a secret value replaced by <redacted>. |
| `gc_collectors` | text[] | The garbage collectors. |
| `os_name` | text | The operating system. |
| `os_version` | text | Its version. |
| `os_arch` | text | Its architecture. |
| `timezone` | text | The time zone. |
| `locale` | text | The locale. |
| `jvm_started_at` | ISO instant | When the JVM started. |
| `jvm_uptime_ms` | ms | The JVM uptime at the record. |

### `config`

The config at the start and after each reload. Secrets (keys, the Packs Server address) show only whether they are set. Rare: also kept 12 months in `year/`. Written by `CairnStart.config, reloaded, reloadFailed`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `reason` | text | start, reload, or reload_failed. |
| `config` | object | Every value by its key in config.yml. |
| `paper_autosave` | object | At the start: Paper's autosave settings (defaults, each world, bukkit_ticks_per_autosave). |
| `changed` | object | At a reload: each key that changed, with old and new. |
| `failure` | text | At a failed reload: why the file broke. |

### `store_open`

The open of the store. Written by `CairnStart.watchOpen`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `open_ms` | ms | Wall time of the open. |
| `since_enable_ms` | ms | Time since the plugin started. |
| `jvm_uptime_ms` | ms | JVM uptime. |
| `store_id` | uuid | The store id. |
| `client_id` | uuid | The client id. |
| `engine` | text | embedded or server. |
| `failure` | text | Why the open failed, or null. |

### `stop`

The stop of the plugin, read before anything closes. Rare: also kept 12 months in `year/`. Written by `OasisPlugin.onDisable`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `since_enable_ms` | ms | Time since the plugin started. |
| `pass_abandoned` | bool | Whether a pass ran and the stop abandoned it. |
| `pass` | pass (live) | The pass the stop abandoned. |
| `rows_pending` | rows | Rows the store still owes. |
| `rows_unsealed` | rows | Rows not sealed. |
| `bytes_pending` | bytes | On Packs Server: bytes not sent. |
| `outbox_bytes` | bytes | Bytes in the outbox. |
| `failure` | text | The failure of the storage, or null. |
| `storage_close_ms` | ms | Wall time of the close of the storage. |
| `day_so_far` | object | The counts of the day so far, as in daily. |

### `error`

One log line of WARNING or above from Oasis or the Packs Server client. Rare: also kept 12 months in `year/`. Written by `CairnEvents.logged`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `level` | text | WARNING or SEVERE. |
| `logger` | text | The logger. |
| `thread` | text | The thread that logged. |
| `message` | text | The message, scrubbed. |
| `thrown` | text | The class of the thrown error. |
| `thrown_message` | text | Its message. |
| `frames` | text[] | The top 8 frames. |
| `causes` | array | Up to 4 causes: thrown, message, at (the top frame). |

### `refusal`

Oasis refused a request. Written by `CairnEvents.refused`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `reason` | text | The reason code. |
| `requester` | uuid | Who asked. |
| `line` | text | The line staff read. |

### `prompt`

Oasis asked staff to confirm. Written by `CairnEvents.prompted`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `prompt` | text | The kind of prompt: CONFIRM, SPEED, EDIT_RUNNING, FORCE, or READ_CONFIG. The chain uses the command's own words for the same prompts in context.plan.prompts[].kind (CONFIRM_LARGE, CONFIRM_WIDE, CONFIRM_UNNAMED, SPEED, EDIT_RUNNING). |
| `requester` | uuid | Who asked. |
| `large` | bool | Over rollback.confirm-above. |
| `unnamed` | bool | The line names no player. |
| `turbo_offered` | bool | The prompt offered turbo. |
| `clickable` | bool | The prompt has a link to accept. |

### `confirm`

The world saved chunks of an operation, and they took its mark. Written by `CairnEvents.confirmed`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `operation` | id | The operation. |
| `chunks` | count | Chunks confirmed now. |
| `chunks_confirmed_so_far_this_start` | count | Chunks confirmed since the start. |
| `since_chain_end_ms` | ms | From the end of the chain, when this start saw it. |

### `saved`

Every finished operation is on disk. Written by `CairnEvents.saved`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `last_chain_operation` | id | The newest chain this start saw. |
| `since_last_chain_end_ms` | ms | From its end to now. |

### `recovery`

The recovery walk at a start ended. Rare: also kept 12 months in `year/`. Written by `CairnEvents.recovered`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `failure` | text | Why it failed, or null. |
| `since_enable_ms` | ms | Time since the plugin started. |
| `jvm_uptime_ms` | ms | JVM uptime. |

### `move`

A move between stores. Written by `CairnEvents.moved`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `move` | text | The step of the move. |
| `requester` | uuid | Who asked. |
| `failure` | text | Why it failed, or null. |
| `retired` | text | The store retired. |
| `state` | object | from, to, known, copying, done_bytes, total_bytes, failure. |

### `import`

A step of a CoreProtect import. Written by `CairnEvents.import*`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `step` | text | config_read, preview, cut, or end. |
| `source` | text | The import source word. |
| `requester` | uuid | Who asked. |
| `rows_left` | rows | Preview: rows left to import. |
| `cut_ms` | epoch ms | Preview: the cut time. |
| `rows` | rows | Cut and end: rows. |
| `cancelled` | bool | End: whether it was cancelled. |
| `failure` | text | Why it failed, or null. |
| `started_ns` | ns | End: the import clock at the start. |
| `cancelled_at_ns` | ns | End: the import clock at the cancel. |
| `ended_ns` | ns | End: the import clock at the end. |
| `wall_ms` | ms | End: wall time. |
| `tables` | array | End: table, imported, rollback_edits, unsupported, unknown, incomplete. |

### `status`

Someone ran /oasis status. Written by `CairnEvents.status`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `requester` | uuid | Who asked. |
| `statistics` | statistics | The store statistics. |
| `history_bytes` | bytes | Bytes of the history. |
| `journal_bytes` | bytes | Bytes of the journal. |
| `other_bytes` | bytes | Other bytes. |
| `fawe_logged` | bool | Whether FAWE edits are logged. |
| `refused_worlds` | object | Each world the store refuses, with why. |

### `world_record`

The world record of a world was saved. Written by `CairnEvents.worldRecordSaved`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `world` | text | The world. |
| `failure` | text | Why it failed, or null. |

### `purge`

A purge: its start, a look every two seconds, and its end. Written by `CairnEvents.purgeStarted, purgeProgress, purged`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `step` | text | start, progress, or end. |
| `requester` | uuid | Who asked. |
| `selection` | selection | Start: what the purge names. |
| `purge` | id | Progress: the purge. |
| `rows_removed` | rows | Progress and end: rows removed. |
| `runs_done` | count | Progress: runs done. |
| `runs_left` | count | Progress: runs left. |
| `done` | bool | Progress: whether it is done. |
| `failure_reason` | text | End: the failure code. |
| `failure` | text | End: the failure. |
| `run_bytes_before` | bytes | Start: bytes of the runs. |
| `run_bytes_after` | bytes | End: bytes of the runs. |
| `logical_bytes` | bytes | Logical bytes. |
| `rows_written` | count | Rows written. |
| `statistics_error` | text | Why the statistics could not be read. |

### `lag`

One tick interval over 250 ms, from the start of one tick to the start of the next. Paper measures the duration of a tick before it calls the tick end handlers, so the interval holds the tick, the tick end work of every plugin (the Oasis pass writes there), and the wait for the next tick. interval_ms - previous_tick_duration_ms - max(previous_tick_time_remaining_ms, 0) is the time lost after the measured work of the tick. Written by `CairnSampler.lag`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `tick` | integer | The tick number. |
| `interval_ms` | ms | The interval. |
| `previous_tick_duration_ms` | ms | What Paper reported for the tick before. |
| `previous_tick_time_remaining_ms` | ms | The time it had left. |
| `gc_pause_collection_ms_in_interval` | ms | Pause time of the collectors in the interval. |
| `gc_pause_collections_in_interval` | count | Pausing collections in the interval. |
| `chunk_loads_in_interval` | count | Chunk loads in the interval. |
| `players_online` | count | Players online. |
| `pass` | pass (live) | The pass that ran, or null. |

### `gc_pause`

One garbage collection pause over 50 ms. `at` is the start of the pause. Written by `CairnSampler.collected`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `collector` | text | The collector. |
| `action` | text | The GC action. |
| `cause` | text | The GC cause. |
| `id` | integer | The collection id. |
| `duration_ms` | ms | The pause. |
| `pools` | object | Each memory pool: used_before_bytes, used_after_bytes, committed_after_bytes. |

### `health`

The health of the server and the store, each minute. A field named since_last_sample covers the window from the last health record. Written by `CairnSampler.health`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `window_ms` | ms | The window since the last sample. |
| `clock_skew_ms_since_last_sample` | ms | How much more the wall clock moved than the monotonic clock. |
| `main_thread_answered` | bool | Whether the server thread answered within 10 s. |
| `main_thread_wait_ms` | ms | The wait for the server thread. |
| `server` | object | tps_1m, tps_5m, tps_15m, mspt_avg_last_100_ticks, tick_ms_max_last_100_ticks, current_tick, players_online, worlds (name, chunks_loaded, entities, tile_entities, players, view_distance, simulation_distance, auto_save). |
| `ticks` | object | Counted since the last sample the server thread answered; absent when it did not answer. intervals_since_last_sample (ticks counted: normalize by it, not by window_ms), interval_ms_max_since_last_sample (the longest start-to-start interval), interval_histogram_since_last_sample (upper_bounds_ms [55, 75, 100, 150, 250, 500, 1000, 2000] and counts; the last bucket is open), intervals_over_55ms/100ms/250ms/1000ms_since_last_sample (late ticks: intervals above the bound; 55 ms is the first histogram bound. Older cairns hold intervals_over_50ms_since_last_sample, which counted on-time ticks too: do not read it), lag_records_since_last_sample, chunk_loads_since_last_sample. |
| `main_thread_stack` | text[] | The stack of the server thread when it did not answer. |
| `storage` | object | The shape "health.storage". Its engine is present on the embedded store only and its packs_server on Packs Server only. |
| `lanes` | array | One entry per storage lane (storage_thread, reader_lane, packs_server_reads): lane, started_since_last_sample (requests started), queue_wait_ms_sum_since_last_sample and queue_wait_ms_max_since_last_sample (ms a request waited for the lane), refused_overloaded_since_last_sample and refused_not_ready_since_last_sample (requests turned away; any count above 0 means staff were told Oasis was busy or not ready). |
| `fawe` | object | The shape "health.fawe". Absent without FAWE. |
| `apply` | object | apply_ms_since_last_sample, applied_rows_since_last_sample, rows_per_apply_ms_estimate. |
| `pass` | pass (live) | The pass that runs, or null. |
| `jvm` | object | The shape "health.jvm", with gc[], memory_pools[], buffer_pools[], threads, and os as shapes of their own. |
| `disk` | array | The shape "health.disk[]": the file system of the plugin folder, the store, and each world. |

### `incident`

A new incident the store raised. `at` is the time of the incident. Rare: also kept 12 months in `year/`. Written by `CairnSampler.incidents`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `text` | text | The incident. |
| `number` | count | Its number in this start. |

### `deadlock`

Threads in a deadlock, with their stacks. Rare: also kept 12 months in `year/`. Written by `CairnSampler.deadlocked`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `threads` | array | name, state, lock, lock_owner, stack. |

### `daily`

The summary of a day, at the first sample after local midnight. Rare: also kept 12 months in `year/`. Written by `CairnSampler.rollover`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `date` | date | The day. |
| `from` | ISO instant | Where the counts start. |
| `to` | ISO instant | Where they end. |
| `health_samples` | count | Health samples. |
| `health_samples_main_thread_unanswered` | count | Samples the server thread did not answer. |
| `mspt_avg_last_100_ticks_max_of_samples` | ms | The highest MSPT of the samples. |
| `tick_interval_ms_max` | ms | The longest tick interval. |
| `tick_intervals_over_250ms` | count | Tick intervals over 250 ms. |
| `tick_intervals_over_1000ms` | count | Tick intervals over 1 s. |
| `lag_records` | count | Lag records. |
| `gc_pauses` | count | GC pauses. |
| `gc_pause_ms_sum` | ms | GC pause time. |
| `gc_pause_ms_max` | ms | The longest GC pause. |
| `incidents` | count | Incidents. |
| `log_warnings` | count | Log warnings. |
| `log_severe` | count | Severe log lines. |
| `chains_ended` | count | Groups that ended. |
| `chains_failed` | count | Groups that failed. |
| `lookups` | count | Lookups. |
| `refusals` | count | Refusals. |
| `packs_server_outages` | count | Lost connections to Packs Server. |

### `stall`

A pass made no progress for the stall time. It holds the queues of the pass and the stacks of the Oasis threads. Written by `OperationApplier (watchdog)`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `no_progress_for_ms` | ms | How long the pass made no progress. |
| `pass` | object | The shape "stall.pass": the queues of the pass now. |
| `storage` | object | The shape "stall.storage": the storage worker now. |
| `threads` | array | Oasis, Packs, and server threads: name, state, lock, lock_owner, stack (24 frames). |

### `chain`

One rollback, restore, or undo group from its start to its end, with every pass and sweep. It is rare when it failed or a recovery ran it. `at` is the end of the group. Written by `OperationApplier.recordChain`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `group` | object | direction, requester, recovered, over_later_edits, turbo, target_tick_ms, engine, started_at (the wall time of the command, before any prompt; with no command, the start of the chain), ended_at, wall_ms (the time of the group itself: it leaves out the prompt wait, so ended_at - started_at = wall_ms + context.plan.prompt_wait_ms), failure, failure_class, cancelled, halted_by_stop, lost_to_outage, operation, group, pass_number, first_pass_wall_ms, passes, sweeps (10 is the bound: the world may still have been changing), edits, edits_by_sweeps, unchanged, left, kept_chunks, newer, rows_walked, rows_total, stopped (text: "cancelled" or the message that stopped the group; stopped of a pass is a bool), failure_reason, owed, sums_over_passes, gc_during_group (collectors, pause_ms), ledger_during_group (ledger), offline_edges_during_group. |
| `context` | object | Absent for a recovery. Who asked for what: shown_as, undone, who, requester, over_later_edits, offers_force, and plan (command, resolve, size, seal, prompts, choice). Through the API: requested_through, forced, target_tick_ms_asked, display. |
| `selection` | selection | The selection. |
| `passes` | pass (chain)[] | Every pass and sweep. |
| `failed_pass` | pass (chain) | The pass that failed before it had a report. |
| `empty_sweep` | object | A final sweep that found no chunk: after, begin, before_sweep (the same fields as before_sweep of a pass). |
| `store_after` | statistics | The store statistics after the group. |
| `store_after_error` | text | Why they could not be read. |

### `lookup`

One lookup page or inspector click. `at` is the ask. Written by `Lookups.record`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `lookup` | text | The kind of lookup. |
| `sender` | text | The name of the sender. |
| `requester` | uuid | The sender. |
| `failure` | text | Why it failed, or null. |
| `request_to_lines_sent_ms` | ms | From the ask to the lines sent. |
| `request_to_event_ms` | ms | From the ask to this record. |
| `label` | text | The label of the session. |
| `selection` | selection | The selection. |
| `at_position` | bool | An inspector lookup at one position. |
| `request_to_page_read_ms` | ms | From the ask to the page in hand. |
| `resolve_ms` | ms | The resolve of the names; null for a page turn. |
| `store_read_ms` | ms | The store's own time for the page. |
| `read_outside_store_ms` | ms | From the call to the store to the page in hand, less store_read_ms. |
| `store_work` | work | The read of the page. |
| `draw_ms` | ms | The drawing of the lines. |
| `page` | count | The page. |
| `page_base` | count | The base of the page. |
| `rows` | count | Rows shown. |
| `reversed` | count | Rows shown as reversed. |
| `more` | bool | Whether more pages follow. |

### `seal`

The embedded store sealed rows into a run. Written by `CairnStore.sealed`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `rows` | rows | Rows sealed. |
| `run_bytes` | bytes | Bytes of the run. |
| `build_ms` | ms | The build. |
| `publish_ms` | ms | The publish. |

### `merge`

The embedded store merged runs. Written by `CairnStore.merged`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `inputs` | count | Runs merged. |
| `rows` | rows | Rows. |
| `ms` | ms | Wall time. |
| `backlog_after` | count | Merges waiting after it. |
| `input_bytes` | bytes | Bytes in. |
| `output_bytes` | bytes | Bytes out. |
| `tier` | integer | 0 to 3, or negative for a rebuild. |
| `tier_name` | text | last_day, last_week, last_month, older, or rebuild. |

### `outage`

The open connection to Packs Server was lost. Rare: also kept 12 months in `year/`. Written by `CairnStore.lost`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `why` | text | Why. |
| `link` | link | The state of the link. |
| `rows_pending` | rows | Rows not sent. |
| `bytes_pending` | bytes | Bytes not sent. |

### `connect`

A handshake with Packs Server. Written by `CairnStore.ready`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `store_id` | uuid | The store Packs Server named. |
| `peer` | object | packs, jar, contract. |
| `link` | link | The state of the link. |
| `outbox_bytes` | bytes | Bytes in the outbox. |

### `share`

A share of the cairn, written when the upload starts. The bundle of that share holds it as its first line, and the service reads the uploader from it. Rare: also kept 12 months in `year/`. Written by `CairnShare.upload`.

| Field | Unit | Meaning |
| --- | --- | --- |
| `share_id` | text | The ID the owner posted, in groups of five. |
| `by` | text | The name of the player who ran /cairn share, or console. |
| `uuid` | uuid | The UUID of that player, or null for the console. |
| `confirmed_at` | ISO instant | When they confirmed the share. |

## Shared shapes

### selection

Every filter of a rollback, restore, lookup, or purge selection (Cairn.selection).

| Field | Unit | Meaning |
| --- | --- | --- |
| `players` | uuid[] | The players named, sorted. Empty means every player. |
| `from` | epoch ms | The start of the time window. |
| `to` | epoch ms | The end of the time window. |
| `worlds` | text[] | The worlds named. |
| `box` | int[6] | minX, minY, minZ, maxX, maxY, maxZ, when the selection has a box. |
| `materials` | text[] | The materials included. |
| `not_materials` | text[] | The materials excluded. |
| `causes` | text[] | The causes included. |
| `not_causes` | text[] | The causes excluded. |
| `kinds` | text[] | The edit kinds included. |
| `not_kinds` | text[] | The edit kinds excluded. |
| `operations` | int[] | The operations named. |
| `rooted_before` | epoch ms | A sweep takes only rows whose root is older than this. |

### work

What one read of the store did (Cairn.work). `ms` is one read, or the sum over reads when the field name says sum. A read of an answer or a lookup page (readWork) leaves out the three count-only fields.

| Field | Unit | Meaning |
| --- | --- | --- |
| `ms` | ms | Wall time of the read, or the sum. |
| `chunks_counted` | count | Chunks counted (count, size, and claim reads only). |
| `run_chunks_covered` | count | Run chunks covered (count, size, and claim reads only). |
| `positions_walked` | count | Positions walked (count, size, and claim reads only). |
| `page_reads` | count | Pages read. |
| `page_rows_read` | count | Rows of the pages read. |
| `pages_decoded` | count | Pages decoded (a page cache miss). |
| `timeline_blocks_decoded` | count | Timeline blocks decoded. |
| `frames_loaded` | count | Frames loaded. |

### statistics

The store statistics, as `/oasis status` reads them (Cairn.store). Held by health.storage.statistics, status.statistics, and chain.store_after.

| Field | Unit | Meaning |
| --- | --- | --- |
| `rows_written` | count | History rows written (rollback edits count too). |
| `rows_last_day` | count | Rows written in the last day. |
| `rows_pending` | count | Rows handed to the store and not yet journaled. |
| `rows_unsealed` | count | Rows journaled and not yet sealed into a run. |
| `players` | count | Players the store knows. |
| `runs` | count | Sealed runs. |
| `run_bytes` | bytes | Bytes of the sealed runs. |
| `logical_bytes` | bytes | Logical bytes of the history. |
| `journal_bytes` | bytes | Bytes of the journal. |
| `cache_bytes` | bytes | Bytes of the page cache. |
| `incidents` | count | Incidents the store raised. |
| `purges_left` | count | Purges not finished. |
| `merge_backlog` | count | Merges waiting. |
| `runs_to_rebuild` | count | Runs waiting for a rebuild. |
| `engine_allocated_bytes_since_start` | bytes | Bytes the engine threads allocated since the start. |
| `engine_gc_pause_ms_since_start` | ms | GC pause time since the start. |
| `page_decodes_since_open` | count | Page decodes since the store opened. |
| `rollbacks_last_week` | count | Rollbacks in the last week. |
| `largest_rollback_30d` | edits | Largest rollback in 30 days. |
| `rollback_rate_30d` | rate | Rollback rate over 30 days. |
| `turbo_last_30d` | bool | Whether a turbo pass ran in the last 30 days. |
| `import_source` | text | The source of the last import, or null. |

### ledger

Where the CPU of the whole process went over an interval (Cairn.ledger). The roles under 1 ms and 1 MB share one line, "other roles", so the lines add up to process_cpu_ms.

| Field | Unit | Meaning |
| --- | --- | --- |
| `wall_ms` | ms | Wall time of the interval. |
| `process_cpu_ms` | ms | CPU of the whole process. |
| `allocated_mb` | MB | Bytes allocated by every thread. |
| `roles` | array | One entry per thread role: role, cpu_ms, allocated_mb, threads. |

### pass (live)

The pass that runs at the moment of the record, or null (CairnSampler.live). Held by lag.pass, health.pass, and stop.pass.

| Field | Unit | Meaning |
| --- | --- | --- |
| `operation` | id | The operation of the pass. |
| `group` | id | The group of the pass. |
| `direction` | text | ROLLBACK, RESTORE, or UNDO. |
| `players` | count | Players the selection names. |
| `phase` | text | The phase the bar shows. |
| `applied_edits` | edits | Edits written so far. |
| `walked_rows` | rows | Rows walked so far. |
| `total_rows` | rows | Rows the pass expects. |
| `exact_total` | bool | Whether total_rows is exact or an upper bound. |
| `applied_chunks` | count | Chunks finished. |
| `total_chunks` | count | Chunks claimed. |

### link

The state of the link to Packs Server, every text scrubbed (CairnStore.diagnostics).

| Field | Unit | Meaning |
| --- | --- | --- |
| `last_error` | text | The last error of the link. |
| `skew_ms` | ms | Clock skew to Packs Server. |
| `outbox_rows` | rows | Rows in the outbox. |
| `outbox_blocks` | count | Blocks in the outbox. |
| `outbox_bytes` | bytes | Bytes in the outbox. |
| `backoff_ms` | ms | The wait before the next connect. |
| `connects_since_open` | count | Connects since the store opened. |
| `losses_since_open` | count | Lost connections since the store opened. |
| `stalls_in_a_row` | count | Stalls in a row. |

### pass (chain)

One pass of a group in chain.passes (CairnPass.json). A stamp that did not come is null, never 0. A field named sum adds steps that can run on several threads, so it can pass the wall time.

| Field | Unit | Meaning |
| --- | --- | --- |
| `kind` | text | The kind of pass: pass (the first), sweep, or replay. |
| `started_at` | ISO instant | Wall time of the request of the pass. |
| `operation` | id | The operation the pass wrote. |
| `group` | id | The group. |
| `pass_number` | count | The number of the pass in its group. |
| `wall_ms` | ms | From the request to the end of the pass. |
| `stopped` | bool | Whether the pass stopped early. |
| `failure_reason` | text | The reason of a failure, or null. |
| `begin` | object | The begin, part after part: write_queue_ms, flush_pump_ms, register_name_ms, open_stream_ms, answer_wait_ms, store_seal_ms, store_queued_ms, store_rebuild_ms, store_claim_ms, store_recorded_ms, store_claim_work (work), unaccounted_ms, start_hop_ms, total_ms. open_stream_ms lies inside answer_wait_ms. |
| `chunks_claimed_live_or_installed_offline` | count | Chunks the pass claimed. |
| `total_rows` | rows | Rows the pass expected. |
| `answers` | object | The answers of the store: first_slice_written_after_start_ms, slice_answer_sum_ms, rows_walked, first_answer_to_write_ms, last_answer_after_start_ms, slices_taken, slice_rows_taken_sum, slice_bytes_taken_sum, store (reads, read_sum_ms, empty_reads, empty_read_sum_ms, read_work_sum, credit_stalls, credit_stall_wall_ms, credit_held_until_release_wall_ms, open_chunks_peak, open_chunks_held_peak_mb, credit_outstanding_peak_mb, entity_prepare_wall_ms, entity_prepare_sum_ms, entity_answers, entity_answer_sum_ms), and slowest_chunks (world, x, z, answer_sum_ms, slices, rows, walked, bytes, answered_to_end), the ten slowest. |
| `apply` | object | The writes: apply_sum_ms, journal_ack_sum_ms, capacity_wait_sum_ms, slices_applied, edits, unchanged, newer, journal_ack_max_ms, capacity_wait_max_ms, capacity_gate (ms each bound held during the run of this pass, start to end; absent for a pass that failed before it opened), answer_to_write_sum_ms, answer_to_write_max_ms. |
| `ticks` | object | The ticks of the pass: ticks, full_ticks, target_ms, task_sum_ms, task_max_ms, apply_sum_ms, claim_sum_ms, rows, late_ticks, late_max, overrun_max_ms, owed_reads_max, owed_acks_max, task_ms_bucket_bounds and task_ms_histogram (the last bucket is open), ticks_not_full_by_cause, refills. |
| `loads` | object | Chunk and world loads: sync_chunk_loads, sync_chunk_load_sum_ms, sync_chunk_load_max_ms, read_ahead_chunk_loads, read_ahead_wait_sum_ms, read_ahead_wait_max_ms, read_ahead_deferred_slices, world_loads, world_load_sum_ms, world_record_wait_ticks. |
| `lookahead` | object | The chunk-load window (loads) and the answer credit in slices (answer_credit_slices): start, cap, final, min, max, mean, fills, fills_at_cap, starved_fills, grows, shrinks, pressures, consumed, idled, latency_mean_ms, latency_max_ms. |
| `phases` | array | The phases before the first write: phase, began_after_request_ms, last_count_after_request_ms, last_done, last_of, count_updates. |
| `routes` | object | live_by_reason (chunks written live, by reason), offline_by_phase, listed_after_start_ms, worlds_ready_after_start_ms, route_after_start_ms, route_store_ms, route_store_sum_ms, route_store_max_ms. |
| `chunk_ends` | object | marked, dropped, left, last_write_to_chunks_done_ms. |
| `slice_continues` | count | Slices continued in a later tick. |
| `settle` | object | The leaf settle: start_after_request_ms, last_write_to_settle_ms, wall_ms, settle_barrier_ms, settle_leaves_sum_ms, settle_to_finish_ms, leaf_batches, leaves_read. |
| `close` | object | wall_ms, hold_ms, record_ms, retries, owes_sweeps, sweep_seal_ms. |
| `before_sweep` | object | For a sweep: sweep_wait_ticks, check_to_fire_ms, hold_ms, preparing_retries, fire_to_request_ms. |
| `problems` | object | newer_rows, unreadable_blocks, unupgraded_blocks, outside_blocks, unreadable_items, unreadable_entities, missing_containers, first_unreadable_block, first_unreadable_item, first_unreadable_entity, turbo, target_tick_ms_asked, replay, unconfirmed_only, failure. |

### health.storage

The state of the storage worker, its statistics, and the engine or the client.

| Field | Unit | Meaning |
| --- | --- | --- |
| `opened` | bool | Whether the store is open. |
| `failure` | text | Present when Oasis stopped logging. Read it first. |
| `rollback_refusal_now` | text | The reason a rollback, restore, or undo started now would be refused: STORE, MOVING, REBUILDING, BUSY, or STARTING while the store is not open yet. Absent when one could start. Older cairns carry refusal_reason_now instead, which read STARTING on a healthy store: do not read it. |
| `moving` | bool | Whether a move between stores runs. |
| `runs_to_rebuild` | count, now | Runs that another build wrote and the store rebuilds before it answers a read. |
| `preparing` | bool | True from a connect to Packs Server until the open turbo preparations resolved. |
| `pass_idle` | bool | True while no pass and no recovery holds the pass gate. |
| `statistics` | statistics | The store statistics of this minute. |
| `statistics_ms` | ms | Wall time of the statistics read. |
| `statistics_error` | text | Present when the read failed or did not end in 10 s. |
| `drain_error` | text | Present when the storage drain failed. |
| `engine` | object | Embedded store only. The shape "health.storage.engine". |
| `packs_server` | object | Packs Server only. The shape "health.storage.packs_server". |

### health.storage.engine

Embedded store only. Reads and writes of the engine since the last sample, and what it holds now. A move or a landing opens another store, whose counts start at zero.

| Field | Unit | Meaning |
| --- | --- | --- |
| `page_reads_since_last_sample` | count | Pages read. |
| `page_rows_read_since_last_sample` | rows | Rows of the pages read. |
| `pages_decoded_since_last_sample` | count | Pages decoded (page cache misses). |
| `closings_decoded_since_last_sample` | count | Closing records decoded. |
| `timeline_blocks_decoded_since_last_sample` | count | Timeline blocks decoded. |
| `candidate_positions_since_last_sample` | count | Positions considered by reads. |
| `column_tables_decoded_since_last_sample` | count | Column tables decoded. |
| `frames_loaded_since_last_sample` | count | Frames loaded. |
| `cache_evictions_since_last_sample` | count | Pages the page cache dropped. |
| `page_cache_hit_ratio_since_last_sample` | 0 to 1 | The share of page reads answered with no decode. Absent when no page was read. |
| `seals_since_last_sample` | count | Seals done. |
| `seal_rows_since_last_sample` | rows | Rows sealed. |
| `seal_run_bytes_since_last_sample` | bytes | Bytes of the runs sealed. |
| `seal_build_ms_sum_since_last_sample` | ms, sum | Build time of the seals. |
| `seal_publish_ms_sum_since_last_sample` | ms, sum | Publish time of the seals. |
| `merges_since_last_sample` | count | Merges done. |
| `rebuilds_since_last_sample` | count | Runs rebuilt. |
| `merge_rows_since_last_sample` | rows | Rows merged. |
| `merge_ms_sum_since_last_sample` | ms, sum | Time of the merges. |
| `merge_input_bytes_since_last_sample` | bytes | Bytes the merges read. |
| `merge_output_bytes_since_last_sample` | bytes | Bytes the merges wrote. |
| `journal_force_count_since_last_sample` | count | Journal syncs (fsync). |
| `journal_force_ms_sum_since_last_sample` | ms, sum | Time of the journal syncs. |
| `journal_force_ms_max_since_last_sample` | ms | The longest journal sync. |
| `journal_drained_count_since_last_sample` | count | Journal drains. |
| `seal_waits_since_open` | count, total | Waits for a seal since the store opened. |
| `seal_wait_ms_ended_since_open` | ms, total | Time of the seal waits that ended. |
| `seal_wait_ms_open_now` | ms, now | Time of the seal wait that has not ended. |
| `merge_holds_now` | count, now | Holds on the merge thread now (a running pass holds it). |
| `tier_runs_now` | count[4], now | Runs by tier: last day, last week, last month, older. |
| `tier_bytes_now` | bytes[4], now | Bytes of the runs by the same tiers. |
| `rows_pending_now` | rows, now | Rows handed to the journal and not yet journaled. FAWE waits above 262,144. |
| `rows_unsealed_now` | rows, now | Rows journaled and not yet sealed. FAWE waits above 2,097,152. |

### health.storage.packs_server

Packs Server only. The client of Packs Server, as the Paper side sees it.

| Field | Unit | Meaning |
| --- | --- | --- |
| `reachable` | bool | Whether the link is up. |
| `rows_pending` | rows, now | Rows sent and not yet journaled by Packs Server. |
| `bytes_pending` | bytes, now | Bytes of those rows. FAWE waits above 16 MiB. |
| `rows_unsealed` | rows, now | Rows journaled and not yet sealed on Packs Server. |
| `rows_acknowledged_since_open` | rows, total | Rows Packs Server acknowledged. |
| `outbox_bytes` | bytes, now | Bytes the link could not send yet. |
| `incompatible` | text | Present when this build and Packs Server cannot work together. |
| `peer` | object | packs (the engine version), jar (the jar), contract (the contract list). |
| `link` | link | The state of the link. |
| `requests_since_last_sample` | array | Per request kind: code, name, count, first_answer_ms_sum (ms, sum), first_answer_ms_max (ms). The mean is the sum divided by the count. |

### health.fawe

Absent without FAWE. The FAWE hook counters. The waits are the time FAWE threads waited on the unsealed bound; they land on FAWE threads by design.

| Field | Unit | Meaning |
| --- | --- | --- |
| `read_failures_since_start` | map, total | Count by kind of read FAWE gave Oasis and Oasis could not use. |
| `sessions_by_cause_since_last_sample` | map | Edit sessions by cause. Only causes with a count show. |
| `handovers_inline_since_last_sample` | count | Hand-overs of rows made on the server thread. |
| `rows_inline_since_last_sample` | rows | Rows handed over on the server thread. |
| `handovers_bulk_since_last_sample` | count | Hand-overs of rows made off the server thread. |
| `rows_bulk_since_last_sample` | rows | Rows handed over off the server thread. |
| `by_actor_since_last_sample` | array | Per actor that handed rows over: actor (UUID), handovers, rows. |
| `queue_waits_since_last_sample` | count | Waits of FAWE threads for room in the store. |
| `queue_wait_ms_sum_since_last_sample` | ms, sum | Time of those waits, over all threads, so it can pass the window. |
| `queue_wait_ms_max_since_last_sample` | ms | The longest single wait. |

### health.jvm

The JVM of this minute. The pieces below have their own shapes.

| Field | Unit | Meaning |
| --- | --- | --- |
| `uptime_ms` | ms, total | JVM uptime. |
| `gc` | array | The shape "health.jvm.gc[]". |
| `gc_pauses_since_last_sample` | count | Pauses of the collectors that pause the world. |
| `gc_pause_ms_sum_since_last_sample` | ms | Time of those pauses. |
| `gc_pause_ms_max_since_last_sample` | ms | The longest of those pauses. |
| `heap` | bytes, now | used_bytes, committed_bytes, max_bytes (-1 means undefined). |
| `non_heap` | bytes, now | The same three figures for the memory outside the heap. |
| `memory_pools` | array | The shape "health.jvm.memory_pools[]". |
| `buffer_pools` | array | The shape "health.jvm.buffer_pools[]". |
| `threads` | object | The shape "health.jvm.threads". |
| `deadlocked` | array | Present only during a deadlock: name, state, lock, lock_owner, stack. |
| `ledger_since_last_sample` | ledger | Where the CPU went. Absent in the first sample of a boot. |
| `classes_loaded` | count, now | Loaded classes. Growth after a plugin reload is a leak. |
| `jit_ms_since_last_sample` | ms, sum | Compile time summed over compiler threads, so it can pass the window. Absent when the JVM does not report it. |
| `os` | object | The shape "health.jvm.os". |

### health.jvm.gc[]

One entry per collector bean.

| Field | Unit | Meaning |
| --- | --- | --- |
| `name` | text | The collector, such as G1 Young Generation or ZGC Pauses. |
| `concurrent` | bool | True for a cycle bean of ZGC or Shenandoah (name holds "Cycles"), which does not pause the world. The "G1 Concurrent GC" bean is false: it reports the Remark and Cleanup pauses. |
| `count_since_last_sample` | count | Collections in the window. |
| `time_ms_since_last_sample` | ms | Collection time in the window. |
| `count_since_jvm_start` | count, total | Collections since the JVM started. |
| `time_ms_since_jvm_start` | ms, total | Collection time since the JVM started. |

### health.jvm.memory_pools[]

One entry per memory pool. Old Gen used is the live set; Eden and Survivor are churn. Metaspace, Compressed Class Space, and the CodeHeap pools are outside the heap. All figures are gauges.

| Field | Unit | Meaning |
| --- | --- | --- |
| `name` | text | The pool, such as G1 Old Gen. |
| `type` | text | HEAP or NON_HEAP. |
| `used_bytes` | bytes, now | Bytes in use. |
| `committed_bytes` | bytes, now | Bytes the JVM reserved from the system. |
| `max_bytes` | bytes, now | The most the pool may grow to; -1 means undefined. |

### health.jvm.buffer_pools[]

One entry per buffer pool: direct (Packs, Netty), mapped, and mapped - non-volatile memory.

| Field | Unit | Meaning |
| --- | --- | --- |
| `name` | text | The pool. |
| `count` | count, now | Buffers in the pool. |
| `used_bytes` | bytes, now | Memory the buffers use. |
| `capacity_bytes` | bytes, now | Total capacity of the buffers. |

### health.jvm.threads

The threads of the JVM.

| Field | Unit | Meaning |
| --- | --- | --- |
| `live` | count, now | Live threads, daemon and not. |
| `peak_since_jvm_start` | count, total | The most live threads at once. |
| `daemon` | count, now | Live daemon threads. |
| `started_since_jvm_start` | count, total | Threads started. Its growth is thread churn. |

### health.jvm.os

The operating system as the JVM reads it. The gauges are of the moment of the sample.

| Field | Unit | Meaning |
| --- | --- | --- |
| `load_average_1m` | number, now | Load average of one minute; -1 on Windows. |
| `process_cpu_load_recent` | 0 to 1, now | CPU share of this process over all CPUs. Noisy. |
| `system_cpu_load_recent` | 0 to 1, now | CPU share of the whole machine. Noisy. |
| `free_memory_bytes` | bytes, now | Free memory. On Linux the page cache keeps it low, so low alone is no alarm. |
| `total_memory_bytes` | bytes, now | Memory of the machine, or of its container. |
| `free_swap_bytes` | bytes, now | Free swap. Compare it with total_swap_bytes; older cairns hold no total, so no swap and a full swap both read 0 there. |
| `total_swap_bytes` | bytes, now | Swap size. 0 means no swap. |
| `committed_virtual_memory_bytes` | bytes, now | Virtual memory the process committed. |
| `open_file_descriptors` | count, now | Open file descriptors. Unix only. |
| `max_file_descriptors` | count, now | The limit of file descriptors. Unix only. |

### health.disk[]

One entry per place: plugins/Oasis, plugins/Oasis/store, and world:<name> for each world. When the server thread did not answer, the worlds are missing.

| Field | Unit | Meaning |
| --- | --- | --- |
| `place` | text | The place. |
| `exists` | bool | Present as false when the folder does not exist. |
| `file_store` | text | The device or volume. |
| `type` | text | The file system type. |
| `usable_bytes` | bytes, now | Space this process may use. |
| `total_bytes` | bytes, now | Size of the file system. |
| `unallocated_bytes` | bytes, now | Unallocated space. |
| `read_only` | bool | Whether the file system is read only. |
| `error` | text | Present when the place could not be read. |

### stall.pass

The queues of the pass at the moment of the stall. Counts are sizes of the queues now. The pass is the one that has made no progress.

| Field | Unit | Meaning |
| --- | --- | --- |
| `operation` | id | The operation. |
| `kind` | text | pass, sweep, or replay. |
| `phase` | text | The phase the bar shows. |
| `ticks` | count | Ticks the pass ran. |
| `open` | count | Chunks open. |
| `ready` | count | Slices ready to write. |
| `writing` | bool | Whether a slice is being written. |
| `incoming` | count | Answers on their way. |
| `read_ahead` | count | Chunk loads read ahead and pending. |
| `read_ahead_window` | count | The chunk-load window. |
| `routing` | count | Chunks being routed. |
| `routed` | count | Chunks routed. |
| `worlds_ready` | count | Worlds ready for the route. |
| `routes_sent` | count | Route requests sent. |
| `route_requests` | count | Route requests open. |
| `route_replies` | count | Route replies waiting. |
| `unresolved` | count | Chunks with no route yet. |
| `offline` | count | Chunks on the offline (turbo) lane. |
| `preparations` | count | Open offline preparations. |
| `preparing_count` | count | Preparations in progress. |
| `offline_saves` | count | Offline saves pending. |
| `stream_ended` | bool | Whether the answer stream ended. |
| `skipping` | count | Chunks skipped. |
| `edge_checks` | count | Edge reads pending. |
| `edge_writes` | count | Edge payments pending. |
| `settling` | bool | Whether the leaf settle runs. |
| `leaf_pages_ahead` | count | Leaf pages read ahead. |
| `leaf_reading` | bool | Whether a leaf page is being read. |
| `leaves_ended` | bool | Whether the leaf rows ended. |
| `evacuation_pending` | bool | Whether players are still being moved out. |
| `stopped` | bool | Whether the pass was stopped. |
| `failure` | text | Present when the pass failed. |
| `first_slice_written` | bool | Whether a slice was written. |
| `applied_edits` | edits | Edits written so far. |
| `finished_chunks` | count | Chunks finished. |
| `chunks` | count | Chunks claimed. |
| `closing` | bool | Whether the pass began to close. |
| `close_held` | bool | Whether the close waits on a hold. |
| `close_recorded` | bool | Whether the close was recorded. |
| `answer_stream` | object or text | The state of the answer stream (the keys are names). On Packs Server it is a text, as the state is on the server; "not open" when no stream is open. |

### stall.storage

The storage worker at the moment of the stall.

| Field | Unit | Meaning |
| --- | --- | --- |
| `reads_in_flight` | count | Reads in flight. |
| `handed_over_not_taken` | count | Blocks the server thread handed over and the worker did not take. |
| `open_streams` | count | Open pass streams. |
| `pass_gate_held` | bool | Whether a pass or a recovery holds the pass gate. |
| `preparing` | bool | Whether open preparations wait to resolve. |
| `unavailable` | text | The failure text, when the store is unavailable. |
| `sealed_capacity_held` | count or text | Durable acknowledgements held for capacity, read on the worker thread. A text when the worker did not answer in 2 s. |

## What the service accepts

Each line is a JSON object with `boot` (8 hex), `seq` (a positive integer), `at` (an ISO instant),
`t_ns` (an integer), and a `kind` from the list above. Keys match `^[a-z0-9_]{1,64}$`, except
under `config`, `changed`, `paper_autosave`, `refused_worlds`, `pools`, `live_by_reason`, `offline_by_phase`, `read_failures_since_start`, `sessions_by_cause_since_last_sample`, `answer_stream`, whose keys are names of up to
64 characters. Depth is at most 8 and arrays hold at most 10,000 items. Strings hold no control
characters and are at most 256 characters, except under
`message`, `thrown_message`, `frames`, `stack`, `main_thread_stack`, `jvm_arguments`, `failure`, `why`, `text`, `line`, `last_error`, `statistics_error`, `store_after_error`, `drain_error`, `incompatible`, `unavailable`, `retired`, `first_unreadable_block`, `first_unreadable_item`, `first_unreadable_entity`, `command`, `label`, `from`, `to`, which may be 64 KB and hold tabs and newlines.
No string holds a run of 120 or more base64 characters. Each `boot`+`seq` appears once.