Observability: what the stack records about an agent, and what it cannot tell you

Parts 3 and 4 asked for evidence: logs kept six months, a person who approved, an action that stayed within its purpose. This part follows one tool call through the stack and looks at what actually gets written down, by which component, in which format, and where it ends up. Four records exist. Together they say what happened and whether it was allowed. None of them can prove it to someone who does not trust the operator.

The four records

1. The trace. The agent harness or the OGX server emits OpenTelemetry spans following the GenAI semantic conventions: one span per model invocation, per tool execution, per agent step, with attributes for the model name, the token counts, the finish reason, optionally the content. This is the richest record and the only one that shows the reasoning chain as a tree. Two caveats. The conventions were still marked "Development" in 2026, so attribute names can change between library versions. And content capture is off by default in most instrumentations: turning it on stores prompts and answers, which is a data-protection decision, not a logging one.

2. The gateway's tool-call log. The MCP gateway of part 1 writes one structured log entry per tool call: the caller's subject from the JWT, the tool, the server, the HTTP status, a request id and an anonymised session id, flagged audit=true, with the trace id attached and exported over OTLP. This is the record that answers "who called what on which system", and it is produced by the choke point itself, not by the agent. It is not in OCSF format; it is a structured log.

3. The verdict and the approval. This is the record telemetry does not produce. The governance toolkit of part 4 writes, for each action, the rule that fired, the verdict, and when a human was involved, who approved, with the digest of the exact action they saw. The entries go into a Merkle tree with a root hash and inclusion proofs, so an auditor can check that a given entry was present and unchanged. It arrives by its own path, not through the collector, and it has to carry the same trace id as records 1 and 2 or it cannot be joined to them.

4. The sandbox's security events. Below the tool call, the OpenShell supervisor emits OCSF events for what the process actually did: files opened, connections attempted, processes spawned, and the policy decision on each. OCSF is the schema SIEMs already ingest, so this record lands where the security team already looks. It answers a question the other three cannot: whether the code behind the tool touched anything it was not supposed to.

What an auditor asks, and which record answers

Where the records are switched on

The files that create record 3, the manifest with its intervention points, the approval section, the intent, and the sector templates, are shown one by one in part 4. Two settings belong here instead, because they decide where the records go rather than what is decided. Both are read from the repository at commit 2e98345 of 21 September 2026.

# The demos point the SDK's OpenTelemetry exporter at a collector by environment variable
environment:
  - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317

# The agent-sre Helm chart exposes the same thing as values
otel:
  enabled: true
  endpoint: http://otel-collector:4317
  serviceName: agent-sre

# Two exports on the audit log: a JSON document that carries the Merkle root with the
# entries, and the same entries as CloudEvents v1.0 envelopes for a bus or a SIEM
def export(self, start_time=None, end_time=None) -> dict:
    entries = self.query(start_time=start_time, end_time=end_time, limit=10000)
    return {
        "exported_at": ...,                          # UTC timestamp of the export
        "merkle_root": self._chain.get_root_hash(),
        "entry_count": len(entries),
        "entries": [e.model_dump() for e in entries],
    }

def export_cloudevents(self, start_time=None, end_time=None) -> list[dict]:
    return [e.to_cloudevent() for e in entries]

Both defaults are conservative. The OpenTelemetry exporter in the Helm chart is off until an endpoint is given, so record 1 does not exist until someone decides where it lives. And the sample policy of part 4 keeps tool arguments out of the audit, include_tool_args: false, because arguments carry personal data: record 3 identifies the action by its digest, not by its content.

The retention problem nobody budgets for

Article 26(6) asks deployers to keep the automatically generated logs for at least six months. Trace backends are built for debugging, and their default retention is measured in days; Prometheus in weeks. Keeping six months of GenAI spans, with or without content, is a storage decision that has to be taken on purpose: a cold tier for traces, the tool-call log and OCSF events shipped to the SIEM the organisation already retains, and the audit tree backed up with its root hashes. The stack produces the records. Keeping them is the operator's job, and it is where the first audit usually finds the first gap.

What a Merkle tree proves, and what it does not

The toolkit's audit tree lets anyone with the root hash verify that an entry belongs to the tree and has not changed. That is real: an operator cannot quietly edit an entry after the fact without changing the root. What the tree does not do is prove to a third party when the root was produced, or that the tree the auditor is shown is the one that existed at the time. The operator holds the root; the operator could regenerate the whole tree. The toolkit's own code says it does not anchor to a transparency log. This is not a flaw specific to that toolkit; it is the limit of any integrity mechanism that stays inside one organisation.

The way out is known from software supply chains: publish the root, or a signed statement about it, to a log the operator does not control, and keep the receipt. Sigstore's Rekor already does this for build artefacts, the IETF's SCITT working group is standardising the receipt format, and RFC 3161 timestamps have done a simpler version of it for twenty years. None of these is wired into the agent stack of parts 1 and 2. For a regulated deployment, it is the last piece of the evidence chain, and the one to plan for before the first audit rather than after.