LLM Observability (LangSmith, Arize, Galileo) vs. Pre-Execution Control
Tracing token latency and prompt chains is essential for debugging—but observing an unauthorized €250,000 ERP mutation after it commits is an incident report, not a control.
Key Takeaways
- LLM observability platforms excel at prompt debugging, eval scoring, and latency tracing during development.
- As documented in ADR-0001, passive telemetry cannot block an unsafe write, pause for maker-checker approval, or prevent duplicate retries.
- ActEscrow sits inline on the write path and exports signed receipts to your observability and SIEM stack.
1. Business & threat context
Engineering teams building with LangChain, LlamaIndex, or CrewAI rely heavily on LangSmith, Arize, or Galileo to inspect agent trajectories. When an agent hallucinates a tool parameter in staging, trace viewers make root-cause analysis fast.
In production finance workflows, however, passive telemetry happens after or alongside execution. An observability SDK cannot hold the sole credential to SAP, pause an HTTP mutation for 4 hours while two finance managers review an IBAN diff, or reconcile an HTTP 504 timeout against the ERP's idempotency log.
ADR-0001: Enforcement, not observability
Keep your existing LLM eval and trace tools. Add ActEscrow solely on the write path to high-consequence business APIs.
2. How ActEscrow solves it (Service 01)
ADR-0001 defines our strict scope boundary: we do not compete with prompt trace visualizers. Instead, your agent keeps its existing LangSmith or OpenTelemetry tracing while routing high-risk writes through the ActEscrow gateway for pre-action blocking, approval, and cryptographic receipt generation.
HTTP/1.1 202 Accepted
Content-Type: application/json
{
"actionId": "act_019a83f2c4e1",
"status": "PendingApproval",
"intentDigest": "sha256:7c91e3a2...09f4",
"requiredApprovals": 2
}3. Production implementation checklist
When evaluating LangSmith / Arize Phoenix / Galileo alongside a customer-owned enforcement boundary, engineering and finance leaders should verify the following operational controls:
- Use LangSmith/Arize for prompt quality, token cost, and trajectory debugging
- Route state-mutating ERP/payment calls exclusively through the Enforcement Gateway
- Correlate OpenTelemetry trace IDs between your agent traces and gateway evidence receipts
A timeline: observing versus blocking
Consider a hypothetical agent that misreads an email and decides to approve invoice INV-1002 for 250000 minor units EUR. With observability alone, the trace shows the model call, the tool call and the ERP response within the same second. An alert may fire a minute later. By then the invoice is approved in the ERP and the next step of the payment run may already have picked it up.
With a pre-execution gateway on the write path, the same tool call arrives at the gateway, matches a RequireApproval rule and returns 202 with a pending status. Nothing has changed in the ERP. The trace still shows everything, and now it also shows a pending action id that a human can reject. The difference is not better telemetry; it is that the write waits for a decision.
Correlating traces with evidence receipts
The two systems are most useful when an engineer can move from one to the other. The gateway participates in OpenTelemetry traces, so the action id and trace id can be recorded together. Keep the split of content clear, though: traces may contain prompts and model output, which is why they belong in your observability tool, while evidence events contain only ids, digests and states.
- Observability: prompts, completions, token costs, latency, eval scores.
- Gateway evidence: action id, digest, policy version, decision, approver ids, outcome.
- Shared key: trace id and action id, so either view can link to the other.
- Never copy bank details or other personal data into evidence to make traces line up.
A simple rule for what lives where
If a field helps an engineer understand why the model did something, it belongs in traces. If it helps an auditor understand who authorized a change and what exactly changed, it belongs in evidence. Keeping that line clear avoids copying prompts into the evidence chain and avoids treating trace retention as an audit record.
Security scope and deployment prerequisites
This is a pilot implementation, not a zero-risk guarantee or compliance certification. Examples describe repository capabilities or deployment recommendations, not proof that a live installation has enabled them.
Every enrolled write fails closed during control-plane outages. Privileged control-plane MFA is mandatory outside Development, but accepted MFA claims do not prove recent authentication or hardware-backed WebAuthn unless the identity provider enforces those properties.
Per-agent deactivation blocks new evaluations and redeems and cancels Pending/Approved actions; it cannot recall an already executing ERP write. Evidence verifies recorded events, not unrecorded ERP activity; independently retained heads and trusted keys are needed to detect truncation or replacement.
Production blockers remain
The static portal still uses sessionStorage tokens (BFF not implemented). Off-host backup signing, immutable backup storage, live IdP MFA/logout testing, restore drills, adapter validation, monitoring, and legal review require deployment-specific work. Anchoring and SIEM forwarding are optional and must be configured and tested. See the repository deployment guide and runbook gap register before using real customer data.
Related deep dives
iPaaS Connectors (Workato, Zapier, Make) vs. Fail-Closed Agent Gateways
Nov 19, 2026 · 3 min readArchitectureAWS Bedrock AgentCore Policy vs. Customer-Owned Action Gateways for Multi-Cloud Finance
Nov 15, 2026 · 3 min readSecurity & AuditGRC Automation (Vanta, Drata) vs. Cryptographic Decision Evidence Vaults
Nov 20, 2026 · 3 min read