Awesome Agent Traceability

Open tools, standards and papers for building AI agent systems whose actions can be traced, audited and replayed.

An agent that acts on real systems should leave a record a person can follow: what it was asked, what it called, what it changed, under which policy, and how to run it again. This list collects open work on that record: tracing and observability, durable and replayable execution, policy enforcement, tamper-evident logs and provenance, and the research that frames them.

Entries are chosen for what they make traceable, not for popularity. Each description says what the project does, in neutral words.

Contents

• Standards and specifications
• Regulation, frameworks and policy
• Tracing and observability
• Gateways and proxies
• Durable and replayable execution
• Policy, guardrails and governance
• MCP and tool security
• Sandboxes and isolation
• Agent memory and state
• Evaluation with recorded runs
• Provenance and tamper-evident logs
• Papers
• From the maintainers

Standards and specifications

• Agent2Agent (A2A) Protocol - Open protocol for communication and interoperability between agent applications, with defined task and message objects exchanged between agents.
• C2PA specification - Specification for attaching signed provenance manifests to media, including content produced or edited by AI.
• CycloneDX ML-BOM - CycloneDX bill-of-materials format for machine learning models, datasets and their provenance.
• in-toto Attestation Framework - Specification for signed, typed statements about software artifacts, used to carry provenance such as SLSA predicates.
• Model Context Protocol specification - Specification of the protocol through which models and agents discover and call external tools, resources and prompts.
• OpenInference - Semantic conventions and OpenTelemetry instrumentation for tracing LLM calls, retrievals, tools and agent steps.
• OpenLineage - Open standard for collecting lineage metadata about jobs, runs and the datasets they read and write.
• OpenTelemetry semantic conventions for agent spans - Span names and attributes for creating and invoking agents, part of the OpenTelemetry generative AI conventions.
• OpenTelemetry semantic conventions for generative AI - Shared attribute and span names for model calls, agent steps and tool invocations, so traces from different frameworks can be read the same way.
• SCITT - IETF working group on Supply Chain Integrity, Transparency and Trust, defining signed statements registered in append-only transparency services.
• SLSA - Framework of levels and requirements for build provenance, describing how an artifact was produced and by what.
• W3C PROV-O - W3C ontology for expressing provenance: which agent carried out which activity that used or generated which entity.

Regulation, frameworks and policy

• EU AI Act, Article 12: Record-keeping - Requires high-risk AI systems to allow the automatic recording of events (logs) over their lifetime.
• EU AI Act, Article 14: Human oversight - Requires high-risk AI systems to be designed so that natural persons can effectively oversee them while in use.
• MITRE ATLAS - Knowledge base of adversary tactics and techniques against AI-enabled systems, drawn from real-world observations and demonstrations.
• NIST AI 100-2 E2025 - NIST taxonomy and terminology of attacks on machine learning systems and their mitigations.
• NIST AI 600-1 - Generative AI profile of the NIST AI Risk Management Framework, listing risks specific to generative AI and suggested actions.
• NIST AI Risk Management Framework - Voluntary framework organised around the Govern, Map, Measure and Manage functions for managing risks of AI systems.
• NIST AI RMF Playbook - Suggested actions, references and documentation practices for each outcome of the NIST AI Risk Management Framework.
• OECD AI Principles - Intergovernmental principles for trustworthy AI, including transparency, explainability and accountability.
• OWASP Agentic AI Threats and Mitigations - OWASP GenAI Security Project guide to threats specific to agentic AI systems and their mitigations.
• OWASP Top 10 for LLM Applications - OWASP list of the most critical security risks for applications built on large language models.

Tracing and observability

• AgentOps - Python SDK for agent monitoring that records agent sessions, LLM calls and costs across common agent frameworks.
• AgentSight - System-level observability for AI agents that uses eBPF to capture model traffic and process activity from outside the agent.
• Evidently - Open-source framework to evaluate, test and monitor ML and LLM systems, with tracing for LLM applications.
• Helicone - Open-source LLM observability platform for monitoring, evaluating and experimenting with logged requests.
• Jaeger - CNCF distributed tracing platform that stores and visualises OpenTelemetry traces.
• Laminar - Open-source observability platform for AI agents, with traces of agent steps and model calls.
• Langfuse - Open-source platform for tracing, evaluating and debugging LLM applications and agents, with self-hosting.
• Langtrace - Open-source, OpenTelemetry-based tracing and evaluation tool for LLM applications and frameworks.
• LangWatch - Open-source platform for tracing, evaluating and testing LLM agents.
• Logfire - Pydantic’s OpenTelemetry-based observability SDK for LLM and agent systems.
• MLflow Tracing - MLflow’s tracing records inputs, outputs and intermediate steps of LLM and agent runs alongside experiments and models.
• OpenAI Agents SDK - Multi-agent framework with built-in tracing of agent runs, model calls, tool calls, handoffs and guardrails.
• OpenLIT - OpenTelemetry-native platform to trace, evaluate and guard LLM and agent applications.
• OpenLLMetry - OpenTelemetry-based instrumentation for LLM providers, vector stores and agent frameworks.
• OpenTelemetry Collector - Vendor-neutral service to receive, process and export traces, metrics and logs, including those from instrumented LLM applications.
• Opik - Comet’s open-source tool to trace, evaluate and monitor LLM applications, RAG systems and agentic workflows.
• Phoenix - Arize’s open-source observability and evaluation tool for LLM applications, built on OpenTelemetry traces.
• RagaAI Catalyst - Python SDK for agent observability that traces agent, LLM and tool calls and shows them as a timeline and execution graph.
• SigNoz - OpenTelemetry-native observability platform for traces, metrics and logs.
• Tempo - Grafana’s distributed tracing backend that stores traces in object storage.
• TruLens - Library for evaluating and tracking LLM experiments and agents, with tracing of each step.
• Weave - Weights & Biases toolkit for tracing and evaluating AI-powered applications.

Gateways and proxies

• agentgateway - Proxy for agent-to-tool and agent-to-agent traffic over MCP and A2A, with authorisation policy and observability.
• LiteLLM - Gateway and SDK that calls many LLM APIs in one format, with logging callbacks, cost tracking and guardrails.
• Portkey Gateway - AI gateway that routes requests to many LLM providers, with guardrails and request logging.
• Traceloop Hub - LLM gateway written in Rust that emits OpenTelemetry traces for each request.

Durable and replayable execution

• Burr - Apache library for building decision-making applications as state machines, with persisted state and a UI to trace and replay each step.
• Cadence - Durable orchestration engine for long-running workflows, with a recorded workflow history.
• Dapr Agents - Framework for agents built on Dapr workflows, with durable execution, state and telemetry.
• DBOS Transact - Python library for durable workflows that checkpoints each step in a PostgreSQL database.
• Golem - Platform for durable agents and applications that persists execution state so work resumes without loss or duplication.
• Hatchet - Orchestration engine for background tasks, AI agents and durable workflows.
• Inngest - Workflow orchestration platform for durable, stateful step functions and AI workflows.
• LangGraph - Agent orchestration library with persisted checkpoints, which allow inspecting, resuming and replaying a run from a saved state.
• Restate - Platform for durable, fault-tolerant services and workflows that journals each step so execution resumes after failures.
• Temporal - Durable execution service whose workflow event history records every step and lets a workflow be replayed deterministically.
• Trigger.dev - Platform for building and running durable AI agents and workflows in TypeScript, with run logs and traces.

Policy, guardrails and governance

• Agent Governance Toolkit - Microsoft’s toolkit for agent policy enforcement, zero-trust identity, execution sandboxing and reliability engineering.
• AI Verify - Testing toolkit from the AI Verify Foundation for checking AI systems against governance principles and producing reports.
• Cedar - Policy language and evaluation engine for authorisation decisions that can be analysed and audited.
• Guardrails - Python framework that validates LLM inputs and outputs against declared validators.
• Invariant - Guardrails for agent systems, written as rules over agent traces and tool calls.
• NeMo Guardrails - NVIDIA’s toolkit for programmable rails on the inputs, outputs and tool use of LLM applications.
• Open Policy Agent - General-purpose policy engine with a declarative language and decision logs that record each policy decision.
• Presidio - Framework for detecting and redacting personal data in text, images and structured data, useful before traces and logs are stored.
• Purple Llama - Meta’s set of tools, including Llama Guard and CyberSecEval, to assess and improve LLM security.

MCP and tool security

• agent-audit-kit - Offline static scanner for MCP-connected agent pipelines that checks configuration and server source against detection rules and writes SARIF.
• agent-scan - Security scanner for AI agents, MCP servers and agent skills.
• MCP Inspector - Visual testing tool for MCP servers that calls their tools, resources and prompts and shows the responses.
• mcp-audit - Proxy for MCP clients and servers that records signed audit trails for tool calls and other protocol messages, with redaction and policy controls.
• ToolHive - Platform for running and managing MCP servers in isolated containers with access controls.

Sandboxes and isolation

• E2B - Open-source sandboxed cloud environments for running agent-generated code.
• Firecracker - Virtual machine monitor for lightweight microVMs that isolate workloads with hardware virtualisation.
• gVisor - Application kernel that runs containers behind a user-space reimplementation of the Linux system call interface.
• Kata Containers - Lightweight virtual machines that run as containers, giving each workload its own kernel.
• Landlock - Linux security module that lets an unprivileged process restrict its own filesystem and network access.
• microsandbox - Local-first microVM runtime and library for running untrusted code.
• nsjail - Process isolation tool built on Linux namespaces, cgroups, resource limits and seccomp-bpf filters.
• Sandbox Runtime - Anthropic’s tool for enforcing filesystem and network restrictions on arbitrary processes at the OS level, without a container.
• Wasmtime - WebAssembly runtime that runs code with only the capabilities the host grants it.

Agent memory and state

• Graphiti - Framework for temporally aware knowledge graphs for agents, which record when each fact held and the episode it came from.
• Letta - Platform for stateful agents whose memory and state are persisted and can be inspected.
• Mem0 - Memory layer for AI agents that stores, updates and retrieves memories, with a history of changes to each memory.

Evaluation with recorded runs

• Agenta - Open-source workspace for building agents and LLM applications with prompt versioning, evaluation and tracing.
• AgentDojo - Environment for evaluating prompt injection attacks and defences on tool-using LLM agents.
• ControlArena - UK AI Security Institute’s settings and protocols for AI control experiments, built on Inspect.
• DeepEval - Framework for unit-testing LLM outputs and agents with metrics, with tracing of components.
• garak - LLM vulnerability scanner that probes models and logs every attempt and result.
• Giskard - Open-source evaluation and testing library for LLM agents.
• HELM - Stanford CRFM framework for holistic, reproducible and transparent evaluation of foundation models.
• Inspect - The UK AI Security Institute’s framework for LLM and agent evaluations, which writes a log of every sample, message and tool call.
• Language Model Evaluation Harness - EleutherAI framework for evaluating language models on many tasks, with options to log every sample.
• OpenAI Evals - Framework and registry of benchmarks for evaluating LLMs and LLM systems.
• promptfoo - CLI and library for testing and red-teaming prompts, agents and RAG pipelines with stored, comparable results.
• PyRIT - Microsoft’s Python Risk Identification Tool for red-teaming generative AI systems, with stored conversations and scores.
• Ragas - Library for evaluating LLM applications, including retrieval-augmented generation and agents.
• τ-bench - Benchmark of tool-agent-user interaction in simulated customer service domains.

Provenance and tamper-evident logs

• c2pa-rs - Rust SDK for reading, writing and validating C2PA provenance manifests.
• Cosign - Sigstore tool for signing and verifying containers, binaries and attestations, with entries recorded in a transparency log.
• Dolt - SQL database with Git-style version control: every write is a commit that can be diffed, branched and reverted.
• DVC - Data Version Control: versions data, models and experiments alongside Git so a result can be reproduced.
• immudb - Database with an immutable, cryptographically verifiable history of every change.
• in-toto - Framework that records signed attestations for each step of a supply chain so the whole chain can be verified.
• lakeFS - Version control for data lakes, with commits, branches and reproducible snapshots of object storage.
• Marquez - Reference implementation of OpenLineage that collects and visualises job and dataset lineage.
• Model Transparency - Library and CLI for signing and verifying machine learning models.
• Rekor - Sigstore’s append-only, verifiable transparency log for signed software supply-chain metadata.
• SLSA Verifier - Tool for verifying SLSA build provenance against an artifact.
• SynthID Text - Google DeepMind’s reference implementation of a watermark for generated text and its detector.
• Tessera - Go library for building tile-based transparency logs.
• Trillian - Verifiable data store built on Merkle trees, used to run transparency logs.

Papers

• A Survey on Trustworthy LLM Agents: Threats and Countermeasures - Yu et al., 2025. Surveys threats to LLM agents and the countermeasures proposed for them.
• Agent Security Bench (ASB) - Zhang et al., 2024. Formalises and benchmarks attacks and defences for LLM-based agents.
• Agent-SafetyBench: Evaluating the Safety of LLM Agents - Zhang et al., 2024. Benchmark of interaction environments and test cases for the safety of LLM agents.
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents - Debenedetti et al., 2024. Introduces an extensible environment of tasks and attacks for tool-using agents.
• AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems - Chan et al., 2024. Captures agent inputs and outputs at each step to predict task performance and correct unsafe behaviour.
• AgentOps: Enabling Observability of LLM Agents - Dong et al., 2024. Taxonomy of the artifacts and data to trace across the lifecycle of LLM agents.
• AI Control: Improving Safety Despite Intentional Subversion - Greenblatt et al., 2023. Evaluates protocols, such as trusted monitoring, that keep a model safe even if it tries to subvert them.
• Auditing large language models: a three-layered approach - Mökander et al., 2023. Proposes governance, model and application audits as complementary layers.
• Black-Box Access is Insufficient for Rigorous AI Audits - Casper et al., 2024. Argues that audits need white-box and outside-the-box access, such as to training and deployment information.
• Closing the AI Accountability Gap - Raji et al., 2020. Defines an end-to-end framework for internal algorithmic auditing across the development lifecycle.
• Datasheets for Datasets - Gebru et al., 2018. Proposes documenting each dataset’s motivation, composition, collection process and recommended uses.
• Defeating Prompt Injections by Design - Debenedetti et al., 2025. Introduces CaMeL, which separates control and data flows so untrusted data cannot change an agent’s program.
• Fully Autonomous AI Agents Should Not be Developed - Mitchell et al., 2025. Argues that risks grow with agent autonomy and that human control should be kept.
• Governing AI Agents - Kolt, 2025. Applies principal-agent theory and agency law to the governance of AI agents, including visibility and monitoring.
• Identifying the Risks of LM Agents with an LM-Emulated Sandbox - Ruan et al., 2023. Introduces ToolEmu, which emulates tool execution to find risky agent behaviour.
• IDs for AI Systems - Chan et al., 2024. Examines identifiers attached to AI system instances so that their outputs and actions can be attributed.
• Infrastructure for AI Agents - Chan et al., 2025. Frames attribution, interaction and response as infrastructure functions for agents acting in the world.
• Model Cards for Model Reporting - Mitchell et al., 2018. Proposes short documents that report a model’s intended use, evaluation and limitations.
• Non-Determinism of “Deterministic” LLM Settings - Atil et al., 2024. Measures how much LLM outputs vary across runs with settings meant to be deterministic.
• R-Judge: Benchmarking Safety Risk Awareness for LLM Agents - Yuan et al., 2024. Benchmarks whether LLMs can judge safety risks from records of agent interactions.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains - Yao et al., 2024. Measures agent reliability over repeated trials with simulated users and domain policies.
• The Data Provenance Initiative - Longpre et al., 2023. Audits the licensing and attribution of fine-tuning datasets and traces their lineage.
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims - Brundage et al., 2020. Recommends institutional, software and hardware mechanisms, including audit trails, for verifiable claims about AI systems.
• TRiSM for Agentic AI - Raza et al., 2025. Reviews trust, risk and security management for LLM-based multi-agent systems.
• Visibility into AI Agents - Chan et al., 2024. Proposes agent identifiers, real-time monitoring and activity logging as measures for visibility into deployed agents.
• Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents - Rombaut et al., 2024. Recovers the implicit reasoning behind agent actions to make it observable for debugging.
• Which Agent Causes Task Failures and When? - Zhang et al., 2025. Defines automated failure attribution, finding the agent and step responsible from multi-agent logs.
• Why Do Multi-Agent LLM Systems Fail? - Cemri et al., 2025. Builds a taxonomy of multi-agent failure modes from annotated execution traces.

From the maintainers

This list is maintained by 100monkeys.ai. We build the three projects below; they are listed here, separately, so that the entries above stay neutral.

• AEGIS by 100monkeys.ai - AGPL-3.0, self-hosted agent orchestrator, in beta, that enforces manifest-declared network and filesystem policy in the orchestrator and runs agents in isolated runtimes with durable workflow state.
• Carbon Forge - Hosted source control and shared record for people and agents, where every revision and changeset is hash-chained and a landing is one all-or-nothing changeset. Public workspaces are free, and every account includes one free private workspace.
• Zaru - Apache-2.0 terminal agent that writes every model reply, tool call, permission decision and tool result to a plain-text transcript. Pre-alpha.

Contributing

Contributions are welcome. Read CONTRIBUTING.md first.