Distributed Tracing with OpenTelemetry

When two agents run in separate processes and call each other over HTTP, OpenTelemetry’s W3C traceparent header carries the trace context across the wire. Permitech joins all spans that share a trace ID into a single trace.

If both services are Python and use the Permitech SDK directly, Distributed Tracing (Beta) is simpler. Use the OTel approach when your services use different agent frameworks (Microsoft Agent Framework, Google ADK, LangChain, etc.) or different languages.

What you’ll see in Permitech

A single user request to Agent A that delegates to Agent B produces one trace:

POST /ask                              ← Agent A: FastAPI server span (workflow)
  invoke_agent OrchestratorAgent       ← Agent A: MAF agent span
    HTTP POST /research                ← Agent A: requests client span (workflow)
      POST /research                   ← Agent B: FastAPI server span (workflow)
        invoke_agent research_agent    ← Agent B: ADK agent span
          gemini.generate_content      ← Agent B: LLM span

Setup

.env

OPENAI_API_KEY=sk-...  # find your key at https://platform.openai.com/api-keys
GOOGLE_API_KEY=...   # find your key at https://aistudio.google.com

PERMITECH_API_KEY=...
PERMITECH_PROJECT=otel-distributed-tracing
PERMITECH_LOG_STREAM=default

AGENT_B_URL=http://localhost:8001

requirements.txt

fastapi
uvicorn[standard]
requests
python-dotenv
permitech[otel]
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-requests
agent-framework
agent-framework-openai
google-adk
openinference-instrumentation-google-adk

PermitechSpanProcessor reads PERMITECH_API_KEY, PERMITECH_PROJECT, and PERMITECH_LOG_STREAM from the environment and exports OTLP directly to Permitech’s OTel endpoint. No collector required.

Agent A: Microsoft Agent Framework, calls Agent B

agent_a.py

import os
import requests
from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel

from permitech.otel import PermitechSpanProcessor, add_permitech_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

from agent_framework import openai, tool
from agent_framework.observability import enable_instrumentation

load_dotenv()

# Set the global tracer provider before any instrumentor runs.
tracer_provider = TracerProvider()
add_permitech_span_processor(tracer_provider, PermitechSpanProcessor())
trace.set_tracer_provider(tracer_provider)

# injects traceparent on outbound HTTP
RequestsInstrumentor().instrument()
# MAF built-in OTel
enable_instrumentation(enable_sensitive_data=True)

AGENT_B_URL = os.environ["AGENT_B_URL"]


@tool(approval_mode="never_require")
def delegate_research(topic: str) -> str:
    """Ask the research agent for facts about a topic."""
    r = requests.post(f"{AGENT_B_URL}/research", json={"topic": topic}, timeout=60)
    r.raise_for_status()
    return r.json()["findings"]


orchestrator = openai.OpenAIChatClient(model="gpt-4.1-mini").as_agent(
    name="OrchestratorAgent",
    instructions="Use delegate_research to gather facts, then answer the user clearly.",
    tools=[delegate_research],
)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)


class AskRequest(BaseModel):
    question: str


@app.post("/ask")
async def ask(req: AskRequest):
    return {"answer": str(await orchestrator.run(req.question))}

Agent B: Google ADK research agent

agent_b.py

from dotenv import load_dotenv
from fastapi import FastAPI
from pydantic import BaseModel

from permitech.otel import PermitechSpanProcessor, add_permitech_span_processor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

from openinference.instrumentation.google_adk import GoogleADKInstrumentor
from google.adk.agents.llm_agent import Agent
from google.adk.runners import InMemoryRunner
from google.genai.types import Content, Part

load_dotenv()

tracer_provider = TracerProvider()
add_permitech_span_processor(tracer_provider, PermitechSpanProcessor())
trace.set_tracer_provider(tracer_provider)

GoogleADKInstrumentor().instrument(tracer_provider=tracer_provider)

agent = Agent(
    model="gemini-2.5-flash",
    name="research_agent",
    instruction="Return 3-5 concise key facts about the topic.",
)
runner = InMemoryRunner(agent=agent, app_name="otel-dt-demo")

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)


class ResearchRequest(BaseModel):
    topic: str


@app.post("/research")
async def research(req: ResearchRequest):
    session = await runner.session_service.create_session(
        app_name="otel-dt-demo", user_id="user"
    )
    findings = ""
    # Drain the full async generator. Returning early closes it from a
    # different asyncio context and breaks OTel context-token cleanup.
    async for event in runner.run_async(
        user_id="user",
        session_id=session.id,
        new_message=Content(role="user", parts=[Part(text=req.topic)]),
    ):
        if event.is_final_response() and event.content and event.content.parts:
            findings = event.content.parts[0].text
    return {"findings": findings}

Run it

pip install -r requirements.txt

# Terminal 1
uvicorn agent_b:app --port 8001

# Terminal 2
uvicorn agent_a:app --port 8000

# Terminal 3
curl -s -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "Tell me about Voyager 1."}' | jq

Open the otel-distributed-tracing project in Permitech, Log stream default. You’ll see one trace covering both agents:

Distributed trace in Permitech console showing both Agent A (Microsoft Agent Framework) and Agent B (Google ADK) spans nested under one trace

Self-hosted Permitech

The OTel endpoint is different from Permitech’s regular API endpoint and is specifically designed to receive telemetry data in the OTLP format. If you are using:

Permitech Cloud at permi.tech, then you don’t need to provide a custom OTel endpoint. The default endpoint https://permi.tech/api/otel/traces will be used automatically.
A self-hosted Permitech deployment, replace the https://permi.tech/api/otel/traces endpoint with your deployment URL. The format of this URL is based on your console URL, replacing console with api and appending /otel/traces.

For example:

if your console URL is https://console.permitech.example.com, the OTel endpoint would be https://api.permitech.example.com/otel/traces
if your console URL is https://console-permitech.apps.mycompany.com, the OTel endpoint would be https://api-permitech.apps.mycompany.com/otel/traces

Set PERMITECH_API_ENDPOINT in .env; PermitechSpanProcessor picks it up automatically.

Distributed Tracing (Beta): Permitech’s native Python SDK distributed mode.
Microsoft Agent Framework integration
Google ADK (OpenTelemetry) integration
: End-to-end runnable example with a Java Spring Boot gateway and a Python FastAPI + LangGraph RAG service.

Overview

Get Started

Observability

Evaluation Metrics

Signals

AI Assistant

Luna Studio

Experiments

Agent Control

Annotations

Integrations

Security

References

Distributed Tracing with OpenTelemetry

What you’ll see in Permitech

Setup

Agent A: Microsoft Agent Framework, calls Agent B

Agent B: Google ADK research agent

Run it

Self-hosted Permitech

​What you’ll see in Permitech

​Setup

​Agent A: Microsoft Agent Framework, calls Agent B

​Agent B: Google ADK research agent

​Run it

​Self-hosted Permitech

​Related

What you’ll see in Permitech

Setup

Agent A: Microsoft Agent Framework, calls Agent B

Agent B: Google ADK research agent

Run it

Self-hosted Permitech

Related