Langfuse

Langfuse is an open-source LLM observability platform — providing tracing, monitoring, cost tracking, and evaluation for AI applications in production. It's the standard starting point for teams that need visibility into what their AI systems are actually doing.

Buy vs Build

Strong buy story: Langfuse Cloud (hosted) gets you up and running in minutes. Strong build story for data-sensitive teams: self-host via Docker with the open-source version. Both options use the same SDK.

Why Engineering Managers Care About LLM Observability

Traditional application monitoring (APM) tells you a request returned a 200 status and took 230ms. That's not enough for AI systems. An LLM can return a 200 in 230ms and still:

• Give completely wrong information
• Generate unsafe content
• Waste tokens on irrelevant reasoning
• Fail silently on an edge case it was never tested on

Langfuse adds the missing layer: it captures what the model was asked, what it answered, how much it cost, and whether the answer was actually good.

What It Tracks

• Traces: The full input → model call → output for each request, including nested tool calls in agentic workflows
• Token costs: Per-request and aggregated cost breakdown across models and users
• Latency: Time to first token, total response time, queue time
• Evaluation scores: Automated LLM-as-a-judge scoring or human annotation via the Langfuse UI
• Sessions: Grouping related traces into a conversation or user session

Key Features

• Works with any LLM framework (LangChain, LangGraph, CrewAI, direct API calls)
• Python and TypeScript SDKs with framework-specific integrations
• Self-hostable (Docker, no external dependencies beyond Postgres + Redis)
• Prompt management: version, A/B test, and deploy prompts from the UI without a code deploy

Getting Started

pip install langfuse

from langfuse.decorators import observe

@observe()  # automatically traces this function
def generate_summary(text: str) -> str:
    response = anthropic.messages.create(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": f"Summarise: {text}"}]
    )
    return response.content[0].text

Key Characteristics

Further Reading

• Langfuse documentation
• Langfuse GitHub (6K+ stars)
• LLM observability concepts