Building an Incident Triage Agent with Full Observability in SigNoz

Short summary

A developer built an incident triage agent for the SigNoz hackathon that chains LLM and tool calls to investigate incidents, score severity, and recommend actions. Each step is instrumented with OpenTelemetry spans, enabling dashboards for latency bottlenecks and severity distribution. The key takeaway: agent observability transforms opaque LLM pipelines into debuggable, measurable systems.

• •Incident triage agent chains 5 LLM/tool calls, each wrapped in OpenTelemetry spans
• •Built with Python, Gemini API, and SigNoz for full trace-level observability
• •Dashboards reveal per-step latency and severity patterns across runs

Generated with AI, which can make mistakes.