Monitoring & Costs
Track agent performance in real time and understand costs at the agent, department, and organization level. Monitor latency, error rates, model usage, and budget utilization.
Performance metrics
Clevername tracks key performance indicators for every agent session: latency (time to first response and total execution time), error rate (percentage of failed work items), throughput (work items completed per hour), and tool usage(which MCP tools and skills are called most frequently).
Cost tracking
Costs are tracked per model call through LiteLLM's telemetry. Each work item's cost is attributed to the agent that processed it and rolled up to the department and organization level. Costs include input tokens, output tokens, and any per-call fees from the underlying provider.
Budget enforcement
Departments can have monthly budgets. When a department's cumulative spend approaches the budget limit, escalation rules determine the response — alerts, throttling, or hard stops. Budget utilization is displayed on department cards and in the monitoring dashboard.
Open the monitoring dashboard
Navigate to AI Company in the dashboard. The top-level stat cards show aggregate metrics. For detailed monitoring, click into a specific agent or department.
View agent-level metrics
Click on an agent card to see its performance metrics: session count, average latency, error rate, and recent work item results. The agent detail view also shows a timeline of recent sessions with their outcomes.
Review department cost breakdown
Open a department's detail view and navigate to the Budget tab. This shows:
- Budget vs. actual spend — bar chart comparing allocated budget to current spend.
- Cost by agent — which agents in the department are consuming the most budget.
- Cost by model — breakdown of spend across different AI models.
- Trend — daily or weekly spend trend over the current billing period.
Check organization-wide telemetry
For a holistic view, use the telemetry endpoints available through the hub API. The telemetry summary shows total costs, request counts, and error rates across all agents and departments for a given time period.
get_telemetry_costs and get_telemetry_summary MCP tools to pull telemetry data programmatically into your own reports.Set up budget alerts
Configure escalation rules on each department to receive alerts at budget thresholds — for example, at 75% and 90% utilization. Alerts can be sent through notification channels (email, Slack) or surfaced as escalations in the dashboard.