Monitoring & Costs
Track agent performance in real time and understand costs at the agent, department, and organization level. Monitor latency, error rates, model usage, and budget utilization.
Performance metrics
Clevername tracks key performance indicators for every agent session: latency (time to first response and total execution time), error rate (percentage of failed work items), throughput (work items completed per hour), and tool usage(which MCP tools and skills are called most frequently).
Cost tracking
Costs are tracked per model call through LiteLLM's telemetry. Each work item's cost is attributed to the agent that processed it and rolled up to the department and organization level. Costs include input tokens, output tokens, and any per-call fees from the underlying provider.
Budget enforcement
Budgets exist at two levels. Per agent: questionnaire answer A1 compiles to budget_limit_usd on the agent's circuit breaker, and the agent detail page has a budget panel with a Test Budget and a Prod Budget (run cap and spend cap in dollars) that admins can edit. When a cap is exhausted the agent page shows a budget-paused banner. Per department: a department created through the API can carry a budget cap; when it is exhausted, new work items for that department are refused (HTTP 402). There is no department budget screen in the dashboard.
Open the monitoring dashboard
Navigate to AI Company in the dashboard. The Overview stats bar shows Agents, Governed, Learning and Alerts for the selected window. For spend, open AI Company → Usage (/dashboard/ai-company/usage): the Usage & Cost view shows Total Requests, Tokens In, Tokens Out and Estimated Cost, split across Governed AI, Shadow AI and Human traffic.
View agent-level metrics
Click on an agent card to see its performance metrics: session count, average latency, error rate, and recent work item results. The agent detail view also shows a timeline of recent sessions with their outcomes.
Review department costs through the API
There is no department screen in the dashboard. For a department roll-up, call GET /hub/ai-company/departments/{dept_id}/costs?org_id=…, which returns spend for the agents filed under that department.
Check organization-wide telemetry
For a holistic view, use the telemetry endpoints in the hub API: GET /telemetry/summary (totals for a period), GET /telemetry/costs (cost breakdown) and GET /telemetry/by-agent (per-agent usage).
get_telemetry_costs and get_telemetry_summary tools in the Clevername MCP server wrap the same endpoints, so an assistant can pull telemetry into your own reports.Set per-agent budgets
Set a monthly spend cap in questionnaire answer A1, or edit the Test Budget / Prod Budget panel on the agent detail page (run cap and spend cap). There are no percentage-threshold alerts; when a cap is exhausted the agent is paused and the detail page shows a budget-paused banner.