Connect
Works with the MCP server your analytics agent already uses
Add it to honest_agent_config.yml with a URL or a local command, and pick the model your agent runs on. One target per agent, like a dbt profile.
honest-agent
Any model
Anthropic or OpenAI
default_target: motherducktargets: motherduck: mcp_url: https://api.motherduck.com/mcp bearer_token_env: MOTHERDUCK_TOKEN model: claude-sonnet-4-5
default_target: snowflaketargets: snowflake: mcp_url: https://<account>.snowflakecomputing.com/api/v2/databases/<db>/schemas/<schema>/mcp-servers/<server> bearer_token_env: SNOWFLAKE_MCP_TOKEN model: claude-sonnet-4-5
default_target: sales-agenttargets: sales-agent: mcp_url: https://mcp.yourcompany.com/mcp bearer_token_env: MCP_TOKEN model: claude-sonnet-4-5
default_target: demotargets: demo: mcp_command: uv run python -m honest_agent.demo_server warehouse.duckdb default_database: warehouse default_schema: main model: claude-sonnet-4-5
$ honest-agent run --target motherduck
$ honest-agent run --target snowflake
$ honest-agent run --target sales-agent
$ honest-agent run --target demo
Close the loop, from test to action
Explorer
evals
example_evals.yml
honest_agent_config.yml
.env
evals/example_evals.yml
version: 1evals: - category: sales grading: method: extract_match # code compares the value min_score: 0.8 provenance: min_score: 0.7 tests: - id: revenue_q4 title: Revenue in Q4 2026 prompt: What was our revenue in Q4 2026, net of discounts and before tax? expected_answer: "307992639.49" provenance: expected_sources: [lineitem, warehouse.main.orders] - id: order_total_2026 title: Order total in 2026 expected_answer: "35055813985.22"
~/my-analytics-project $ honest-agent run
Target: sales-agent (honest_agent_config.yml)
Starting eval run 7d2e41a0-3c55-4f0b-9a61-2b8e5d90c417 (4 evals)...
Evaluating agent: sales-agent
Model: claude-sonnet-4-5 · grading model: claude-sonnet-4-5
- revenue_q4: 'What was our revenue in Q4 2026, net of discounts and before tax?'
- order_total_2026: 'What was the total price of all the orders placed in 2026?'
- top_segment: 'Which market segment has the most customers?'
- top_nations_last_year: 'Which 5 nations placed the most orders last year?'
Wrote 4 result(s) to ./honest_agent_results/results.duckdb
2/4 eval(s) below threshold this run:
- revenue_q4: accuracy 0.00 < 0.8, provenance 0.50 < 0.7
- order_total_2026: provenance 0.50 < 0.7
Run 7d2e41a0-3c55-4f0b-9a61-2b8e5d90c417 complete. Next: `honest-agent report` / `honest-agent notify`.
~/my-analytics-project $ honest-agent report
Wrote honest_agent_report/. View it with `honest-agent serve --out honest_agent_report`.
~/my-analytics-project $ honest-agent serve
Serving honest_agent_report/ at http://127.0.0.1:8000/ (Ctrl+C to stop)
Overview
Last eval run Oct 10, 08:02
sales-agentclaude-sonnet-4-5
Overall
73.8%
−2.9 ppvs previous run
Accuracy
72.5%
+2.5 ppvs previous run
Provenance
75.0%
−8.3 ppvs previous run
Passed thresholds
2 / 4
2 below min score
Scores by eval run
Hover a cell to see its scores
CombinedAccuracyProvenance
| Category / Eval | 2026-10-0710-07 | 2026-10-0810-08 | 2026-10-0910-09 | 2026-10-1010-10 | Overall |
|---|---|---|---|---|---|
sales 2 evals | sales 2026-10-07 Combined75.0% Accuracy100.0% Provenance50.0% Mostly right | sales 2026-10-08 Combined100.0% Accuracy100.0% Provenance100.0% Correct | sales 2026-10-09 Combined75.0% Accuracy100.0% Provenance50.0% Mostly right | sales 2026-10-10 Combined50.0% Accuracy50.0% Provenance50.0% Partly right | sales Overall Combined75.0% Accuracy87.5% Provenance62.5% Mostly right |
revenue_q4 Revenue in Q4 2026 | revenue_q4 2026-10-07 No run | revenue_q4 2026-10-08 No run | revenue_q4 2026-10-09 No run | revenue_q4 2026-10-10 Combined25.0% Accuracy0.0% Provenance50.0% Wrong | revenue_q4 Overall Combined25.0% Accuracy0.0% Provenance50.0% Wrong |
order_total_2026 Order total in 2026 | order_total_2026 2026-10-07 Combined75.0% Accuracy100.0% Provenance50.0% Mostly right | order_total_2026 2026-10-08 Combined100.0% Accuracy100.0% Provenance100.0% Correct | order_total_2026 2026-10-09 Combined75.0% Accuracy100.0% Provenance50.0% Mostly right | order_total_2026 2026-10-10 Combined75.0% Accuracy100.0% Provenance50.0% Mostly right | order_total_2026 Overall Combined81.2% Accuracy100.0% Provenance62.5% Mostly right |
customers 2 evals | customers 2026-10-07 Combined65.0% Accuracy50.0% Provenance80.0% Partly right | customers 2026-10-08 Combined77.5% Accuracy55.0% Provenance100.0% Mostly right | customers 2026-10-09 Combined77.5% Accuracy55.0% Provenance100.0% Mostly right | customers 2026-10-10 Combined97.5% Accuracy95.0% Provenance100.0% Correct | customers Overall Combined79.4% Accuracy63.7% Provenance95.0% Mostly right |
top_segment Market segment with the most customers | top_segment 2026-10-07 Combined100.0% Accuracy100.0% Provenance100.0% Correct | top_segment 2026-10-08 Combined100.0% Accuracy100.0% Provenance100.0% Correct | top_segment 2026-10-09 Combined100.0% Accuracy100.0% Provenance100.0% Correct | top_segment 2026-10-10 Combined100.0% Accuracy100.0% Provenance100.0% Correct | top_segment Overall Combined100.0% Accuracy100.0% Provenance100.0% Correct |
top_nations_last_year Nations with the most orders last year | top_nations_last_year 2026-10-07 Combined30.0% Accuracy0.0% Provenance60.0% Wrong | top_nations_last_year 2026-10-08 Combined55.0% Accuracy10.0% Provenance100.0% Partly right | top_nations_last_year 2026-10-09 Combined55.0% Accuracy10.0% Provenance100.0% Partly right | top_nations_last_year 2026-10-10 Combined95.0% Accuracy90.0% Provenance100.0% Correct | top_nations_last_year Overall Combined58.8% Accuracy27.5% Provenance90.0% Partly right |
Overall | Overall 2026-10-07 Combined68.3% Accuracy66.7% Provenance70.0% Partly right | Overall 2026-10-08 Combined85.0% Accuracy70.0% Provenance100.0% Mostly right | Overall 2026-10-09 Combined76.7% Accuracy70.0% Provenance83.3% Mostly right | Overall 2026-10-10 Combined73.8% Accuracy72.5% Provenance75.0% Partly right | Overall Overall Combined75.9% Accuracy69.8% Provenance82.1% Mostly right |
Below threshold
Latest eval run, Oct 10 · 2 evals failed accuracy or provenance min score
| Eval | Category | Accuracy | Provenance | |
|---|---|---|---|---|
Revenue in Q4 2026 revenue_q4 | sales | 0.0% | 50.0% | AccuracyProvenance |
Order total in 2026 order_total_2026 | sales | 100.0% | 50.0% | Provenance |
# analytics-alertsAlerts when the analytics agent fails its checks
Today
honest-agentAPP8:04 AM
🔴 2 of 4 evals below threshold
sales-agent · claude-sonnet-4-5 · run started 2026-10-10 08:02 UTC
Open the report · honest-agent
👀 2✅ 1
MR 1 reply Last reply today at 8:21 AM
Message #analytics-alerts
BI@☺
1Define
Define what a right answer looks like
Every eval is a question you already know the answer to. honest-agent always checks that the answer is correct (accuracy), and can also check that it came from the right sources (provenance).
contains
Exact text or numbers
Passes when the expected answer appears in the reply. No model call.
evals: - category: customers grading: method: contains tests: - id: top_segment title: Market segment with the most customers prompt: Which market segment has the most customers? expected_answer: BUILDING
extract_match
Numbers and single values
A model pulls the value out of the agent's answer, then code compares it with the expected one, within an optional tolerance.
evals: - category: sales grading: method: extract_match min_score: 0.8 tests: - id: revenue_q4 title: Revenue in Q4 2026 prompt: What was our revenue in Q4 2026, net of discounts and before tax? expected_answer: "307992639.49" grading: tolerance: 0.01 # an absolute difference that still counts as equal
llm_judge
Free-text answers
The grading model scores the agent's answer against the expected one, from 0 to 1, and explains why.
evals: - category: customers grading: method: llm_judge min_score: 0.8 tests: - id: top_nations_last_year title: Top 5 nations by orders last year prompt: Which 5 nations placed the most orders last year? expected_answer: "Germany, France, Brazil, Japan and Canada placed the most orders." provenance: expected_sources: [orders, customer, nation]
Provenance
expected_sources
Checks that your agent queried the sources you expect.
evals: - category: sales provenance: min_score: 0.7 tests: - id: revenue_q4 title: Revenue in Q4 2026 prompt: What was our revenue in Q4 2026, net of discounts and before tax? expected_answer: "307992639.49" provenance: expected_sources: [lineitem, warehouse.main.orders] # database.schema.table
~/my-analytics-project $ honest-agent run
Target: sales-agent (honest_agent_config.yml)
Starting eval run 7d2e41a0-3c55-4f0b-9a61-2b8e5d90c417 (4 evals)...
Evaluating agent: sales-agent
Model: claude-sonnet-4-5 · grading model: claude-sonnet-4-5
- revenue_q4: 'What was our revenue in Q4 2026, net of discounts and before tax?'
- order_total_2026: 'What was the total price of all the orders placed in 2026?'
- top_segment: 'Which market segment has the most customers?'
- top_nations_last_year: 'Which 5 nations placed the most orders last year?'
Wrote 4 result(s) to ./honest_agent_results/results.duckdb
2/4 eval(s) below threshold this run:
- revenue_q4: accuracy 0.00 < 0.8, provenance 0.50 < 0.7
- order_total_2026: provenance 0.50 < 0.7
Run 7d2e41a0-3c55-4f0b-9a61-2b8e5d90c417 complete. Next: `honest-agent report` / `honest-agent notify`.
~/my-analytics-project $ honest-agent run --select category:sales
Target: sales-agent (honest_agent_config.yml)
Starting eval run b81f06c2-94d7-4e3a-8f25-61c0a7e3d952 (2 evals)...
Evaluating agent: sales-agent
Model: claude-sonnet-4-5 · grading model: claude-sonnet-4-5
- revenue_q4: 'What was our revenue in Q4 2026, net of discounts and before tax?'
- order_total_2026: 'What was the total price of all the orders placed in 2026?'
Wrote 2 result(s) to ./honest_agent_results/results.duckdb
2/2 eval(s) below threshold this run:
- revenue_q4: accuracy 0.00 < 0.8, provenance 0.50 < 0.7
- order_total_2026: provenance 0.50 < 0.7
Run b81f06c2-94d7-4e3a-8f25-61c0a7e3d952 complete. Next: `honest-agent report` / `honest-agent notify`.
~/my-analytics-project $ honest-agent run --select tag:critical
Target: sales-agent (honest_agent_config.yml)
Starting eval run 2c9a7e15-6b04-4d8f-b3e1-9f50d2a6c8b7 (2 evals)...
Evaluating agent: sales-agent
Model: claude-sonnet-4-5 · grading model: claude-sonnet-4-5
- revenue_q4: 'What was our revenue in Q4 2026, net of discounts and before tax?'
- top_segment: 'Which market segment has the most customers?'
Wrote 2 result(s) to ./honest_agent_results/results.duckdb
1/2 eval(s) below threshold this run:
- revenue_q4: accuracy 0.00 < 0.8, provenance 0.50 < 0.7
Run 2c9a7e15-6b04-4d8f-b3e1-9f50d2a6c8b7 complete. Next: `honest-agent report` / `honest-agent notify`.
~/my-analytics-project $ honest-agent run --select revenue_q4
Target: sales-agent (honest_agent_config.yml)
Starting eval run e45b3d98-1a7c-4c26-8e0f-73d1b9a2f604 (1 eval)...
Evaluating agent: sales-agent
Model: claude-sonnet-4-5 · grading model: claude-sonnet-4-5
- revenue_q4: 'What was our revenue in Q4 2026, net of discounts and before tax?'
Wrote 1 result(s) to ./honest_agent_results/results.duckdb
1/1 eval(s) below threshold this run:
- revenue_q4: accuracy 0.00 < 0.8, provenance 0.50 < 0.7
Run e45b3d98-1a7c-4c26-8e0f-73d1b9a2f604 complete. Next: `honest-agent report` / `honest-agent notify`.
~/my-analytics-project $ honest-agent report
Wrote honest_agent_report/. View it with `honest-agent serve --out honest_agent_report`.
~/my-analytics-project $
~/my-analytics-project $ honest-agent serve
Serving honest_agent_report/ at http://127.0.0.1:8000/ (Ctrl+C to stop)
127.0.0.1 - - [12/Oct/2026 12:40:02] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [12/Oct/2026 12:40:02] "GET /assets/index-9rCiGK-N.js HTTP/1.1" 200 -
127.0.0.1 - - [12/Oct/2026 12:40:02] "GET /assets/index-DEJ_0_VZ.css HTTP/1.1" 200 -
127.0.0.1 - - [12/Oct/2026 12:40:03] "GET /data/report.json HTTP/1.1" 200 -
127.0.0.1 - - [12/Oct/2026 12:40:03] "GET /logo-mark.svg HTTP/1.1" 200 -
2Run
Run your evals in three simple commands
Run your evals, build the report, and open it in your browser.
The full flow
→→
Run all your evals, or pick a category, a tag or a single eval.
Builds the report from your recent runs into a folder, honest_agent_report by default.
Opens the report in your browser, or host it for your team.
3Review
See what failed, and why
The report shows every run of your evals, compares models, and opens each answer.
# analytics-alertsAlerts when the analytics agent fails its checks
Today
honest-agentAPP8:04 AM
🔴 2 of 4 evals below threshold
sales-agent · claude-sonnet-4-5 · run started 2026-10-10 08:02 UTC
Open the report · honest-agent
MR
Maya R.8:21 AM
The agent filtered by ship date and never read orders. Pointed it at orders.o_orderdate in our semantic layer. Rerunning.Message #analytics-alerts
4Improve
Catch it, fix it, prove it
honest-agent tells you when an eval breaks. After you change your agent, one run proves the fix.
Read the docs on setting up alerts →