Skip to content
The Honest Agent robothonest-agent

CLI commands

Run any command with --help to see the same in your terminal. A flag beats an environment variable, which beats honest_agent_config.yml, which beats the default.

Options for every command

Section titled “Options for every command”

These go before the command, like honest-agent --env-file prod.env run.

  • --version: Show the version and exit.
  • --env-file FILE: Load settings and API keys from this file instead of the .env found from the current directory (or its parents). Variables already set in your shell take precedence. (environment variable HONEST_AGENT_ENV_FILE)
  • --config-file FILE: Read settings and targets from this file instead of the honest_agent_config.yml found from the current directory (or its parents). (environment variable HONEST_AGENT_CONFIG_FILE)

debug

Section titled “debug”

Check a project’s setup, like dbt debug: the config file and target, the variables they need, the evals, the MCP server and the results store. Calls no model and runs no query: it lists the server’s tools without calling one. Exits non-zero if a check fails.

Terminal window
honest-agent debug [OPTIONS]
  • -t, --target TEXT: Target (agent) from honest_agent_config.yml to check. Defaults to its default_target.
  • --evals-dir DIRECTORY: Directory of eval YAML files. Defaults to the target’s evals_dir, else ./evals.
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --model TEXT: Model to evaluate (the agent under test): a Claude model, or an OpenAI one (gpt-*, o3, o4-mini, …). Only the API key for its provider is needed. Defaults to claude-haiku-4-5 if ANTHROPIC_API_KEY is set, else gpt-5.4-mini if OPENAI_API_KEY is set. (environment variable HONEST_AGENT_MODEL)
  • --grading-model TEXT: Model for grading: it pulls the value out of the answer for extract_match evals and judges the answer for llm_judge ones (contains needs none). Defaults to –model. Keeping it the same across runs keeps comparisons between agent models fair. (environment variable HONEST_AGENT_GRADING_MODEL)
  • --mcp-command TEXT: Shell command launching a local MCP server over stdio, e.g. “python mcp_server/server.py”. Mutually exclusive with –mcp-url; when given on the command line, it overrides an MCP_URL set in the environment. (environment variable MCP_COMMAND)
  • --mcp-url TEXT: URL of a remote MCP server’s streamable-HTTP endpoint. Mutually exclusive with –mcp-command; when given on the command line, it overrides an MCP_COMMAND set in the environment. (environment variable MCP_URL)
  • --mcp-bearer-token TEXT: [–mcp-url only] Bearer token sent as the Authorization header. (environment variable MCP_BEARER_TOKEN)
  • --mcp-env NAME: [–mcp-command only] Environment variable the local MCP server needs, e.g. MOTHERDUCK_TOKEN. Repeat for several. The server gets only these plus a minimal environment (PATH, HOME, …), never the rest of .env. Defaults to the target’s mcp_env.

init

Section titled “init”

Write a new project’s starter files: config, a first eval and .env.

Terminal window
honest-agent init [OPTIONS]
  • --server [snowflake|motherduck|url|local]: The MCP server your agent uses.
  • --mcp-command TEXT: The command that starts a local MCP server (–server local).
  • --mcp-url TEXT: The MCP server’s URL (any other –server); it goes into .env.
  • --target-name TEXT: The target’s name. Defaults to the server’s name.
  • --model-provider [anthropic|openai]: anthropic (Claude) or openai.
  • --results [local|motherduck|s3]: Where results live.
  • --results-db TEXT: The MotherDuck database for results (–results motherduck).
  • --no-input: Ask nothing: use the flags given and the defaults.
  • --example: Instead, create honest-agent-example/: a demo agent over sample data with six evals.
  • --with-skill: Add the setup skill for coding agents (asked otherwise). In a folder that already has honest_agent_config.yml, only adds it, or replaces it with this version’s.

logs

Section titled “logs”

Print what each matching eval sent and received, call by call, as recorded: every model call, tool call and grading call with its request, response and timing. Scores and SQL summaries are in the report (honest-agent report).

Terminal window
honest-agent logs [OPTIONS]
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --run-id TEXT: Only show logs for this run.
  • --eval-id TEXT: Only show logs for this eval id.
  • --json: Print the raw records as JSON, exactly as stored.

ls

Section titled “ls”

List the evals a run would pick, with their id, category and tags, like dbt ls. Reads only the eval files: no model or MCP server.

Terminal window
honest-agent ls [OPTIONS]
  • -t, --target TEXT: Target from honest_agent_config.yml whose evals to list.
  • --evals-dir DIRECTORY: Directory of eval YAML files. Defaults to the target’s evals_dir, else ./evals.
  • -s, --select TEXT: Only these evals, dbt---select-style: an eval id (e.g. top_segment), tag:<tag> or category:<category>. Spaces mean OR, commas mean AND: ‘–select “category:sales,tag:smoke top_segment”’ picks the sales evals tagged smoke, plus top_segment. honest-agent ls lists ids, categories and tags.
  • --exclude TEXT: Leave out these evals (same syntax as –select), after –select.

notify

Section titled “notify”

Send a Slack alert if the target’s most recent run had any eval below its threshold.

Terminal window
honest-agent notify [OPTIONS]
  • --target TEXT: Target (agent) from honest_agent_config.yml to alert on. Defaults to its default_target.
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --webhook-url TEXT: The Slack incoming webhook to post the alert to. (environment variable SLACK_WEBHOOK_URL)
  • --report-url TEXT: Where the report is published, so the alert links to each failing eval. (environment variable HONEST_AGENT_REPORT_URL)

rebuild

Section titled “rebuild”

Re-derive results, SQL calls and traces from the raw record of past runs, after a change to how honest-agent scores or reads them. Calls no model: it costs nothing and uses each run’s own settings and eval definitions. Runs recorded before honest-agent kept raw records, and evals stopped by an error, are left as they are.

Terminal window
honest-agent rebuild [OPTIONS]
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --run-id TEXT: Only rebuild this run.

report

Section titled “report”

Write the report (web UI + recent results) to a folder.

Terminal window
honest-agent report [OPTIONS]
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --out DIRECTORY: Report folder (web UI + data/report.json). (default honest_agent_report)
  • --days INTEGER RANGE: Include results from this many days before the latest run. Older results stay in the results file. (default 30)

run

Section titled “run”

Run every eval, grade the answers, and write results to storage. Exits 1 if any eval falls below its threshold, like a failing test.

Terminal window
honest-agent run [OPTIONS]
  • -t, --target TEXT: Target (agent) from honest_agent_config.yml to evaluate. Defaults to its default_target.
  • --evals-dir DIRECTORY: Directory of eval YAML files. Defaults to the target’s evals_dir, else ./evals.
  • --results-path TEXT: Local DuckDB file where results are stored (created on first run), or a MotherDuck database as md:<name>, opened with HONEST_AGENT_RESULTS_TOKEN.
  • --model TEXT: Model to evaluate (the agent under test): a Claude model, or an OpenAI one (gpt-*, o3, o4-mini, …). Only the API key for its provider is needed. Defaults to claude-haiku-4-5 if ANTHROPIC_API_KEY is set, else gpt-5.4-mini if OPENAI_API_KEY is set. (environment variable HONEST_AGENT_MODEL)
  • --grading-model TEXT: Model for grading: it pulls the value out of the answer for extract_match evals and judges the answer for llm_judge ones (contains needs none). Defaults to –model. Keeping it the same across runs keeps comparisons between agent models fair. (environment variable HONEST_AGENT_GRADING_MODEL)
  • --max-tool-steps INTEGER RANGE: Max steps per eval before giving up: each step is one model API call and the tool calls it asks for. If the model still wants to call tools when this is hit, that eval fails with a clear error instead of silently returning an empty answer. (default 5)
  • --mcp-command TEXT: Shell command launching a local MCP server over stdio, e.g. “python mcp_server/server.py”. Mutually exclusive with –mcp-url; when given on the command line, it overrides an MCP_URL set in the environment. (environment variable MCP_COMMAND)
  • --mcp-url TEXT: URL of a remote MCP server’s streamable-HTTP endpoint. Mutually exclusive with –mcp-command; when given on the command line, it overrides an MCP_COMMAND set in the environment. (environment variable MCP_URL)
  • --mcp-bearer-token TEXT: [–mcp-url only] Bearer token sent as the Authorization header. (environment variable MCP_BEARER_TOKEN)
  • --mcp-env NAME: [–mcp-command only] Environment variable the local MCP server needs, e.g. MOTHERDUCK_TOKEN. Repeat for several. The server gets only these plus a minimal environment (PATH, HOME, …), never the rest of .env. Defaults to the target’s mcp_env.
  • --agent-name TEXT: Label for the agent being evaluated (e.g. snowflake, motherduck), stored with every result so reports can filter by agent. Defaults to the target’s agent_name or name, else the name the MCP server reports about itself. (environment variable HONEST_AGENT_AGENT_NAME)
  • -s, --select TEXT: Only these evals, dbt---select-style: an eval id (e.g. top_segment), tag:<tag> or category:<category>. Spaces mean OR, commas mean AND: ‘–select “category:sales,tag:smoke top_segment”’ picks the sales evals tagged smoke, plus top_segment. honest-agent ls lists ids, categories and tags.
  • --exclude TEXT: Leave out these evals (same syntax as –select), after –select.

serve

Section titled “serve”

Serve a generated report folder locally, the same way dbt docs serve does.

Terminal window
honest-agent serve [OPTIONS]
  • --out DIRECTORY: Report folder (web UI + data/report.json). (default honest_agent_report)
  • --port INTEGER: Port to serve the report on, at http://127.0.0.1:&lt;port>/. (default 8000)
  • --open-browser / --no-open-browser: Open the report in your browser. (default --open-browser)