Providers Configuration

Providers define which agent or LLM backend to evaluate. AgentV uses one composable config graph across project manifests and eval files:

• .agentv/config.yaml is the project-local discovery and composition root. It can hold providers, tests, defaults, execution policy, results settings, and repo-local project policy.
• $AGENTV_HOME/config.yaml is the user/operator config. Use it for defaults that apply across projects, project registry data, default result locations, and provider defaults that should not be copied into each repo.
• eval.yaml is a focused, shareable slice of the same graph. Use it for a suite-specific provider, grader, tests, evaluator settings, or run controls that should travel with the eval.

Any supported top-level field can stay inline or become a direct field reference such as providers: file://providers.yaml. Both forms normalize to the same config graph.

Structure

providers:
  - id: openai
    label: local-openai
    runtime: host
    config:
      api_format: chat
      base_url: "{{ env.LOCAL_OPENAI_PROXY_BASE_URL }}"
      api_key: "{{ env.LOCAL_OPENAI_PROXY_API_KEY }}"
      model: "{{ env.LOCAL_OPENAI_PROXY_MODEL }}"

  - id: codex-app-server
    label: codex-local
    runtime: host
    config:
      command: ["codex", "app-server"]
      model: gpt-5-codex

  - id: openai
    label: openai-grader
    runtime: host
    config:
      model: gpt-5-mini

defaults:
  provider: codex-local
  grader: openai-grader

Use id for the provider backend/spec and label for the stable AgentV selection identity. runtime describes where the provider runs; use host as the shorthand for the current machine, or object form when you need mode: host | profile | sandbox plus runtime-specific settings. Provider settings belong under config. Process-backed coding-agent providers use config.command as a non-empty argv array.

A grader is not a separate kind of entity — it is a provider selected for a grading role, either through defaults.grader (shown above) or an options.provider / assertion-level provider override. default_test.options.provider sets a shared grader fallback for tests, tests[].options.provider overrides it for one test, and an assertion-level provider wins for that assertion. There is no separate graders: list; authoring one is a hard error telling you to move each entry into providers.

Runtime Modes

Use runtime: host when you want AgentV to run the provider exactly as it is installed on the current machine. This is the best fit for local research, subscription-auth workflows, and evaluating the same CLI profile an engineer uses manually.

Use runtime.mode: profile when the provider still runs as a host process but should use an isolated home/config directory, such as a dedicated CODEX_HOME or HOME.

Use runtime.mode: sandbox when the provider should run inside a separate execution substrate. The built-in sandbox runner currently supports Docker for id: cli; coding-agent adapters such as codex-cli, claude-cli, copilot-cli, and pi-cli return a structured unsupported provider error until their transcript parsers are wired through sandbox-aware runners.

providers:
  - id: codex-cli
    label: codex-sandbox
    runtime:
      mode: sandbox
      engine: docker
      image: ghcr.io/acme/codex-agent:sha256
      workdir: /workspace
      network: none
      mounts:
        - source: ./workspace
          target: /workspace
          access: rw
        - source: ./.agentv/results
          target: /results
          access: rw
      env:
        AGENTV_RESULT_DIR: /results
      secrets:
        OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}"
    config:
      command: ["codex", "exec", "--json"]
      timeout_seconds: 300

Sandbox mode does not inherit host credentials by default. Mount only the workspace, results, cache, or credential paths the provider needs, and pass only the environment variables and secrets listed under runtime.env and runtime.secrets. Install the provider CLI by using an image that already contains it or by adding explicit setup under runtime.setup; locate the CLI with config.command.

For CI, API-key or explicitly injected secret auth is the most reproducible path. Subscription OAuth can work in a sandbox only when you intentionally mount or seed the relevant profile directory into the sandbox. That makes the run less portable than API-key CI and should be reserved for workflows where matching a local subscription profile is the point of the evaluation.

Inline and decomposed forms are equivalent. This single-file config:

providers:
  - id: codex-app-server
    label: codex-local
    runtime: host
    config:
      command: ["codex", "app-server"]
      model: gpt-5-codex

  - id: openai
    label: openai-grader
    runtime: host
    config:
      model: gpt-5-mini

prompts:
  - "{{ input }}"

tests:
  - id: smoke
    vars:
      input: Fix the failing test.

defaults:
  provider: codex-local
  grader: openai-grader

can be decomposed like this:

providers: file://providers.yaml
tests: file://tests.yaml
defaults: file://defaults.yaml

Referenced field files contain the field value directly. providers.yaml contains a bare array, not an object wrapped in providers::

.agentv/providers.yaml

- id: codex-app-server
  label: codex-local
  runtime: host
  config:
    command: ["codex", "app-server"]
    model: gpt-5-codex

- id: openai
  label: openai-grader
  runtime: host
  config:
    model: gpt-5-mini

.agentv/defaults.yaml

provider: codex-local
grader: openai-grader

File refs are optional. Use them when a config field is large, reused, or owned by a separate team; keep fields inline when that is easier to read.

Environment Variables

Use {{ env.VARIABLE_NAME }} syntax to reference values from your environment. AgentV reads exported process environment variables directly, and it also loads .env files from the eval directory hierarchy when present:

providers:
  - id: anthropic
    label: my-provider
    runtime: host
    config:
      api_key: "{{ env.ANTHROPIC_API_KEY }}"
      model: "{{ env.ANTHROPIC_MODEL }}"

This keeps secrets out of version-controlled files and avoids requiring a CI step that rewrites already-exported secrets into .env.

Loading Environment Variables from Config

Set env_path and/or env_from in .agentv/config.yaml to load environment variables before validation and eval run, so {{ env.* }} references resolve for both agentv validate and agentv eval:

.agentv/config.yaml

env_path:
  - .env
  - .env.local

env_from:
  - command: ["bun", "scripts/load-secrets.ts"]
    format: shell_exports
  - command: ["node", "scripts/print-env-json.mjs"]
    format: json

• env_path loads one or more dotenv-style files, relative to the project directory (the parent of .agentv/) unless the path is absolute. A missing file logs a warning and is skipped.
• env_from runs one or more argv commands and injects their parsed stdout. command must be a non-empty argv array — shell command strings are not accepted. format defaults to shell_exports (lines like export KEY=value or KEY=value); format: json expects a flat JSON object of string values. A failing command aborts the CLI invocation with an error.
• Both singular (env_path: .env, env_from: {command: [...]}) and list forms are accepted.
• Values already present in the process environment are never overwritten, and injected values are never printed to logs.
• An argv command that explicitly invokes a shell, such as ["sh", "-c", "..."], is still accepted — the requirement is an argv array, not a ban on shells. Only implicit shell command strings (command: "bun scripts/load-secrets.ts") are rejected.

hooks.before_session still runs for backward compatibility but is deprecated; migrate its command to env_from (or a .env file plus env_path) instead.

Known limitations:

• env_path/env_from are discovered from the current working directory at CLI startup, the same as legacy hooks.before_session. In a multi-project workspace, a nested project’s own .agentv/config.yaml env_from only runs when a command is invoked from within (or below) that project’s directory, not when run from a parent directory that resolves to a different .agentv/config.yaml.
• A .agentv/config.yaml’s own fields (for example results.repo) are interpolated before that same file’s env_path/env_from runs, so they cannot reference values produced by its own env_path/env_from. Use an already-exported process environment variable, or a value loaded by an ancestor project’s config, for fields inside config.yaml itself.

Supported Providers

Provider	Type	Description
azure	LLM	Azure OpenAI
anthropic	LLM	Anthropic Claude API
gemini	LLM	Google Gemini
claude-cli	Agent	Claude CLI subprocess
claude-sdk	Agent	Claude Agent SDK in an isolated child runner
codex-cli	Agent	Codex CLI subprocess
codex-app-server	Agent	Codex app-server subprocess
codex-sdk	Agent	Codex SDK in an isolated child runner
copilot-cli	Agent	Copilot CLI subprocess
copilot-sdk	Agent	Copilot SDK in an isolated child runner
pi-sdk	Agent	Pi SDK in an isolated child runner
pi-cli	Agent	Pi CLI subprocess
pi-rpc	Agent	Pi RPC subprocess over stdio
vscode	Agent	VS Code with Copilot
vscode-insiders	Agent	VS Code Insiders
cli	Agent	Any CLI command — see CLI Provider
mock	Testing	Explicit mock provider for examples and tests

Referencing Providers in Evals

Select the system under test with defaults.provider, top-level providers, or CLI --provider, depending on the command flow. Test cases do not choose providers; split provider-specific cases into separate eval suites, select them with tags/filters, or run the same eval with different --provider values.

providers:
  - local-openai

tests:
  - id: test-1
  - id: test-2

The string is a configured provider label or id. Use object form when an eval needs a local provider variant:

providers:
  - id: codex-app-server
    label: codex-high-reasoning
  runtime: host
  config:
    command: ["codex", "app-server"]
    model: gpt-5-codex
    reasoning_effort: high

Use defaults.grader for the project default grader. default_test.options.provider and tests[].options.provider can choose a grader provider for LLM-backed assertions before an assertion-level provider override takes final precedence.

Environment And Lifecycle Extensions

Use environment for the authored coding-agent testbed recipe: host or Docker placement, workdir, setup argv, fixtures, services, and repository materialization scripts. Use top-level extensions for Promptfoo-style lifecycle callbacks and instrumentation that run after the environment is prepared. Use provider hooks only for runner-specific setup.

extensions:
  - file://scripts/workspace.mjs:beforeAll
  - file://scripts/workspace.mjs:beforeEach
  - file://scripts/workspace.mjs:afterEach
  - file://scripts/workspace.mjs:afterAll
  - id: agentv:agent-rules
    hook: beforeAll
    skills: agent-rules/skills
    rules: agent-rules/AGENTS.md

environment:
  type: host
  workdir: ./workspaces/my-project
  setup:
    command: ["bash", "-lc", "bun install && bun run build"]
    cwd: "."
    timeout_ms: 120000

Field	Description
environment.type	host or docker
environment.workdir	Cwd used by providers and graders inside the prepared testbed
environment.setup.command	Non-empty argv setup command; use ["bash", "-lc", "..."] for shell behavior
Top-level env	Provider/eval variables and template inputs
extensions[]	file://...:beforeAll, beforeEach, afterEach, afterAll, or agentv:agent-rules

Lifecycle order: resolve environment → prepare host or Docker testbed → environment.setup.command → extensions.beforeAll → provider hooks.before_all → git baseline → (extensions.beforeEach → provider hooks.before_each → agent runs → file changes captured → provider hooks.after_each → extensions.afterEach) × N tests → provider hooks.after_all → extensions.afterAll → cleanup

Error handling:

• beforeAll / beforeEach extension failure aborts the affected run with an error result
• afterAll / afterEach extension failure is non-fatal

File hook context: Exported functions receive a JSON-compatible object with case context:

{
  "workspace_path": "/home/user/.agentv/workspaces/run-123/case-01",
  "test_id": "case-01",
  "eval_run_id": "run-123",
  "case_input": "Fix the bug",
  "case_metadata": { "repo": "sympy/sympy", "source_commit": "abc123" }
}

extensions are callbacks, not the canonical place to hide repository materialization, Docker/image selection, cwd, or testbed provenance. Put that contract in environment.

Repository Setup

Materialize git repositories through the environment recipe. A setup command can call your normal clone, checkout, fixture, or build script with explicit argv inputs:

environment:
  type: host
  workdir: ./workspaces/my-repo
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - ./workspaces/my-repo
      - https://github.com/org/repo.git
      - main
    cwd: "."
    timeout_ms: 120000

Keep the same repo URL and commit in tests[].metadata only when you also need audit fields in results or extensions. Metadata by itself is not an operational checkout.

Field	Description
setup.command[0]	Executable
setup.command[1...]	Arguments passed unchanged to the executable
setup.cwd	Directory for the setup command
setup.timeout_ms	Setup timeout in milliseconds

Existing local workspaces: do not commit local paths in eval YAML. Use --workspace-path /path/to/workspace for a one-off run, or put execution.workspace_path in .agentv/config.local.yaml.

Cleanup Behavior

Default finish behavior:

• Success: cleanup
• Failure: keep

CLI overrides:

• --retain-on-success keep|cleanup
• --retain-on-failure keep|cleanup

cwd

Use cwd on a provider to run in an existing directory (shared across tests). If not set, the eval file’s directory is used as the working directory.

Provider Hooks

Eval files can define per-provider hooks that run setup/teardown scripts to customize each provider variant. This enables comparing different harness configurations, such as baseline vs with-plugins, in a single eval file.

Providers do not declare testbed setup. The shared environment prepares the world so every provider runs against the same files, fixtures, cwd, and services. Provider hooks customize the harness under evaluation, such as enabling wrappers or changing provider-local config.

Provider hooks can be scoped to an eval-local provider object:

providers:
  - id: codex-app-server
    label: codex-with-skills
    extends: default
    hooks:
      before_each:
        command: ["setup-plugins.sh", "skills"]

Hook execution order

Provider hooks run after environment setup and lifecycle extensions on setup. Teardown runs provider hooks before lifecycle extensions:

1. Extension beforeAll
2. Provider before_all
3. For each test:
   • Extension beforeEach
   • Provider before_each
   • Test executes
   • Provider after_each
   • Extension afterEach
4. Provider after_all
5. Extension afterAll

Hook schema

Provider hooks follow the same command schema as lifecycle hook commands:

hooks:
  before_all:
    command: ["bash", "./setup.sh"] # Command argv
    timeout_ms: 60000           # Optional timeout
    cwd: "./scripts"            # Optional working directory
  before_each:
    command: ["bash", "-lc", "echo setup"]
  after_each:
    command: ["bash", "./cleanup.sh"]
  after_all:
    command: ["bash", "./teardown.sh"]