Check Skill Reply Regressions with Agent Response Regression Kit

mochi-agent-eval-kit (Agent Response Regression Kit) is a small offline checker for already recorded agent replies. It can assert that text contains a required phrase, omits a forbidden phrase, or exactly matches a string. It does not run an agent, call a model, or judge meaning, factual accuracy, or whether the task was truly completed. Treat it as a text assertion tool, not a complete agent-evaluation platform.^[1]

Know what it checks before installing

The reviewed release package is version 1.0.0 under MIT-0. It requires the OpenClaw CLI and Python 3.10 or later; the helper uses Python's standard library. Run these commands in a terminal:

openclaw skills install @hznetnewpower/mochi-agent-eval-kit --version 1.0.0
openclaw skills list
openclaw skills check

OpenClaw's CLI reference documents installation by owner/slug with an optional version, list for local skills, and check for missing requirements or configuration.^[2] If installation raises a source-policy or trust warning, stop and inspect the package's SKILL.md and scripts before deciding. Do not bypass the warning with a --force option. Once the skill appears in the list, ask it in an OpenClaw conversation to check recorded fixtures; do not ask it to imply that a model run occurred.

Make a first check with fictional replies

Create a sanitized JSON request. This is a tutorial fixture, not real account data or a result produced by this site:

{
  "cases": [
    {
      "id": "timezone-label",
      "actual": "The report runs at 09:00 UTC.",
      "mustContain": ["09:00 UTC"],
      "mustNotContain": ["API_KEY"]
    },
    {
      "id": "exact-status",
      "actual": "done",
      "exact": "done"
    }
  ]
}

actual is the reply text to inspect. Each case needs a unique id and at least one of mustContain, mustNotContain, or exact. These are literal substring or equality checks; they do not recognize paraphrases. In the example, the exact text 09:00 UTC must appear for the first containment check to pass.

Save the file as request.json in the current agent workspace. In OpenClaw, you can ask:

Use mochi-agent-eval-kit to check the two sanitized reply fixtures in request.json.
Run only the local evaluator from the installed skill. Show the exit status, per-case result, report path, and any failed expectation. Do not claim that an agent or model was run; these are recorded strings only.

The package documents this local command template:

python3 "{baseDir}/scripts/run.py" --workspace "CURRENT_WORKSPACE" --input "request.json"

The agent should replace {baseDir} with the installed skill directory and CURRENT_WORKSPACE with the actual workspace path; do not paste the placeholders unchanged into a terminal. A useful first-run check is to confirm the process exit status, overall JSON ok, per-case results, and a report in a new exports/mochi-…/ directory. Open report.json to see which expectation failed, then check the output filenames, byte counts, and SHA-256 values. Overall success means this local transformation completed; it does not mean an agent's behavior passed or an online feature was verified.^[1]

Add it to a real regression workflow

Capture a small set of representative replies from your own agent runs and define their intended checks before editing the skill. Cover a normal answer, a missing field, a sensitive marker, a boundary request, and a similar request that should not trigger the skill. Keep the skill version, case ID, and reply provenance with each fixture. Reserve separate held-out examples so the same examples do not both shape and validate the instructions.

Use deterministic assertions for explicit constraints: a reply must include an order ID, must not expose an internal token prefix, or must return exactly done for a status field. For summary accuracy, sensible steps, correct skill selection, and instruction following, inspect traces, ask for human review, or use a suitable evaluator. OpenAI's eval guide describes a loop of prompt, captured run and artifacts, a small set of checks, and a comparable score. AWS's skill-evaluation guide separates choosing the right skill from following its instructions. This kit covers only the narrowest text-assertion layer of that broader process.^{[3] [4]}

When comparing a skill change, run both versions on the same inputs. Keep a few unseen held-out cases to catch instructions tailored to the visible test answers. When a check fails, inspect the case and report: an overly broad assertion, an unrealistic fixture, and an agent skipping a step are different problems. Do not let this tool rewrite SKILL.md automatically, and do not rely on an aggregate pass rate that hides a specific failure.

Common issues and limits

• The skill is not available: Confirm it with openclaw skills list, then use openclaw skills check to inspect missing requirements. Check that the agent uses the expected workspace and that a higher-precedence copy with the same name is not taking effect. Install Python 3.10+ if needed.
• The input or report path fails: Make sure request.json is valid and inside the workspace. Do not use ../, absolute paths, or symlinks pointing outside it. The helper writes into a new exports/mochi-…/ directory instead of replacing an existing one.
• Every check passes but the answer is still wrong: Literal string checks do not evaluate meaning. A wrong response may still contain the required phrase. Add structured-output assertions, human review, or trace-based evaluation.
• You need to know whether an agent invoked a skill: This tool consumes recorded strings; it does not collect model traces. Run the agent separately and save its trace. Use a deterministic invocation check for a fixed routing requirement and a suitable evaluator for meaning or instruction following.
• Fixtures contain customer data or secrets: Sanitize them first. Even with an offline helper, protect real replies according to the access controls on the local workspace, backups, and reports.

The package includes smoke tests, but this guide did not execute the candidate package's scripts or tests. Read the release's SKILL.md and scripts/ yourself before installation. Do not present the tutorial fixtures as tests performed by this site. This guide reflects the 1.0.0 package; verify a newer release's input fields, output, and requirements against its current source.

References

1. ^[1]
2. ^[2]
3. ^[3]
4. ^[4]