A copy/paste questionnaire for reviewing AI-agent skills before they get file, browser, memory, SaaS, messaging, workflow, or external-action authority.
Agent Skill Safety Vendor Questionnaire
12 questions to ask before buying, installing, listing, or delegating an AI-agent skill
About this template
Use this as a copy/paste RFP, marketplace submission form, or internal vendor-review checklist when a reusable AI-agent skill will receive file, browser, memory, SaaS, messaging, workflow, or external-action authority.
Quick win: in 20 minutes, you can separate safe-to-pilot skills from skills that need contract fixes before they touch production systems.
When to use it
Use this questionnaire when:
- A vendor, creator, internal team, or marketplace listing claims an agent skill is “ready to run.”
- The skill can read private context, write files, use browser sessions, call SaaS APIs, message people, spend money, update memory, or trigger workflows.
- You need a redaction-safe review packet before approving a pilot.
- You want evidence for an install / install-with-limits / ask-for-changes / reject decision.
Do not send credentials, private prompts, customer data, regulated data, or proprietary internals with this questionnaire. Ask for redacted examples and permission boundaries instead.
Vendor / creator questions
1. What exact job is the skill allowed to do?
Ask for:
- Primary job-to-be-done
- Explicit non-goals
- Intended users or agent runners
- Supported runtimes, e.g. OpenClaw, Claude Code, Hermes, Codex-style agents, browser-only agents
Good answer: names a narrow workflow and non-goals.
Risky answer: “automates your business,” “handles anything,” or unclear runtime assumptions.
2. What inputs does the skill require, and what inputs are forbidden?
Ask for:
- Required inputs
- Optional inputs
- Forbidden inputs
- Private-data handling rules
- Source freshness requirements
Good answer: lists allowed input classes and tells users what not to paste.
Risky answer: invites secrets, full database dumps, private customer records, or unrestricted browser/session access.
3. What tools, APIs, browsers, files, or SaaS systems may it use?
Ask for a permission matrix:
| Surface | Read? | Write? | External action? | Approval required? | Notes |
|---|---|---|---|---|---|
| Files | |||||
| Browser | |||||
| SaaS/API | |||||
| Messages/email | |||||
| Payments/spend | |||||
| Memory |
Good answer: separates read, write, external-send, destructive, and spend authority.
Risky answer: says “needs all tools,” “admin access,” or “approval handled by the LLM.”
4. What actions are always out of scope?
Ask for hard bans, such as:
- No spending money
- No deleting records
- No sending public posts or customer emails without approval
- No credential capture
- No bypassing CAPTCHA, 2FA, policy controls, or rate limits
- No writing long-term memory without a review step
Good answer: contains explicit deny rules.
Risky answer: relies on vague “be careful” instructions.
5. What output should the skill produce?
Ask for:
- Expected artifact or schema
- Required evidence/citations
- Confidence notes
- Uncertainty format
- What counts as done
Good answer: includes a concrete output contract and done criteria.
Risky answer: produces unstructured prose with no evidence trail.
6. What receipt does each run generate?
Ask for a run receipt containing:
- Inputs used, with sensitive data redacted
- Tools/actions attempted
- Files, URLs, records, or messages changed
- Errors and retries
- Approval checkpoints
- Before/after state where relevant
- Human review notes
Good answer: produces an audit trail a supervising agent or human can inspect.
Risky answer: only says “task completed.”
7. What memory may the skill read, write, update, or propose?
Ask for:
- Memory read scope
- Memory write scope
- Promotion rules for long-term memory
- Expiry/retention rules
- Conflict resolution rules
- “Never store” categories
Good answer: treats memory writes as governed state changes.
Risky answer: silently updates memory or stores private user data by default.
8. How does the skill fail safely?
Ask for:
- Stop conditions
- Escalation triggers
- Draft-vs-send rules
- Exception queue format
- Human checkpoint requirements
- Retry limits
Good answer: knows when to stop and produce an exception packet.
Risky answer: keeps trying indefinitely or improvises external actions.
9. What evals have been run?
Ask for evidence of:
- Happy-path test
- Missing-input test
- Bad-source test
- Tool-denied test
- Private-data redaction test
- External-action approval test
- Rollback/repair test
- Prompt-injection or malicious-input test
Good answer: includes concrete test cases and results.
Risky answer: “we tested it manually” with no cases.
10. What rollback or repair path exists?
Ask for:
- Reversible actions
- Compensating actions for non-reversible changes
- Required backups or snapshots
- Owner of repair work
- User-visible apology/notification flow if external users are affected
Good answer: distinguishes reversible, compensatable, and irreversible actions.
Risky answer: has no plan for bad writes, bad sends, or corrupted memory.
11. What review verdict do you recommend for first install?
Ask the vendor to choose one:
- Install — safe for intended use with documented controls
- Install with limits — safe only with restricted tools/data/actions
- Ask for changes — missing controls before pilot
- Reject for now — unacceptable risk or unclear authority
Ask them to justify the verdict in 5 bullets.
Good answer: recommends limits where appropriate.
Risky answer: always recommends unrestricted install.
12. What should a buyer inspect before renewal or wider rollout?
Ask for:
- Run metrics
- Incident/exception log
- False-positive/false-negative examples
- Human override frequency
- Scope creep indicators
- Cost/spend records
- Memory diffs
- Version changelog
Good answer: defines an operating review, not just install review.
Risky answer: treats install as the final safety step.
10-minute scoring rubric
Give each category 0, 1, or 2 points.
| Category | 0 = missing | 1 = partial | 2 = clear |
| Scope and non-goals | |||
| Input/privacy contract | |||
| Tool permission matrix | |||
| Hard deny rules | |||
| Output/done contract | |||
| Run receipts | |||
| Memory policy | |||
| Safe failure/escalation | |||
| Eval evidence | |||
| Rollback/repair plan |
Verdict guide:
- 17–20: install or pilot with normal controls
- 13–16: install with limits; request fixes before production
- 8–12: ask for changes before pilot
- 0–7: reject for now