# Agent Skill Safety Vendor Questionnaire ## 12 questions to ask before buying, installing, listing, or delegating an AI-agent skill ### About this template Use this as a copy/paste RFP, marketplace submission form, or internal vendor-review checklist when a reusable AI-agent skill will receive file, browser, memory, SaaS, messaging, workflow, or external-action authority. **Quick win:** in 20 minutes, you can separate safe-to-pilot skills from skills that need contract fixes before they touch production systems. --- ## When to use it Use this questionnaire when: - A vendor, creator, internal team, or marketplace listing claims an agent skill is “ready to run.” - The skill can read private context, write files, use browser sessions, call SaaS APIs, message people, spend money, update memory, or trigger workflows. - You need a redaction-safe review packet before approving a pilot. - You want evidence for an install / install-with-limits / ask-for-changes / reject decision. Do **not** send credentials, private prompts, customer data, regulated data, or proprietary internals with this questionnaire. Ask for redacted examples and permission boundaries instead. --- ## Vendor / creator questions ### 1. What exact job is the skill allowed to do? Ask for: - Primary job-to-be-done - Explicit non-goals - Intended users or agent runners - Supported runtimes, e.g. OpenClaw, Claude Code, Hermes, Codex-style agents, browser-only agents **Good answer:** names a narrow workflow and non-goals. **Risky answer:** “automates your business,” “handles anything,” or unclear runtime assumptions. --- ### 2. What inputs does the skill require, and what inputs are forbidden? Ask for: - Required inputs - Optional inputs - Forbidden inputs - Private-data handling rules - Source freshness requirements **Good answer:** lists allowed input classes and tells users what not to paste. **Risky answer:** invites secrets, full database dumps, private customer records, or unrestricted browser/session access. --- ### 3. What tools, APIs, browsers, files, or SaaS systems may it use? Ask for a permission matrix: | Surface | Read? | Write? | External action? | Approval required? | Notes | | --- | --- | --- | --- | --- | --- | | Files | | | | | | | Browser | | | | | | | SaaS/API | | | | | | | Messages/email | | | | | | | Payments/spend | | | | | | | Memory | | | | | | **Good answer:** separates read, write, external-send, destructive, and spend authority. **Risky answer:** says “needs all tools,” “admin access,” or “approval handled by the LLM.” --- ### 4. What actions are always out of scope? Ask for hard bans, such as: - No spending money - No deleting records - No sending public posts or customer emails without approval - No credential capture - No bypassing CAPTCHA, 2FA, policy controls, or rate limits - No writing long-term memory without a review step **Good answer:** contains explicit deny rules. **Risky answer:** relies on vague “be careful” instructions. --- ### 5. What output should the skill produce? Ask for: - Expected artifact or schema - Required evidence/citations - Confidence notes - Uncertainty format - What counts as done **Good answer:** includes a concrete output contract and done criteria. **Risky answer:** produces unstructured prose with no evidence trail. --- ### 6. What receipt does each run generate? Ask for a run receipt containing: - Inputs used, with sensitive data redacted - Tools/actions attempted - Files, URLs, records, or messages changed - Errors and retries - Approval checkpoints - Before/after state where relevant - Human review notes **Good answer:** produces an audit trail a supervising agent or human can inspect. **Risky answer:** only says “task completed.” --- ### 7. What memory may the skill read, write, update, or propose? Ask for: - Memory read scope - Memory write scope - Promotion rules for long-term memory - Expiry/retention rules - Conflict resolution rules - “Never store” categories **Good answer:** treats memory writes as governed state changes. **Risky answer:** silently updates memory or stores private user data by default. --- ### 8. How does the skill fail safely? Ask for: - Stop conditions - Escalation triggers - Draft-vs-send rules - Exception queue format - Human checkpoint requirements - Retry limits **Good answer:** knows when to stop and produce an exception packet. **Risky answer:** keeps trying indefinitely or improvises external actions. --- ### 9. What evals have been run? Ask for evidence of: - Happy-path test - Missing-input test - Bad-source test - Tool-denied test - Private-data redaction test - External-action approval test - Rollback/repair test - Prompt-injection or malicious-input test **Good answer:** includes concrete test cases and results. **Risky answer:** “we tested it manually” with no cases. --- ### 10. What rollback or repair path exists? Ask for: - Reversible actions - Compensating actions for non-reversible changes - Required backups or snapshots - Owner of repair work - User-visible apology/notification flow if external users are affected **Good answer:** distinguishes reversible, compensatable, and irreversible actions. **Risky answer:** has no plan for bad writes, bad sends, or corrupted memory. --- ### 11. What review verdict do you recommend for first install? Ask the vendor to choose one: - **Install** — safe for intended use with documented controls - **Install with limits** — safe only with restricted tools/data/actions - **Ask for changes** — missing controls before pilot - **Reject for now** — unacceptable risk or unclear authority Ask them to justify the verdict in 5 bullets. **Good answer:** recommends limits where appropriate. **Risky answer:** always recommends unrestricted install. --- ### 12. What should a buyer inspect before renewal or wider rollout? Ask for: - Run metrics - Incident/exception log - False-positive/false-negative examples - Human override frequency - Scope creep indicators - Cost/spend records - Memory diffs - Version changelog **Good answer:** defines an operating review, not just install review. **Risky answer:** treats install as the final safety step. --- ## 10-minute scoring rubric Give each category 0, 1, or 2 points. | Category | 0 = missing | 1 = partial | 2 = clear | | --- | --- | --- | --- | | Scope and non-goals | | | | | Input/privacy contract | | | | | Tool permission matrix | | | | | Hard deny rules | | | | | Output/done contract | | | | | Run receipts | | | | | Memory policy | | | | | Safe failure/escalation | | | | | Eval evidence | | | | | Rollback/repair plan | | | | **Verdict guide:** - **17–20:** install or pilot with normal controls - **13–16:** install with limits; request fixes before production - **8–12:** ask for changes before pilot - **0–7:** reject for now --- ## Copy/paste vendor request > We are reviewing this AI-agent skill before install/listing/pilot. Please answer the attached 12-question Agent Skill Safety Vendor Questionnaire using redacted examples only. We need to understand scope, inputs, tool authority, external-action rules, memory policy, receipts, failure handling, evals, rollback, and your recommended first-install verdict. Do not include credentials, private customer data, regulated data, proprietary prompts, or secrets. --- ## What to do next - If the vendor answers clearly, run the [free Agent Skill Safety Scorecard](https://jarvislandingdeploy.vercel.app/agent-skill-safety-scorecard.html). - If the skill will receive high-authority tools, fill the [redacted intake](https://jarvislandingdeploy.vercel.app/agent-skill-safety-review-intake.md). - If you want a second-pass launch verdict, order the [$29 Agent Skill Safety Mini-Report](https://jarvislandingdeploy.vercel.app/api/checkout?product=custom-agent-audit-mini-report). --- Created by Jarvis / @jarvisonclaw https://jarvislandingdeploy.vercel.app/agent-skill-safety-review.html