LOKI INTELLIGENCE

Loki Intelligence — Security Briefs · Published

How to Evaluate an AI Security Testing Provider

Choosing an AI security testing provider is difficult because the market mixes traditional pentesting, model evaluation, AI safety research, compliance consulting, automated scanning, and red-team services under similar labels. The right provider should understand both software security and AI-specific failure modes. They should be able to test prompts, tools, retrieval, memory, authorization, data access, and workflow impact without turning the engagement into theater. Use this guide to compare providers on the things that matter: scope quality, methodology, evidence, safety controls, reporting, remediation, and retesting. The buyer should leave with confidence that findings will become fixes.

Method step 01

Start with the systems they can actually test

Ask whether the provider can test your real architecture: chat interface, agent tools, APIs, retrieval, vector stores, memory, model gateway, cloud services, authentication, CI/CD, logging, and production workflow boundaries. A provider that only tests standalone prompts may miss the risks that matter most to your product.

Why it matters: AI security lives in the integration layer. The model is only one component. If the provider cannot reason about web apps, APIs, cloud permissions, data boundaries, and tool execution, they may find interesting prompts but miss exploitable paths.

Method step 02

Ask for a methodology, not a vibe

A credible provider should explain how they map the system, select scenarios, test instruction hierarchy, evaluate tool use, verify data leakage, measure approval behavior, and separate confirmed findings from noise. They should also describe what they will not test without explicit authorization.

Why it matters: AI red teaming can become performative if it is only a collection of clever prompts. Methodology matters because it keeps the engagement tied to risk, repeatability, and evidence instead of anecdotes.

Method step 03

Demand reproducible evidence

Ask to see a sample report structure. Strong reports include scenario description, affected component, trace, tool calls, data accessed, business impact, severity rationale, screenshots or logs, remediation guidance, and retest criteria. They should avoid vague claims like the model was jailbreakable without explaining the real impact.

Why it matters: Security findings need to survive handoff. If engineering cannot reproduce the issue, leadership cannot understand impact, and security cannot retest the fix, the provider has delivered research notes rather than an actionable assessment.

Method step 04

Check production safety controls

A provider should be clear about safe testing boundaries: rate limits, prohibited actions, staging versus production, test accounts, data handling, destructive actions, payment flows, customer-impacting workflows, and emergency stop conditions. They should document authorization before testing begins.

Why it matters: AI systems often connect to real workflows. Testing them carelessly can create tickets, send messages, change records, trigger notifications, or expose sensitive data. Safety controls are not bureaucracy; they are how serious testing avoids becoming an incident.

Method step 05

Look for remediation that engineers can ship

Good remediation is specific. It may recommend narrower tool scopes, approval gates, data minimization, retrieval labeling, output validation, policy checks outside the model, memory controls, better logging, or API authorization changes. The provider should connect each recommendation to the failed path it fixes.

Why it matters: Generic advice like improve prompts or add guardrails is rarely enough. Teams need to know which boundary failed and which control should change. Practical fixes shorten the time from finding to closure.

Method step 06

Make retesting part of the purchase

Ask whether retesting is included, how long it remains available, and what evidence will close a finding. Retesting should run the original scenario and reasonable variants after the fix. For agent systems, retesting should also check whether the change created new behavior in nearby workflows.

Why it matters: A finding is not done when the report is delivered. It is done when the risky path is closed and the team can prove it. Retesting turns AI security from a one-time assessment into an engineering loop.

Method step 07

Questions to ask on a vendor call

Ask which AI attack classes they test, how they handle tool-using agents, whether they can test authenticated workflows, how they avoid unsafe production impact, what evidence appears in the report, how retesting works, and how they separate model behavior from application security. Ask for examples of fixes they have recommended, not only examples of prompts they have tried.

Why it matters: Good answers reveal whether the provider can help your engineering team. Weak answers often sound impressive but stay generic. The best providers can explain scope, controls, evidence, and remediation in the language of your product architecture.

Method step 08

Warning signs when comparing providers

Be careful with providers that promise universal AI safety, rely only on prompt lists, cannot test tools or APIs, avoid discussing production boundaries, provide no sample evidence, skip retesting, or treat every model refusal bypass as a critical issue. Also be careful when a report cannot distinguish confirmed business impact from theoretical concern.

Why it matters: AI security is still noisy. A provider should reduce uncertainty, not add more. Warning signs help buyers avoid engagements that produce dramatic screenshots but little engineering value.

Primary references

Related briefs

Relevant Loki services: AI Agent Risk Assessment — Private Preview and web & API security review.