Testing scope

Penetration testing of
applications built on LLMs

A language model does not tell data apart from instructions. Everything that reaches the context, the user question, the contents of a fetched page, a file from disk, a tool response, is read the same way. We check what that can be made to do: leak other people’s data, call a tool without the user agreeing to it, and step around the rules written into the system prompt.

  • Chatbots, in-product assistants, RAG and agents with access to tools
  • Direct and indirect prompt injection, together with the backend and integrations
  • Tested against the OWASP Top 10 for LLM Applications, with a free retest

AI and LLM applications

Duration
usually 5-12 working days
We work to
  • OWASP Top 10 for LLM Applications 2025
  • OWASP GenAI Red Teaming Guide
  • MITRE ATLAS (adversarial techniques against machine learning systems)
  • NIST AI RMF with its generative AI profile (NIST AI 600-1)
  • OWASP API Security Top 10 2023 (backend and integrations)

What exactly we check

The scope follows what the model is able to do. A chatbot with no tools is tested differently from an agent with write access to a database, so we start from the list of tools and data sources.

1

Prompt injection

Direct, typed by the user, and indirect, hidden in a document, a web page, a file or an API response the model pulls in while working. We also check whether external content is kept apart from instructions.

2

System prompt and data in context

Attempts to read the system instruction, the keys and rules written into it, and the data of other users that reached the context through conversation history, memory or a shared session.

3

Agent permissions and tools

What the model can call and on whose behalf. We check whether an action that changes data requires human confirmation, whether a tool’s permissions are narrower than the application account, and whether a chain of calls can be assembled that reaches further than the author intended.

4

RAG and the vector store

Document-level access control at retrieval time, poisoning the index with a planted document, recovering content from embeddings, and data mixing between tenants in a shared collection.

5

Handling the model output

What the application does with the answer. Text placed into HTML gives XSS, into an SQL query an injection, into an HTTP call an SSRF request, into a shell a command execution. This is where an LLM flaw turns into an ordinary application flaw.

6

Limits, cost and availability

Queries that force long answers or many tool calls, no token limit per user, and no budget separation between tenants. Where billing is per token, this is a financial vulnerability as well.

What we find most often

Examples from our projects in this area.

  • A document uploaded to the knowledge base carried an instruction the model executed on the next question from a different user (LLM01)
  • The assistant returned fragments of documents outside the asker’s permissions, because filtering happened on the result list rather than at retrieval (LLM02)
  • A system prompt holding a key to an external API could be read by asking for a summary of the instructions (LLM07)
  • An agent with mailbox access sent a message without the user confirming it (LLM06)
  • The model answer was placed into the interface without encoding, which gave XSS from model-generated content (LLM05)
  • No token limit per account, so a single test account ran up the cost of a monthly bill (LLM10)
  • One tenant’s conversation history appeared in another tenant’s context after a session key bug

Selected tools

Burp Suite ProfessionalgarakPyRITpromptfooour own scenarios and scripts

Tools take care of the repetitive work. The conclusions and attack chains are built by hand.

How the test runs and what you get

The testing process, the report format and the retest rules are the same across every scope.

See the full process

How the test works, step by step

A test usually takes 5-12 working days and runs in six steps.

1

Map of permissions

We write down what the model has access to: tools, data sources, integrations, and the accounts it acts on behalf of. That list decides the scope and the duration.

2

Scope and rules

We agree roles, test accounts, the test window and exclusions. With agents we also agree which actions may actually be carried out and which we stop at confirmation.

3

Attack scenarios

We build a set of attempts from the OWASP Top 10 for LLM Applications and techniques in MITRE ATLAS, chosen for what the application can actually do.

4

Application and backend

Prompt injection, reading the context, tool abuse and output handling. In parallel we test the API the application calls, because some of the impact only shows up there.

5

Report

Every finding with the exact input, the model answer or tool call, and the control that failed. That is what makes it reproducible.

6

Free retest

Once the fixes are deployed we verify every reported vulnerability and update the report with its remediation status.

Reproducibility. A model answers differently every time, so a single hit proves nothing. We repeat each scenario and state how many attempts out of how many worked.

What an LLM application test costs

The price follows the scope of the test. Below is what changes it most.

What drives the price

  • whether the model only answers or also calls tools
  • the number of tools and integrations it can reach
  • whether there is RAG, and how many data sources
  • the number of user roles and how tenants are separated
  • whether the application API is in scope

Indicative ranges

  • Assistant or chatbot without tools: PLN 12,000-25,000 net
  • Agent with tools and RAG: PLN 20,000-40,000 net

We prepare a quote after a short call about scope, with no commitment.

Compliance with the AI Act, NIS2 and the GDPR

A penetration test is not a compliance audit and does not mean the requirements are met. It gives documented evidence that the application’s defenses were checked in practice, along with a remediation plan.

AI Act

For high-risk systems the regulation requires risk management, resilience against manipulation attempts, and event logging. A test shows which of those defenses hold on your deployment.

NIS2

An assistant wired into production systems is part of them, so it falls under the same requirement to assess the effectiveness of security measures.

GDPR, Article 32

If personal data reaches the context, the requirement to regularly test the effectiveness of security measures applies. A leak through prompt injection is a breach like any other.

The description above is informational and is not legal advice.

Why PWNONE

We build agent systems ourselves. DA1MON is a swarm of agents that runs reconnaissance in continuous testing, so we have made the same decisions on our own side: what permissions a single tool gets, what is allowed into the context, and what the application does with the model answer.

Frequently asked questions

How does testing an LLM application differ from testing an ordinary web application?

One layer is added, the one where data and instructions are the same text. Classic testing assumes input can be separated from logic, and with a language model that assumption does not hold. The rest stays: backend, API, access control and output handling are tested as always, because that is usually where the impact materialises.

What does an LLM application test cost?

Roughly from PLN 12,000 net for an assistant without tools and from PLN 20,000 net for an agent with tools and RAG. The price depends above all on what the model can call and how many data sources it has. We prepare an exact quote after a short call about scope.

Do you test the model itself or the application around it?

The application. A vendor model such as GPT, Claude or Gemini is out of scope and we do not assess its internal safeguards. We test your deployment: the system prompt, the data sources, tool permissions, access control and how the answer is handled. That is where the risk you can influence lives.

Can prompt injection be fixed at all?

It cannot be removed with an input filter, because the model always reads external content as text. The impact can be limited: narrow the tool permissions, require confirmation for actions that change data, filter documents before retrieval rather than after, and treat the model output as user input. The report says which of those layers did not hold in your case.

Will the test affect production and API costs?

Scope, the test window and test accounts are agreed before we start. Limit and cost testing runs inside agreed bounds, and with agents that can write we agree which actions may actually be carried out. Some attempts we stop at confirmation so that production data is not changed.

A model answers differently every time. How do you report that?

We repeat each scenario and state how many attempts out of how many worked, along with the exact input and answer. A one-off hit is described as exactly that. Your team can then repeat the attempt and check whether a fix actually changed anything.

Does the test cover the API and the backend?

Yes, the API the application uses is part of this scope. Some of the impact of prompt injection only becomes visible on the backend, for example when the model assembles a database query or calls an internal service.

Do you need access to the code and the system prompts?

It is not required, but it shortens the test. Without them we work as an attacker would and part of the time goes into reconstructing the rules. With the prompts and the tool list in hand we know what is worth attacking straight away, so more time goes into scenarios.

Other testing scopes

We combine scopes into a single project whenever that works better for your environment.

Let us check it before somebody else does

Tell us what you want tested. You will get a scope, a date and a price, with no obligation.

Prefer to talk? Call +48 500 480 420 or write to contact@pwnone.com.