# Your AI assistant can leak files people aren't allowed to see — here's how to fix it

> **TL;DR**
> 
> *   Vector search ignores your permission model: an engineer can ask your RAG assistant about board-level financials and get them, because semantically close = retrieved, and a restricted chunk in the prompt can't be un-read.
>     
> *   Fix it with a **pre-filter**: Authorizer's embedded OpenFGA answers `list_permissions` with the *user's* token, and that allow-list becomes a Qdrant `must` payload filter — forbidden chunks are never scored.
>     
> *   Revocation is one tuple deletion, effective on the next question. No re-indexing, no re-login.
>     
> *   Verified live: 63 tests green (including a dedicated fail-closed permission suite), official `quay.io/authorizer/authorizer:2.3.0` image, real grant/revoke cycle. Run it: `docker compose up -d && python scripts/fga_seed.py && python scripts/fga_demo.py --llm`.
>     

* * *

There's a quiet security regression happening in almost every company shipping AI features right now.

You spent years getting access control right. Row-level security in Postgres. Folder permissions in Drive. Channel membership in Slack. Then someone embedded all of it into a vector database, pointed an LLM at it, and called it a knowledge assistant.

The assistant doesn't know about your permission model. Similarity search returns whatever is *semantically close* — the intern asking "how do raises work?" gets chunks of the compensation memo, because cosine similarity has no concept of "need to know." And once a restricted chunk lands in the prompt, it's over: you cannot redact what the model has already read.

The common fix is a *post-filter*: run the vector search first, **then** check each result against your permission system and drop what the user shouldn't see. Most off-the-shelf "permission-aware retriever" wrappers work exactly this way. It's better than nothing, but the forbidden chunks were still scored, fetched, and held in your application's memory before being discarded — and if 3 of your top-4 results get dropped, your agent answers from a starved context. When the checks run against a hosted authorization service, you also pay a network round trip per result, and your company's entire who-can-see-what graph lives on someone else's infrastructure.

We built it the other way around — as a **pre-filter**, fully self-hosted, on an open-source stack — and shipped it as a runnable demo. The demo's knowledge base belongs to a fictional **ACME Corp** and holds four documents: a public onboarding guide, an engineering tech-stack doc, a finance-only Q4 financial report, and a security policy. Picture the most relatable version of the leak: **an engineer asking the assistant about company financials.** Same question, two users, different answers. Here's the actual output:

```plaintext
👤 alice@example.com   (engineering)   may view: onboarding_guide.txt, tech_stack.txt
   ❓ What was our Q4 revenue and cash runway?
   💬 I don't have enough information in the knowledge base to answer this question.

👤 carol@example.com   (finance)       may view: financial_report.txt, onboarding_guide.txt
   ❓ What was our Q4 revenue and cash runway?
   💬 The Q4 revenue for ACME Corp was $14.2M, and the monthly cash burn is $1.1M —
      a 26-month runway at current burn.
```

Alice isn't being told "access denied." ACME's financial report simply does not exist in her world — it was never a search candidate, never retrieved, never in the prompt. The model cannot leak what it never saw. Carol, who's on the finance team, gets the number.

Here's the use case at a glance — the same question, the two outcomes, and the open-source stack underneath:

![](https://cdn.hashnode.com/uploads/covers/580f234f5fec191d85b1524a/dba13d70-2675-4b7d-a39b-7b171df2f0df.svg align="center")

## The stack — open source end to end

Every piece runs on your hardware, and each one earns its place:

*   [**Authorizer**](https://github.com/authorizerdev/authorizer) (≥ v2.3.0) — open-source authentication *and* authorization in one Go binary. It embeds an [OpenFGA](https://openfga.dev) engine (the open-source implementation of Google's Zanzibar relationship-based access control), so the server that issues your users' tokens also answers "can this user view this document?" — in-process, no extra service to deploy.
    
*   [**Qdrant**](https://qdrant.tech) — the vector database whose [payload filters](https://qdrant.tech/documentation/concepts/filtering/) make true pre-filtering possible: a `must` condition is evaluated *during* the HNSW vector search, not bolted on after. This single feature is what lets the permission boundary live inside the retrieval step. Its built-in dashboard (`/dashboard`) also lets you inspect exactly which chunks and payloads exist.
    
*   [**FastEmbed**](https://qdrant.github.io/fastembed/) — local ONNX embeddings (`BAAI/bge-small-en-v1.5`, 384 dimensions, ~25 MB). No embedding API, no per-token costs, works offline after the first download.
    
*   [**Ollama**](https://ollama.com) — the local LLM (`llama3.2` in the demo). Prompts containing your internal documents never leave the machine.
    
*   [**Gradio**](https://gradio.app) — the demo's web UI, now with a login row so each person chats as themselves.
    

Nothing in this chain phones home: not the documents, not the tokens, not the permission graph, not the prompts.

```bash
docker compose up -d              # Qdrant + Authorizer (quay.io/authorizer/authorizer:2.3.0)
python scripts/fga_seed.py        # demo users, authorization model, grants
python scripts/fga_demo.py --llm  # watch two users get different answers
```

## Permissions are relationships, not roles

The document side of the authorization model is six lines of OpenFGA DSL:

```dsl
type document
  relations
    define owner: [user]
    define viewer: [user, user:*, team#member]
    define blocked: [user]
    define can_view: (viewer or owner) but not blocked
```

Access is granted by writing *facts*, not code:

```text
user:*                    viewer   document:onboarding_guide.txt   # public
team:engineering#member   viewer   document:tech_stack.txt         # a team, not a user list
team:finance#member       viewer   document:financial_report.txt   # finance only
team:security#member      viewer   document:security_policy.txt
user:<alice>              member   team:engineering
user:<carol>              member   team:finance
```

That `team:finance#member viewer` tuple is the whole ballgame for our example: Alice is a member of `team:engineering`, not `team:finance`, so she has no path to `document:financial_report.txt`. Carol does.

Notice what's *not* here: no changes to the ingestion pipeline, no per-user collections, no ACL columns bolted onto chunk metadata. Each Qdrant chunk already carries its source filename in the payload — the FGA object is simply `document:<that filename>`. The payload you already have is the join key.

## How does it work? The architecture

Under the hood it's one Python app process talking to three local servers. The important part is the *order*: the permission check happens **before** the vector search, so off-limits files are never even embedded into the candidate set.

![](https://cdn.hashnode.com/uploads/covers/580f234f5fec191d85b1524a/543cdfb5-1189-4eb7-8560-6cdfcf0845f5.svg align="center")

It comes down to two API calls.

**One:** before searching, ask Authorizer what this user may see — authenticated with the *user's own token*:

```python
allowed = authz.allowed_documents(user_token)
# GraphQL: list_permissions(relation: "can_view", object_type: "document")
# Alice (engineering) → ["onboarding_guide.txt", "tech_stack.txt"]   # no financial_report
```

**Two:** hand that allow-list to Qdrant as a `must` condition:

```python
FieldCondition(key="source", match=MatchAny(any=allowed))
```

Qdrant evaluates the condition during HNSW traversal. Forbidden vectors are never candidates — top-k stays full of the best chunks the user *may* see, instead of being decimated by after-the-fact redaction.

And because the permission call carries the end user's JWT, Authorizer pins the subject server-side. An explicit "check as someone else" is rejected unless the caller is a super-admin. A fully prompt-injected agent has nothing to escalate with: it asks as Alice, it gets Alice's answer — and Alice can't see finance.

## Fail closed, or don't bother

The demo treats every ambiguous state as *deny*:

*   Authorizer unreachable → `AuthorizationError`, not an unrestricted search.
    
*   Authorization configured but no token supplied → error, not a silent bypass.
    
*   Permission list truncated → refuse, a partial allow-list is not the full one.
    
*   Empty allow-list → refusal answer *without ever calling the LLM*.
    
*   And after generation, the cited sources are batch re-checked with `check_permissions` — a grant revoked mid-request raises instead of leaking.
    

That last list is about 30 lines of code total. It's the difference between a security feature and a security theater feature.

## Revocation that actually revokes

The demo's second act grants Bob engineering access — one tuple write — and his very next question retrieves the tech-stack doc. Then it deletes the tuple:

```plaintext
⚡ Granting: bob → member → team:engineering
👤 bob   may view: onboarding_guide.txt, tech_stack.txt    ← immediately

⚡ Revoking the same tuple
👤 bob   may view: onboarding_guide.txt                    ← immediately
```

No re-embedding. No re-indexing. No re-login. No "takes effect after the nightly sync." Permissions live in tuples, not in the vector index — offboarding a contractor is one `_fga_delete_tuples` call, and their next question is denied.

## What we verified (and what we'd tell you anyway)

This isn't a whiteboard architecture. The demo ships a dedicated fail-closed permission suite (authz client, Qdrant pre-filter semantics, pipeline enforcement) — the full suite is **63 tests, all green** — and the whole flow ran live against the official `quay.io/authorizer/authorizer:2.3.0` image, including the grant/revocation cycle above.

Honest notes from the build: `list_permissions` caps at 1,000 objects (the client fails closed on truncation — model access via teams/folders to stay under it); contextual tuples work on checks but not on the list call; and Authorizer's CSRF guard requires an `Origin` header on server-side POSTs — your reverse proxy or HTTP client needs to send one.

## FAQ

### Do I need to re-index Qdrant when permissions change?

No. Permissions live in OpenFGA relationship tuples, not in the vector index. Granting access is one `_fga_write_tuples` call and revoking is one `_fga_delete_tuples` call — both take effect on the user's very next question. The embeddings, payloads, and collections in Qdrant are never touched.

### What happens if Authorizer is unreachable mid-request?

The pipeline fails closed: the permission client raises an error and the user gets no documents — never all documents. The same applies to expired tokens, missing tokens when authorization is enabled, and truncated permission lists. Treating every ambiguous state as "deny" is roughly 30 lines of code in the demo.

### Can a prompt-injected agent bypass the filter?

No. The agent calls Authorizer with the end user's own JWT, and the server pins the permission subject from that token — a "check as someone else" parameter is rejected unless the caller is a super-admin. A hijacked agent holds no privileged credential, so it asks as the user and gets the user's answer: denied.

### Does this work with LangChain or LlamaIndex?

The demo's permission calls go through the official [`authorizer-py`](https://github.com/authorizerdev/authorizer-py) SDK — `login`, `check_permissions`, `list_permissions`, all typed and fail-closed. There's no dedicated LangChain/LlamaIndex retriever wrapper yet, but the pattern (allow-list → vector-store metadata filter) ports directly to any retriever that supports pre-search filtering — Qdrant's LangChain and LlamaIndex integrations both expose payload filters. The demo's `authz.py` is the thin glue, and packaging it as a reusable retriever is the obvious next step.

### What are the current limits?

`list_permissions` caps at 1,000 objects per call (the client fails closed on truncation — model access via teams or folders to stay under it). Contextual tuples work on `check_permissions` but not on the list call, parameterized CEL conditions aren't exposed through the API, and the embedded FGA engine needs a SQL datastore.

## Run the demo yourself, in four steps

**Step 1 — Start the tools.** One command brings up Qdrant and Authorizer (with its embedded OpenFGA engine):

```bash
docker compose up -d        # Qdrant on :6333, Authorizer on :8080
```

**Step 2 — Seed ACME's users and permissions.** The seed script is idempotent: it signs up the three demo users, installs the authorization model, and writes the relationship tuples — all skipped on re-runs.

```bash
python scripts/fga_seed.py
# ✓ Created users alice (engineering), carol (finance), bob (new hire)
# ✓ Installed authorization model
# ✓ Granted: team:finance#member viewer document:financial_report.txt …
```

This produces ACME's access matrix:

| document | alice (eng) | bob (new hire) | carol (finance) |
| --- | --- | --- | --- |
| `onboarding_guide.txt` | ✅ | ✅ | ✅ |
| `tech_stack.txt` | ✅ | ❌ | ❌ |
| `financial_report.txt` | ❌ | ❌ | ✅ |
| `security_policy.txt` | ❌ | ❌ | ❌ |

**Step 3 — Log in.** Launch the UI; it asks you to log in first (it can't answer anything until it knows who you are), with the demo accounts shown right on the page:

```bash
python src/app.py --authorizer http://localhost:8080
```

> Log in as `alice@example.com` (engineering) or `carol@example.com` (finance), password `Demo@Pass123`. The login screen is shown first — the credentials are verified against Authorizer, and the chat only appears once you're authenticated.

**Step 4 — Ask the same question as different people.** Ask *"What was our Q4 revenue?"* as Alice and you get *"I don't have enough information"* — the financial report was never retrieved. Ask Carol and she gets ACME's $14.2M. Prefer the terminal? `python scripts/fga_demo.py --llm` runs the whole comparison, including a live grant-and-revoke.

### Links

*   **Demo**: https://github.com/lakhansamani/qdrant-rag-llm-example/tree/main
    
*   **FGA guide** (models, patterns, full DSL reference): https://docs.authorizer.dev/core/fga-guide
    
*   **Authorization API**: https://docs.authorizer.dev/core/graphql-api
    
*   **GitHub**: https://github.com/authorizerdev/authorizer ⭐
    
*   **SDKs**: [Python](https://github.com/authorizerdev/authorizer-py) (used in this demo) · [Go](https://docs.authorizer.dev/sdks/authorizer-go) · [JavaScript](https://docs.authorizer.dev/sdks/authorizer-js) — `check_permissions` / `list_permissions` Ship in all three.
    
*   **Discord**: https://discord.gg/3Jse9RtS
    

Your agents are only as trustworthy as the boundaries you give them. Give them real ones — on your own hardware.