For your technical reviewers
The parts your engineers will actually interrogate
How a passage is found, how a citation is built, what we measure and what we have not measured yet, and how much of the stack you could swap out. Written for the people who will be asked to sign off on it, which is why it also says where we are weakest.
Retrieval
Where most of the engineering went. A good answer starts with the right passage.
Any model can write fluent prose. What decides whether it is right is what it was handed to read. So before a model sees anything, each question is planned, searched two ways, filtered, reranked and checked for gaps.
Illustrative example · one question, three requirements
The question
“What does alarm E-217 mean on the chiller, how do I reset it, and who approved the last firmware change?”
Requirement 01
What alarm E-217 means
Search · semantic phrasing, plus “E-217” matched literally
Rerank · 3 passages clear the threshold
Covered
Requirement 02
How to reset it
Search · reset procedure, same filters
Rerank · two procedures, for two chiller models
Needs clarification
Requirement 03
Who approved the firmware change
Search · approvals, change records
Rerank · nothing clears the threshold
Missing from your documents
The answer explains E-217 with citations, asks which chiller model before giving a reset procedure, and states plainly that no document records the firmware approval.
Five mechanisms, one job: hand the model the right passages.
Hybrid search that searches two different texts
Dense semantic search, multilingual, runs beside BM25 lexical search, and the two are fused with reciprocal rank fusion. The semantic half gets the conceptual phrasing; the lexical half gets the same query with the exact identifiers added. An error code or a part number is not “understood”. It has to match literally.
Query planning, not query rewriting
Each message becomes a typed plan: intent, language, the separate factual requirements and any format constraints. Every requirement gets its own search and its own reranking, so a compound question gets every part answered, not just the first.
Metadata filtering
Searches narrow by access group, source type, document and page range. The group filter runs inside each search, before results are fused, so the model never sees a passage the user isn't allowed to read. How access is enforced.
Reranking with a relevance threshold
A dedicated cross-encoder rescores the candidates for each requirement. If nothing clears the threshold, you are told that it isn't in your documents, instead of being handed the closest wrong passage.
A coverage ledger
Every requirement is marked covered, missing from the documentation or needs clarification, and the model is instructed to declare the gaps in its answer. A missing part is named, not quietly skipped.
Watch one question get searched
Two searches, fused, reranked, then gated.
Dense · semantic
0.81 0.74 0.58BM25 · lexical, “E-217” matched literally
0.92 0.61 0.44Fused by reciprocal rank fusion, then reranked against a threshold
Three passages clear the threshold and reach the model. The two that do not are never read into the answer, and if none had cleared it, you would be told the answer is not in your documents.
Accuracy & citations
Every claim points at its page. And a straight account of what we measure.
A citation you can open is worth more than a promise of accuracy. So the product shows its sources down to the fragment, is built so that gaps get declared rather than filled, and we tell you exactly which parts of its quality we measure today.
A cited claim
Alarm E-217 is a condenser high-pressure trip, raised when discharge pressure exceeds the set limit.
Chiller_Service_Manual.pdf · § 6.3 Alarm codes · p. 48 · scanned, OCR
“…E-217: condenser high-pressure trip. Unit stops; manual reset required…”
Illustrative example.
-
01
Claim to fragment
Each claim links to the document, section and page, plus the exact fragment that supports it. Not the whole chunk. -
02
Scanned pages included
Page numbers are real provenance from parsing, including documents that were read with OCR. -
03
Sources travel with the file
In a generated PDF or Word document, the references become a Sources section. -
04
“Where exactly does it say that?”
Literal search tools return offsets in the indexed text, not an approximate match.
Hallucination is handled in the architecture, not in a disclaimer
A relevance threshold
It prefers “that isn't in your documents” to building an answer on a weak passage.
A coverage ledger
Gaps are declared per requirement, so a missing part can't be papered over.
Deduplication
One passage can't appear as two sources confirming each other.
An optional verifier
Before delivery, it audits the answer sentence by sentence and removes what isn't supported.
Measurement
What we measure, and what we don't yet.
In use, we see far fewer hallucinations than from the same model used on its own. That is an observation, not a measurement, so it carries no number here.
Measured today
Retrieval quality, as a gate
Recall and MRR against a hand-labelled set. Thresholds block any retrieval change that makes them worse.
Latency, stage by stage
Time spent in planning, embedding, retrieval, reranking and generation, plus time to first token and tokens per second.
Behaviour under load
Load tests with p50 to p99 percentiles, and the cost of each request.
Not measured yet
Automated answer scoring
No automated scoring yet for groundedness, answer relevance or hallucination rate.
A larger evaluation set
The labelled set is small today, and we say so.
Next on the roadmap
Both are the next priority. Until they exist, we don't quote an accuracy figure.
Architecture
Built to change providers. It already runs on more than one.
The platform has run entirely on self-hosted models, with generation, embeddings and reranking all local, and on AWS Bedrock, without losing a feature. Here is how modular each layer is, including the one that isn't.
-
Generation model
Fully abstracted behind a neutral internal contract. Bedrock and Ollama are adapters; the tools never see the provider.
Swap the adapter (3 of 3)
No reindexing
-
Tools
Excel formulas, document generation, charts and web page reading are ours, described in standard JSON Schema.
Model-independent (3 of 3)
Nothing to change
-
Embeddings & reranking
Swappable by editing a module, not through a plug-in.
Edit a module (2 of 3)
A new embedding model means re-indexing the corpus
-
Vector store
Qdrant, used directly for hybrid search, RRF fusion and payload filtering. The most coupled layer.
A real refactor (1 of 3)
Not a configuration change
Your cloud: AWS Bedrock, in your own account
Inference runs in your own account and region. Bedrock does not retain what it is sent or train on it, and no third-party AI vendor sees your documents.
Your servers, or anywhere else: Ollama and other hosts
Run the model on your own servers with Ollama, or on any other infrastructure that can serve the model you pick, and nothing leaves at all. We configure it with you during deployment.
Bring the person who will say no. These are their questions.
Where permissions are enforced, what reaches the model, what is retained and what we have not measured. Nobody here will ask you to take it on trust.
30 minutes, no commitment. Your security team is the audience we want.