How to Tell Whether Your Context Layer Is Actually Working

A context layer is working when it retrieves the right content, retrieves only content the person asking is entitled to, produces answers supported by what it retrieved, and reflects what the source systems say today. Those are four separate measurements, and a system can pass any one of them while failing the other three. Most teams track none of them, because the demo answered the question that was asked and nobody wrote down what “answered well” would have meant.

Every vendor in this category sells the layer. Almost nobody publishes how to tell whether the one you bought, or built, is any good. What follows is the measurement reference.

Four questions, four different measurements

The most common failure in evaluating these systems is collapsing all four questions into one judgment, usually “the answers seem good,” which is unfalsifiable and unactionable. When an answer is wrong you need to know which stage failed, because the fixes live with different teams.

Question	What you measure	How you measure it	What a bad reading means
Did it find the right content?	Retrieval quality: recall@k, precision@k, NDCG, MRR	Score a labeled set of real queries, graded for relevance, against what retrieval returned	The generator never saw the answer, and no prompt work or model upgrade fixes that.
Did it find only what this person may see?	Permission correctness: over-retrieval and over-restriction rates	Replay the retrieved set against the source systems’ own access decisions for that user	Over-retrieval is an access incident waiting to be found; over-restriction is a confident answer nobody reports.
Is the answer supported by what was retrieved?	Groundedness, answer relevance, and citation accuracy	Claim-level checks against the retrieved passages: human review on a sample, automated judging at volume	The system is fabricating, or citing real documents that do not contain the claim.
Is the context current?	Freshness: index lag, permission lag, and content drift	Timestamps compared between source and index, plus scheduled permission-change probes	Answers are correct about a world that no longer exists, and permission lag fails unsafely.

Treat that table as the contract. When a vendor, or your own team, reports a single accuracy score, ask which rows it covers. The honest answer is usually part of row one and part of row three.

Retrieval quality: did it find the right content

Four metrics carry most of the weight.

• Recall@k is the share of all relevant documents for a query that appear in the top k results. It answers whether the right material made it into the window at all.
• Precision@k is the share of the top k results that are relevant, so it measures how much noise the generator reads past.
• NDCG (normalized discounted cumulative gain) scores the ranking itself, rewarding highly relevant results near the top and discounting ones buried lower, normalized so a perfect ordering scores 1.
• MRR (mean reciprocal rank) averages one divided by the position of the first relevant result, measuring how fast a reader reaches something useful.

Recall matters most for a context layer, because a model can ignore an irrelevant passage but cannot invent one that was never retrieved. Precision and ranking still matter, since context windows are finite and noise costs money and latency, but a recall failure is unrecoverable downstream. That inverts the instinct of teams who come from classic site search.

Use graded relevance rather than binary judgments. A binary label throws away the difference between a passage that fully answers the question, one that contains part of the answer, and one merely on the right topic. A four-point scale keeps that information, which is what NDCG needs to mean anything; NDCG over binary labels is a weaker metric wearing a sophisticated name.

When retrieval is poor, the cause is usually upstream of the ranker: passages split so the answer straddles a boundary, which is why chunking is the foundation for accurate retrieval, or a vocabulary mismatch that only hybrid keyword and vector matching resolves, as this guide to semantic search explains. Measure first, check ingestion second, tune the ranker third. Teams reliably do this in reverse.

Permission correctness: did it find only what this person may see

Permission correctness is, of the documents retrieved for a request, the share the asking user was actually entitled to open, measured by replaying the retrieval set against the source systems’ own access decisions for that user. That last phrase is the whole method: you are not re-implementing SharePoint’s permission model in your test, which would only prove two of your implementations agree. It also assumes the request carried the asking person’s identity in the first place, which is a separate problem and a common point of failure. You ask the system of record, per document and per user, whether access would be granted, and compare that to what retrieval returned.

Over-retrieval is the agent seeing what it should not: a restricted document reaches the context window, and its content or a summary of it reaches someone not entitled to it. It is the failure everyone imagines, and it eventually announces itself, because the recipient often recognizes they should not have it.

Over-restriction is the opposite: content the person is fully entitled to is withheld, usually because a group membership did not expand, an inherited permission was not captured at ingestion, or a conservative default dropped a document whose access control list could not be resolved. The answer comes back confident, fluent, cited, and wrong by omission, because an answer built from eight of eleven relevant documents looks exactly like one built from all eleven.

Over-restriction is the quieter failure and, in most deployments, the more common one, because every ambiguous case in an ingestion pipeline is resolved in the safe direction by design. Teams only ever catch the first kind. Report two numbers, since a single permission accuracy figure hides the trade: you can drive over-retrieval to zero by returning almost nothing.

Sample continuously rather than testing once. A permission model is not a property of your software; it is your software plus the current state of every group, role, and sharing decision in every connected system, and that state changes daily. Replay a rolling sample on a schedule and alert on movement, because a certification done once during procurement describes a configuration that no longer exists.

Groundedness: is the answer actually supported by what was retrieved

Groundedness asks whether every claim in the answer is supported by a retrieved passage. Answer relevance asks whether the answer addresses the question. They need separate measurement, because an answer can be perfectly grounded and useless, faithfully summarizing three documents unrelated to the question, or highly relevant and entirely fabricated. The remedies differ: groundedness failures point at the generator, relevance failures point back at retrieval.

Measure groundedness per claim rather than per answer: decompose the generated text into individual assertions and check each against the retrieved set. What you want is a rate plus the list of unsupported claims, which is more useful to an engineer than a score. All of this depends on the retrieved set having been recorded at the moment of the answer, which is the first thing an answer log has to hold.

Citation accuracy is its own check. A citation pointing at a real document that does not contain the claim is worse than no citation, because it converts healthy skepticism into misplaced confidence; nobody opens the fourth source. Measure the share of citations where the cited passage supports the specific claim attached to it, and treat a drop as urgent even when groundedness looks stable, since the two move independently when a system attributes correct claims to the wrong source. Being able to see why a result ranked where it did, the aim behind making search rankings explainable, gives you somewhere to look when a citation misfires.

Freshness: is the context current

Index lag is the time between a change at source and that change being retrievable. Measure it per connector and report a distribution rather than an average, because one slow connector disappears into a blended figure and the tail is what produces the incident.

Content drift is the share of indexed documents whose source has since changed, been superseded, or been deleted. It catches documents withdrawn at source that still answer questions.

Permission lag is the time between an access change at source and that change taking effect in retrieval, and it is the dangerous one, because it fails silently and in the unsafe direction. Someone changes role on Monday morning, their directory groups update, and the index does not know until the next permission sync, so the agent keeps answering from content they lost access to hours ago. Nothing errors, nothing logs, and the answer is excellent. Measure it separately from index lag, since the permission schedule is almost always the slower of the two.

Building an evaluation set you will actually maintain

Everything above depends on an evaluation set, which teams skip because it is unglamorous and nobody’s job. Build it from real query logs, sampling three bands: head queries many people ask, torso queries that recur across a team, and tail queries one person asked once. The tail is where the system fails and where invented benchmarks never go, because people writing test questions produce head queries without realizing it.

A few hundred queries is the right order of magnitude, for practical rather than statistical reasons. It is small enough that domain experts will finish labeling and agree to refresh it, and large enough that a real change in retrieval moves the aggregate beyond labeling noise. Below about a hundred, single disagreements swing your metrics; above a thousand, labeling quality falls off.

Labeling belongs to people who know the domain, not to the team that built the system. Have two people label an overlapping subset and examine disagreement instead of averaging it away: a query two experts score differently usually reveals an ambiguous question or a conflict in the corpus, and both are findings.

Refresh quarterly, and sooner after a large migration or reorganization. Be honest that the set decays as the corpus, the business, and the questions change. An unmaintained evaluation set is worse than none, because it produces stable, reassuring numbers that correspond to nothing and get reported upward with unearned confidence.

Where LLM-as-judge helps and where it drifts

Using a model to grade output makes volume affordable, and it has documented failure modes. Position bias: judges favor whichever candidate is shown first, so score in both orders and average. Verbosity bias: longer, more fluent answers score higher regardless of correctness, which is backwards for grounded enterprise answers. Self-preference: a judge prefers output from its own model family, so when judge and generator share one you are partly measuring stylistic affinity. Prompt sensitivity: scores move with small rewordings, so version the judge prompt and treat a change to it as a change to the instrument.

Calibrate against human labels on a subset every cycle and report the agreement rate next to the judged scores. Agreement on four cases in five makes the judged number a usable proxy for movement; three in five means you are reporting the judge’s taste.

It is genuinely good at groundedness and citation support checking at volume, because asking whether this claim is supported by this passage is a narrow, local, verifiable judgment that models make well and cheaply. It is unreliable at fine-grained relevance grading, where separating a passage that fully answers a question from one that partially answers it depends on what the asker already knows, what the organization means by particular terms, and which of two conflicting documents is authoritative. That is domain judgment. Keep humans on relevance, use models for support checking.

What the numbers look like in practice

This article will not give you target numbers, because a number quoted without the corpus and evaluation set that produced it is decoration. NDCG of 0.82 on a clean public benchmark and 0.82 on a sprawling corpus of overlapping policy documents are not comparable quantities, and the second is far harder. There is no universal good score.

Read your own numbers instead. Establish a baseline before changing anything, since without it every later improvement is an anecdote. Measure deltas on the same set with the labeling method held constant. Set thresholds internally, such as a maximum over-retrieval rate or a permission lag ceiling, derived from your regulatory position rather than someone’s benchmark.

Treat any absolute threshold quoted by a vendor with suspicion, including ours. Ask what corpus produced it, how relevance was labeled and by whom, and whether the same method on your content would give a comparable figure.

Lucidworks’ position is that measurement belongs inside the platform rather than beside it: permissions resolve at query time so correctness can be replayed, chunking and ranking are inspectable so a retrieval failure traces to a cause, and Lucidworks AI exposes the retrieval set behind every answer so groundedness can be checked against what was actually retrieved. That is the design intent behind the AI Context Data Platform, the Lucidworks name for its approach in this category. Hold it to the same four questions as anyone else’s.

A thirty-day measurement plan

Week	What you do	What you have at the end
Week 1	Sample real query logs across head, torso, and tail. Agree the graded scale and recruit two or three domain labelers. Log the retrieved set per request, not just the answer.	A query sample, a scale, named labelers, and the logging everything else depends on.
Week 2	Label the set, with an overlapping subset for agreement, and adjudicate disagreements. Compute recall@k, precision@k, NDCG, and MRR.	A labeled set and a retrieval baseline you can defend.
Week 3	Build the replay harness: for a sample of requests, ask each source system whether the asking user may open each retrieved document. Compute over-retrieval and over-restriction separately, and measure both lag types per connector.	Two permission numbers, two lag numbers, and your first real surprise.
Week 4	Run groundedness and citation support checks, automated at volume and human-reviewed on a sample, with the judge calibrated against those labels. Assemble one scorecard.	A baseline across all four dimensions, a threshold for each, and a named owner.
Ongoing	Re-run the set on every material change to ingestion, retrieval, or prompting. Sample permission correctness continuously, and refresh the set quarterly.	Deltas rather than anecdotes, and a number to show when someone asks whether it is working.

Frequently asked questions

What is a good NDCG score?

There is no universal threshold, and anyone offering one is not describing your corpus. NDCG depends on how relevance was graded, how many relevant documents exist per query, how deep k is, and how much near-duplicate content sits in the index, so one system scores very differently on two corpora. It is meaningful only as a delta against your own baseline.

How do we measure permission correctness without causing a security incident?

Replay rather than probe. Capture the documents retrieved for a real request, then ask each source system, in a test harness, whether that user would be granted access to each one, and compare the two sets. Nothing additional is exposed, because the harness checks decisions already made and returns a boolean rather than content. Run it on a rolling sample and record both failure directions.

Can we just ask users whether the answers are good?

User feedback is a useful signal and insufficient alone. Users can tell you an answer was unhelpful or obviously wrong, which is worth collecting. They cannot tell you it was missing three documents they were entitled to but never saw, that a citation points at a document which does not contain the claim, or that the content was six weeks stale, because all three look identical to a correct answer from their side.

How many queries does an evaluation set need?

A few hundred, sampled across head, torso, and tail, suits most enterprise deployments. Below about a hundred, a handful of label disagreements swing the aggregate; above a thousand, labeling quality degrades and the set stops being refreshed. The binding constraint is whether domain experts will label it again in three months.

How often should we re-run the evaluation?

Run the full set on every material change to ingestion, chunking, retrieval, or prompting, and treat it as a release gate rather than a report. Sample permission correctness and freshness continuously, because they depend on connected systems rather than your code and can degrade on a day you shipped nothing. Refresh the set quarterly, and after any reorganization that changes group structure.

Who should own this, the search team or the security team?

Split it by dimension and name a person for each. Retrieval quality and groundedness sit with the platform team, who can act on the result. Permission correctness sits jointly with security and identity management, because the fix usually lives in group mapping or connector configuration rather than retrieval, and freshness sits with whoever owns the connectors.

Does a vendor benchmark tell us anything useful?

It tells you what the vendor chose to measure, which is informative in its own way. Ask what corpus produced the number, how relevance was labeled and by whom, and whether the same method run against your content would give a comparable figure. A vendor who answers all three and offers to run the method on your data is being useful; a single accuracy percentage with no method behind it is marketing.