What AI Actually Knows About Your Project - And Why Context Quality Determines Your Results
The status report that’s accurate one week and off the next. The risk summary that reads well in testing and drifts in production. Most AI output inconsistency traces back to the same variable: the model didn’t have the right information.
Not a model problem. Not a prompt problem. A context problem - and the fix isn’t about using less. It’s about loading what the task actually needs.
What Goes In Determines What Comes Out
Context windows are now large enough to load your entire project history and still have room left over. That’s made a quiet problem worse: more fits, so more gets loaded, and nobody stops to ask whether it should.
The problem is that more isn’t neutral. Unverified information from web searches sits alongside accurate project data. Documents that were current six months ago get loaded alongside this week’s status update. Previous session outputs accumulate without a clean reset. The model synthesises whatever it receives - and nothing flags that the synthesis is partially built on stale or irrelevant input.
The output looks right. The logic holds. The figures don’t stack up when someone checks the source.
Agents Accumulate Problems You Can’t See
The issue compounds when agents are involved. A single-session prompt has a clean context - you provide it, you see it, you control it. An agent running across multiple sessions, pulling from web searches, accessing live project data, building on previous outputs - that agent is accumulating context you never reviewed.
Ben Tossell of “Ben’s Bites”, who has done serious work on AI agent design, describes it as context contamination. An agent runs a web search to validate a risk. The search returns a result with outdated data. That data gets loaded into context alongside your accurate project information. The agent synthesises both. Nothing flags that the synthesis is partially wrong.
The failure is invisible until it isn’t. And by the time you notice - a steering committee pack with figures that don’t stack up, a risk assessment based on a vendor document that was superseded six months ago - the output has already done its damage.
This isn’t a reason not to use agents. It’s a reason to be deliberate about what enters their context at each stage.
Context Quality Doesn’t Manage Itself
Context quality isn’t something you monitor passively; it’s something you design actively. And the tool for that already exists in every PM’s toolkit: a simple specification document.
A context design doc maps what information enters each workflow step, at what stage, in what format, and how it’s validated before the model sees it. It’s the same thinking as a data flow diagram - inputs, transformation, outputs - applied to an AI session rather than a system architecture.
In practice, it answers a small set of questions for each workflow:
What is the model’s knowledge boundary for this task? What it knows from training, what you’re providing, and what gaps remain.
What’s the freshness requirement? A project status prompt needs data from this week. A lessons learned summary can draw on the full project history. The scope of the task should determine the scope of the input.
How large is the input, and does it fit cleanly? Context windows are now large enough that fitting everything in rarely feels like the constraint - which makes it easier to load too much without noticing. Large document sets - technical specs, contracts, historical reports - are usually better summarised into a single reference file than loaded in full. The question isn’t whether it fits. It’s whether it belongs.
What’s the validation step before the output goes anywhere? Not a full review of everything, but a defined check. For a status report, it might be verifying the RAG status against the source data. For a risk synthesis, confirming the top three risks match what’s actually in the register.
What the Document Looks Like in Practice
The context design doc doesn’t need to be elaborate. A one-page table per workflow covers it: step name, inputs required, freshness, size constraint, and validation checkpoint.
The value isn’t in the document itself - it’s in what writing it forces you to do. You can’t fill in the “inputs required” column without confirming you actually have those inputs in a usable format. You can’t fill in the “freshness” column without deciding how stale is too stale. You can’t fill in the validation checkpoint without knowing what a correct output should look like.
Input requirements left implicit is where the variation comes from. The model works with whatever it receives.
This Is Already PM Work
What I’m describing is requirements definition applied to a different kind of workflow. When we specify a system, we document the data inputs and outputs. When we design a process, we define the triggers and checkpoints. A context design doc is the same thinking applied to the AI layer - familiar discipline, unfamiliar domain.
The PMs getting consistent results from AI workflows have mostly figured this out. Not by finding better tools, but by treating context as a managed input - something defined before the model is asked to produce, not assembled on the fly each time.
Frequently Asked Questions
Why does my AI workflow give inconsistent results even when I use the same prompt? Inconsistent results usually mean inconsistent inputs. The prompt is the same, but the context - the data, documents, or conversation history you load into each session - varies. Defining your context inputs explicitly for each workflow step is the most reliable fix.
What is a context window and why does it matter for AI project management tools? A context window is the amount of text an AI model can process in a single session. Staying within roughly 60% of the available window keeps output quality reliable. Loading too much - especially unverified content from web searches or long document sets - can degrade results without producing any obvious error.
How do I manage context in an AI agent workflow? Map what information enters each step of the workflow, verify its freshness and accuracy before it’s loaded, and keep each session focused on a defined scope. Summarise large documents into a single reference file rather than loading them in full. Build a validation checkpoint into the final step before any output is acted on.
Do I need technical knowledge to manage AI context quality? No. It’s a documentation problem, not a technical one. The same instinct that makes a PM good at specifying system requirements or documenting data flows applies directly here. The tool is a simple table.
The AI tools in your workflow are only as good as what you give them to work with. That’s been true of every system PMs have ever managed - the principle didn’t change, just the application.
How deliberately are you designing the inputs to your AI workflows right now - or are you mostly trusting that the model will work it out? I’d be curious to hear where people are drawing the line between ‘good enough’ and ‘needs a spec’.
Yes, AI helped me to write this :)