Build a forensic trail for every AI decision
Something you can use today.
Build a forensic trail for every AI decision
The Weekly Build
Every AI system in production will eventually be asked a hard question. Not "what can you do?" but "why did you do that?". Right now, most of those questions have no good answer. The decisions happen, outputs get returned, and six months later someone in legal or compliance wants to know the reasoning. There is nothing to show them.
That is the gap we are filling today.
We are building a logging layer. When an AI model receives a prompt and returns an answer, we capture the full context: the exact prompt, the model used, the tools called, the timestamp and a structured reason field. Everything goes into a local JSON file you can inspect later. This is not explainability in the academic sense. This is forensics. When something goes wrong, or when someone asks for accountability, you have the data.
The setup takes about twenty minutes.
Create a folder called `ai-forensics` and initialise a Node project. You need `dotenv` for handling API keys cleanly. Run these commands:
mkdir ai-forensics
cd ai-forensics
npm init -y
npm i dotenvCreate a `.env` file and add your API key and model name. The model name is important because you want to know later whether version changes affected outputs.
Create `run.js` with the logging function. It checks if the log file exists, reads it, appends the new entry and writes everything back. The entry structure includes a UUID for each run, an ISO timestamp, the system prompt, the user prompt, any tools called and the response text.
Replace the placeholder response section with your actual model call. The logging structure stays fixed. When you add new models or change providers, the log format does not change.
Now add the reason field. After your model returns text, ask it to return structured JSON with three fields: the answer, a reasoning summary and a confidence level of low, medium or high. Capture that verbatim. Later, when auditing decisions, you want to see not just what it said but what it thought and how certain it was.
Run the script with a prompt:
node run.js "Write a short risk note for an insurance claim review based on this summary: [paste]"Open `ai-run-logs.json`. You will see every run recorded with a complete forensic trail.
This is the foundation. If you later add a prompt injection detector to your pipeline, the logging layer is already in place to show what was attempted and how it was handled.
This Week's Challenge
If you already use AI in your business, do not build anything new. Instead, find one automated AI process you run regularly and write down what you would need to explain that decision later. The exact prompt, the data used, what the output was. If you cannot answer those three questions from memory, that process needs the logging layer above.
Try it and reply with what happened. I read every one.
Forward this to one person who should be using AI better than they are. Reply with what you built, tried or broke this week. I read every one.
Gareth, founder of The Anthropic Stack (theanthropicstack.com)