Thoughts / 04

Why your AI is guessing about your business.

Try this before you read any further. Open whichever AI assistant you use and ask it something only your own records could answer. How many jobs did we book last month. Which of our posts did best this year. What did we spend with that supplier.

You will get one of two things. Either it tells you it has no way to know, which is the honest answer. Or it gives you a number, and the number is fluent, specific, formatted like a real figure, and completely invented.

Why it does that

The model has read an enormous amount of the public internet. It has never seen your booking system, your point of sale, or the spreadsheet on the machine behind the counter. When you ask about your business, it has nothing to look at, so it does the only thing it can do: produces the shape of an answer that questions like yours usually have.

The uncomfortable part is that it cannot tell the difference between knowing and guessing. It is not being evasive. It genuinely has no signal that this particular question was one it should have declined.

What changed

Connecting an assistant to your actual systems used to be bespoke work every time. Every tool, every assistant, its own wiring.

There is now a common standard for it, called the Model Context Protocol, usually shortened to MCP. It is worth about one paragraph of your attention and then you can forget the name.

Think of it as an agreed plug shape. Rather than every AI needing custom wiring into every system, a system can expose its data once, in a standard way, and any assistant that speaks the same standard can use it. It is an open standard rather than one company's product, which matters mostly because it means you are not betting your setup on a single AI vendor still being your preference in two years.

What it actually changes

The difference is between an assistant that can talk about your industry and one that can answer about your business.

The second kind is a different tool. You can ask it the sort of thing you would ask a well informed employee. Which jobs ran over the estimate this quarter. Whether that supplier's prices moved before or after we renegotiated. What we were doing differently in the month everything worked. Questions you currently answer by exporting something and staring at it, or more often by not answering them at all.

What it does not fix

It does not make the assistant trustworthy in general. It narrows what it can be wrong about.

An assistant that can look up the number stops inventing the number. It can still misread what the number means, miss the context that made that month unusual, or answer a subtly different question from the one you asked. Somebody still reads the answer. What goes away is an entire category of confident fiction, which is the failure most likely to cost you something, because invented figures look exactly like real ones.

Two honest limits

Your data has to be somewhere reachable. A system with a modern interface, or a database somebody can query. If the answers live in a filing cabinet, or in one spreadsheet on one laptop, that is a different and earlier project, and worth doing first for its own reasons.

Something has to be built and looked after. This is a small piece of software, not a setting. It needs to keep working when the systems it connects to change, which they will.

The question everyone asks next

You are giving an assistant access to business data, so the sensible next question is where that data goes.

The useful thing to understand is that you decide what the connection can see. It can be read only, so nothing can be changed by accident. It can be scoped to specific data rather than everything, which usually means totals and trends without customer names attached. And what it exposes is a decision made while building it rather than a property of the technology.

Whatever it does expose is readable by whichever assistant is asking, so that choice deserves the same thought you would give any other vendor holding your data. My own default is to expose the minimum that answers the question, and to leave personal details out unless there is a specific reason they need to be there.

Where to start

Do the test at the top, so you have seen the problem rather than taken my word for it. Then write down the two or three systems that hold the answers you most often want and cannot easily get.

That list is the project. I build these, and the build is usually smaller than people expect, but the list is worth making whether or not you ever hand it to anyone.

Recognise your own week in that?

Bring the task. Thirty minutes, no cost, and a straight answer about whether it is worth building anything.