The new fashion in business AI: "ask your ERP in plain English". The owner types "how much did we make on customer X in August" into a chat window, the model translates it into SQL, hits the database, returns a number. In a demo it looks like magic. In production it can return a number that simply doesn't exist.
This isn't an article about AI "not working". Text-to-SQL works, and in specific use cases it's genuinely useful. The point is different: there's a condition without which all that magic turns into a very expensive generator of plausibly-looking nonsense. Almost no SME meets that condition out of the box — and the vendor selling it usually doesn't mention it.
Where the magic breaks
Text-to-SQL fails in SMEs not because the language model is too weak. It fails for three reasons, and they're always the same.
First — the tables in your ERP are cryptic. Comarch Optima, enova365, Subiekt, Microsoft Dynamics AX hold data in tables named things like CDN.TraNag, K_KLIENT, TWG_ID. Columns are called Kwo, TrN_TrNId, Typ_dokumentu. The model has no idea that Kwo is a net amount or that TrN_TrNId is a transaction header ID. It'll guess — sometimes correctly, sometimes not. Nobody checks, because the answer looks convincing.
Second — the definition of "revenue" in your company doesn't exist unambiguously. Ask three people in a 30-person business "how much did we sell in August" and you'll get four answers. Because "sold" can mean: amount on invoices (what about credit notes?), on orders (what about cancelled ones?), on warehouse releases (what about returns?), by sale date or invoice date (what about invoices issued late?). The model doesn't know. It'll query the first field that looks reasonable — and hand you a number.
Third — the model hallucinates relationships between tables. Classic case: the question "how much did we make on customer X" requires joining invoices with purchase price (to compute margin). In most ERPs the purchase price lives in a completely different table than sales, and the join key isn't obvious (product code with suffix, variant, unit of measure). The model glues it together by column-name similarity. You get a number that looks like a margin but isn't a margin.
A concrete example from this year
At one of our clients — a 40-person wholesaler of furniture parts near Wrocław, running Comarch Optima — the owner bought a "smart reporting assistant" from an outside vendor. The pilot ran three weeks and cost 28,000 PLN.
For the first two weeks everyone was thrilled. "How much of item X did we sell in July" — worked. "Which salesperson had the most orders last month" — worked. "Show me the top 10 customers by turnover" — worked. In the third week the owner asked about the margin on a specific product group. The model replied: "24.3%". A nice, concrete number.
Three days later the accountant calculated the same figure by hand for a report to the bank. It came out at 11.8%. The gap is because the model joined sales with the latest purchase price from the product record — but purchase prices in this company had risen by a low double-digit percentage over the year. Actual margin is calculated from the purchase price of the specific delivery the goods were shipped from. The model didn't know that, because nobody told it. It guessed. It guessed wrong.
Had the pricing decision for September gone out based on that number, the company would have been subsidizing half its assortment for a full quarter.
When text-to-SQL actually works
After that lesson, we got the same company to a sensible pilot. It took six weeks and required three things.
A reporting layer, not raw ERP tables. Instead of showing the model 400 Comarch Optima tables, we built six "reporting views" (SQL views) with clearly named columns: net_sales_document, purchase_price_from_delivery, margin_amount, issue_date, sale_date. The model only sees those views. It can't query the wrong table, because for it there are no wrong tables.
A written glossary of business terms. We attached a one-page document to the pilot: what exactly "sales", "margin", "turnover", "active customer" mean here. Without that document, even the version with views returns different numbers to the same question, because the model interprets words its own way. That glossary came out of a four-hour meeting between the owner and the accountant — and was the hardest part of the project, because along the way it turned out they had disagreed with each other all along.
A human between the model and the decision. Every number the company uses to change pricing, negotiate with a supplier, or sign a contract above 20,000 PLN goes through the accountant's eyes. The model speeds work up — it doesn't replace accountability. That's mundane, but you'd be surprised how many rollouts "forget" it.
After this pilot, the owner gets an answer in a minute to 80% of the typical questions that used to take 2–3 hours of the accountant's Excel work. The remaining 20% (complex multi-period analyses, corrections, edge-case documents) still goes through the accountant — and that's fine.
What to do before the pilot
If someone offers you text-to-SQL for your ERP, before you sign, ask four things.
- What data layer will the model actually work on? Raw ERP tables (run) or prepared views and semantic models (keep talking)?
- Who will prepare the glossary of business terms, and when? If the answer is "the model will figure it out from context" — it won't. That's work someone who knows your company has to do.
- How are the numbers the model returns validated? For the first month, is every answer compared with a manual calculation? Is there an audit that catches "24.3% instead of 11.8%"?
- What happens when the model answers wrong? Does the interface show the generated SQL so the accountant can verify which tables were used? Is there a "this is wrong, fix the definition" button?
If the vendor waves those questions off with "AI will handle it", you're looking at a demo, not a product. In Polish SMEs the demo works for three weeks. Then someone makes a decision on it, and the trouble starts.
What next
Text-to-SQL isn't nonsense. It's a tool that, set up right, genuinely saves the owner and the accounting team over ten hours a month. But "set up right" isn't a license and a login to the ERP. It's a reporting layer, a glossary of terms, and a human making sure the numbers add up at the end of the day.
If someone is already offering you something like this, or you're considering a pilot yourself — it's worth half an hour with someone who has stood these things up in other SMEs and seen where they collapse. First conversation is free.
A language model doesn't know your company. It knows column names — and guesses what they mean. Without a glossary and a reporting layer, it will guess wrong.