LLM vs a
deterministic workflow.

Two ways to read
the same page.

This is a real engagement in the construction and real-estate world. The client is a national homebuilder, so the specifics are anonymised, but the numbers are exactly as we found them. Before the comparison makes sense, the two systems being compared need to be clear, because people use "AI" for both and they are not the same animal.

One is a large language model. It looks at the rendered page, the way a person would, and reasons about what the page means. The other is a deterministic workflow: it breaks the page into signals, what objects are in the image and what words are in the text, and runs fixed rules over them. Here is the honest line between them.

One clarification, because it trips people up: both systems use machine learning. The deterministic workflow leans on a computer-vision model to detect objects and an embedding model to compare text. The dividing line is not "AI versus not AI." It is whether a generative model makes the final call, or whether machine learning only produces signals that fixed, readable rules then judge. That second shape is what makes a workflow deterministic: the same page in, the same verdict out, every run.

Want how each system was actually built, step by step? That is the project write-up. This page is about the results of putting the two head to head.

The check nobody
wants to do by hand.

The client runs thousands of promotional emails and landing pages: new-community launches, limited-time rate offers, "explore homes" grids. Every creative carries a wall of legal fine print. Rate disclosures. Square-footage caveats. Fair-housing language. In regulated real-estate advertising, the wrong disclaimers missing is a compliance exposure with real teeth.

The catch is that the correct disclaimers depend on what is actually in the creative. Advertise a promotional interest rate, and you need the rate disclosures. Show a photo of people, and fair-housing language kicks in. Mention square footage, quote a price, depict an amenity, describe a self-guided tour: each triggers its own required block of text, drawn from a rulebook of well over a hundred options. A human reviewer reads each creative, works out which disclaimers apply, and assembles the fine print. It is slow, repetitive, rules-heavy judgement work, exactly what teams reach for AI to automate.

The client already had a version running: it sent a screenshot of each creative to an LLM, which read the rendered image and returned the required disclaimers. It worked well enough to be in production. But it cost money on every creative and it was slow, close to a minute per document. So they asked a reasonable question. Could a deterministic workflow, with no model call in the decision, do the same job?

Getting the deterministic workflow to that point took the slow kind of work. We built an audit corpus of 150 records and ran more than 2,000 test passes over them to settle the final, no-LLM process, one setting at a time. The comparison here uses the outcomes from the most recent run.

To make it, we pointed both systems at the same 83 production creatives and diffed every decision they made. (89 had been run through both; six deterministic runs failed closed on source pages that no longer loaded, expired promos and dead preview links, so we compared the 83 where both produced a full audit. That fail-closed behaviour is a feature: the deterministic system refused to publish an audit of a blank page, where the LLM had happily audited those same pages while they were live.)

Fifty-nine percent.

Across those 83 creatives, the two systems made 495 individual disclaimer decisions, each one a specific option considered on a specific creative. They landed on the same answer 294 times. That is 59%.

If you are evaluating a cheaper replacement for a production system, and it agrees with the incumbent barely three times in five, the conversation is over. You do not ship that. We were most of the way to writing it up as a negative result. Then we did the thing that changed everything: instead of trusting the headline number, we looked at what the 201 disagreements actually were. And the number inverted.

Same agreements.
Two denominators.

When two systems each produce a list of applicable disclaimers, there are three ways they can differ on any given option, not one. They can contradict each other: both considered it, one said required, the other said not. That is a real disagreement. Or one raised it and the other never did: not a contradiction, a coverage gap, one engine considering something the other never brought up. Or they agree.

The raw 59% treats every coverage gap as a disagreement. It should not. A silent option is not a wrong answer, it is a different question being asked. When we separated the two, the 201 disagreements broke down like this.

Recompute agreement on the decisions where both systems actually made a call, where a comparison is even meaningful, and it moves from 294 out of 495 to 294 out of 305.

That reframe alone was worth the engagement. The dashboard had not measured whether the deterministic engine was wrong. It had measured whether the two engines look at the same set of things, which is a completely different property. But it set up a second trap, subtler and more dangerous.

More is not
better.

With 96% concordance in hand, we started writing the satisfying version of the story. It went like this: the two systems have clean, complementary specialties. The deterministic engine dominates anything written in the copy. The LLM dominates anything shown in the picture. So ship both, route each disclaimer to whichever engine is stronger, and you get the best of both. Tidy. We had the charts drawn. Then the whole thing fell apart on one question: does detecting more actually mean being right?

"The LLM flagged people 23 times" is not a point in the LLM's favour. It is a bigger number. If those extra flags are correct, they are valuable coverage. If they are wrong, they are noise, false positives a human reviewer now has to notice and clear, which is worse than useless, because it erodes trust in the whole system. A count tells you how often an engine fired. It tells you nothing about whether it should have. We had built our entire comparison out of counts. And "which system is better" is a question about right, not about did. So we stopped comparing the two systems to each other and started comparing each one to the ground.

They fail in
opposite directions.

Here is where the two systems part ways the hardest. Read it top to bottom and the crossover is the whole story: the LLM fires more on anything in the picture, the deterministic engine fires more on anything in the text. Neither is winning. They are over-firing in different places, for opposite reasons.

Where the LLM over-reads the picture

The single largest gap is People Depicted, the fair-housing disclaimer required when a creative shows photos of people. The LLM fired it on 23 creatives, the deterministic engine on 5. On the surface, that looks like the LLM catching what a dumb image-labeller is blind to. It is the opposite. The written policy has an explicit carve-out: the disclaimer is not required when the only people shown are the sales team's headshots in the contact module, the "Contact Us" block that ends nearly every one of these emails. The deterministic engine handles this precisely, because the vision model returns bounding boxes, so the workflow localises every face and checks whether it sits inside the contact module. Its log literally reads 0 people outside the contact module / 8 total, and it correctly suppresses the disclaimer. The LLM sees faces and fires.

We checked every one of the LLM's extra flags. All 18 land on creatives where every detected person is inside the contact module. Eighteen for eighteen. Against the written exception, those are eighteen false positives: the sophisticated model ignoring a documented rule that the "dumb" workflow encodes exactly. The same pattern shows up on Off-Property Scenes, 16 LLM firings to 6, including a plain rate-table email with no off-property photograph anywhere on it.

But sometimes the LLM is genuinely right

Here is the part that keeps you honest. On Community & Amenities, a disclaimer required when amenities are described or depicted, the LLM again fired more, 36 to 24, and this time it was often correct. On a creative whose copy touted "resort-style amenities, pool, pickleball and basketball courts, dog park, playground," the LLM fired the disclaimer. The deterministic engine looked at the image, a staged interior shot, saw no amenities in it, and missed the ones plainly written in the copy. Better recall, same system, opposite direction. So the LLM's higher count is a false positive on one disclaimer and a genuine catch on the next, and you cannot tell which from the number. That is the whole lesson in miniature.

Where the rules engine over-matches the text

Now the mirror image. Take Limited Inventory / Select Homesites, the disclaimer for genuine scarcity claims. The deterministic engine fired it on 29 creatives, and not one of them contains the documented trigger ("limited inventory," "select homesites"). What they contain instead is "offer available on select new homes" and "limited funds are available," rate-buydown boilerplate that has nothing to do with how many houses are left. The words "select" and "limited" collided with promotional language and fired a scarcity disclaimer the creative never earned. The LLM fired it 10 times and, reading meaning rather than tokens, mostly did not take the bait. Market Conditions / Interest Rates is the same shape: the deterministic engine fired on 24 plain promotional-rate ads with zero market-commentary phrases in them, precisely the case the guideline says not to fire on. The LLM fired it twice.

One is a precision failure born of literalness, the other a precision failure born of over-interpretation. Neither is "better." They fail in opposite directions.

The counts that
are actually wrong.

Where a documented rule exists, we can catch a system firing against it. That is not a legal ruling, it is a floor on explainability: a firing with no documented trigger anywhere in the text is, at minimum, unexplained by the rules that are supposed to govern it. Three of those floors are stark.

These are not opinions. They are each system contradicting its own rulebook. Everything past that, the 11 true conflicts and the "better recall" cases, needs a human adjudicator, and that is exactly the queue a reviewer should see: small, high-value, and precisely where the systems are least trustworthy.

Clean signal,
near-perfect agreement.

The reassuring half of the result: where the trigger is clean and unambiguous, both systems are well-grounded and they agree almost perfectly. The disagreements cluster exactly where the rules are ambiguous or the judgement is visual. Where the signal is clean, the cheap system and the expensive system are indistinguishable, and you only learn that by adjudicating rather than counting.

Even the deterministic engine's 35 firings on a bundled-features program disclaimer, which looked over-eager at first, held up: 33 of them carry that program's standard-features tagline. It was reading the program block correctly, not hallucinating.

Nobody had graded
a single page.

Step back and here is what the whole exercise actually established. Every headline number, the 59%, the 96%, the 23-versus-5, the entire scoreboard, describes behaviour: how often each system fires, how broadly it matches, how the two overlap. Not one of them describes accuracy, because the thing that would establish accuracy did not exist in the data. The compliance tool has fields for exactly this, an operator-review status and a legal-feedback column, and on all 1,827 runs in the system they were empty. No lawyer had scored a single creative.

This is the trap stated plainly, and it generalises far past this one engagement: an evaluation that ranks two AI systems by counting and comparing their outputs will always crown the more trigger-happy one. More output looks like more coverage, and coverage looks like winning. The only thing that converts a count into a grade is an adjudicator with ground truth. Until one exists, "which system is better" is a question you have not answered, no matter how many decimal places your agreement metric carries.

Five rules for every
"which AI is better" call.

This engagement sharpened a checklist we now apply whenever a client asks us to compare two AI approaches. If you are running your own evaluation, these are worth stealing.

The system whose
mistakes have an address.

We recommended the deterministic workflow. Not because the scoreboard said 96%, by then we did not trust scoreboards, but because of a property that only surfaced once the counting stopped. On the calls it gets wrong, you can see why. A keyword collision on "select new homes" has an address: you can find the rule, tighten it, and re-run in about two seconds. A confident error from a model on a blank page does not have an address; you cannot open it up and fix the reason, because the reason lives in weights.

For a compliance workflow, where the failure mode that matters is a wrong disclaimer nobody can explain, and where the whole value is auditability, a system whose mistakes are legible and cheap to correct is worth more than one that is marginally more clever and completely opaque. That is an operability judgement, and notice when we made it: after the counting stopped, which is the only place a judgement like it can honestly be made. We also pushed hard on closing the loop, turning those empty review fields into a real human-adjudicated dataset, because without it the client has two AI systems that disagree in interesting ways and no way to ever know which one is right.

The scoreboard felt like an answer. It was a to-do list.

Common questions.