Why We Trained Our Own Plan-Reading Models (Instead of Wrapping a Vision API)

There's a fast way to put "AI" in a takeoff product: screenshot the plan, send it to a general-purpose vision model, print whatever comes back. It demos beautifully. A wave of construction tools launched exactly this way over the last two years.

We went the slow way instead: we train our own neural networks, in-house, on thousands of real U.S. permit plans, and we ship them as tools that produce measurable geometry on your sheet. This post is about why that difference isn't a technical footnote — it's the whole ballgame for whether you can bid the number.

What a wrapper can't do

General vision models are astonishing at describing pictures. But a takeoff isn't a description — it's a measurement. Three things break when the "AI" is a wrapper:

1. No scale. A vision API sees pixels. It doesn't know your sheet is 1/4″ = 1′-0″, doesn't know the E-sheets are a different scale than the site plan, and can't return coordinates in the drawing's own system. Whatever length it claims, nothing ties it to the sheet.

2. No geometry, no audit. When a wrapper says "the master bedroom is about 210 square feet," where did that come from? You can't see the boundary it assumed, so you can't check it. An estimator who can't verify a quantity can't sign a bid on it. Our tools return the actual polygon or wall lines on the sheet — the number and the geometry that produced it, together.

3. Wrong training diet. General models learned from the internet's photos. Permit drawings are a different visual language: wall poché, dimension strings, leaders, hatching, schedules, title blocks — dense, monochrome, and full of things that look like lines but aren't walls. A model that hasn't lived on this material confuses decoration for structure, exactly where the money is.

What owning the models buys

Our one-click wall tracer and one-click Room SF tool are segmentation networks we trained ourselves. Owning the whole stack changes what we can promise:

• Trained on the real thing. Thousands of cleanly-labeled floor plans, drawn from a much larger library of actual permit sets — the messy, dense drawings contractors bid every day. The hardest part was never the network; it was building that labeled dataset at scale. We developed our own way to produce those labels from real plans (that method stays in-house — it's the moat), which let us train on far more real-world variety than hand-labeling ever could.
• Purpose-built post-processing. The model's raw prediction becomes estimator-grade geometry through steps a wrapper can't take: thinning wall predictions to centerlines and walking them junction-by-junction so corners survive; straightening runs against the plan's own tilt instead of forcing everything square (scanned sheets are rarely true); topologically testing which walls actually face outdoors so exterior vs. interior LF is exact on courtyards and bump-outs, not guessed.
• Fixable failure modes. When a tool misreads a plan, we can find out why and fix that — retrain, adjust the post-processing, add the case to the dataset. With someone else's API you file a ticket and hope. When we upgraded our room model, we compared candidates room-by-room against dimensioned plans and shipped the one with tighter boundaries, not the better leaderboard score. You can only make that call when the model is yours.
• A flywheel wrappers don't get. Every trace a user accepts or corrects becomes a new human-checked example on real working plans. The models sharpen on exactly the drawings our customers bid. That loop — real plans in, corrections back — takes both the data and the users, which is why it's hard to copy.

Where we're honest about limits

Purpose-built doesn't mean perfect, and we won't pretend otherwise (the wall model page says the same in public). Unusual drafting styles and rough scans can still confuse the models. That's why every AI-produced quantity in Foreman AI lands as editable geometry you review — the AI does the clicking, you keep the judgment. We think that's the only responsible way to put machine learning into a number someone signs.

The test that settles it

Don't take the architecture argument on faith — run the drawing test. Take a real plan from your backlog (not a clean marketing sample), and ask any AI takeoff tool three questions: Can I see the geometry behind the number? Is it in my sheet's real scale? Can I fix it when it's wrong? Three yeses and you've found a tool. Fewer, and you've found a demo.

Try ours on your plans — free trial, no charge today.