FloorDrafter Team Shares Six Fixes for AI Models That Ignore Reference Images

Developers at FloorDrafter, an AI floor plan tool, discovered that their model was not ignoring user instructions but was instead being overridden by conflicting language within their own prompt. The conflict arose because style preset strings containing phrases like 'clear room labels' were injected into the same prompt block as user commands, causing the model to prioritize preset rules over the requested change. The team identified six key prompt-engineering fixes, including separating form from content, explicitly granting requested changes, and avoiding blanket prohibitions on existing elements. They also built a suite of 28 automated tests that verify the prompt still contains specific clauses tied to known failure modes, preventing regressions. The engineers note these issues are not unique to floor plans and apply broadly to any AI task that combines a reference image with a targeted instruction.