They have reported West Norfolk Council's planning department to a fraud helpline as part of a dispute over the three-storey property on Wodehouse Road, an exclusive street in Old Hunstanton known as Millionaire's Row.
The street is named after author PG Wodehouse, a frequent visitor to the village and who drew inspiration from its leafy surroundings for his Jeeves and Wooster series.
Neighbours say the new build has broken planning rules because it is vastly different from the original application which was approved by councillors - with an extra storey on top.
And they claim the council itself has acted fraudulently by misrepresenting the original plans and insisting the extra floor was always supposed to be there.
The fraud probe has been revealed as councillors met to decide what action to take over the property.
The old Hunstanton home before demolition (Image: West Norfolk Council)
It follows homeowner Henry Middleton making a retrospective application to WNC in the hope of gaining approval for the design of the house after it was........