Owners of a 1960s bungalow on a historic Herefordshire estate have been told it need not be occupied by a farmer.