Technically he IS correct... Sort of... If he owns the land and objects to you flying over it the best thing to do is apologise, land and fly elsewhere rather than it turning ugly.
He doesnt own the airspace above the land but if he calls the Police they will more than likely ask you to move...