A property owner only owns the airspace to the tallest structure. (Tree top, roof, antenna tower)
But I would stay as high as I can and still under 400'. Don't be pointing your camera directly at house/in windows.
I am a drone owner, so I understand them. But if I didn't, I would probably be...