IMHO even if the guy doesn't have any creepy intent, I think the obligation of being a good neighbor should prevail, and he should fly somewhere else.
I just try to act as I'd like to be treated if the situation was reversed. Whether it's a drone, a noisy dog, people parking where they aren't...