that old (and boring I think) chestnut of PfAW and aerial "rights" has me confused (easily done

)
I use my Phantom to check factory roofs are ok for installing solar panels for our company.
I go up, go forward, turn round, come back and land. I'm not into sweeping aerial shots or artistic scenes, just use it as a useful tool. So, I'm on private property (owned by the factory owner), have their permission and am not within 50 m of other persons not under my control etc. etc. I'm a safe flyer, I take sensible precautions and even joined the BMFA to get their 3rd party insurance (probably invalid in this case, but useful for my training flights at a friend's farm).
So my question Mr CAA is - since when did you have the rights to come onto private property and tell me (actually the factory owner) what I can do on my land??
Is the law an *** in this case? Or has the CAA an automatic right to enforce anything not bolted to the ground, including the £15 minicopter I bought my small nephew that's flown in my living room?
The key issue is this idea of "reward". I can understand if I'm charging someone for a wedding video or a commercial advertising shoot over Stonehenge etc., but I'm quietly minding my own business with the full cooperation of the property owner. I guess if I give him control for 10 seconds he can claim it's his flight and therefore no "consideration" will take place.
I think the PfAW criteria need to be clearer - in the end it's whether the CAA can be bothered to enforce it.