It's probably just you seeing thing more as you advance. Take it up quickly to 6 feet and then release the sticks. It may bounce and drift a little but it will quickly right itself
This is a known issue with the latest firmware, try downgrading your firmware and wait to upgrade until the next release or I believe that we can turn VPS off now, I haven't tried it myself but disabling VPS should fix the issue as well