Welcome Felix!
I use Adam's Paint Sealant often, but I have never had an issue removing it. Perhaps you put it on too thick? You stated you applied with a buffer at low speed (I use it at about setting 2), which usually makes it go on thin. I apply to a the whole vehicle, then remove, and have not had an problem removing it, even when the humidity and/or temps are high.
I would have suggested spraying the tougher areas with Detail Spray or even to work a little more product in the areas, then wipe right away.