I am by far no expert, but I just clayed my car for the first time a couple of days ago. My thought is that the baggie REALLY amplifies EVERYTHING sitting on the paint. I too clayed my heart out only to have the baggie test tell me that there was still "stuff" on my paint. I think it could even be dust that has settled in the time you went to go get the baggie. My neighbor was using a leaf blower when I was claying...I am sure that junk blowing around was settling on my car.
I would just get it as good as I could and call it good.