K5/TBSS,
After getting similar results using various mits and sponges I recently began using a natural sea wool sponge to wash the car (a friend of mine was detailing his Porsche with one). It is by far the best product I have ever used to wash my car.
1. It's the softest material I've used yet
2. It holds more soap than other products
3. Releases what it picks up off the car better than anything else (what may be causing the swirl marks)
4. They last a long time - I have been using the same one for over 6 months and it still seams brand new.
No matter how dirty the sponge gets it always becomes clean after rinsing, always, which means whatever is in the sponge is being released and decreasing the odds of swirl marks.
Below is a good article I came across that explains the simple characteristics you want in a car wash mit or sponge.
How to Choose the Best Car Cleaning Cloth, Sponge or Mit
Hope this helps