This is from a rider with nearly 40 years experience riding. I was Director of our HOG Club for seven years and a Road Captain for 10. Riding a Harley is the most fun on two wheels that I have ever had. And, I've owned motocross and the "go fast" sport bikes. Riding a motorcycle has been a great stress reliever for me.
However, w/ young kids, I recommend you give this decision a bit more thought. The Sportster will be a "me" toy and a "me" stress reliever. If you make the Sporster yours, I suggest you make sure you have plenty of disability insurance in place in case you're hurt and can't work. If you don't have one, I also suggest you consider having a family toy that will involve everyone.
The best piece of advice I rec'd came when I was just learning to ride and I think of it often -
"Ride like you're invisible".
Too many folks, young and old alike are in too big of a hurry and simply don't see motorcyclists.