I totally agree. It's fun while you are doing it at your own pace. Keep it small and with as little overhead as possible. If you still love it in a year or two, go big if that is where you still want to be. I love doing it evenings and weekends. I have two customers that I visit once a week every week. (they get boring), but I still love it.
Good luck and yes, I'd get insurance. Even friends & family have problems that sometimes can't be worked out.