I jumped straight back into school after being out for ten years. It's one of the best things that I have done. I highly recommend one or two night classes to start. I never stopped reading or learning in between. Plus, I don't have children, and I had other types of training and school that I had recently done before jumping in. Most of the adults that are returning to school at my school are only doing it a little at a time. A class or two. Get your feet wet first, then jump right in if you want to. Like any major decision, it should include your entire family and be taken seriously.
All of that aside.
GO FOR IT!!!!!!!!!!!!!!!
It is not scary at all. It is a great environment full of good and supportive people who are there for the same reasons that you are (and a few kids who don't care, but the majority are people like you).
"Every child asks the questions which hold the answers to the secrets of the universe, WHAT?, and WHY?". --Me