I feel this pain. I worked for 25 years in IT. When I reached 51 I got made redundant, second time it had happened to me. I took stock, thought I was too old to be getting another IT job, and dropped out.
I'd always loved writing, and music, from being a kid. I'd done the odd review for fanzines and such. So, I used my redundancy money to live on for a while, and started networking around all the contacts I have. Now, seven years later, I've got five years editing a magazine behind me (granted, on a voluntary basis, but loads of free music and guest passes to gigs), I've written for several mags, and have published four books, with numbers five and six coming this Autumn.
I've interviewed and worked with so many people who used to be heroes of mine. Paul Rodgers, Greg Lake, Jack Bruce, David Coverdale, Glenn Hughes, Ritchie Blackmore, Leslie West, Ian Anderson, Roger Hodgson, Steve Hackett, Justin Hayward, Carl Palmer, Rick Wakeman, Chris Squire etc etc. I'm the authorised biographer of Gordon Giltrap, and I worked with him for three years on that.
I make my own hours, do everything from home. I may not have the money to go on holidays and the like now, but my quality of life is immeasurably richer. If I hadn't taken the plunge when I did I'd never have done it. If you have a dream, follow it.