Do what your heart desires. Don't do what all your friends are doing and think that is the path you need to take. Leaders never follow.
I've been there. I've made the big leap myself. It seems like a complicated switch, but it really isn't much more difficult than getting the job that you got working in finance.
Okay, so here's the deal. You probably spent 4 years in college taking economics and finance classes. You probably had a few internships in finance and then began working full-time right after school. You had to prove to people that despite only your academic experience, you would make for a good fit. You convinced people that based on your book knowledge, you would be able to handle real-life scenarios. Now take a step back...we're talking about spreadsheets.
The same applies for a career in UX. The good news is that it's still a relatively new field, which means there aren't a whole lot of domain experts. This means that with limited experience, you can brand yourself as a very knowledgeable resource in the field. And since this field of work is so new, it's not something that will require you to prove your credentials with fancy degrees. Rather, you need to prove yourself with a combination of personal projects that you've worked on and potentially anything else you've done for work that might relate, although if you're plowing through spreadsheets, that might be limited.
These personal projects will make up your portfolio, which you will need to share during interviews. If you do not have any projects to share (i.e. things you've worked on for your friends, something you might have designed for yourself, etc...), then you're only dreaming of what you want to do and haven't really put in the effort to deserve it. Remember, you spent years studying finance to get a job in finance. You can't expect to walk into a place saying you like UX with absolutely no experience. However, the experience you need can all be self-taught by working on some projects. You can even challenge yourself to redesign features of existing websites or services, simply to build up portfolio pieces.
With a willingness to learn and self-motivation, you will easily be able to make the switch from finance to UX.
Hope this helps!