Is my decision to leave him too drastic?

I am 23 and have a four year old and i am currently 19 weeks pregnant and both children are with my partner of 7 years. I love him greatly but i need more back. He will not marry me, he will not come on our holiday we have just booked i have to take my son on my own (he has done this before he always changes his mind last minute) he will not take us anywhere on day trips etc and i am tired of trying to tak to him about our problems as he is too busy too listen to me. To be honest i have spent the last seven years making him happy (he doesnt even run his own bath) so he doesnt leave and now i have come to the point where i know deep down inside it is over i want more for me and my children. I have this immense feeling of guilt as my son idolises his dad, and i know my partner will be gutted about my decision (i am his personal slave.) But i cannot spend the rest of my life with such a selfish person. i want a partner who doesnt take constantly and loves me enough to marry me. I was just going to go on holiday just me and my son and take the time to think i do worry but being alone at 23 with two kids but i will manage. Am i doing the right thing? and how do i tell him? Any advice would be nice.