Should i leave my boyfreind?

he has a daughter from an ex and a son thats been adopted from the same ex and then we have 2 children.he doesnt work but he doesnt help me..he expects me to look after hes daughter when hes never in and she lives with hes mother as they have shared custody of her but his mother drops her round willy nilly and the poor little girl doesnt have a proper scedule she just gets dropped off picked up dropped off.im fed up with her being dumped on me when her father is out with friends.should i leave him?we always argue..mainly about his daughter and its because i think if im looking after her it should be on my terms but his mother is the boss of it all.it feels like his mother rules my life cos i have to wait in for her to drop her off and be in for hewr to pick her up.wy cant she just drop her off on friday and pick her up on sunday.because my partner wont back me up and agrees with his mum that everything is up to his mum when it comes to his daughter.my partner doesnt help with our children who are babies either even though he doesnt work.we argue everyday.should i just leave him and all the stress behind?