No, but I think that's an extreme opinion not shared by many. I think the only time that might be OK if the baby has no chance to live and the woman's life is in danger. I am thinking more of pro-lifers who think it's never ok to have an abortion even less than a week into the pregnancy.
I believe pro-life is, at least sub-consciously, tied to the idea that it's sinful for women to have sex outside of marriage, but less so for men. So, I think abortion would be viewed differently if men were the carriers. Also, like you said, men don't want to have babies after casual sex. I also think men would also not want to be inconvenienced by having to carry a baby for nine months.