Libertarians simply reject the initiation of violence against peaceful people and their property.
I think it’s common to focus on other tangential topics or act like libertarianism is about something else, because critics don’t want to admit that they support initiating violence against peaceful people.