Mill said the only reason to restrict someone is to stop them harming others

In On Liberty, John Stuart Mill argued the only justification for a society to exercise power over an individual against their will is to prevent harm to others — not to make them happier, wiser, or more moral, even by their own reckoning. Purely self-regarding choices, however unwise, are off-limits to interference. The principle became a foundation of liberal political philosophy and modern debates over paternalism.

— John Stuart Mill, On Liberty — 1859
Go deeper: get the book →