The New South Wales opposition leader has quit, a week after Labor's poor showing in a state by-election.