About 4,000 residents from Mallacoota town fled to waterside as winds pushed wildfire towards their homes.