Almond growers in New South Wales are warning they're facing the worst case scenario, after varroa mite was detected as far south as the Victorian border.