Victoria's environment protection authority has warned swimmers to avoid taking a dip in the Yarra River and all of Port Phillip Bay, after this week's heavy rain.