The 49ers defense and coordinator Steve Wilks aren't the only ones who need to be held accountable for the team's woes.