The QB spoke about how the game gives him a certain kind of peace and it's difficult being without it even for a week off