I think this year's worst-to-first performance is a good indication that chemistry does indeed matter. It might not be everything, but I think it's important.
Many will point to teams like the mid-70s A's or the Yankees of a few years later as "proof" that chemistry doesn't matter, but that isn't proof of anything. Maybe those teams would have been even better if they'd gotten along. Or maybe the particular alchemy of those specific teams, and their various animosities, was conducive to winning; i.e. those particular relationships drove each other to "one-up" the other guy. Who knows?
I know there's no way to prove my point, but it just seems to me when you get a group of people together, it helps a great deal if they're all on the same page and pulling together toward the same goal. And I believe it hurts when the group is disjointed and doesn't get along, and is undermining efforts to achieve a goal.
And I think this year's Red Sox team shows just that. The crap that went on last year, intangible as it may have been, cost the team victories on the field, irrespective of the players' talent.