A conversation at com-prac has brought back some memories. Quite a few years ago, when I was first trying to formalise what we learned at Macuarium for use beyond our forums (and specifically at Knowledgeboard.com), we attempted to do a community-tool evaluation mechanism that allowed for sensible comparison.
The result was the Pentagrafos project, which didn’t get any traction beyond Macuarium :-). There was quite a bit of interest in the methodology from the most surprising corners of the world, but nobody troubled to send in any of their own evaluations, so I closed it. That doesn’t mean we don’t use it… it simply is no longer public, or being actively worked on.
You can have a look at the example, flash-based user interface we were working on here.