Harvesting the community: a Holy Grail tool… built.

Well, this is the kind of news I really didn’t dream of having.

If you’re into communities at all, you have probably often run into the problem of «reification»: how to pick the relevant, interesting conversations in a forum or list, select the pieces among the stream, graft them, and make a coherent article or reference piece that concentrates the meaning of the thread. Those «emergent» pieces are often the best outcome that a community produces, the lighthouse that attracts new members and brings newcomers up to speed, the reference for new conversations… indeed, the «shoulders» on which the community can stand and further work can be built.

Yes, it can be done, but it usually requires inordinate amounts of time and work to select the pieces, edit them, and publish them in a way that integrates the new «piece» in the conversation. Not just intellectual work, but also (technology being what it is) a majestic waste of time due to having to mingle different applications that were not designed for this use. This is one of the reasons that this work is often the province of hired professionals and not left to volunteers, who tend to dislike it (and thus avoid it).

Trawling the forums for news, reviews, tips and guides has been a core task of Macuarium moderation from the start, so we know how upslope it is… and how much great content goes unremarked this way (well, at least less remarked than it deserves).

For the last few weeks, we set finally out to see if we could change this. We designed a tool that looked like it could make the harvesting’s drudgery part (getting the relevant messages in the best order, stitching them, publishing them, getting everything cross-linked) a breeze. It should be usable by any authorised member without permission hassles, complex preferences or whatnots (everything predefined for the task and configurable from the backend according to our user experience). Of course, there’s still the part of pruning and stitching the selections… but I believe that should be human work.

Then we were lucky enough to find a developer who could turn that vision into pieces of code that link our forums (made by Invision) to our content management tool (Joomla), spanning everything including different character encoding on each side :-). I’m really finicky with these things but to cut a long story short, he’s done great work. And for a price tag we can afford.

We’re now in final testing stages and we’re so proud we’re weighting the idea of packaging the little beast into a product, maybe spanning more CMS and forum types. We’re even thinking names for it (how does «Community Voice» sound?).

But that’s beside the point. I just wanted to share the news that the «holy grail» of making community harvesting a humane task is doable: it has just been done :-). And boy, do I look forward to using it.