Sidetracked...

[ellenmillion]

So... I'm working on this not-so-secret project, and I'm today, I'm thinking about quality control.

Very basic background: it will be a shared-world, collaborative project. Members can create and control characters, write stories, make artwork, etc. There will be a subscription system, and people will be able to use their original work commercially. Membership is open to everyone.

What I don't want: to be glutted with poor and mediocre quality work. That's a very real risk with open membership. (I'm sure we can all think of several examples...)



I have already decided that there will be Canon and Non-canon material. Canon work is... well, canon. It's accepted as the basis for further work - storylines portray events that actually "happen" in our continuity, artwork is accurate to the creatures and characters. Canon work also meets basic quality control - only artwork that is finished will be available in these galleries, only stories that have been well-edited and have good use of language, character and plot. Only Canon work will be public at the site. Members will also have access to non-canon material - sketches, stories that might even be good, but just don't fit into the timeline as it's been established - including cross-overs, or characters in our own modern day setting, etc.

The question I keep coming back to: how to sift the Canon from the Non-Canon.

Editing

Now, the experiences I have come mostly from these fanclubs: Kadanzer and River Twine (and it's predecessor, Rushwater). In both places, there is a story review board that edits and reviews all stories to make sure that they fit into continuity. They go back and forth with the author until the story is 'right'. In both places, being fanclubs, they try not to be too harsh on the quality issue, but they do shoulder all of the editing responsibility, they 'force' every story to conform to the setting, and it's a lot of work. I'm allergic to that much work. I veto this.

I've also observed that the first person to edit a story usually does the largest chunk of the work - it's very frequent that if you have a decent editorial staff, the first one picks up the bulk of the grammatical problems and typos, and points out sticky places. The second set of eyes (and up) tend to also catch a few, as well as add valuable input about those sticky issues, but I think that probably 85%+ of the errors are picked up by the first editor, if they're paying attention.

I am a big fan of editors. I think they are invaluable, and really deserve greater spotlight. So, I've already decided that editors will get billing on stories. And that their attribution will be chosen by the collaborators who submit the stories. Editors, as far as I'm concerned, are part of the story-writing team. Writers will choose the editors THEY want to use. Or none, if they're that good (or think they are...). I've already coded this in and have even made sure that someone can show up as both writer and editor on a story, since sometimes you do end up playing both roles.

So, without a storyboard doing the editing, how do we control what gets filtered to the Canon areas of the site? How does storyline and NPC character use approval occur? For that matter, how does owned character use occur?

Character Use

Okay, for owned characters, I'm getting comfortable enough with the coding end of things to think that I can actually MAKE an approval system. Owners of characters would be able to review the story and approve the character use through the site.

Tentative flowchart: The story is submitted, linked to characters involved from the database. That story, unapproved at the site, is available only to the owners of those linked characters (and admins). Ideally, the site alerts those owners that there is material to review. Story cannot go online until those character uses are approved. And it's approved by admins. (No hate-stories, check the rating, verify that yes, it is a Torn World story, not a shopping list, etc.) This can be made to work for artwork, too.

Something similar could also be done with NPC use. Some member of a story review board would have to approve those uses. Would each member of this board have independent ability to do this? Is this something that multiple board members should agree on? I am inclined to say that 1 member is enough - that anyone on the board would have the ability to okay NPC use in a non-canon fashion. (Even non-canon should not allow character 'abuse' or use without approval.)

Canon nominations

So, with character approvals, and very basic checks for rating and categories, etc, pretty much any story or piece of artwork can be approved to the site in a non-canon fashion.

From there, to get to canon, I am thinking that pieces must be nominated by the story board. (I am thinking of making 1 board for story AND art, and calling them the continuity board.) Pieces of excellent quality and suitability are elected from non-canon to canon. I'm thinking a board of five or six people, and 3 'votes' are required to move something to Canon. You could probably get one or two softies to vote for something borderline, but convincing 3 independent people that something is worth being part of our shared world... that will take some goodness, I think. Note that this does not require the entire board to be active at the same time. Fantastic Portfolios has taught me the value of this!


Question - should members of the board be able to see if something has already been voted on? It could be kind of freeing to have to make every one of those decisions blind - will your vote bump it to canon? Will you be the only one voting on it? Who knows! (The Continuity board would have a private thread at the forums where they could discuss things if they had questions.)

Question - how do editors and writers 'hook up?' In the forum, probably? Editors could also volunteer, if they see something in non-canon that is *this close* to being canon? That connection should probably be made on a personal level, not a management level. I see non-canon as almost being 'works in progress', with potential for being polished into canon, though not all of it will be. There should be a good, non-threatening way for people to express interest in editing or being edited...

Question - Who should be able to view non-canon? ONLY folks signed up as contributors? Or all registered readers? Should their be an optional step between registered readers and contributors? Should it be a setting in a person's account whether they wish to see that or not?

Those are enough thoughts for now. I've got some tea to work on!

Comments

[puffbird.livejournal.com]

I think the editing and submissions process are what make Kadanzer and RiverTwine (though I've never been involved there) work as collaborative writing sites. But I can understand why you'd not want that kind of intensive review process. :) It is a lot of work!

Question the first: I'd lean towards voting blind, just to keep it objective. :) And yes to having a private thread to discuss things.

Question the second: I suggest having a roster of people willing to be editors, and writers can contact them directly to connect. I also think allowing editors to volunteer edits is reasonable... however, some writers don't respond well to being given unrequested edits. That said, I would think that a writer's goal here would be for their work to be accepted as canon, in the which case they couldn't avoid an editorial process. So I don't know.

Question the third: That's a harder one. I can see some people wanting to see all submissions and others wanting only to see canon. I think it should be optional.

[ellenmillion.livejournal.com]

I think that the Kaz and RTH editing process are absolute gold - I don't want to belittle them. They really create a much, much better site than just about anything out there. But I want to put more of that effort on the contributor(s), and not on a volunteer board. Burnout on the story editing teams were a big problem, and I'd like to see that alleviated, as much as possible, and make the job a little less intimidating.

I'm leaning towards blind voting, too.

Mm! I like the idea of a roster of willing editors. Maybe they can 'sign up' for a story - if they see something they like but are itching to polish to canon-quality, then they can indicate their interest in helping with that in some not-pushy fashion. Hmm...

I've already got all the canon vs. non-canon material separated out into separate pages, so you wouldn't be browsing them mixed up together, ever. And the non-canon links don't show up unless you're logged in. AND, on every non-canon page, there's a disclaimer. I'm wondering if that will suffice.

[natsunekko.livejournal.com]

How will the art approval process work? (I am not a writer. At all.)

The setting is very appealing, at least, what I've seen you draw from it. I'm looking forward to seeing more about it.

[ellenmillion.livejournal.com]

I'm thinking that it will work very similarly to writing, except no editors.

An artist would submit artwork at the site, linking it to any characters who appear it in (they are drop-down options). It would need approval for that character use, and an administrator would check it for basic okay-ness. (Right rating, most importantly... it will be filtered for any R or PG-13 content, only G content will show up publicly.)

That would okay it for the non-canon gallery automatically, and then, canon work could be nominated from those ranks by the continuity team.

That's what I'm planning right now, anyway!

I am SO excited you're thinking about participating! :)

[natsunekko.livejournal.com]

Unless an artist is working digitally, though, changing the artwork to make to "Cannon" isn't really do-able. Perhaps a forum or something where one can submit the line-drawing and/or color roughs, before doing the actual painting?

Will it be easy to contact the "owner" of the characters outside the forums, to get feedback and information?

[ellenmillion.livejournal.com]

Oh yes, there will definitely be WIP forums for both artwork and stories, for people who want in-progress input. The forum will also allow attachments in private messages, in case people want to keep their sketches private. Right now, all contributors and character owners will be able to see each others email addresses so that communication can be made at every level, though I'm open to the option that I could also put in a little contact form, for folks who don't want their emails to be public.

Hmm...

[ysabetwordsmith.livejournal.com]

It depends on the type of art, and the stage of development. Most of the artists I know definitely fiddle around with composition choices before finalizing the art. At the light sketch stage, they can change character position and major accoutrements. At the detailed sketch stage, they can refine expressions, garment details, and other little things. If it's going to be colored, they might do a color test on several thumbnails to decide one the palette. And all those things can involve input from other people.

I've done this with Ellen. We went through several rounds of discussion when she illustrated my poem "The String of Beads" - what the ice spiders should look like, where characters would be, etc. Then she ran into some challenges with the technical things she was attempting, and had to work around those. The end picture looks terrific.

[ellenmillion.livejournal.com]

It might be an option, too, to code into the site a way to submit a WIP for the character owner to approve. That would be pretty slick - but kind of complicated to code. I am going to put it on my 'copious free time' wishlist for the site, but I'm probably not going to get to it before the site opens.

Thoughts

[ysabetwordsmith.livejournal.com]

>>So, without a storyboard doing the editing, how do we control what gets filtered to the Canon areas of the site? How does storyline and NPC character use approval occur? For that matter, how does owned character use occur?<<

First, I recommend creating a set of guidelines. Anything that violates the guidelines gets ruled out immediately. This can save you an amazing amount of work. Things you might want to include in the guidelines:

* Must be a Torn World story with readily identifiable characters/settings belonging there.
* Must be legibly written above a gradeschool level: Stories should use complete sentences, with a subject and predicate in them, beginning with capital letters and ending with appropriate punctuation. Spelling and grammar should be generally accurate: a handful of mistakes, not mistakes every line or two.
* May not contain things that could lead to lawsuits, such as libelous caricatures of real people, misuse of copyrighted material, etc.

Basically, a slush pile is going to contain a fair amount of stuff that is simply unsalvagable, and you can save a lot of energy by throwing that out immediately. Then there will be some stuff that's mediocre but not objectionable. Toward the top are the important levels: stuff by up-and-coming writers that shows promise but needs a fair amount of work, stuff by experienced writers that just needs a normal amount of polishing, and the rare gem that is perfect already (usually because the author did several rounds of revision with first-readers before submission).

If you set your standards high, people will bitch at you. If you set them low, well, the quality will match but there will be less friction. I think we need a way for authors and editors to communicate their respects tastes in this regard. At one editing job, I had a lot of people hate me because I made them work, but there were a few who really appreciated how much they learned from me. At another editing job, most of the writers were semi-pro or pro and I built up a very enthusiastic group. You want the author and editor to be on the same page, and one way to do that is offer different options such as proofreading (mainly for spelling/grammar/punctuation errors), copyediting or line editing (add word choice, sentence structure, general readability), content editing (add structural and technique issues, plus continuity for our purposes), and commenting (explaining why things need to be changed and suggesting ways to fix problems).

>>Something similar could also be done with NPC use. Some member of a story review board would have to approve those uses. Would each member of this board have independent ability to do this? Is this something that multiple board members should agree on?<<

This depends on how good your board members are and how much you trust them to do a thorough and precise job. If you have good or excellent people, one may be enough. If you have average people, or you're making do with whomever is willing, you'll probably need to have them cross-check each other's work, so get 2+ to agree on things.

>>From there, to get to canon, I am thinking that pieces must be nominated by the story board.<<

You might also ask creators to indicate whether 1) they are just writing for fun, and not particularly aiming for canon; 2) they would enjoy having their work become canon, but aren't heartset on it; or 3) they seriously intend this piece to be canon and are willing to work hard to make that happen, if it needs a lot of editing. This will help focus attention in places where the author is most willing and interested in polishing. It would also give creators a sense that they're really getting a chance to aim for the canon, not just be picked based on someone else's choice.

Re: Thoughts

[ysabetwordsmith.livejournal.com]

>>You could probably get one or two softies to vote for something borderline, but convincing 3 independent people that something is worth being part of our shared world... that will take some goodness, I think. <<

It might be helpful to provide guidelines for the board, too. There are scoring rubrics for fiction, poetry, and art that help distinguish what it is good and how good it is. Then you could set a threshold somewhere in there and say, "For canon we want above-average or excellent work. For non-canon we can accept average or fair-effort work. Poor work we don't accept." Other supporting editorial materials might also be helpful. I have some stuff from prior projects that I might be able to adapt, if you're interested.

>>Question - should members of the board be able to see if something has already been voted on?<<

I think of the board and editors as needing teamwork, so I'd like to see the votes.

>>Question - how do editors and writers 'hook up?'<<

I'd say make a list of editors, and for gods' sake ask for credentials and/or samples of their work before approving people as editors. Then when writers need an editor, they can go down that list to find someone. Also editors should be able to view a list of stories needing attention. If different levels of editing are available, writers can indicate how deep they want to go and editors can advertise what their skills are. You'll probably get more people who can proofread than content edit, and probably more writers wanting lighter edits than are willing to do heavy rewrites.

Happily you have a built-in way to minimize the risk of editorial burnout: payment. With the system already in place to pay writers/artists, it should be simple to loop in the editors. That way, the ones who are in highest demand for best skills can get more from writers, while writers can pay less for just proofreading if they think their story is already pretty clean.

>>Question - Who should be able to view non-canon? ONLY folks signed up as contributors? Or all registered readers? <<

I think more than just contributors. Maybe all registered readers, but you could also make non-canon material a perk for supporting members who are supposed to get extra reading/viewing material anyhow. People like fanfic and they like seeing tidbit sketches; not everyone is obsessed about top-quality material. As long as you have some brilliant stuff, people enjoy other stuff too.

>>Should it be a setting in a person's account whether they wish to see that or not? <<

That's a good idea. Some people may find it distracting and want to shut it off.

Re: Thoughts

[ellenmillion.livejournal.com]

Some great ideas here - I have to run to a dinner with Jake and his workfolks and will have to hash over these later, but in particular I think that last suggestion is gold. It will help a lot in offering advice/edits/suggestions if we know what the creator's goal with the piece is.

Re: Thoughts

[ysabetwordsmith.livejournal.com]

I'm glad I could help. In editing, I've found that some writers love working with editors and are deeply grateful for the improvements and what they learn in the process. One of the things I enjoy most as an editor is coaching the up-and-coming folks who show promise and diligence. But some people hate editing, hate revision, resent being asked to change things -- and not all of them are just nuisances, some of them are writing for fun and it's not fun if they have to make lots of changes. Clear communication improves the chance of everyone enjoying themselves.

Re: Thoughts

[ellenmillion.livejournal.com]

I'm firmly in the *love working with editors* camp. It's extremely valuable! But, I've also encountered those folks who just... don't like changing things. I think that pinpointing who is who on this scale will be critical for a smoothly running community.

Re: Thoughts

[ysabetwordsmith.livejournal.com]

I emailed you several resources today: a fiction feedback sheet, a fiction quality control piece, and a list of contribtor resources. Please let me know if those arrive intact.

Re: Thoughts

[ellenmillion.livejournal.com]

I got them! Should have a chance to look them over in depth tonight, I hope! Thank you so much. :)

Re: Thoughts

[ysabetwordsmith.livejournal.com]

Yay! I look forward to hearing what you think.

Re: Thoughts

[ellenmillion.livejournal.com]

I hadn't thought about the slush level stuff, but yes, there is definitely a possibility for that - I think that's a good start to guidelines that would prevent that.

Re: single/double admin approval. It would very much depend on the dedication and reliability of the board... which could easily fluxuate over time. Probably it is smartest to get 2 approvals. It's another layer of complication in the coding, but I keep telling myself the automation of this process will be worth it.

I'm also thinking there may be a 'canon-flaw' flag that can be checked - some nit-picky detail that prevents it from ever being canon, non matter how great a piece it is, that may not be noticed at a cursory look.

Re: Thoughts

[ysabetwordsmith.livejournal.com]

>> I hadn't thought about the slush level stuff, but yes, there is definitely a possibility for that - I think that's a good start to guidelines that would prevent that. <<

If you open a market to submissions, you will get slush. If you work mainly with hobby and semi-pro writers, you will get a LOT of slush, including some that will make your eyeballs itch.

>>I'm also thinking there may be a 'canon-flaw' flag that can be checked - some nit-picky detail that prevents it from ever being canon, non matter how great a piece it is, that may not be noticed at a cursory look.<<

Excellent idea!

Also, I figured out one possible way to address the issue of having different Continuity Board Members nominating a piece for canon, and maybe waffling over whether or not to vote on a piece because that might tip it over and it's borderline. Scale the nominations:

Most highly recommended. Everyone must read it now!
Highly recommended. This is quite good.
Recommended. No disqualifying flaws; a decent story.
Not recommended. Just not good enough for canon.
Ineligible. Has something that disqualifies it from canon.

That way, people can indicate how much they really like a story. If it attracts only lukewarm support, maybe it's not worth canonizing after all. But if a couple people love it and one hates it, then it's probably worth canonizing (if the hater can't finger a fatal flaw).

If you wanted to be fancy about this, you could assign points to the top three categories, let the "not recommended" category deduct a point, and let the "ineligible" category block a story from canon. Then set a threshold of points that would require at least two votes -- but allow a great story to get in with fewer votes, and an okay story to get in with more. Frex, if MHR = 3, HR = 2, and R = 1 with a canon threshold of 5, a story would qualify with any of these:
One member votes MHR, one member votes HR
Three members vote HR
Five members vote R

Re: Thoughts

[ellenmillion.livejournal.com]

The more I think about this, the more I like it.

I would amend it a little - have the 'can't be canon' flag be separate from the 'canon-quality' rating - sometimes that which prevents it from being canon has nothing to do with how good it is.

So, the Never-canon flag would be a checkbox and a text box - you would have to explain WHY it would qualify for this: cross-over, wrong time physics, conflicts with established plots, etc. Actual factual, not stylistic things. That flag would prevent it from ever being canon.

For the rating - I like a five-choice scale: -2 through +2.

Highly recommended: +2. Must read! Very obviously canon quality.
Recommended: +1. Enjoyable, well-done and would contribute to the project.
Neutral: 0. No major flaws, but not really outstanding, either. On the fence!
Not recommended: -1. Some flaws. While it's not awful, exactly, it's not something you'd want to display as an example of the site.
Strongly not recommended: -2. Lots of flaws, poor quality.

Now, calculating what becomes canon still comes with a few choices:

Option 1: A 4 point threshold seems reasonable: two people REALLY like it, or four people like it a bit, with no negatives. Or, two people really like it, one sort of likes it and one sort of doesn't.

Option 2: Three people are required to vote - any three CB members. Any positive score passes to canon. (Does that include completely neutral scores?)

Option 1 may allow a piece that's been sitting around for several months to suddenly become canon as a CB member gets around to reading it.

Option 2 would be slightly easier to code.

Option 2 could be coded along with the NPC approval, which would probably save me some effort.

After all that thinking out loud, Option 2 sounds much better!

So - include the 0 scores? One person likes it, one doesn't, but neither feels strongly, and a third is neutral.

Re: Thoughts

[ellenmillion.livejournal.com]

I'll add to this that I'm thinking about saying that a -2 score (average of all 3 votes) should be considered rejected - not published on the site at all.

Is that enough of a slush-filter, I wonder? It does require three people to agree that it's awful!

If so, I would amend the bottom 2 ratings to be:

*Not recommended for Canon: -1. Some flaws. Not worthy of being canon.
*Recommended for rejection: -2. Massive flaws, very poor quality. Doesn't deserve to be published even in the non-canon gallery.

Or, possibly make the scoring a 6-step rating, with a -3 that is 'reject entirely.' Anything that rates less than a -2 average would be rejected. (Two people think it's really bad, one thinks it's really, really bad.)

Hmm.

Re: Thoughts

[ysabetwordsmith.livejournal.com]

>>I would amend it a little - have the 'can't be canon' flag be separate from the 'canon-quality' rating - sometimes that which prevents it from being canon has nothing to do with how good it is.<<

Agreed.

>>Option 1: A 4 point threshold seems reasonable: two people REALLY like it, or four people like it a bit, with no negatives. Or, two people really like it, one sort of likes it and one sort of doesn't.

Option 2: Three people are required to vote - any three CB members. Any positive score passes to canon. (Does that include completely neutral scores?) <<

I really prefer Option 1. It's a little more work (which to be fair, you'd be doing and I wouldn't) but I think it would do a much better job of quality control, canonizing better material. One point could put a story over the bar in Option 2, compared to 4 in Option 1.

>> Option 1 may allow a piece that's been sitting around for several months to suddenly become canon as a CB member gets around to reading it. <<

I like this idea.

>>So - include the 0 scores? One person likes it, one doesn't, but neither feels strongly, and a third is neutral.<<

I'm ambivalent about neutral scores. In Option 1, they would have no effect, other than to indicate someone read the story and scored it. In Option 2, a neutral vote would basically enable canonization if somebody else voted positive: one 0 vote and one +1 vote would qualify a story. I'm not keen on that. However, I can see the value in allowing people to indicate that they have mixed feelings or don't care about a story.

I'm also a little leery of allowing -2 scores. If people feel like others are undoing their work, they might be less likely to score the stories. But this is something that we could try and see how well it works; if it does cause a problem, we could change it.

Re: Thoughts

[ellenmillion.livejournal.com]

In Option 2, a neutral vote would basically enable canonization if somebody else voted positive: one 0 vote and one +1 vote would qualify a story. I'm not keen on that.

I'm not sure I follow this - to get through approval, 3 board members would have to vote. A 0 vote and a +1 vote wouldn't cut it - someone else would have to weigh in also. A -1 vote puts it at completely neutral, which is the tipping point. We could make 0 scores Non-canon.

I'm leaning more towards Option 2 the longer I stare at the code. I may have underestimated how complex Option 1 would be to implement! We use Option 2 almost verbatim for selecting work for EMG-Zine right now, and it works pretty well. Option 1 has some nice points, but weighed against the workload... it's not going to happen right away. If it turns out to be clunky or feel inaccurate, we can always revisit the idea later. A lot of this will probably shake out as we actually start using the system in July...