The recent growth of global crowdsourcing platforms has enabled businesses to leverage the time and expertise of workers world-wide with low overhead and at low cost. In order to utilize such platforms, one must decompose work into tasks that can be distributed to crowd workers. To this end, platform vendors provide task interfaces at varying degrees of granularity, from short, simple microtasks (e.g., Amazon’s Mechanical Turk) to multi-hour, context-heavy tasks that require training (e.g., oDesk). Most research in quality control in crowdsourced workflows has focused on microtasks, wherein quality can be improved by assigning tasks to multiple workers and interpreting the output as a function of workers’ agreement. Not all work fits into microtask frameworks, however, especially work that requires significant training or time per task. Such work is not amenable to simple voting schemes, as redundancy can be expensive, worker agreement can be difficult to define, and training can limit worker availability. Nevertheless, the same characteristics that limit the effectiveness of known quality control techniques offer unique opportunities for other forms of quality improvement. For example, systems with worker training modules also possess a wealth of context about individuals and their work history on specific tasks. In such a context-heavy crowd work system with limited budget for task redundancy, we propose three novel techniques for reducing task error:

• A self-policing crowd hierarchy in which trusted workers review, correct, and improve entry-level workers’ output.
• Predictive modeling of task error that improves data qual- ity through targeted redundancy. When workers complete tasks, we can allocate spot-checked reviews to the tasks with the highest predicted error. This technique allows us to capture 23% more errors given our reviewing budget.
• Holistic modeling of worker performance that supports crowd management strategies designed to improve av- erage crowd worker quality and allocate training to the workers that need the most assistance.