Chapter 7 of Statera's HITL volume covers crowdsourcing for advanced human-AI systems

A chapter from the STATERA Library's Human-in-the-Loop Learning volume examines crowdsourcing as a distributed method for gathering human knowledge, judgment, annotation, and feedback to improve AI systems. It explains that complex problems are split into small tasks such as image labeling, text classification, and preference comparison, which coordination platforms assign, collect, verify, and feed into AI workflows. The text stresses that task decomposition, contributor selection, and quality assurance are central, since larger crowds do not automatically yield better accuracy. It also covers quality-control tools such as redundant annotation, qualification tests, and inter-annotator agreement metrics, along with consensus aggregation methods like majority and weighted voting. The chapter concludes that combining crowdsourcing with active learning can reduce labeling costs, though its benefits depend on query quality.