cs549: Crowdsourcing Research Seminar
- Winter quarter 2011, 2 units
- Fridays 2:15-4:30pm
- Gates 392
Instructor: Steven Dow
Crowdsourcing and human computation are rapidly evolving topics of research in Computer Science. This course includes a literature review and laboratory portion. We will survey and discuss the latest literature, covering the following overarching themes: HCI applications of crowdsourcing, patterns and systems for crowd programming; organizational behavior and economic aspects of crowdsourcing; and crowdsourcing demographics & labor issues. The laboratory component will focus on developing concrete crowdsourcing research projects.
For each seminar period, we will spend 30 minutes discussing the papers' main findings or principles. The remaining time will be devoted to engineering two crowdsourcing research projects. Students must sign up to lead the discussion for at least one week during the quarter.
Projects and Links
Jan 7: Introduction
Exercise: Discuss class and project goals
Exercise: Login to wiki, create account
Exercise: Architecture discussion, code setup
Homework (due next class): Sign up for a discussant date
Homework (optional): Perform 5 HITs on Amazon Mechanical Turk
Jan 14: Surveys and taxonomies
Paper 1: A Taxonomy of Distributed Human Computation. Alexander J. Quinn, Benjamin B. Bederson. UMD Tech Report 2009-23.
Paper 2: Analyzing the Mechanical Turk Marketplace. Panagiotis G. Ipeirotis, ACM XRDS, December 2010.
Jan 21: Applications in HCI
Paper 1: VizWiz: nearly real-time answers to visual questions. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. UIST 2010.
Paper 2: Soylent: a word processor with a crowd inside. Michael S. Bernstein , Greg Little , Robert C. Miller , Björn Hartmann , Mark S. Ackerman , David R. Karger , David Crowell , Katrina Panovich. UIST 2010.
Discussant: Vignan Pattamatta
Jan 28: Applications for Data Annotation
Paper 1: Utility data annotation with Amazon Mechanical Turk. Alexander Sorokin, David Forsyth. First IEEE Workshop on Internet Vision at CVPR 2008.
Paper 2: Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08).
Discussant: Julie Fortuna
Feb 4: Running Experiments with Online Crowds
Paper 1: The Online Laboratory: Conducting Experiments in a Real Labor Market. Horton, John J., Rand, David G. and Zeckhauser, Richard J. SSRN, 2010.
Paper 2: Crowdsourcing User Studies with Mechanical Turk. Aniket Kittur, Ed Chi, Bongwon Sun, CHI 2008.
Discussant: Truc Nguyen
Special guest: Niki Kittur
Feb 11: Task and Workflow Design
Paper 1: Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets. Dana Chandler and Adam Kapelner, CrowdFlower blog, March 2010.
Paper 2: Exploring Iterative and Parallel Human Computation Processes. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. HCOMP 2010.
Discussant: Charlie Guo
Special guests: CrowdFlower's Lukas Biewald and Mollie Allick
Feb 18: Economics
Paper 1: Financial Incentives and the "Performance of Crowds". Mason, W., & Watts, D. J. HCOMP 2009.
Paper 2: Algorithmic Wage Negotiations: Applications to Paid Crowdsourcing. John Horton and Richard Zeckhauser. CrowdConf, 2010
Feb 25: Labor Issues
Paper 1: Who are the crowdworkers?: shifting demographics in mechanical turk. Joel Ross , Lilly Irani , M. Six Silberman , Andrew Zaldivar , Bill Tomlinson. CHI Extended Abstracts, 2010.
Paper 2: Sellers' problems in human computation markets. M. Six Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. HCOMP 2010.
Optional Paper: Web Workers Unite! Addressing Challenges of Online Laborers. Ben Bederson and Alexander Quinn. CHI 2011 Extended Abstracts.
Discussant: Pao Siangliulue
Mar 4: Crowd Programming Tools
Paper 1: TurKit: Human Computation Algorithms on Mechanical Turk. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. UIST 2010.
Paper 2: Generalized task markets for human and machine computation. Dafna Shahaf and Eric Horvitz. 2010.
Discussant: Kingston Tam
Mar 11: Last class
Exercise: demo preparation
Mar 18: Demo day