Understanding the experience of neurodivergent workers in image and text data annotation

Metadata sources used for this reference

OpenAlex Crossref BibTeX OpenCitations
Learn more about the sources used

Garrison, E., Singh, D., Hantula, D., Tincani, M., Nosek, J., Hong, S. R., Dragut, E., & Vucetic, S. (2023). Understanding the experience of neurodivergent workers in image and text data annotation. Computers in Human Behavior Reports, 11, 100318. https://doi.org/10.1016/j.chbr.2023.100318

Publication date: 1 Aug 2023 Added to AutiHub: 29 Sep 2026 Type: Article Article language: English
Publisher source (DOI) OpenAlex source

This publication is integrated into AutiHub through:

Authors

Publication authors
8
Publication authors identified as autistic
0 / 8 (0.0%)
  1. Elizabeth Garrison
  2. Dalvir Singh
  3. Donald Hantula
  4. Matt Tincani
  1. John Nosek
  2. Sungsoo Ray Hong
  3. Eduard Dragut
  4. Slobodan Vucetic

Abstract

With the rise of large-scale data-driven innovation in AI, data annotation tasks found in digital work environments present an employment opportunity for neurodivergent individuals. Though work in data annotation can potentially ease the high unemployment rate of neurodivergent individuals, limited research focuses on the experience of neurodivergent workers in data annotation micro-tasks. To aid in understanding the experience of neurodivergent crowd workers, we conducted a user study with ten neurodivergent workers between the ages of 18–30. Participants completed three types of micro-tasks in a custom web-based data annotation platform. With the data collected from the platform, we examined individual responses within data annotations, work completion, the time to complete work, and calculated a potential “effective hourly wage,” for each participant based on their responses. Through a survey and semi-structured interview following each task, we learned about the experience of all participants regarding each of the data annotation tasks. Results of the study show: 1) our participants provide diverse annotations that are valuable for employers in digital data annotation work environments; 2) when calculating the “effective hourly wage” of all participants per task, most of our participants would earn less than minimum hourly wage on tasks; and 3) participant perceptions of the tasks matched their responses in the tasks presented.

Bibliography cited by this reference

Cited references are imported from external metadata sources when they are available. The list may be partial.

Cited bibliography overview

These indicators describe the bibliography cited by this publication. An author name is counted each time it appears in one cited reference, so the same person can be counted more than once. Names not yet linked to an author already present in AutiHub are treated as unknown, not as non-autistic. Last computed: 2 Oct 2026 08:13.

Cited references
62
With a DOI
32
Without a DOI, from raw bibliography text
27
0 / 62 (0.0%) cited references include at least one author identified as autistic.
References with data to complete
6 / 62 (9.7%)
References with detected author names
56 / 62 (90.3%)
Without detected author names
6
Without a structured title
1
Without a stable identifier
25
References with raw author names still to review
1
References with external metadata lookup issues
0
These indicators apply to cited references displayed on this page, after technical duplicates have been merged. A reference without a DOI can still support author statistics when a title and author names are available.
Author names detected in the cited bibliography
133
From DOI or external metadata
133
From validated raw bibliography text
0
Raw names already validated
0
Raw names still to review
1
10 / 133 (7.5%) names are linked to an author already present in AutiHub. 123 / 133 (92.5%) names are not yet linked.
Names linked to a person identified as autistic
0 / 133 (0.0%)
Calculated across all author names detected in the cited bibliography. Among names linked to an author already present in AutiHub: 0 / 10 (0.0%). Distinct people identified as autistic: 0 / 128 (0.0%).
  1. Simon Baron-Cohen (2000). Theory of mind and autism: A review . International Review of Research in Mental Retardation, 23, 169-184. Elsevier.
    Type: Article DOI: 10.1016/s0074-7750(00)80010-5 OpenAlex: https://openalex.org/W1597602009
    Crossref OpenAlex OpenCitations
  2. Alm (2011). Subjective natural language problems: Motivations, applications, characterizations, and implications .
    Type: Other
    Crossref
  3. Edith Walsh , Helena Lydon , Jennifer Holloway (2020). An Evaluation of Assistive Technology in Determining Job-Specific Preference for Adults With Autism and Intellectual Disabilities . Behavior Analysis in Practice, 13(2), 434-444. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s40617-019-00380-3 OpenAlex: https://openalex.org/W2964971475
    Crossref OpenAlex OpenCitations
  4. Alexandra Birch , Miles Osborne (2011). Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies . The Association for Computational Linguistics.
    Type: Article OpenAlex: https://openalex.org/W1607035479
    OpenAlex
  5. Lars Hornuf , Daniel Vrankar (2022). Hourly Wages in Crowdworking: A Meta-Analysis . Business & Information Systems Engineering, 64(5), 553-573. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s12599-022-00769-5 OpenAlex: https://openalex.org/W4293561951
    Crossref OpenAlex OpenCitations
  6. Chris Callison-Burch (2014). Crowd-Workers: Aggregating Information Across Turkers to Help Them Find Higher Paying Work . Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 2, 8-9. Association for the Advancement of Artificial Intelligence (AAAI).
    Type: Article DOI: 10.1609/hcomp.v2i1.13198 OpenAlex: https://openalex.org/W1681309541
    Crossref OpenAlex OpenCitations
  7. Rebecca Brewer , Federica Biotti , Caroline Catmur , Clare Press , Francesca Happé , Richard Cook et al. (2016). Can Neurotypical Individuals Read Autistic Facial Expressions? Atypical Production of Emotional Facial Expressions in Autism Spectrum Disorders . Autism Research, 9(2), 262-271. Wiley.
    Type: Article DOI: 10.1002/aur.1508 OpenAlex: https://openalex.org/W2159450620
    Crossref OpenAlex OpenCitations
  8. Mary Wagner , Lynn Newman , Renée Cameto , Phyllis Levine , Nicolle Garza (2006). An overview of findings from wave 2 of the national longitudinal transition study-2 (NLTS2). NCSER 2006-3004 .
    Type: Article OpenAlex: https://openalex.org/W1705719549
    Crossref OpenAlex
  9. Cai (2016). Chain reactions: The impact of order on microtask chains .
    Type: Other
    Crossref
  10. Carol Schall , Jennifer Todd McDonough (2010). Autism spectrum disorders in adolescence and early adulthood: Characteristics and issues . Journal of Vocational Rehabilitation, 32(2), 81-88. SAGE Publications.
    Type: Article DOI: 10.3233/jvr-2010-0503 OpenAlex: https://openalex.org/W2133101211
    Crossref OpenAlex OpenCitations
  11. Steven Paul Woods , David W. Lovejoy , J.D. Ball (2002). Neuropsychological Characteristics of Adults with ADHD: A Comprehensive Review of Initial Studies . The Clinical Neuropsychologist, 16(1), 12-34. Informa UK Limited.
    Type: Article DOI: 10.1076/clin.16.1.12.8336 OpenAlex: https://openalex.org/W1998066562
    Crossref OpenAlex OpenCitations
  12. Paul Shattuck , Sarah Carter Narendorf , Benjamin Cooper , Paul R. Sterzing , Mary Wagner , Julie Lounds Taylor (2012). Postsecondary Education and Employment Among Youth With an Autism Spectrum Disorder . Pediatrics, 129(6), 1042-1049. American Academy of Pediatrics (AAP).
    Type: Article DOI: 10.1542/peds.2011-2864 OpenAlex: https://openalex.org/W2128138260
    Crossref OpenAlex OpenCitations
  13. Nikolay Pavlov (2014). User interface for people with autism spectrum disorders . Journal of Software Engineering and Applications, 2014(02), 128-134. Scientific Research Publishing, Inc..
    Type: Article DOI: 10.4236/jsea.2014.72014 OpenAlex: https://openalex.org/W2041512992
    Crossref OpenAlex
  14. Maitraye Das , John Tang , Kathryn E. Ringland , Anne Marie Piper (2021). Towards Accessible Remote Work . Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1-30. Association for Computing Machinery (ACM).
    Type: Article DOI: 10.1145/3449282 OpenAlex: https://openalex.org/W3128311906
    Crossref OpenAlex OpenCitations
  15. Gao (2005). Latin squares in experimental design .
    Type: Other
    Crossref
  16. V. K. Manam , Alexander Quinn (2018). WingIt: Efficient Refinement of Unclear Task Instructions . Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6, 108-116. Association for the Advancement of Artificial Intelligence (AAAI).
    Type: Article DOI: 10.1609/hcomp.v6i1.13338 OpenAlex: https://openalex.org/W2809806295
    Crossref OpenAlex OpenCitations
  17. Yuyang Gao , Tong Steven Sun , Liang Zhao , Sungsoo Ray Hong (2022). Aligning eyes between humans and deep neural network through interactive attention alignment . Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), 1-28. Association for Computing Machinery (ACM).
    Type: Article DOI: 10.1145/3555590 OpenAlex: https://openalex.org/W4309619902
    Crossref OpenAlex
  18. Chanelle T. Gordon , Gregory A. Fabiano (2019). The Transition of Youth with ADHD into the Workforce: Review and Future Directions . Clinical Child and Family Psychology Review, 22(3), 316-347. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s10567-019-00274-4 OpenAlex: https://openalex.org/W2912703214
    Crossref OpenAlex OpenCitations
  19. Sandra G. Hart (2006). NASA-task load index (NASA-TLX); 20 years later . Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50(9), 904. SAGE Publications.
    Type: Article DOI: 10.1177/154193120605000909 OpenAlex: https://openalex.org/W2151905266
    Crossref OpenAlex
  20. Anhong Guo , Ece Kamar , Jennifer Wortman Vaughan , Hanna Wallach , Meredith Ringel Morris (2020). Toward fairness in AI for people with disabilities SBG@a research roadmap . ACM SIGACCESS Accessibility and Computing, (125), 1-1. Association for Computing Machinery (ACM).
    Type: Article DOI: 10.1145/3386296.3386298 OpenAlex: https://openalex.org/W3009306974
    Crossref OpenAlex OpenCitations
  21. Ludmila Praslova , Liana Bernard , Stefanie Fox , Aviva Legatt (2023). Don’t tell me what to do: Neurodiversity inclusion beyond the occupational typecasting . Industrial and Organizational Psychology, 16(1), 66-69. Cambridge University Press (CUP).
    Type: Article DOI: 10.1017/iop.2022.105 OpenAlex: https://openalex.org/W4323663029
    Crossref OpenAlex OpenCitations
  22. Dan Cosley , Andrea Forte , Luigina Ciolfi , David W. McDonald (2015). Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing .
    Type: Article OpenAlex: https://openalex.org/W2262972580
    OpenAlex
  23. Hara (2018). A data-driven analysis of workers' earnings on Amazon Mechanical Turk .
    Type: Other
    Crossref
  24. Rose LeFevre-Levy , Arturia Melson-Silimon , Rebecca Harmata , Anna L. Hulett , Nathan T. Carter (2023). Neurodiversity in the workplace: Considering neuroatypicality as a form of diversity . Industrial and Organizational Psychology, 16(1), 1-19. Cambridge University Press (CUP).
    Type: Article DOI: 10.1017/iop.2022.86 OpenAlex: https://openalex.org/W4323663412
    Crossref OpenAlex OpenCitations
  25. Ranjay Krishna , Yuke Zhu , Oliver Groth , Justin Johnson , Kenji Hata , Joshua Kravitz et al. (2017). Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations . International Journal of Computer Vision, 123(1), 32-73. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s11263-016-0981-7 OpenAlex: https://openalex.org/W2277195237
    Crossref OpenAlex OpenCitations
  26. Hara (2019). Worker demographics and earnings on amazon mechanical turk: An exploratory analysis .
    Type: Other
    Crossref
  27. M. Venkatesan , K. Anders Ericsson , Herbert A. Simon (1986). Protocol Analysis: Verbal Reports as Data . Journal of Marketing Research, 23(3), 306. JSTOR.
    Type: Article DOI: 10.2307/3151491 OpenAlex: https://openalex.org/W2333057202
    OpenAlex
  28. Hara (2017). Introducing people with ASD to crowd work .
    Type: Other
    Crossref
  29. Ghaidaa Khalifa , Zonera Sharif , Madiha Sultan , Briano Di Rezze (2020). Workplace accommodations for adults with autism spectrum disorder: a scoping review . Disability and Rehabilitation, 42(9), 1316-1331. Informa UK Limited.
    Type: Article DOI: 10.1080/09638288.2018.1527952 OpenAlex: https://openalex.org/W2913924606
    Crossref OpenAlex OpenCitations
  30. Yuan-mei Xiao , Zhi-ming Wang , Mian-zhen Wang , Ya-jia Lan (2005). [The appraisal of reliability and validity of subjective workload assessment technique and NASA-task load index] . Chinese Journal of Industrial Hygiene and Occupational Diseases, 23(3), 178.
    Type: Article OpenAlex: https://openalex.org/W2369666883
    Crossref OpenAlex
  31. (2023). Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . ACM.
    Type: Other DOI: 10.1145/3544548 OpenAlex: https://openalex.org/W2768343520
    OpenAlex
  32. Sungsoo Ray Hong (2018). Collaborative dynamic queries: Supporting distributed small group decision-making .
    Type: Other
    Crossref
  33. Carlos Toxtli , Siddharth Suri , Saiph Savage (2021). Quantifying the Invisible Labor in Crowd Work . Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1-26. Association for Computing Machinery (ACM).
    Type: Article DOI: 10.1145/3476060 OpenAlex: https://openalex.org/W3203959757
    Crossref OpenAlex OpenCitations
  34. Bernhaupt, Regina , Mueller, Florian , Verweij, David , Andres, Josh (2020). Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . ACM.
    Type: Other DOI: 10.1145/3313831 OpenAlex: https://openalex.org/W2783539613
    OpenAlex
  35. Carol Schall , Paul Wehman , Lauren Avellone , Joshua P. Taylor (2020). Competitive Integrated Employment for Youth and Adults with Autism . Psychiatric Clinics of North America, 43(4), 701-722. Elsevier BV.
    Type: Article DOI: 10.1016/j.psc.2020.08.007 OpenAlex: https://openalex.org/W3094721248
    Crossref OpenAlex OpenCitations
  36. Hube (2019). Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments .
    Type: Other
    Crossref
  37. (2020). Proceedings of the 38th ACM International Conference on Design of Communication . ACM.
    Type: Other DOI: 10.1145/3380851 OpenAlex: https://openalex.org/W2911344542
    OpenAlex
  38. (n.d.). Janík, P. [n.d.]. Receipt database with OCR scan. ([n. d.])..
    Type: Other
    Crossref
  39. (2018). Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility . ACM.
    Type: Other DOI: 10.1145/3234695 OpenAlex: https://openalex.org/W2911480189
    OpenAlex
  40. Kacorri (2017). People with visual impairment training personal object recognizers: Feasibility and challenges .
    Type: Other
    Crossref
  41. Sri Kurniawan , John T. Richards (2014). Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility . Conference on Computers and Accessibility.
    Type: Other OpenAlex: https://openalex.org/W2912011982
    OpenAlex
  42. Laffal (1985). Protocol analysis: Verbal reports as data .
    Type: Other
    Crossref
  43. Hiniker A. (2017). Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems . Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM.
    Type: Other DOI: 10.1145/3025453 OpenAlex: https://openalex.org/W2914783728
    OpenAlex
  44. Maas (2011). Learning word vectors for sentiment analysis .
    Type: Other
    Crossref
  45. (2019). Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems . ACM.
    Type: Other DOI: 10.1145/3290607 OpenAlex: https://openalex.org/W2981917765
    OpenAlex
  46. QUSAY SARHAN (2019). Best practices and recommendations for writing good software . Journal of Donghua University, 22(1), 90. University of Duhok.
    Type: Article DOI: 10.26682/sjuod.2019.22.1.11 OpenAlex: https://openalex.org/W2986402160
    Crossref OpenAlex
  47. Medina (2022). Human-computer interaction in artificial intelligence for blind and vision impairment: An interpretative literature review based on bibliometrics .
    Type: Other
    Crossref
  48. Motti (2019). Designing emerging technologies for and with neurodiverse users .
    Type: Other
    Crossref
  49. Murray (2018). Monotropism: An interest-based account of autism .
    Type: Other
    Crossref
  50. Margari L. , de Giambattista C. , Ventura P (2015). Encyclopedia of Autism Spectrum Disorders . Springer New York.
    Type: Book DOI: 10.1007/978-1-4614-6435-8 OpenAlex: https://openalex.org/W3155009214
    OpenAlex
  51. Rapp (2019). Designing mobile technologies for neurodiversity: Challenges and opportunities .
    Type: Other
    Crossref
  52. Sam (2022). No such thing as normal: Exploration and opportunities for neurodiversity in research. CHI ‘22 .
    Type: Other
    Crossref
  53. (2019). Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services . ACM.
    Type: Other DOI: 10.1145/3338286 OpenAlex: https://openalex.org/W4231109226
    OpenAlex
  54. Jaeyoun You , Daemin Park , Joo-yeong Song , Bongwon Suh (2022). A labeling task design for supporting algorithmic needs: Facilitating worker diversity and reducing AI bias . arXiv (Cornell University). arXiv.
    Type: Preprint DOI: 10.48550/arxiv.2205.08076 OpenAlex: https://openalex.org/W4280605301
    Crossref OpenAlex
  55. Spiel (2020). Nothing about us without us: Investigating the role of critical disability studies in HCI .
    Type: Other
    Crossref
  56. Suzuki (2016). Atelier: Repurposing expert crowdsourcing tasks as micro-internships .
    Type: Other
    Crossref
  57. Tavanapour (2019). Towards supportive mechanisms for crowd collaboration–design guidelines for platform developers .
    Type: Other
    Crossref
  58. Tomczak (2023). Toward an understanding of occupational burnout among employees with autism–the Job Demands-Resources theory experience . Current Psychology, 1.
    Type: Other
    Crossref
  59. Uzor (2021). Investigating the accessibility of crowdwork tasks on Mechanical Turk .
    Type: Other
    Crossref
  60. Yan (2022). FlatMagic: Improving flat colorization through AI-driven design for digital comic professionals .
    Type: Other
    Crossref
  61. Yaneva (2015). Accessible texts for autism: An eye-tracking study .
    Type: Other
    Crossref
  62. Zyskowski (2015). Accessible crowdwork? Understanding the value in and challenge of microtask employment for people with disabilities .
    Type: Other
    Crossref