Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli

Metadata sources used for this reference

OpenAlex Crossref
Learn more about the sources used

Isaev, D., Major, S., Carpenter, K. L. H., Grapel, J., Chang, Z., Martino, J. M. D., Carlson, D., Dawson, G., & Sapiro, G. (2025). Use of computer vision analysis for labeling inattention periods in EEG recordings with visual stimuli. Scientific Reports, 15(1), 30963-30963. https://doi.org/10.1038/s41598-025-10511-2

Publication date: 22 Aug 2025 Added to AutiHub: 5 Jul 2026 Type: Article Article language: English
OpenAlex source Publisher source (DOI) Full text

This publication is integrated into AutiHub through:

Authors

Publication authors
9
Publication authors identified as autistic
1 / 9 (11.1%)
  1. Dmitry Isaev
  2. Samantha Major
  3. Kimberly L. H. Carpenter
  4. Jordan Grapel *
  5. Zhuoqing Chang
  1. J. Matías Di Martino
  2. David Carlson
  3. Géraldine Dawson
  4. Guillermo Sapiro

Abstract

Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant's attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children with and without co-occurring ADHD (attention-deficit/hyperactivity disorder) aged 49-95 months, and training on additional 2560 labeled frames (equivalent to 85.3 s of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984-0.993) and the median inter-rater reliability (Cohen's kappa) with a trained human annotator was 0.888. Agreement with human annotations for nine participants was in the 0.616-0.944 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community.

Bibliography cited by this reference

Cited references are imported from external metadata sources when they are available. The list may be partial.

Bibliography inclusion overview

These indicators describe the cited bibliography imported for this publication. Cited-reference metrics use the cited-reference total as denominator. Cited-author metrics state whether they use all cited-author occurrences or only occurrences linked to authors already integrated in the AutiHub database. They use cached links between cited authors and authors integrated in the AutiHub database. Last computed: 16 Aug 2026 11:30.

Cited references
47
Total cited references integrated for this publication.
Cited references with an identified autistic author
0 / 47 (0.0%)
Cited author occurrences identified as autistic
0 / 324 (0.0%)
Among occurrences linked to AutiHub author records: 0 / 69 (0.0%). Distinct cited authors identified as autistic: 0 / 261 (0.0%).
Cited author occurrences linked to AutiHub author records
69 / 324 (21.3%)
Distinct linked cited authors: 26 / 261 (10.0%)
Linked cited-author occurrences not identified as autistic
69 / 69 (100.0%)
Among linked cited-author occurrences only. Across all cited-author occurrences: 69 / 324 (21.3%). Distinct linked cited authors not identified as autistic: 26 / 26 (100.0%).
  1. Trevor Hastie , Jerome Friedman , Robert Tibshirani (2001). The Elements of Statistical Learning . Springer New York.
    Type: Book DOI: 10.1007/978-0-387-21606-5 OpenAlex: https://openalex.org/W1480376833
    Crossref OpenAlex
  2. (n.d.). DeBoer, T., Scott, L. & Nelson, C. Methods for Acquiring and Analyzing Infant Event-Related Potentials in Infant EEG and Event-Related Potentials. 5–38 (Psychology, 2013).
    Type: Other
    Crossref
  3. Emily Werner , Géraldine Dawson , Julie Osterling , Nuhad Dinno (2000). Brief Report: Recognition of Autism Spectrum Disorder Before One Year of Age: A Retrospective Study Based on Home Videotapes . Journal of Autism and Developmental Disorders, 30(2), 157-162. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1023/a:1005463707029 OpenAlex: https://openalex.org/W1942840158
    Crossref OpenAlex
  4. Guillaume Thierry (2005). The use of event-related potentials in the study of early cognitive development . Infant and Child Development, 14(1), 85-94. Wiley.
    Type: Article DOI: 10.1002/icd.353 OpenAlex: https://openalex.org/W2089016957
    Crossref OpenAlex
  5. Mary L McHugh (2012). Interrater reliability: the kappa statistic. PubMed, 22(3), 276-82.
    Type: Article OpenAlex: https://openalex.org/W1966976587
    OpenAlex
  6. Dmitry Isaev , Samantha Major , Michael Murias , Kimberly L. H. Carpenter , David Carlson , Guillermo Sapiro et al. (2020). Relative Average Look Duration and its Association with Neurophysiological Activity in Young Children with Autism Spectrum Disorder . Scientific Reports, 10(1), 1912-1912. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1038/s41598-020-57902-1 OpenAlex: https://openalex.org/W3004416636
    Crossref OpenAlex
  7. E.V. Orekhova , T.A. Stroganova , I.N. Posikera , M. Elam (2006). EEG theta rhythm in infants and preschool children . Clinical Neurophysiology, 117(5), 1047-1062. Elsevier BV.
    Type: Article DOI: 10.1016/j.clinph.2005.12.027 OpenAlex: https://openalex.org/W1972174341
    Crossref OpenAlex
  8. Sara Jane Webb , Raphael Bernier , Heather A. Henderson , Mark H. Johnson , Emily J.H. Jones , Matthew D Lerner et al. (2015). Guidelines and Best Practices for Electrophysiological Data Collection, Analysis and Reporting in Autism . Journal of Autism and Developmental Disorders, 45(2), 425-443. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s10803-013-1916-6 OpenAlex: https://openalex.org/W1998424804
    Crossref OpenAlex
  9. Mayada Elsabbagh , Janice Fernandes , Sara Jane Webb , Géraldine Dawson , Tony Charman , Mark H. Johnson (2013). Disengagement of Visual Attention in Infancy is Associated with Emerging Autism in Toddlerhood . Biological Psychiatry, 74(3), 189-194. Elsevier BV.
    Type: Article DOI: 10.1016/j.biopsych.2012.11.030 OpenAlex: https://openalex.org/W1992386489
    Crossref OpenAlex
  10. Manuela Stets , Daniel Stahl , Vincent M. Reid (2012). A Meta-Analysis Investigating Factors Underlying Attrition Rates in Infant ERP Studies . Developmental Neuropsychology, 37(3), 226-252. Informa UK Limited.
    Type: Article DOI: 10.1080/87565641.2012.654867 OpenAlex: https://openalex.org/W2066514016
    Crossref OpenAlex
  11. Martha Ann Bell , Kimberly Cuevas (2012). Using EEG to Study Cognitive Development: Issues and Practices . Journal of Cognition and Development, 13(3), 281-294. Informa UK Limited.
    Type: Article DOI: 10.1080/15248372.2012.691143 OpenAlex: https://openalex.org/W2080883821
    Crossref OpenAlex
  12. Ann E. Ellis , Charles A. Nelson (1999). Category prototypicality judgments in adults and children: Behavioral and electrophysiological correlates . Developmental Neuropsychology, 15(2), 193-211. Informa UK Limited.
    Type: Article DOI: 10.1080/87565649909540745 OpenAlex: https://openalex.org/W1998858591
    Crossref OpenAlex
  13. Tatiana A Stroganova , Elena V. Orekhova , Irina N Posikera (1998). Externally and internally controlled attention in infants: an EEG study . International Journal of Psychophysiology, 30(3), 339-351. Elsevier BV.
    Type: Article DOI: 10.1016/s0167-8760(98)00026-9 OpenAlex: https://openalex.org/W2006601933
    Crossref OpenAlex
  14. Rebecca M. Todd , Marc D. Lewis , Liesel-Ann Meusel , Philip David Zelazo (2008). The time course of social-emotional processing in early childhood: ERP responses to facial affect and personal familiarity in a Go-Nogo task . Neuropsychologia, 46(2), 595-613. Elsevier BV.
    Type: Article DOI: 10.1016/j.neuropsychologia.2007.10.011 OpenAlex: https://openalex.org/W2133314703
    Crossref OpenAlex
  15. Michael A. Regan , Charlene Hallett , Craig P. Gordon (2011). Driver distraction and driver inattention: Definition, relationship and taxonomy . Accident Analysis & Prevention, 43(5), 1771-1781. Elsevier BV.
    Type: Article DOI: 10.1016/j.aap.2011.04.008 OpenAlex: https://openalex.org/W2009193978
    Crossref OpenAlex
  16. Michael Murias , Samantha Major , Katherine Davlantis , Lauren Franz , Adrianne Harris , Benjamin Rardin et al. (2018). Validation of eye‐tracking measures of social attention as a potential biomarker for autism clinical trials . Autism Research, 11(1), 166-174. Wiley.
    Type: Article DOI: 10.1002/aur.1894 OpenAlex: https://openalex.org/W2770305551
    Crossref OpenAlex
  17. James C. McPartland , Sara Jane Webb , Brandon Keehn , Géraldine Dawson (2011). Patterns of Visual Attention to Faces and Objects in Autism Spectrum Disorder . Journal of Autism and Developmental Disorders, 41(2), 148-157. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s10803-010-1033-8 OpenAlex: https://openalex.org/W2017848939
    Crossref OpenAlex
  18. Géraldine Dawson , Emily J.H. Jones , Kristen Merkle , Kaitlin Venema , Rachel Lowy , Susan Faja et al. (2012). Early Behavioral Intervention Is Associated With Normalized Brain Activity in Young Children With Autism . Journal of the American Academy of Child & Adolescent Psychiatry, 51(11), 1150-1159. Elsevier BV.
    Type: Article DOI: 10.1016/j.jaac.2012.08.018 OpenAlex: https://openalex.org/W2111874514
    Crossref OpenAlex
  19. Mandy J. Maguire , Grant Magnon , Anna E. Fitzhugh (2014). Improving data retention in EEG research with children using child-centered eye tracking . Journal of Neuroscience Methods, 238, 78-81. Elsevier BV.
    Type: Article DOI: 10.1016/j.jneumeth.2014.09.014 OpenAlex: https://openalex.org/W2023009248
    Crossref OpenAlex
  20. Katherine Gotham , SUSAN RISI , Géraldine Dawson , HELEN TAGER-FLUSBERG , ROBERT JOSEPH , ALICE CARTER et al. (2008). A Replication of the Autism Diagnostic Observation Schedule (ADOS) Revised Algorithms . Journal of the American Academy of Child & Adolescent Psychiatry, 47(6), 642-651. Elsevier BV.
    Type: Article DOI: 10.1097/chi.0b013e31816bffb7 OpenAlex: https://openalex.org/W2034529431
    Crossref OpenAlex
  21. Michael Murias , Samantha Major , Scott N. Compton , Jessica Buttinger , Jessica M. Sun , Joanne Kurtzberg et al. (2018). Electrophysiological Biomarkers Predict Clinical Improvement in an Open-Label Trial Assessing Efficacy of Autologous Umbilical Cord Blood for Treatment of Autism . Stem Cells Translational Medicine, 7(11), 783-791. Oxford University Press (OUP).
    Type: Article DOI: 10.1002/sctm.18-0090 OpenAlex: https://openalex.org/W2886530524
    Crossref OpenAlex
  22. Jacob Cohen (1960). A Coefficient of Agreement for Nominal Scales . Educational and Psychological Measurement, 20(1), 37-46. SAGE Publications.
    Type: Article DOI: 10.1177/001316446002000104 OpenAlex: https://openalex.org/W2053154970
    Crossref OpenAlex
  23. Yotam Erel , Katherine Adams Shannon , Junyi Chu , Kim Scott , Melissa Kline Struhl , Peng Cao et al. (2023). iCatcher+: Robust and Automated Annotation of Infants’ and Young Children’s Gaze Behavior From Videos Collected in Laboratory, Field, and Online Studies . Advances in Methods and Practices in Psychological Science, 6(2), 251524592211472. SAGE Publications.
    Type: Article DOI: 10.1177/25152459221147250 OpenAlex: https://openalex.org/W4366425751
    Crossref OpenAlex
  24. Anna Kaiser , Pascal-M. Aggensteiner , Martin Holtmann , Andreas Fallgatter , Marcel Romanos , Karina Abenova et al. (2021). EEG Data Quality: Determinants and Impact in a Multicenter Study of Children, Adolescents, and Adults with Attention-Deficit/Hyperactivity Disorder (ADHD) . Brain Sciences, 11(2), 214. MDPI AG.
    Type: Article DOI: 10.3390/brainsci11020214 OpenAlex: https://openalex.org/W3126813410
    Crossref OpenAlex
  25. Brandon Keehn , Ralph-Axel Müller , Jeanne Townsend (2013). Atypical attentional networks and the emergence of autism . Neuroscience & Biobehavioral Reviews, 37(2), 164-183. Elsevier BV.
    Type: Article DOI: 10.1016/j.neubiorev.2012.11.014 OpenAlex: https://openalex.org/W2067020152
    Crossref OpenAlex
  26. Sara Jane Webb , Frederick Shic , Michael Murias , Catherine A. Sugar , Adam J. Naples , Erin Barney et al. (2020). Biomarker Acquisition and Quality Control for Multi-Site Studies: The Autism Biomarkers Consortium for Clinical Trials . Frontiers in Integrative Neuroscience, 13, 71-71. Frontiers Media SA.
    Type: Article DOI: 10.3389/fnint.2019.00071 OpenAlex: https://openalex.org/W2991600992
    Crossref OpenAlex
  27. Jill MacLaren Chorney , C. Meghan McMurtry , Christine T. Chambers , Roger Bakeman (2015). Developing and Modifying Behavioral Coding Schemes in Pediatric Psychology: A Practical Guide . Journal of Pediatric Psychology, 40(1), 154-164. Oxford University Press (OUP).
    Type: Article DOI: 10.1093/jpepsy/jsu099 OpenAlex: https://openalex.org/W2123417791
    Crossref OpenAlex
  28. The BASIS Team , Elena V Orekhova , Mayada Elsabbagh , Emily J.H. Jones , Géraldine Dawson , Tony Charman et al. (2014). EEG hyper-connectivity in high-risk infants is associated with later autism . Journal of Neurodevelopmental Disorders, 6(1), 40. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1186/1866-1955-6-40 OpenAlex: https://openalex.org/W2154205996
    Crossref OpenAlex
  29. Kyle Krafka , Aditya Khosla , Petr Kellnhofer , Harini Kannan , Suchendra Bhandarkar , Wojciech Matusik et al. (2016). Eye Tracking for Everyone . 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2176-2184. IEEE.
    Type: Article DOI: 10.1109/cvpr.2016.239 OpenAlex: https://openalex.org/W2468114283
    Crossref OpenAlex
  30. Iuliia Kotseruba , John K. Tsotsos (2022). Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets . IEEE Transactions on Intelligent Transportation Systems, 23(11), 19907-19928. Institute of Electrical and Electronics Engineers (IEEE).
    Type: Article DOI: 10.1109/tits.2022.3186613 OpenAlex: https://openalex.org/W4285231587
    Crossref OpenAlex
  31. Eero Ahtola , Susanna Stjerna , Nathan Stevenson , Sampsa Vanhatalo (2017). Use of eye tracking improves the detection of evoked responses to complex visual stimuli during EEG in infants . Clinical Neurophysiology Practice, 2, 81-90. Elsevier BV.
    Type: Article DOI: 10.1016/j.cnp.2017.03.002 OpenAlex: https://openalex.org/W2603026798
    Crossref OpenAlex
  32. Wanli Li , Jing Huang , Guoqi Xie , Fakhri Karray , Renfa Li (2021). A survey on vision-based driver distraction analysis . Journal of Systems Architecture, 121, 102319. Elsevier BV.
    Type: Article DOI: 10.1016/j.sysarc.2021.102319 OpenAlex: https://openalex.org/W3208798367
    Crossref OpenAlex
  33. Tadas Baltrusaitis , Amir Zadeh , Yao Chong Lim , Louis-Philippe Morency (2018). OpenFace 2.0: Facial Behavior Analysis Toolkit . 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 59-66. IEEE.
    Type: Article DOI: 10.1109/fg.2018.00019 OpenAlex: https://openalex.org/W2807126412
    Crossref OpenAlex
  34. Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library . arXiv (Cornell University), 32, 8026-8037. arXiv.
    Type: Preprint DOI: 10.48550/arxiv.1912.01703 OpenAlex: https://openalex.org/W2970971581
    OpenAlex
  35. Camillo Lugaresi , Jiuqiang Tang , Hadon Nash , Chris McClanahan , Esha Uboweja , Michael Hays et al. (2019). MediaPipe: A Framework for Building Perception Pipelines . arXiv (Cornell University). arXiv.
    Type: Preprint DOI: 10.48550/arxiv.1906.08172 OpenAlex: https://openalex.org/W2949520286
    Crossref
  36. Fernando De la Torre , Wen-Sheng Chu , Xuehan Xiong , Francisco Vicente , Xiaoyu Ding , Jeffrey Cohn (2015). IntraFace . 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 1(61), 1-8. IEEE.
    Type: Article DOI: 10.1109/fg.2015.7163082 OpenAlex: https://openalex.org/W4238319993
    Crossref OpenAlex
  37. Sam Perochon , J. Matías Di Martino , Rachel Aiello , Jeffrey Baker , Kimberly L. H. Carpenter , Zhuoqing Chang et al. (2021). A scalable computational approach to assessing response to name in toddlers with autism . Journal of Child Psychology and Psychiatry, 62(9), 1120-1131. Wiley.
    Type: Article DOI: 10.1111/jcpp.13381 OpenAlex: https://openalex.org/W3135851158
    Crossref OpenAlex
  38. Zhuoqing Chang , J. Matías Di Martino , Rachel Aiello , Jeffrey Baker , Kimberly L. H. Carpenter , Scott N. Compton et al. (2021). Computational Methods to Measure Patterns of Gaze in Toddlers With Autism Spectrum Disorder . JAMA Pediatrics, 175(8), 827. American Medical Association (AMA).
    Type: Article DOI: 10.1001/jamapediatrics.2021.0530 OpenAlex: https://openalex.org/W3157659231
    Crossref OpenAlex
  39. Xuesheng Qian , Mianjie Wang , Xinyue Wang , Yihang Wang , Weihui Dai (2022). Intelligent Method for Real-Time Portable EEG Artifact Annotation in Semiconstrained Environment Based on Computer Vision . Computational Intelligence and Neuroscience, 2022, 1-14. Wiley.
    Type: Article DOI: 10.1155/2022/9590411 OpenAlex: https://openalex.org/W4213352249
    Crossref OpenAlex
  40. Colin D. Elliott (2021). Differential Ability Scales--Second Edition . PsycTESTS Dataset. American Psychological Association (APA).
    Type: Dataset DOI: 10.1037/t15074-000 OpenAlex: https://openalex.org/W4205285540
    Crossref OpenAlex
  41. DE King (2009). King, D. E. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009). J. Mach. Learn. Res., 10, 1755.
    Type: Other
    Crossref
  42. Naftali Tishby , Noga Zaslavsky (2015). Deep learning and the information bottleneck principle . 2015 IEEE Information Theory Workshop (ITW), 1-5. IEEE.
    Type: Preprint DOI: 10.1109/itw.2015.7133169 OpenAlex: https://openalex.org/W2096516049
    Crossref
  43. Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In 34th International Conference on Machine Learning, ICML 2017. Vol. 3. 2130–2143 (2017).
    Type: Other
    Crossref
  44. Marry L. McHugh (2012). Interrater reliability: the kappa statistic . Biochemia Medica, 22(3), 276-282. Croatian Society for Medical Biochemistry and Laboratory Medicine.
    Type: Article DOI: 10.11613/bm.2012.031 OpenAlex: https://openalex.org/W4294214983
    Crossref OpenAlex
  45. Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization. http://arxiv.org/abs/1412 . 6980 (2015).
    Type: Other
    Crossref
  46. Alexandra L. Bey , Maura Sabatos-DeVito , Kimberly L. H. Carpenter , Lauren Franz , Jill Howard , Saritha Vermeer et al. (2024). Automated Video Tracking of Autistic Children’s Movement During Caregiver-Child Interaction: An Exploratory Study . Journal of Autism and Developmental Disorders, 54(10), 3706-3718. Springer Science and Business Media LLC.
    Type: Article DOI: 10.1007/s10803-023-06107-2 OpenAlex: https://openalex.org/W4386254050
    Crossref OpenAlex
  47. (n.d.). Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Article 721. (Curran Associates Inc., 2019).
    Type: Other
    Crossref