Human-AI Interactive and Continuous Sensemaking: A Case Study of Image Classification using Scribble Attention Maps

Haifeng Shen, Kewen Liao, Zhibin Liao, Job Doornberg, Maoying Qiao, Anton Van Den Hengel, Johan W. Verjans

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Advances in Artificial Intelligence (AI), especially the stunning achievements of Deep Learning (DL) in recent years, have shown AI/DL models possess remarkable understanding towards the logic reasoning behind the solved tasks. However, human understanding towards what knowledge is captured by deep neural networks is still elementary and this has a detrimental effect on human's trust in the decisions made by AI systems. Explainable AI (XAI) is a hot topic in both AI and HCI communities in order to open up the blackbox to elucidate the reasoning processes of AI algorithms in such a way that makes sense to humans. However, XAI is only half of human-AI interaction and research on the other half - human's feedback on AI explanations together with AI making sense of the feedback - is generally lacking. Human cognition is also a blackbox to AI and effective human-AI interaction requires unveiling both blackboxes to each other for mutual sensemaking. The main contribution of this paper is a conceptual framework for supporting effective human-AI interaction, referred to as interactive and continuous sensemaking (HAICS). We further implement this framework in an image classification application using deep Convolutional Neural Network (CNN) classifiers as a browser-based tool that displays network attention maps to the human for explainability and collects human's feedback in the form of scribble annotations overlaid onto the maps. Experimental results using a real-world dataset has shown significant improvement of classification accuracy (the AI performance) with the HAICS framework.

Original languageEnglish
Title of host publicationExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021
PublisherAssociation for Computing Machinery
Number of pages8
ISBN (Electronic)9781450380959
Publication statusPublished or Issued - 8 May 2021
Event2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021 - Virtual, Online, Japan
Duration: 8 May 202113 May 2021

Publication series

NameConference on Human Factors in Computing Systems - Proceedings


Conference2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI EA 2021
CityVirtual, Online


  • attention map
  • explainable AI
  • image classification
  • interactive sensemaking
  • scribble interaction

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this