Image Text Integration project

Table of Contents

Project Leads:
Sameer Antani Dina Demner-Fushman, George Thoma

Project Members:
Glenn Ford, Matthew Simpson, Suchet Chachra, Michael Kushnir, Md Rahman, Zhiyun Xue, Daekeun You

Project Support:
Sonya Shooshan, Laritza Taft, Joseph Chow, Michael Chung

Introduction

This project seeks to improve information retrieval from collections of full-text biomedical articles, images, and patient cases, by moving beyond conventional text-based searching to combining both text and visual features to:

In addition to developing these tools, we test them in two related initiatives that seek to:

Test the protoype image search engine Open-I

See the technical report to the LHNCBC Board of Scientific Counselors (September 2010) for details.

Processes

  • Text processing:
    • Caption extraction and segmentation
    • Mention extraction
    • Biomedical terminology extraction
  • Image processing:
    • multi-panel figure segmentation
    • text and symbols localization
    • color and texture features computation
  • Image classification using supervised machine learning
  • Image annotation:
    • automatic UMLS-based medium-level annotation using text references to image regions of interest and mark-yp

Evaluation

Open-I was evaluated in the ImageCLEF medical image retrieval tasks.

Publications

  1. Demner-Fushman D, Antani S, Simpson MS, Thoma GR. Design and development of a multimodal biomedical information retrieval system. JCSE v.6, no.2, 2012:68-177.
  2. Simpson M, You D, Rahman M, Antani S, Thoma G, Demner-Fushman D. Towards the Creation of a Visual Ontology of Biomedical Imaging Entities. AMIA 2012 Fall Symposium
  3. Xue Z, Antani S, Long R, Demner-Fushman D, Thoma G. Window Classification of Brain CT Images in Biomedical Articles. AMIA 2012 Fall Symposium
  4. Simpson M, Demner-Fushman D, Thoma G. Evaluating the Importance of Image-related Text for Ad-hoc and Case-based Biomedical Article Retrieval. Proceedings of the 2010 Annual Symposium of the American Medical Information Association (AMIA 2010), Washington, DC, November 2010
  5. Simpson M, Sneiderman C, Demner-Fushman D, Thoma G, Knowledge Acquisition from Clinical Case Reports: Quality and Utility for Case-based Biomedical Article Retrieval. Proceedings of the 2010 Annual Symposium of the American Medical Information Association (AMIA 2010), Washington, DC, November 2010
  6. Stanley RJ, De S, Demner-Fushman D, Antani S, Thoma GR. An image feature-based approach to automatically find images for application to clinical decision support. Comput Med Imaging Graph. 2010 Dec 6.
  7. Rahman MM, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Multi-Modal Query Expansion Based On Local Analysis For Medical Image Retrieval. Lecture Notes in Computer Science. First MICCAI International Workshop on Medical Content-Based Retrieval for Clinical Decision Support, International Conference on Medical Image Computing and Computer Assisted Interventio February 2010;5853/2010(doi: 10.1007/978-3-642-11769-5):110-9.
  8. You D, Antani S, Demner-Fushman D, Rahman M, Govindaraju V, Thoma G. Biomedical Article Retrieval Using Multimodal Features and Image Annotations In Region-based CBIR. Document Recognition and Retrieval XVII. Proceedings of the SPIE. San Jose, CA. January 2010;7534:75340V-75340V-12.
  9. Simpson M, Demner-Fushman D, Antani S, Thoma GR. Design and Implementation of a Biomedical Image Retrieval System. NIH Research Festival. October 6-9, 2009
  10. Simpson M, Rahman MM, Demner-Fushman D, Antani S, Thoma GR. Text- and Content-based Approaches to Image Retrieval for the ImageCLEF
    2009 Medical Retrieval Track. CLEF2009 Working Notes. CLEF 2009 Workshop 30 September - 2 October, Corfu, Greece, in conjunction with ECDL2009.
  11. Demner-Fushman D, Antani S, Simpson M, Thoma GR. Annotation and retrieval of clinically relevant images. Int J Med Inform. 2009 Jun 20.
  12. Apostolova E, Demner-Fushman D. Towards Automatic Image Region Annotation - Image Region Textual Coreference Resolution. In Proceedings of the 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics. June 2009. Boulder, Colorado. Association for Computational Linguistics, p 41--44.
  13. You D, Apostolova E, Antani S, Demner-Fushman D, Thoma G. Figure content analysis for improved biomedical article retrieval. Proc. SPIE, Vol. 7247, 72470V (2009)
  14. Simpson M, Demner-Fushman D, Sneiderman C, Antani SK Thoma GR. Using Non-lexical Features to Identify Effective Indexing Terms for Biomedical Illustrations Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL-09),Athens, Greece, April 2009
  15. Sneiderman C, Demner-Fushman D, Fung KW, Bray B. UMLS-based Automatic Image Indexing.
    Proceedings of the 2008 Annual Symposium of the American Medical Information Association (AMIA 2008), Washington, DC, November 2008
  16. Demner-Fushman D, Antani S, Simpson M, Thoma GR. Combining Medical Domain Ontological Knowledge and Low-level Image Features for Multimedia Indexing
    LREC 2008 (Sixth International Conference on Language Resources and Evaluation), OntoImage Workshop, Marrakech, Morocco, May 2008
  17. Antani S, Demner-Fushman D, Li J, Srinivasan BV, Thoma GR. Exploring use of images in clinical articles for decision support in Evidence-Based Medicine. Proc. SPIE-IS&T Electronic Imaging. San Jose, CA. January 2008;6815:68150Q(1-10)
  18. Demner-Fushman D, Antani SK, Thoma GR. Automatically Finding Images for Clinical Decision Support.
    Proceedings of IEEE International Workshop on Data Mining in Medicine (DM-Med 2007). Omaha, NE. October 2007;:139-44
  19. Li J, Demner-Fushman D, Antani SK, Thoma GR. Localizing Text and Symbols in Images from Biomedical Journal Articles.
    Poster at 20th NIH Research Festival. National Institutes of Health September 2007
  20. Srinivasan B, Antani SK, Demner-Fushman D, Thoma GR. Identification and Segmentation of Multi-Panel Images in Biomedical Journal Articles
    Poster at 20th NIH Research Festival. National Institutes of Health. September 2007