Open-i®

Open-i screenshot

The Open-I® project is the result of combining text processing, image analysis and machine learning to create a system that enables about 10,000 users a day to retrieve relevant images and expanded citations from the open-access biomedical literature, and from clinical and historic image collections as well. Searching may be done by text as well as image queries. Images include a wide range of clinical imaging modalities, graphs, charts, photographs and other illustrations. The images are indexed by text in captions and mentions in the article, as well as by image features.

The underlying research in natural language processing, biomedical image analysis, and informatics leading to the design, development and practical implementation of this system are described in the following articles:
Design and Development of a Multimodal Biomedical Information Retrieval System
Combining Text and Visual Features for Biomedical Information Retrieval

Publications/Tools

Candemir S, Antani SK, Xue Z, Thoma GR. Novel Method for Storyboarding Biomedical Videos for Medical Informatics. 30th IEEE International Symposium on Computer-Based Medical Systems
Abstract | PDF

Ben Abacha A, De Herrera A, Gayen S, Demner-Fushman D, Antani SK. NLM at ImageCLEF 2017 Caption Task. International Conference of the Cross-Language Evaluation Forum for European Languages 2017 Sep 11 (pp. 358-360). Springer, Cham.
Abstract | PDF

Xue Z, Antani SK, Long LR, Thoma GR. Automatic multi-label annotation of abdominal CT images using CBIR. Proc. SPIE 10138, Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications, 1013807 (March 13, 2017); doi:10.1117/12.2254368.
Abstract | PDF

Chachra S, Ben Abacha A, Shooshan SE, Rodriguez L, Demner-Fushman D. A Hybrid Approach to Generation of Missing Abstracts in Biomedical Literature. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers: 1093-1100.
Abstract | PDF

De Herrera A, Schaer R, Antani SK, Müller H. Using Crowdsourcing for Multi-label Biomedical Compound Figure Annotation. In: Carneiro G. et al. (eds) Deep Learning and Data Labeling for Medical Applications. LABELS 2016, DLMIA 2016. Lecture Notes in Computer Science, vol 10008. Springer, Cham
Abstract | PDF

De Herrera A, Long LR, Antani SK. Content-Based fMRI Brain Maps Retrieval. International Conference on Brain and Health Informatics, Omaha, NE, USA, October 13-16, 2016.
Abstract | PDF

Narum R, Zou J, Antani SK. Semi-Automated Ground-Truth Data Collection and Annotation for Journal Figure Analysis [Poster]. 2016 NIH Research Festival
Abstract | PDF

Xue Z, Rahman M, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Modality Classification for Searching Figures in Biomedical Literature. Proceedings of the IEEE 29th International Symposium on Computer-Based Medical Systems, pp. 152-157, 2016. doi:10.1109/CBMS.2016.29.
Abstract | PDF | URL: https://www.computer.org/csdl/proceedings/cbms/2016/9036/00/9036a152-abs.html

Demner-Fushman D, Antani SK, Chachra S, Kushnir M, Gayen S. Open-i Imaging informatics, natural language processing and multi-modal information retrieval – research and development Technical Report to the LHNCBC Board of Scientific Counselors September 2016
Abstract | PDF

Ben Abacha A, Demner-Fushman D. Meta-Learning with Selective Data Augmentation for Medical Entity Recognition. International Journal of Computational Linguistics and Applications (IJCLA) 7(2): 167-182 (2016)
Abstract | PDF

Xue Z, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Improving face image extraction by using deep learning technique. Proc. SPIE 9789, Medical Imaging 2016: PACS and Imaging Informatics: Next Generation and Innovations, 97890J (March 25, 2016); doi:10.1117/12.2216278.
Abstract | PDF

Demner-Fushman D, Shooshan SE, Rodriguez L, Antani SK, Thoma GR. Annotation of Chest Radiology Reports for Indexing and Retrieval. Multimodal Retrieval in the Medical Domain 2015 (MRMD 2015), Vienna, Austria, March 29, 2015
Abstract | PDF

Antani SK, You D, Simpson M, Rahman M, Demner-Fushman D, Thoma GR. The role of image modality and visual characteristics in archiving biomedical images. Proc of IS&T Archiving Conference. Volume 2013, Number 1, 2013, pp. 31-35.
Abstract | PDF

You D, Antani SK, Demner-Fushman D, Thoma GR. Does Figure-Text Improve Biomedical Article Retrieval? A Pilot Study. CBMS 2014. May 2014.
Abstract

Rahman MM, You D, Simpson MS, Antani SK, Demner-Fushman D. Interactive Cross and Multimodal Biomedical Image Retrieval Based on Automatic Region-Of-Interest (ROI) Identification and Classification. International Journal of Multimedia Information Retrieval (MMIR) 2014 Sep;3(3):131-146.
Abstract

Vajda S. Label the many with a few: Semi-automatic medical image modality discovery in a large image collection. IEEE Symposium Series on Computational Intelligence, Orlando, FL, Dec 9-12, 2014
Abstract | PDF

Xue Z, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Body Segment Classification for Visible Human Cross Section Slices. IEEE 27th International Symposium on Computer-Based Medical Systems (CBMS), New York, NY, May 2014: pp. 199-204
Abstract | PDF | URL: http://dx.doi.org/10.1109/CBMS.2014.55

Kilicoglu H, Demner-Fushman D. Coreference Resolution for Structured Drug Product Labels Proceedings of BioNLP 2014. pp. 44-53.
Abstract | PDF

Chachra S, Xue Z, Antani SK, Demner-Fushman D, Thoma GR. Extraction and Labeling High-resolution Images from PDF Documents. Proc. SPIE 9021, Document Recognition and Retrieval XXI, 90210Q (December 27, 2013). doi:10.1117/12.2042336.
Abstract | PDF

Xue Z, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Classification of CT Figures in Biomedical Articles Based on Body Segments. 2013 IEEE International Conference on Healthcare Informatics (ICHI), pp.264,268, 9-11 Sept. 2013. doi: 10.1109/ICHI.2013.17.
Abstract | PDF

Apostolova E, You D, Xue Z, Antani SK, Demner-Fushman D, Thoma GR. Image Retrieval From Scientific Publications: Text and Image Content Processing To Separate Multipanel Figures. JASIST (Journal of the American Society for Information Science and Technology), 64(5):893–908, 2013.
Abstract | PDF

Xue Z, You D, Antani SK, Long LR, Demner-Fushman D, Thoma GR. Classification of Visual Signs in Abdominal CT Image Figures in Biomedical Literature. Proc. SPIE. 9039. Medical Imaging 2014: PACS and Imaging Informatics: Next Generation and Innovations. 90390T.
Abstract | PDF

Xue Z, Long LR, Antani SK, Thoma GR. Pathology-based vertebral image retrieval. Proceedings of the 8th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, ISBI 2011, March 30 – April 2, 2011, Chicago, Illinois, USA. pages 1893-1896, IEEE, 2011. doi: 10.1109/ISBI.2011.5872778.
Abstract | URL: http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5872778

Demner-Fushman D, Antani SK, Simpson M. Web Interface: Open-I
Abstract

Cheng B, Antani SK, Stanley J, Demner-Fushman D, Thoma GR. Automatic segmentation of subfigure image panels for multimodal biomedical document retrieval. Proceeding of: Document Recognition and Retrieval XVIII – DRR 2011, 18th Document Recognition and Retrieval Conference, part of the IS&T-SPIE Electronic Imaging Symposium, San Jose, CA, USA, January 24-29, 2011, Proceedings.
Abstract | PDF

You D, Antani SK, Demner-Fushman D, Rahman MM, Govindaraju V, Thoma GR. Automatic identification of ROI in figure images toward improving hybrid (text and image) biomedical document retrieval. Proceedings of SPIE Electronic Imaging Science and Technology, Document Retrieval and Recognition XVIII. San Francisco, CA. January 2011;7874:78740K.
Abstract | PDF

You D, Antani SK, Demner-Fushman D, Govindaraju V, Thoma GR. Detecting Figure-Panel Labels In Medical Journal Articles Using MRF. Proceedings of 11th International Conference on Document Analysis and Recognition (ICDAR 2011). Beijing, China. September 2011:967-971.
Abstract | PDF

Rahman MM, You D, Simpson M. An interactive image retrieval framework for biomedical articles based on visual region-of-interest (ROI) identification and classification [Abstract]. The 2nd IEEE Conference on Healthcare Informatics, Imaging, and Systems Biology Analyzing Big Data for Healthcare and Biomedical Sciences (HISB 2012). La Jolla, CA. September 2012.
Abstract | PDF

Simpson M, You D, Rahman MM, Demner-Fushman D, Antani SK, Thoma GR. ITI’s participation in the ImageCLEF 2012 medical retrieval and classification tasks. CLEF 2012 Working Notes. Rome, Italy. September 2012.
Abstract | PDF | URL: http://ims-sites.dei.unipd.it/documents/71612/155385/CLEF2012wn-ImageCLEF-Simpso…

You D, Simpson M, Antani SK, Demner-Fushman D, Thoma GR. A robust pointer segmentation in biomedical images toward building a visual ontology for biomedical article retrieval. Proc. SPIE 8658, Document Recognition and Retrieval XX, 86580Q (February 4, 2013); doi:10.1117/12.2005934.
Abstract | PDF | URL: http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1568647

You D, Rahman MM, Antani SK, Demner-Fushman D, Thoma GR. Text- and content-based biomedical image modality classification. Proc. SPIE Medical Imaging. Orlando, FL. February 2013;8674-21.
Abstract

Long LR, Antani SK, Thoma GR, Deserno TM. Content-based Image Retrieval For Advancing Medical Diagnostics, Treatment, and Education New Technologies for Advancing Healthcare and Clinical Practices. June 2011:1-17.
Abstract

Simpson MS, You D, Rahman MM, Antani SK, Thoma GR, Demner-Fushman D. Towards the creation of a visual ontology of biomedical imaging entities. AMIA Annu Symp Proc. 2012;2012:866-75. Epub 2012 Nov 3.
Abstract | PDF | PMID: 23304361 | PMCID: PMC3540530

Demner-Fushman D, Antani SK, Simpson M, Thoma GR. Design and Development of a Multimodal Biomedical Information Retrieval System JCSE. June 2012;6(2):168-177.
Abstract | PDF

Hsu W, Antani SK, Long LR, Neve L, Thoma GR. SPIRS: a Web-based image retrieval system for large biomedical databases. Int J Med Inform. 2009 Apr;78 Suppl 1:S13-24. doi: 10.1016/j.ijmedinf.2008.09.006. Epub 2008 Nov 8.
Abstract | PMID: 18996737 | PMCID: PMC2693318

Stanley RJ, De S, Demner-Fushman D, Antani SK, Thoma GR. An image feature-based approach to automatically find images for application to clinical decision support. Comput Med Imaging Graph. 2011 Jul;35(5):365-72. doi: 10.1016/j.compmedimag.2010.11.008. Epub 2010 Dec 8.
Abstract | PMID: 21144707 | PMCID: PMC3084330

Simpson M, Demner-Fushman D, Thoma GR. Evaluating the Importance of Image-related Text for Ad-hoc and Case-based Biomedical Article Retrieval. AMIA Annu Symp Proc. 2010 Nov 13;2010:752-6.
Abstract | PMID: 21347079 | PMCID: PMC3041350

Rahman MM, Antani S, Long LR, Demner-Fushman D, Thoma GR. Multi-Modal Query Expansion Based on Local Analysis for Medical Image Retrieval. Lecture Notes in Computer Science. First MICCAI International Workshop on Medical Content-Based Retrieval for Clinical Decision Support (MCBR-CDS 2009); part of the 12th International Conference on Medical Image Computing and Computer Assisted Interventio February 2010;5853/2010(doi: 10.1007/978-3-642-11769-5):110-9..
Abstract | PDF

You D, Antani S, Demner-Fushman D, Rahman MM, Govindarju V, Thoma GR. Biomedical Article Retrieval Using Multimodal Features and Image Annotations In Region-based CBIR. Document Recognition and Retrieval XVII. Edited by Likforman-Sulem, Laurence; Agam, Gady. Proceedings of the SPIE. San Jose, CA. January 2010;7534:75340V-75340V-12.
Abstract | PDF

Demner-Fushman D, Simpson M, Antani S, Thoma GR. Annotation and retrieval of clinically relevant images. Int J Med Inform. 2009 Dec;78(12):e59-67. doi: 10.1016/j.ijmedinf.2009.05.003. Epub 2009 Jul 9.
Abstract | PDF | PMID: 19546026 | PMCID: PMC2783743

Deserno TM, Antani S, Long LR. Content-based image retrieval for scientific literature access. Methods Inf Med. 2009;48(4):371-80. doi: 10.3414/ME0561. Epub 2009 Jul 20.
Abstract | PDF | PMID: 19621115

Deserno TM, Antani S, Long LR. Ontology of gaps in content-based image retrieval. J Digit Imaging. 2009 Apr;22(2):202-15. doi: 10.1007/s10278-007-9092-x. Epub 2008 Feb 1.
Abstract | PDF | PMID: 18239964 | PMCID: PMC3043678

Rahman MM, Antani SK, Thoma GR. A Classification-Driven Similarity Matching Framework For Retrieval of Biomedical Images. 11th ACM International Conference on Multimedia Information Retrieval (MIR 2010). 2010:147-154.
Abstract | PDF

Lee DJ, Antani S, Chang Y, Long LR, Christensen P, Jeronimo J. CBIR of Spine X-ray Images On Inter-vertebral Disc Space and Shape Profiles Using Feature Ranking and Voting Consensus. Data & Knowledge Engineering, Special Issue on Knowledge Discovery in Medicine. 2009.
Abstract | PDF

Rahman MM, Antani SK, Thoma GR. Biomedical Image Retrieval In a Fuzzy Feature Space With Affine Region Detection and Vector Quantization of a Scale-Invariant Descriptor. 6th International Symposium on Visual Computing (ISVC) 2010. Las Vegas, NV. November 2010.
Abstract | PDF

Rahman MM, Antani SK, Thoma GR. Bag of Keypoints-Based Biomedical Image Search With Affine Covariant Region Detection and Correlation-Enhanced Similarity Matching. 23rd IEEE International Symposium on Computer-Based Medical Systems (CBMS ) 2010. Perth, Australia. October 2010:261-6.
Abstract | PDF

Demner-Fushman D, Antani SA, Simpson M, Rahman MM. Combining Text and Visual Features for Biomedical Information Retrieval and Ontologies. September 2010 Technical Report to the LHNCBC Board of Scientific Counselors.
Abstract | PDF