AI & Digital Health

Overview

Established in 2020, the SERI AI & Digital Health Research Group brings together a multi-disciplinary team consisting of experienced ophthalmologists and clinician scientists, vision scientists and researchers, as well as computer scientists, artificial intelligence (AI), and block chain technology experts.

The strengths of this multi-disciplinary team include:

Clinical expertise in ophthalmology to drive translational research
Large clinical and epidemiological datasets
Domain technical expertise in ocular imaging, artificial intelligence, machine learning, deep learning, and block chain technology
Experience in innovation, entrepreneurship and industry

Our research goals are to:

Develop innovative AI and digital solutions to address unmet clinical needs
Harness the power of big data analytics and AI to improve patient care and outcomes
Develop useful AI tools for diagnostic evaluation, outcome prediction and prognostication, and to guide patient management and treatment
To provide a clinically simulated digital environment to trial different AI and digital solutions in ophthalmology

International AI Committees

American Academy of Ophthalmology AI Task Force, USA
Standards for Reporting of Diagnostic Accuracy Studies for Artificial Intelligence (STARD-AI)

Editorial Boards

Progress in Retinal and Eye Research, Ophthalmology, Ophthalmology Retina, JAMA Ophthalmology, British Journal of Ophthalmology, Asia-Pacific Journal of Ophthalmology, Frontiers in Medicine

Projects

Development, validation and testing of Singapore Eye Lesion Analzyer Plus (SELENA+) (Ting et al, JAMA 2017) URL: https://jamanetwork.com/journals/jama/fullarticle/2665775
Deep Learning in Estimating Prevalence and Systemic Risk Factors for Diabetic Retinopathy: A Multi-ethnic Study (Ting et al. NPJ Digital Medicine 2018)
URL: https://www.nature.com/articles/s41746-019-0097-x
Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study (Bellemo et al, Lancet Digital Health, 2020)
URL: https://www.thelancet.com/journals/landig/article/PIIS2589-7500%2819%2930004-4/fulltext
Technical and imaging factors influencing performance of deep learning systems for diabetic retinopathy (Yip et al. NPJ Digital Medicine 2020)
URL: https://www.nature.com/articles/s41746-020-0247-1
Artificial Intelligence for Teleophthalmology-based Diabetic Retinopathy Screening in a National Program: A Modelled Economic Analysis Study (Xie et al. Lancet Digital Health 2020)
URL: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(20)30060-1/fulltext
Artificial Intelligence to Detect Papilledema from Ocular Fundus Photographs (Milea et al, NEJM, 2020) URL: https://www.nejm.org/doi/full/10.1056/NEJMoa1917130
A deep learning algorithm to detect chronic kidney disease from retinal photographs in community-based populations (Sabanayagam et al, Lancet Digital Health, 2020)
URL: https://www.thelancet.com/journals/landig/article/PIIS2589-7500(20)30063-7/fulltext

1. Development, Validation and Testing of SELENA+

In collaboration with 30 physicians and computer scientists worldwide, we conducted a Deep Eye Study, a multi-center AI study involving multiple countries and retinal cameras. This study involves the development and testing of a deep learning system using a customised convolutional neural network (CNN) and approximately 500,000 retinal images, in detecting 3 major blinding eye conditions, namely the diabetic retinopathy (DR), glaucoma and age-related macular degenerations. In the study, we utilised a total of 10 external validation datasets to further validate the generalisability of this system, using varying reference standards, retinal cameras, races, age groups, gender and glycemic controls. This deep learning system has been patented, approved by the Singapore Health Service Authority (HSA), European CE Mark.

For real-world implementation, we have been working closely with the National Healthcare Group (NHG), National University Hospital System (NUHS) and Integrated Health Information System (IHIS) to clinical translation, aiming to clinically integrate SELENA into the existing Singapore Integrated Diabetic Retinopathy Programme.

Source: PM Office https://www.pmo.gov.sg/Newsroom/DPM-Heng-Swee-Keat-at-SFF-X-SWITCH-2019

SELENA+ has also been included as one of the national AI strategies in Singapore. This initiative was announced by Deputy Prime Minister Heng Swee Keat at the Singapore FinTech Festival (SFF) x the Singapore Week of Innovation and Technology (SWITCH) conference.

2. Deep Learning in Estimating Prevalence and Systemic Risk Factors for Diabetic Retinopathy: A Multi-ethnic Study

Using SELENA+, we have conducted a follow-up study involving 19,000 patients of 5 races (Chinese, Malay, Indians, American Blacks and Caucasian White) from 4 countries (Singapore, USA, China and Australia). We showed that the deep learning system (DLS) had comparable performance to the grading performed by a total of 17 human assessors – 10 eye specialists and 7 non-medical graders in detecting prevalence of any DR, referable DR and vision-threatening DR (VTDR), but using a significantly much shorter time as compared to human assessors (1-month vs 2 years).

In addition, DLS was also as equally sensitive in detecting the systemic risk factors associated with referable DR and VTDR, namely younger age, increased HbA1c and increased systolic blood pressure.

3. Artificial Intelligence Using Deep Learning to Screen for Referable and Vision-Threatening Diabetic Retinopathy in Africa: A Clinical Validation Study

In collaboration with Kitwe Central Eye Hospital and Lusaka University Teaching Hospital in Zambia, and Frimley Park Hospital and Moorfields Eye Hospital in United Kingdom. This study involves approximately 1.5k Zambians population with diabetes recruited prospectively in the past. To date, this is one of the first studies that demonstrated the potential application AI in the under-resourced countries such as Africa. Zambia is a low-to-middle income country that is ranked 159th (out of 194 countries) for gross domestic product (GDP) per capita in 2018. The life expectancy of all population in sub-Saharan countries, for example Zambia, are much lower compared to the other parts of the world. In Africa, the prevalence of diabetes was estimated to be as high as 15%, and of whom, 30% may develop diabetic retinopathy at some stage in their lifetime.

“The ratio of the ophthalmologists to Zambian population is less than 3 to 1 million population, as opposed to 80:1 in high-income countries. The shortage of ophthalmologists in this region may result in severe delay in diabetes eye detection and treatment, resulting in irreversible blindness.

Trained with 76,000 multi-ethnic retinal images from Singapore, this AI system using deep learning technique, was able to achieve >90% diagnostic performance to detect referable DR, vision-threatening DR and diabetic macular edema. The results showed the potential of AI to be used as a DR screening tool in Africa, although more work is required to assess the feasibility of the supporting infrastructures and manpower expertise in dealing with the patients who require referrals for further management and intervention.

4. Technical and Imaging Factors Influencing Performance of Deep Learning Systems for Diabetic Retinopathy

Deep learning (DL) has been shown to be effective in developing diabetic retinopathy (DR) algorithms, possibly tackling financial and manpower challenges hindering implementation of DR screening. However, our systematic review of the literature reveals few studies studied the impact of different factors on these DL algorithms that are important for clinical deployment in real-world settings.

Using 455,491 retinal images, we evaluated two technical and three image-related factors in detection of referable DR. For technical factors, the performances of four DL models (VGGNet, ResNet, DenseNet, Ensemble) and two computational frameworks (Caffe and TensorFlow) were evaluated while for image-related factors, we evaluated image compression levels (reducing image size, 350KB, 300KB, 250KB, 200KB and 150KB), number of fields (7-field, 2-field, 1-field) and media clarity (pseudophakic vs phakic).

In detection of referable DR, four DL models showed comparable diagnostic performance (AUC 0.936-0.944). To develop the VGGNet model, two computational frameworks had similar AUC (0.936). The DL performance dropped when image size decreased below 250KB (AUC 0.936, 0.900, p<0.001). The DL performance performed better when there were increased number of fields (dataset 1: 2-field vs 1-field - AUC 0.936 vs 0.908, p<0.001; dataset 2: 7-field vs 2-field vs 1-field, AUC 0.949 vs 0.911 vs 0.895). DL performed better in the pseudophakic than phakic eyes (AUC 0.918 vs 0.833, p<0.001).

Various image-related factors play more significant roles than technical factors in determining the diagnostic performance, suggesting the importance of having robust training and testing datasets for DL training and deployment in the real-world settings.

Heat maps Generated for Compressed Images

5. Artificial Intelligence for Teleophthalmology-based Diabetic Retinopathy Screening in a National Programme: A Modelled Economic Analysis Study

Deep learning is a novel machine learning technique that has been shown to be as effective as human graders in detecting diabetic retinopathy (DR) from fundus photographs. In this study, we utilised a cost miminisation analysis to evaluate the potential savings of two deep learning approaches as compared to the current human assessment: 1) a semi-automated deep learning model as a “triage” filter prior to secondary human assessment, and 2) a fully-automated deep learning model without human assessment.

Using 39,006 consecutive patients with diabetes in a national DR screening programme in Singapore in 2015, we utilised a decision tree and TreeAge Pro to compare the actual cost of screening this cohort with human graders against the simulated cost for the semi- and fully- automated models. Model parameters included DR prevalence rates, DR screening costs under each screening model, cost of medical consultation; and diagnostic performance (i.e. sensitivity and specificity). The primary outcome is total cost for each screening model. Deterministic sensitivity analyses were performed to gauge the sensitivity of the results to key model assumptions.

From the health system perspective, the semi-automated screening model is the least expensive at US$62/patient/year compared to the fully-automated model at US$66/patient/year, and human assessment at US$77/patient/year. The current savings to the Singapore health system associated with switching to the semi-automated model is estimated to be US$ 489,000, which is roughly 20% of the current annual screening cost. By 2050, Singapore is projected to have 1M people with diabetes. At this time, annual savings is forecast to approach US$15M.

This study provides a strong economic rationale for using DLS as an assistive tool to screen for DR.

Tornado diagram showing how individual parameters affect incremental cost

6. Artificial Intelligence to Detect Papilledema from Ocular Fundus Photographs

Physicians do not confidently perform direct ophthalmoscopy. The use of artificial intelligence to detect papilledema and other optic disc abnormalities from fundus photographs has not been well studied. We trained, validated, and externally tested a deep learning system to classify the appearance of optic discs as normal, papilledema, or other abnormalities from 15,846 retrospectively collected ocular fundus photographs of multi-ethnic populations obtained with pupillary dilation and various digital cameras. Of these photographs, 14,341 from 19 sites in 11 countries were used for training and validation, and 1,505 photographs from 5 other sites in 5 countries were used for external testing. Performance at classifying the optic disc appearance was evaluated by calculating the area under the receiver operating characteristic curve (AUC), sensitivity and specificity, compared to a reference standard of clinical diagnoses by neuro-ophthalmologists.

The training and validation datasets from 7,532 patients included 1431 photographs; 9,156 of normal discs, 2,148 of papilledema, and 3,037 of other optic disc abnormalities. The percentage classified as normal and as papilledema varied across sites from 9.8% to 100% and from 0% to 59.5%, respectively. In the validation set, the system discriminated papilledema from normal and other optic disc abnormalities with AUC 0.99 (95% confidence interval [CI], 0.98-0.99), and normal from abnormal optic discs with AUC 0.99 (95% CI, 0.99-0.99). The external validation set used 1505 photographs and the AUC for the detection of papilledema was 0.96 (95%CI, 0.95-0.97), sensitivity 96.4% (95%CI, 93.9%-98.3%) and specificity 84.7% (95%CI, 82.3%-87.1%). A deep learning system using fundus photographs with dilated pupils differentiated among papilledema, normal optic discs, and non-papilledema optic disc abnormalities.

7. A Deep Learning Algorithm to Detect Chronic Kidney Disease from Retinal Photographs in Community-based Populations

Screening for chronic kidney disease (CKD) is challenging in community and primary care settings, even in high-income countries. We developed an artificial intelligence deep learning algorithm (DLA) to detect CKD from retinal images, which may add to existing CKD screening strategies.

We utilised data from three population-based multi-ethnic cross-sectional studies in Singapore and China. The Singapore Epidemiology of Eye Diseases study (SEED, aged ≥40 years) was used to develop (n=5188) and validate (n=1297) the DLA. External testing was conducted on two independent datasets: the Singapore Prospective Study Programme (SP2, n=3735, aged ≥25 years) and the Beijing Eye Study (BES, n= 1538, aged ≥40 years). CKD was defined as an estimated glomerular filtration rate [eGFR] <60 mL/min/1.73m2. Three models were trained: 1) image DLA, 2) risk factors (RF) including age, sex, ethnicity, diabetes and hypertension and 3) hybrid DLA combining image and RF. Model performances were evaluated using the area under the receiver operating characteristic curve (AUC).

In the SEED validation dataset, the AUC (95% confidence interval) was 0.911 (0.886-0.936), 0.916 (0.891-0.941) and 0.938 (0.917-0.959) for image, RF and hybrid DLA. Corresponding estimates in SP2 and BES testing datasets were 0.733, 0.829, 0.810 and 0.835, 0.887, 0.858. AUC estimates were similar in subgroups of people with diabetes (0.889, 0.899, 0.925) and hypertension (0.889, 0.889, 0.918).

A retinal image DLA shows good performance for estimating CKD, underlying the feasibility of using retinal photography as an adjunctive and/or opportunistic screening tool for CKD in community populations.

Publications

AI & Digital Innovations Publications

1. Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA. 2017 12;318(22):2211–23.

2. Ting DSW, Cheung CY, Nguyen Q, Sabanayagam C, Lim G, Lim ZW, et al. Deep learning in estimating prevalence and systemic risk factors for diabetic retinopathy: a multi-ethnic study. NPJ Digit Med. 2019;2:24.

3. Milea D, Najjar RP, Zhubo J, Ting D, Vasseneix C, Xu X, et al. Artificial Intelligence to Detect Papilledema from Ocular Fundus Photographs. N Engl J Med. 2020 30;382(18):1687–95.

4. Bellemo V, Lim ZW, Lim G, Nguyen QD, Xie Y, Yip MYT, et al. Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: a clinical validation study. The Lancet Digital Health. 2019 May 1;1(1):e35–44.

5. Yip M, Lim G, Lim ZW, Nguyen Q, Chong C, Yu M, Bellemo V, Xie Y, Lee XQ, Hamzah H, Ho J, Tan T, Sabanayagam C, Grzybowski A, Tan GSW, Hsu W, Lee ML, Wong TY, Ting DSW. Technical and Image-related Factors Influencing Performance of Clinical Deployment of Deep Learning Algorithms for Diabetic Retinopathy Screening. NPJ Digital Medicine. 2020. 3:40

6. Xie YC, Nguyen Q, Hamzah H, Lim G, Bellemo V, Gunasekeran D, Yip M, Lee XQ, Hsu W, Lee ML, Tan C, Wong HT, Lamoureux E, Tan GSW, Wong TY, Finkelstein, Ting DSW. Artificial Intelligence for Teleophthalmolgy-based Diabetic Retinopathy Screening in a National Program: A Modelled Economic Analysis Study. 2020. Lancet Digital Health. [Accepted]

7. Lim G, Ting DSW, Cheung CY, Tan GS, Rudyanto R, Gan ATL, et al. Deep Learning System for Screening of Diabetic Retinopathy, Glaucoma and Age-related Macular Degeneration Using Retinal Photographs: The DEEP EYE Study. Invest Ophthalmol Vis Sci. 2017 Jun 23;58(8):683–683.

8. Ting D, Cheng C-Y, Cheung CY, Lim G, Tan G, Hsu W, et al. Classic Risk Factors for Diabetic Retinopathy: Deep Learning versus Human Graders. Invest Ophthalmol Vis Sci. 2018 Jul 13;59(9):1706–1706.

9. Bellemo V, Yip MYT, Xie Y, Lee XQ, Nguyen QD, Hamzah H, et al. Artificial Intelligence Using Deep Learning in Classifying Side of the Eyes and Width of Field for Retinal Fundus Photographs. In: Carneiro G, You S, editors. Computer Vision – ACCV 2018 Workshops. Cham: Springer International Publishing; 2019. p. 309–15.

10. Li S, Liu Y, Sui X, Chen C, Tjio G, Ting D, et al. Multi-Instance Multi-Scale CNN for Medical Image Classification. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2019. Springer International Publishing; 2019. p. 531–9.

11. Tjio G, Li S, Xu X, Ting DSW, Liu Y, Goh RSM. Multi-Discriminator Generative Adversarial Networks for Improves Thin Retinal Vessel Segmentation. 2019. MICCAI. [In Press]

12. Li S, Liu Y, Su X, Chen C, Tjio G, Ting DSW, Goh RSM. Multi-Instance Multi-Scale CNN for Medical Image Classification. 2019. MICCAI. [In Press]

13. Huang CY, Kuo RJ, Li CH, Ting DSW, Kang EY, Lai CC, et al. Prediction of visual outcomes by an artificial neural network following intravitreal injection and laser therapy for retinopathy of prematurity. Br J Ophthalmol. 2019. [In Press]

14. Yu M, Rim T, Tham YC, Ting DSW, Wong TY, Cheng CY. Recommendations on Summarizing and Reporting Findings on the Performance of Deep Learning Algorithms in Health Outcomes Research. 2019. Lancet Digital Health. [In Press]

15. Liu H, Liu L, Wormstone I, Qiao C, Zhang C, Liu P, Li S, Wang H, Mou D, Pang R, Yang D, Lai J, Chen Y, Hu M, Xu Y, Kang H, Ji X, Chang R, Tham C, Cheung C, Ting DSW, Wong TY, Wang Z, Weinreb R, Xu M, Wang NL. Development and Validation of a Deep Learning System to Detect Glaucomatous Optic Neuropathy Using Fundus Photographs, JAMA Ophthalmology. 2019. doi: 10.1001/jamaophthalmol.2019.3501. [Epub ahead of print]

16. Sabanayagam C, Xu D, Ting DSW, et al. A Deep Learning Algorithm to Detect Chronic Kidney Disease from Retinal Photographs. 2020. Lancet Digital Health

Review Articles & Editorials

17. Ting DSJ, Foo V, Yang L, Sia J, Chodosh J, Ang M, Mehta M, Ting DSW. Artificial Intelligence for Anterior Segment Disease. 2020. British Journal of Ophthalmology. [In Press]

18. Lim G, Bellemo V, Xie Y, Lee XQ, Yip M, Ting DSW. Different Fundus Imaging Modalities and Technical Factors in AI Screening for Diabetic Retinopathy: A Review. 2020. Eye and Vision. [In Press]

19. Xie Y, Gunasekeran D, Balaskas K, Keane P, Sim D, Bachmann L, Macrae C, Ting DSW. Health Economic and Safety Considerations for Artificial Intelligence Applications in Diabetic Retinopathy Screening. 2020. TVST. [Accepted]

20. Abramoff M, Leng T, Ting DSW, Rhee K, Horton M, Brady C, Chiang M. Automated and Computer Assisted Detection, Detection, Classification and Diagnosis of Diabetic Retinopathy. 2020. TMJ. [In Press]

21. Yanagihara R, Lee CS, Ting DSW, Lee AY. Methodological Challenges of Deep Learning in Optical Coherence Tomography for Retinal Diseases: A Review. 2019. TVST. [In Press]

22. Ruamviboonsuk, P, Cheung C, Zhang X, Raman R, Park S, Ting DSW. Artificial Intelligence in Ophthalmology: Evolutions in Asia. 2020. Asia-Pacific Journal of Ophthalmology. [Accepted]

23. Grzybowski A, Brona P, Lim G, Ruamviboonsuk P, Tan G, Abramoff M, Ting DSW. Artificial Intelligence for Diabetic Retinopathy Screening: A Review. 2019. Eye. [In Press]

24. Bellemo V, Lim G, Rim, T, Tan G, Cheung C, Sadda S, He M, Tufail A, Lee ML, Hsu W, Ting DSW. Artificial Intelligence Screening for Diabetic Retinopathy: The Real-world Application. 2019. Current Diabetes Report. [In Press]

25. Ting DSW, Peng L, Varadarajan A, Keane P, Burlina B, Chiang M, Schmetterer L, Pasquale L, Bressler N, Abramoff M, Wong TY. Deep Learning in Ophthalmology: Clinical and Technical Considerations. Progress in Retinal and Eye Research. 2019; 72:100759. doi: 10.1016/j.preteyeres.2019.04.003

26. Ting DSW, Pasquale L, Peng L, et al. Artificial Intelligence and Deep Learning in Ophthalmology. Br J Ophthalmol. 2018. doi: 10.1136/bjophthalmol-2018-313173

27. Liu A, Farsiu S, Ting DSW. Generative Adversarial Networks (GANs) to Predict Treatment Response for Neovascular Age-related Macular Degeneration: Interesting, but is it useful? 2020. British Journal of Ophthalmology. [In Press]

28. Ting DSW, Carin L, Wong TY. Digital Technology and COVID-19. 2020. Nature Medicine. [In Press]

29. Sounderajah V, Ashrafian H, Aggarwal R, De Fauw J, Denniston AK, Greaves F, Karthikesalingam A, King D, Liu X, Markar SR, McInnes MDF, Panch T, Pearson-Stuttard J, Ting DSW, Golub R, Moher D, Bossuyt P, Darzi A. Developing Specific Reporting Guidelines for Diagnostic Accuracy Studies Asessing Artificial Intelligence Intervetions: the STARD-AI Steering Group. 2020. Nature Medicine. [In Press]

30. Joen S, Liu Y, Li JPO, Webster D, Peng L, Ting DSW. AI in Ophthalmology Made Simple. 2020. Eye. [In Press]

31. Ting DSW, Lin H, Ruamviboonsuk P, Wong TY, Sim D. Artificial Intelligence, the Internet of Things and Virtual Clinics: Ophthalmology at the Digital Translation Forefront. 2019. Lancet Digital Health. [In Press]

32. Ting DSW, Lee A, Wong TY. An Ophthalmologist’s Guide to Deciphering Studies in Artificial Intelligence. 2019. Ophthalmology. [In press]

33. Ting DSW, Dinesh VG, Louisa W, Wong TY. Next Generation Telemedicine Platforms to Screen and Triage Major Eye Diseases. 2019. British Journal of Ophthalmology. [In press]

34. Ting DSW, Carin L, Abramoff M. Observations and Lessons Learned from the Artificial Intelligence Studies for Diabetic Retinopathy Screening. JAMA Ophthalmology. 2019; 137(9):994-995. doi: 10.001/jamaophthalmo.2019.1997.

35. Ting DSJ, Ang M, Mehta J, Ting DSW. Artificial Intelligence-Assisted Tele-Medicine Platform for Cataract Screening and Management: A Potential Model of Care for Global Eye Health. 2019. British Journal of Ophthalmology.

36. Ting DSW, Tan TE and Lim TCC. Development and Validation of a Deep Learning System for Detection of Active Pulmonary Tuberculosis on Chest Radiographs: Clinical and Technical Considerations. 2018. Clinical Infectious Disease. doi: 10.1093/cid/ciy969. [Epub ahead of print]

37. Ting DSW, Wei-Chi Wu, Toth C. Deep Learning for Retinopathy of Prematurity Screening. 2018. Br J Ophthalmol. doi: 10.1136/bjophthalmol-2018-313290

38. Ting DSW, Yong L, Burlina P, Xu X, Bressler N, Wong TY. AI in Medical Imaging Goes Deep. Nature Medicine. 2018; 24: 539-540.

39. Ting DSW, Wong TY. Deep Learning Tehcnology using Retinal Images for Predicting Cardiovascular Risk Factors. Nature Biomedical Engineering. 2018; 2: 140-141.

40. Ting DSW, Yi P, Hui F. Clinical Applicability of Deep Learning System in Detecting Tuberculosis using Chest Radiography. Radiology. 2018: 286(2):729-731

41. Bakshi S, Lin S, Ting DSW, Chiang M, Chodosh J. The Era of Artificial Intelligence and Virtual Reality: Transforming Surgical Education in Ophthalmology. 2020. BJO. [In Press]