Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

Page 1

Enhancing Antibiotic Stewardship using a Natural Language Approach for

Better Feature Representation

Simon A. Lee 1 Trevor Brokowski 2 3 Jeffrey N. Chiang 1 4

Abstract

The rapid emergence of antibiotic-resistant bac-

teria is recognized as a global healthcare crisis,

undermining the efficacy of life-saving antibiotics.

This crisis is driven by the improper and overuse

of antibiotics, which escalates bacterial resistance.

In response, this study explores the use of clinical

decision support systems, enhanced through the

integration of electronic health records (EHRs),

to improve antibiotic stewardship. However, EHR

systems present numerous data-level challenges,

complicating the effective synthesis and utiliza-

tion of data. In this work, we transform EHR data

into a serialized textual representation and employ

pretrained foundation models to demonstrate how

this enhanced feature representation can aid in

antibiotic susceptibility predictions. Our results

suggest that this text representation, combined

with foundation models, provides a valuable tool

to increase interpretability and support antibiotic

stewardship efforts.

1. Introduction

The Centers for Disease Control and Prevention (CDC) has

declared the rapid emergence of resistant bacteria a global

healthcare crisis, threatening the efficacy of antibiotics that

have saved millions of lives (Ventola, 2015; Golkar et al.,

2014; Gould & Bal, 2013; Sengupta et al., 2013; Nature,

2013; Lushniak, 2014). This crisis is primarily driven by the

mishandling and overuse of these antibiotics, which leads to

bacteria developing resistance through repetitive exposure

(Viswanathan, 2014; Read & Woods, 2014). These resistant

bacteria impose significant clinical and financial burdens on

*Equal contribution 1Department of Computational Medicine,

UCLA 2Yale School of Medicine 3Biomedical Informatics & Data

Science, Yale University 4Department of Neurosurgery, UCLA.

Correspondence to: Simon A. Lee <simonlee711@g.ucla.edu>,

Jeffrey Chiang <njchiang@g.ucla.edu>.

Proceedings of the 41st International Conference on Machine

the author(s).

healthcare systems, as well as on patients and their families

worldwide (Bartlett et al., 2013).

Clinical decision support systems hold substantial poten-

tial to assist healthcare providers in adhering to antibiotic

stewardship practices. This potential is largely facilitated by

electronic health record (EHR) software, which allows for

the seamless integration of patient health histories in digital

form (Evans, 2016; Cowie et al., 2017; Hoerbst & Ammen-

werth, 2010). The integration with EHRs enables the use

of continuously updated and deployed machine learning

models for clinical decision-making. However, EHR data

in its raw form presents numerous challenges, such as data

ingestion and feature representation (Wu et al., 2010).

In this work, we present a methodology that converts EHR

data into a serialized text form called pseudo-notes to predict

antibiotic susceptibility. This conversion from tabular to text

format facilitates the creation of interpretable data inputs for

pretrained foundation models, known for their rich feature

representation. Our primary objective is to develop a predic-

tive model that incorporates this representation strategy in

conjunction with foundation models. This approach aims to

enhance decision support systems and accurately identify

the most suitable antibiotics for patients, thereby offering a

data-driven solution to combat antibiotic resistance.

2. Related Works

Medical Representation Learning Medical representa-

tion learning on Electronic Health Records (EHRs) has

emerged as a critical area in healthcare research, focusing

on transforming complex medical data for enhanced clinical

decision-making. Initially, this involved extensive feature

engineering to convert raw EHR data into formats suitable

for traditional machine learning models (Tang et al., 2020;

Ferrao et al., 2016). However, this approach can be labor-

intensive and varies significantly across research groups due

to the lack of a standard protocol.

Recently, the focus has shifted to advanced foundation mod-

els that learn to represent medical data by analyzing exten-

sive text corpora, including clinical notes, medical literature,

and records. These models, primarily based on the BERT

architecture, generate rich, contextual latent representations

arXiv:2405.20419v1 [cs.LG] 30 May 2024

Page 2

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

of patient histories, significantly reducing the manual effort

in feature engineering (Rasmy et al., 2021; Liu et al., 2021;

Alsentzer et al., 2019; Lee et al., 2020). Moreover, new

techniques have been developed to incorporate the longitu-

dinal nature of EHR data, leveraging a patients progression

to predict outcomes (Steinberg et al., 2023; Wornow et al.,

2024; Pang et al., 2021; Li et al., 2022).

Clinical Decision Support Machine learning enhances

clinical decision support systems by analyzing vast datasets

to provide evidence-based recommendations that improve

patient care outcomes (Sutton et al., 2020). Since the acces-

sibility to EHR systems, there have been numerous such use

cases in predicting diseases (Liu et al., 2018; Cheng et al.,

2016), and various other patient outcomes (Lee et al., 2024;

Suter et al., 1994; Churpek et al., 2014) across all institu-

tions within the healthcare system. Furthermore, the inte-

gration of predictive models into clinical workflows enables

researchers to preemptively manage chronic conditions and

mitigate potential health crises before they escalate (Li et al.,

2020; Goldstein et al., 2017; Hohman et al., 2023). By con-

tinuously learning from new data, these systems evolve to

provide more accurate assessments and recommendations,

which support ongoing improvements in medical practices

and patient management strategies. However, much work

remains to be done on addressing the generalization and bi-

ases prevalent in many of these predictive algorithms (Goetz

et al., 2024; Agniel et al., 2018).

3. Methods

EHR Electronic Health Records are digital versions of a

patient’s medical history, maintained over time by health-

care providers. These records are valuable but consist of

heterogeneous tabular datasets organized into separate ta-

bles such as diagnostics, demographics, and medication,

presenting numerous challenges for researchers.

The primary challenge with EHRs is the heterogeneous na-

ture of the data, which includes numerical, categorical, and

free-text formats that are difficult to integrate and convert

into machine-readable formats. Furthermore, categorical

data fields often contain a large number of classes, poten-

tially distorting their original representation. For instance,

employing feature engineering techniques such as dummy

coding for categorical variables can introduce collinearity,

increase dimensionality, and result in sparse data represen-

tations. These modifications can complicate simple table

readouts and require more memory capacity for statistical

models to function effectively.

Pseudo-notes: Clinical Notes Generation from Tabular

Data Recent research has introduced a methodology for

serializing tabular data into text using text templates (Hegsel-

mann et al., 2023). This approach significantly enhances our

work by enabling uniform representation of all data in the

EHR as human-readable and interpretable text, rather than

as a collection of merged tables. Moreover, it creates an

interface to use foundation models pre-trained on large text

corpora, facilitating rich feature representation. In our work,

we use a mapping function f : T → S that turns individual

tables into serialized text, where T stands for individual

tables and S for serialized text. For each patient, we convert

each of their N tables—which cover different aspects like

diagnostics, medications, and vitals—into text segments.

These segments are then joined to create a single, detailed

paragraph per patient. This method combines all pertinent

patient information from various sources into one unified

narrative, effectively transforming the data structure into

S = J

i=1 f(Ti), where Ti is the ith table row concerning

the patient.

Data Source and Inclusion Criteria We sourced data

from the Medical Information Mart for Intensive Care IV

(MIMIC-IV) and MIMIC-IV Emergency Department (ED)

databases (Johnson et al., 2020; 2023). This study focused

on ED patients presumed to have staph infections, selected

based on specific inclusion criteria. Eligible participants

included those with any microbiological culture testing posi-

tive for a staph-related organism, sourced from bodily fluids

such as blood, urine, cerebral spinal fluid, pleural cavity, or

joint fluid, accompanied by a prescribed antibiotic whose

susceptibility was subsequently tested (Tong et al., 2015;

Kwiecinski & Horswill, 2020). From these criteria, we

identified 5976 unique prescriptions in our database. Addi-

tionally, patients with multiple ED admissions that met the

criteria were analyzed separately but were grouped within

the same train/test divisions to prevent test set contamina-

tion. This cohort included 10 unique antibiotics, whose

prevalences are shown in Table 1. A demographic overview

of our cohort is presented in Appendix Section B.

Table 1. Antibiotic Prevalence in MIMIC IV Cohort

ANTIBIOTIC

TRAIN

TEST

TOTAL PREVALENCE (%)

CLINDAMYCIN

2645

624

54.69%

DAPTOMYCIN

1815

425

37.51%

ERYTHROMYCIN

2626

639

54.59%

GENTAMICIN

4549

1127

94.89%

LEVOFLOXACIN

2866

715

60.00%

OXACILLIN

2702

667

56.32%

RIFAMPIN

1929

459

39.96%

TETRACYCLINE

3747

909

76.57%

TRIMETHOPRIM/SUL

3671

908

71.66%

VANCOMYCIN

2529

611

52.53%

To motivate our experimental setup, we examine the infor-

mation available about a patient at their time of arrival in the

emergency department. To predict antibiotic use, we utilize

Page 3

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

Figure 1. Area under the Receiver Operating Characteristic (AUROC) curves for each antibiotic classification shows that, despite the

nuances, BioMegatron performs the best.

six clinical modalities from the MIMIC ED Database. These

EHR modalities include arrival and triage information, med-

ication reconciliation (medrecon), diagnostic codes (ICD-

9/10), vital signs, and Pyxis data. All these data points are

linked to antibiotic labels from the MIMIC database using

a patient ID, visit, and Hospital Admission ID (Hadm id),

allowing us to accurately identify the patients and their tests

in which certain antibiotics were effective.

Experiments In this work, we benchmark different rep-

resentation strategies of EHR to identify the most effective

method for predicting antibiotic susceptibility. We approach

this problem as a multilabel binary classification, where

we train the same base model (Light Gradient Boosted Ma-

chines) using various representation startegies of the input.

These representations include: raw tabular data; EHR-shot

(Wornow et al., 2024), a foundation model for tabular EHR;

and three text-based representations: word2vec, a generic

language model, and a medical language model, BioMega-

tron (Shin et al., 2020). Additionally, we conduct a clus-

tering of our pseudonotes using the BERTopic algorithm

(Grootendorst, 2022) to determine if these embeddings can

naturally cluster patients. Identifying these clusters can pro-

vide insights into their potential performance in settings like

zero-shot learning and provide insights into the decision

making process.

4. Results

Antibiotic Susceptibility Prediction In our analysis of

antibiotic prediction, we measure the Area Under the Re-

ceiver Operating Characteristic curve (AUROC) and Area

Under the Precision-Recall Curve (AUPRC). Additionally,

we bootstrap 1,000 times to generate 95% confidence in-

tervals. Our AUROC and AUPRC results are displayed

in Figures 1 and 2. We also measure additional F1 scores

and Matthews correlation coefficients, with a whole table

readout which are included in the appendix.

Clustering Experiment In our clustering experiments,

we aim to identify clusters using the BERTopic algorithm.

By identifying clusters based on embeddings, we believe

this approach can form the basis for zero-shot applications

across various clinical tasks. Additionally, finding similar

embeddings could provide insights into decision-making

processes in these black-box models. We showcase the

similarity matrix of our patient clusters in Figure 3.

5. Discussion

Clinical Notes with Foundation Models Provide the best

representation and interpretability From Figures 1 and

2, we observe that the foundation models operating on our

pseudo-notes method provide the best overall performance

across most of the antibiotics. While a generic founda-

tion model and EHR-shot excel with some antibiotics, the

clinical foundation model consistently shows superior per-

formance across both AUROC and AUPRC metrics.

Beyond enhanced predictive abilities, an advantage of our

pseudo-notes method over EHR foundation models and

tabular representations is its interpretability. Compared to

the specific structuring required by EHR-shot, our method

offers a simpler and more effective interface to understand

the data that is being modeled which can improve the trust

between AI and healthcare professionals.

Interoperability Another advantage of pseudo-notes over

EHR foundation models is their interoperability with pro-

prietary healthcare systems. Our method offers a straight-

Page 4

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

Figure 2. Area under the Precision Recall Curve (AUPRC) curves for each antibiotic classification shows that, despite the nuances,

BioMegatron performs the best.

Figure 3. Identifying clusters from our patient embeddings is indi-

cated by the squares forming along the diagonal of our similarity

matrix.

forward interface for converting any EHR tabular data from

tables to text. Our conversion also facilitates the use of

off-the-shelf open-source foundation models. As improved

models are continually developed, this data format provides

an easy interface to adapt and swap the backbone for im-

proved representation of our clinical text. Additionally,

EHR foundation models do not operate on non-OMOP vo-

cabularies, which limits its effectiveness on datasets like

MIMIC-IV that utilize these specialized vocabularies.

Patient Similarity One final advantage of our pseudo-

notes is illustrated in Figure 3, which demonstrates the

capability to perform a similarity search on our patient em-

beddings. From this analysis, we identified clusters related

to sepsis, diabetes, stomach acid issues, anxiety, painkillers,

respiratory conditions, and antidepressants. This opens up

potential use cases for this data representation strategy to

be used in zero-shot learning studies and offers insights

into decision-making processes based on the embeddings.

Further research is needed to explore both of these areas.

6. Conclusion

In this work, we introduced a methodology called pseudo-

notes, which converts EHR tabular data into text to achieve

an optimal representation strategy. We discovered that

pseudo-notes outperformed various representation strategies

and remains a highly flexible framework, compatible with

the ongoing development of foundation model backbones,

which could further enhance its performance. Additionally,

we found that pseudo-notes can identify patient clusters

within the EHR, opening up promising avenues for future

studies in zero-shot learning and model interpretation.

From an application perspective, we demonstrated how a

straightforward data transformation has emerged as an easy

interface for making EHR data synergized before integrat-

ing it into machine learning models. We think that this

strategy could be a better way to work with EHR data for

future research and help build trust due to its interpretabil-

ity. Particularly in this study, we illustrated its potential by

identifying suitable antibiotics for patients arriving at the

ED, where timely and accurate decisions are critical. We as

Page 5

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

a group have highlighted the importance of improving an-

tibiotic stewardship and showcase the impact of data-driven

strategies in addressing pressing healthcare challenges.

Impact Statement

The goal of this work is to advance the field of Machine

Learning in Healthcare, and thus presents a novel potential

interface between modern NLP (Foundation models) and

clinical data.

References

Agniel, D., Kohane, I. S., and Weber, G. M. Biases in

electronic health record data due to processes within the

healthcare system: retrospective observational study. Bmj,

361, 2018.

Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D.,

Naumann, T., and McDermott, M. Publicly available clin-

ical bert embeddings. arXiv preprint arXiv:1904.03323,

2019.

Bartlett, J. G., Gilbert, D. N., and Spellberg, B. Seven ways

to preserve the miracle of antibiotics. Clinical infectious

diseases, 56(10):1445–1450, 2013.

Cheng, Y., Wang, F., Zhang, P., and Hu, J. Risk prediction

with electronic health records: A deep learning approach.

In Proceedings of the 2016 SIAM international confer-

ence on data mining, pp. 432–440. SIAM, 2016.

Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R.,

and Edelson, D. P. Using electronic health record data

to develop and validate a prediction model for adverse

outcomes in the wards. Critical care medicine, 42(4):

841–848, 2014.

Cowie, M. R., Blomster, J. I., Curtis, L. H., Duclaux, S.,

Ford, I., Fritz, F., Goldman, S., Janmohamed, S., Kreuzer,

J., Leenay, M., et al. Electronic health records to facilitate

clinical research. Clinical Research in Cardiology, 106:

1–9, 2017.

Evans, R. S. Electronic health records: then, now, and in

the future. Yearbook of medical informatics, 25(S 01):

S48–S61, 2016.

Ferrao, J. C., Oliveira, M. D., Janela, F., and Martins, H. M.

Preprocessing structured clinical data for predictive mod-

eling and decision support. Applied clinical informatics,

7(04):1135–1153, 2016.

Goetz, L., Seedat, N., Vandersluis, R., and van der Schaar,

M. Generalization—a key challenge for responsible ai in

patient-facing clinical applications. npj Digital Medicine,

7(1):1–4, 2024.

Goldstein, B. A., Navar, A. M., Pencina, M. J., and Ioan-

nidis, J. P. Opportunities and challenges in developing

risk prediction models with electronic health records data:

a systematic review. Journal of the American Medical

Informatics Association: JAMIA, 24(1):198, 2017.

Golkar, Z., Bagasra, O., and Pace, D. G. Bacteriophage

therapy: a potential solution for the antibiotic resistance

crisis. The Journal of Infection in Developing Countries,

8(02):129–136, 2014.

Gould, I. M. and Bal, A. M. New antibiotic agents in

the pipeline and how they can help overcome microbial

resistance. Virulence, 4(2):185–191, 2013.

Grootendorst, M. Bertopic: Neural topic modeling

with a class-based tf-idf procedure. arXiv preprint

arXiv:2203.05794, 2022.

Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang,

X., and Sontag, D. Tabllm: Few-shot classification of

tabular data with large language models. In International

Conference on Artificial Intelligence and Statistics, pp.

5549–5581. PMLR, 2023.

Hoerbst, A. and Ammenwerth, E. Electronic health records.

Methods of information in medicine, 49(04):320–336,

2010.

Hohman, K. H., Martinez, A. K., Klompas, M., Kraus,

E. M., Li, W., Carton, T. W., Cocoros, N. M., Jack-

son, S. L., Karras, B. T., Wiltz, J. L., et al. Leveraging

electronic health record data for timely chronic disease

surveillance: the multi-state ehr-based network for dis-

ease surveillance. Journal of Public Health Management

and Practice, 29(2):162–173, 2023.

Johnson, A., Bulgarelli, L., Pollard, T., Horng, S.,

Celi, L. A., and Mark, R.

Mimic-iv.

Phy-

sioNet. Available online at:

https://physionet.

org/content/mimiciv/1.0/(accessed August 23, 2021), pp.

49–55, 2020.

Johnson, A. E., Bulgarelli, L., Shen, L., Gayles, A., Sham-

mout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B.,

Gow, B., et al. Mimic-iv, a freely accessible electronic

health record dataset. Scientific data, 10(1):1, 2023.

Kwiecinski, J. M. and Horswill, A. R. Staphylococcus au-

reus bloodstream infections: pathogenesis and regulatory

mechanisms. Current opinion in microbiology, 53:51–60,

2020.

Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H.,

and Kang, J. Biobert: a pre-trained biomedical language

representation model for biomedical text mining. Bioin-

formatics, 36(4):1234–1240, 2020.

Page 6

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

Lee, S. A., Jain, S., Chen, A., Ono, K., Fang, J., Rudas,

A., and Chiang, J. N. Emergency department decision

support using clinical pseudo-notes, 2024.

Li, R., Chen, Y., Ritchie, M. D., and Moore, J. H. Electronic

health records and polygenic risk scores for predicting

disease risk. Nature Reviews Genetics, 21(8):493–502,

2020.

Li, Y., Mamouei, M., Salimi-Khorshidi, G., Rao, S., Has-

saine, A., Canoy, D., Lukasiewicz, T., and Rahimi, K. Hi-

behrt: hierarchical transformer-based model for accurate

prediction of clinical events using multimodal longitudi-

nal electronic health records. IEEE journal of biomedical

and health informatics, 27(2):1106–1117, 2022.

Liu, J., Zhang, Z., and Razavian, N. Deep ehr: Chronic dis-

ease prediction using medical notes. In Machine Learning

for Healthcare Conference, pp. 440–464. PMLR, 2018.

Liu, N., Hu, Q., Xu, H., Xu, X., and Chen, M. Med-bert: A

pretraining framework for medical records named entity

recognition. IEEE Transactions on Industrial Informatics,

18(8):5600–5608, 2021.

Lushniak, B. D. Antibiotic resistance: a public health crisis.

Public Health Reports, 129(4):314–316, 2014.

Nature, E. The antibiotic alarm. Nature, 495(7440):141,

2013.

Pang, C., Jiang, X., Kalluri, K. S., Spotnitz, M., Chen, R.,

Perotte, A., and Natarajan, K. Cehr-bert: Incorporating

temporal information from structured ehr data to improve

prediction tasks. In Machine Learning for Health, pp.

239–260. PMLR, 2021.

Rasmy, L., Xiang, Y., Xie, Z., Tao, C., and Zhi, D. Med-

bert: pretrained contextualized embeddings on large-scale

structured electronic health records for disease prediction.

NPJ digital medicine, 4(1):86, 2021.

Read, A. F. and Woods, R. J. Antibiotic resistance manage-

ment. Evolution, medicine, and public health, 2014(1):

147, 2014.

Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert,

a distilled version of bert: smaller, faster, cheaper and

lighter. arXiv preprint arXiv:1910.01108, 2019.

Sengupta, S., Chattopadhyay, M. K., and Grossart, H.-P. The

multifaceted roles of antibiotics and antibiotic resistance

in nature. Frontiers in microbiology, 4:47, 2013.

Shin, H.-C., Zhang, Y., Bakhturina, E., Puri, R., Patwary,

M., Shoeybi, M., and Mani, R. Biomegatron: Larger

biomedical domain language model, 2020.

Steinberg, E., Fries, J., Xu, Y., and Shah, N. Motor: A

time-to-event foundation model for structured medical

records. arXiv preprint arXiv:2301.03150, 2023.

Suter, P., Armaganidis, A., Beaufils, F., Bonfill, X., Bur-

chardi, H., Cook, D., Fagot-Largeault, A., Thijs, L.,

Vesconi, S., Williams, A., et al. Predicting outcome

in icu patients. Intensive Care Medicine, 20:390–397,

1994.

Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski,

D. C., Fedorak, R. N., and Kroeker, K. I. An overview

of clinical decision support systems: benefits, risks, and

strategies for success. NPJ digital medicine, 3(1):17,

2020.

Tang, S., Davarmanesh, P., Song, Y., Koutra, D., Sjoding,

M. W., and Wiens, J. Democratizing ehr analyses with

fiddle: a flexible data-driven preprocessing pipeline for

structured clinical data. Journal of the American Medical

Informatics Association, 27(12):1921–1934, 2020.

Tong, S. Y., Davis, J. S., Eichenberger, E., Holland, T. L.,

and Fowler Jr, V. G. Staphylococcus aureus infections:

epidemiology, pathophysiology, clinical manifestations,

and management. Clinical microbiology reviews, 28(3):

603–661, 2015.

Ventola, C. L. The antibiotic resistance crisis: part 1: causes

and threats. Pharmacy and therapeutics, 40(4):277, 2015.

Viswanathan, V. Off-label abuse of antibiotics by bacteria.

Gut microbes, 5(1):3–4, 2014.

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C.,

Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,

et al. Huggingface’s transformers: State-of-the-art natural

language processing. arXiv preprint arXiv:1910.03771,

2019.

Wornow, M., Thapa, R., Steinberg, E., Fries, J., and Shah,

N. Ehrshot: An ehr benchmark for few-shot evaluation

of foundation models. Advances in Neural Information

Processing Systems, 36, 2024.

Wu, J., Roy, J., and Stewart, W. F. Prediction modeling

using ehr data: challenges, strategies, and a comparison

of machine learning approaches. Medical care, 48(6):

S106–S113, 2010.

Page 7

Enhancing Antibiotic Stewardship using a Natural Language Approach for Better Feature Representation

A. Appendix

A.1. Additional Commentary

Limitations Some limitations of this work include the variability in patients’ histories and the 512 sequence length

limitation imposed by the DistilBERT (Sanh et al., 2019) and BioMegatron models (Shin et al., 2020). Consequently,

portions of a patient’s medical history may be truncated depending on the length of that history. Tokenization strategies (e.g.,

sub-word tokenization) can significantly influence how we handle the analysis.

Future Work Future work in our group, from a methodological perspective, aims to explore how these notes can enhance

studies in model interpretability and zero-shot or few-shot frameworks. From an application standpoint, we are interested in

applying this methodology across various departments and applications. We plan to collaborate with clinicians throughout

our institution to determine the types of clinical decision support models that are most needed and to assess how AI can

benefit these healthcare facilities.

Additionally, future work will include benchmarking the plethora of foundation models available on the Huggingface

Platform (Wolf et al., 2019). This will help us identify the best foundation model for specific tasks and determine whether

these embeddings are task-agnostic.

B. Dataset Characteristics

B.1. Patient Demographics

Table 2. MIMIC IV Cohort Data Overview

DESCRIPTION