Introduction

Artificial Intelligence (AI) refers to software that imitates aspects of intelligent human behavior, including technologies such as machine learning (ML), natural language processing (NLP), and generative AI, which is focused on creating new content.^1–3 AI usage is prevalent in healthcare, where it is used to reduce costs, increase access, improve outcomes, and support innovations in disease diagnosis and treatment, personalized medicine, and predictive modeling.⁴ AI applications include detecting atrial fibrillation, predicting cardiovascular disease, and image-based diagnosis.⁵ The rapid diffusion of AI into healthcare has led to health information professionals focusing on developing AI literacy, which is the ability to understand, use, and evaluate AI systems.^6,7

Recent advancements in generative AI have transformed healthcare, particularly in electronic health records (EHRs), medical language processing, and personalized care. By using speech recognition and NLP, generative AI summarizes patient conversations for integration into EHRs, improving care accessibility and continuity.⁸ Its benefits include enhanced diagnosis and treatment accuracy, improved patient care support, increased engagement, advanced medical training, task automation, and streamlined radiology reports, which reduce human errors, redundancy, and healthcare costs, impacting the Health Information (HI) field.⁴

Clinical Decision Support Systems (CDSSs) improve outcomes by providing real-time, individualized medical advice and alerts, demonstrating early adoption of AI technologies in health information (HI). Computer Assisted Coding (CAC) uses NLP to assign billing codes from documentation, and after 20 years, more advanced models can enhance this process.⁹ CAC’s rule-based algorithms analyze medical records and suggest codes, allowing coders to focus on auditing and accuracy, thus improving efficiency and patient care by accelerating the revenue cycle and reducing the number of coding errors.⁹ AI systems that are more advanced than CAC can potentially enable autonomous clinical coding.

To address AI’s benefits in healthcare and education, HI students and professionals need basic knowledge of AI concepts.⁶ HI is an interdisciplinary field that applies computer science, information technology, healthcare research, public health, and patient care to improve outcomes.¹⁰ HI educators could incorporate AI into curricula to teach data mining for cancer detection¹¹ and predictive analytics for improving medication compliance, with advanced classes focused on creating and testing AI applications.¹²

As AI adoption in healthcare increases, concerns arise that healthcare education programs may not be aligned with job market needs, leaving HI professionals lacking the skills to support AI adoption and to assess the clinical effects of AI.¹³ Low AI literacy, which includes social and technical skills, can be a barrier to adoption and implementation of clinical AI applications.^12–14 Assessing AI literacy in HI professionals is essential for identifying knowledge gaps in AI education, which can inform curriculum design. Addressing AI challenges, especially ethical, privacy, bias, and quality issues, offers opportunities for HI professionals to reduce these challenges and improve AI systems.

Recent research has offered self-reported assessments of medical students’ and professionals’ understanding of generative AI, primarily focused on clinicians using Likert-scale surveys, leaving a gap in understanding the perspectives of HI students and professionals.^3,13,15 Our study addresses this gap by combining self-reported questionnaires with objective AI assessments to identify discrepancies between perceived and actual AI understanding across various domains, including socio-technical aspects.

As such, this study was undertaken to inform the International Federation of Health Information Management Associations (IFHIMA) about AI knowledge among HI professionals and provide data for AI training planning, with an understanding that results would also be applicable to other US health information bodies as well. For educators, we sought to identify knowledge gaps and strategies to better prepare students to align their educational programs with the job market demands, ensuring individuals are well-prepared to enter the workforce. For professionals and employers, we assessed AI comprehension to tailor professional development materials and ensure workforce readiness.

Methods

Study Design

An exploratory, descriptive, adapted online survey was conducted to measure the AI literacy of IFHIMA HI professionals. We conducted Chi-square, ANOVA, and Tukey honestly significant difference (HSD) post-hoc tests to assess the associations between the categorical response variables and the subjective survey question measures. Effect sizes were measured by Cramer’s V for Chi-square tests and Cohen’s d for some of the ANOVAs.

Survey Instrument

The validated questionnaires and assessments used in this study are adapted from two instruments, the AI Literacy Scale from Pinski and Benlian (2023)¹⁶ and the objective AI literacy scale from Weber, Pinski, and Baum (2023).¹⁷ The Pinski and Benlian AI literacy scale¹⁶ has thirteen items that measure the five dimensions of human socio-technical competencies (technology knowledge, actors in AI, steps knowledge, AI usage experience, AI design experience, and AI literacy). The objective AI literacy scale from Weber, Pinski, and Baum¹⁷ has 16 multiple-choice items with four response choices.

The online web-based questionnaire for this study had three sections. Section 1 asked demographic questions on association membership, participants’ country of residence and work country, age, education level, job title, credentials, and certifications. Section 2 collected information on self-assessed knowledge and perceptions of AI with a five-point Likert rating scale (strongly disagree, disagree, neutral, agree, strongly agree) questions presented in a rating matrix. Section 3 collected information on objective evaluations of AI for aggregate assessments with multiple choice questions with four response choices, and one open-ended question inviting comments related to AI education for HI professionals. The survey questions are presented in Tables 2-4.

Data Collection

Data were collected with convenience sampling. IRB approval was obtained on November 27, 2004 with exempt status (HSC-SBMI-24-1140 - Global AI Literacy Survey for Health Information Professionals) from the Committee for the Protection of Human Subjects of the study organization. Data were collected between December 5, 2024 and February 28, 2025, using a self-administered Qualtrics (Seattle, Washington) online survey link distributed by email to HI professionals who were members of an IFHIMA organization. The survey link was also posted on LinkedIn by multiple IFHIMA members. A total of 176 participants began the survey. Data were cleaned to exclude 48 incomplete responses where all the demographics and AI self-assessment questions were not answered. After data cleaning, 128 complete and valid responses were retained for analysis.

Data Analysis

Data were analyzed with Microsoft Excel (version 16.95.1; Redmond, Washington) and Python statistical software (version 1.96.2) for Macintosh computers.

Results

Sample descriptive statistics

Table 1 displays the respondents’ demographics. Most respondents (45; 35.2%) were aged 50 to 59 years. The respondents’ predominant country of residence was the United States (60; 46.9%), followed by Australia (15; 11.7 %). Most had education levels of ISCED 7 (Master) (53; 41.4%) or ISCED 8 (Doctorate) (37; 28.9%). Most respondents reported association membership in AHIMA (57; 44.5 %) or Other (47, 36.7%). The most-frequently reported job titles were Academics / Education (70; 54.7%), Coder (44; 34.4%), and Information Systems and Technologies (43; 33.6%). Less than half (53; 41.4%) of respondents reported holding a non-HI credential or certification.

Descriptive Analysis of Survey Questions

Table 2 displays the subjective survey question responses for Section 2 in which information on self-assessed knowledge and perceptions of AI was collected. Respondents’ experience and familiarity with AI in healthcare were evaluated with Likert-scale items rated on a five-point scale ranging from strongly disagree to strongly agree. When asked about their experience interacting with different types of AI systems, 46.1% agreed (40; 31.3%) or strongly agreed (19; 14.8%), 35.2% strongly disagreed (17; 13.3%) or disagreed (28, 21.9%), and (24;18.8%) were neutral in response to the question beginning, “I have experience in interaction with different types of AI.” Most (64.1% (82/128)) agreed (54; 42.2%) or strongly agreed (28; 21.9%) that they had frequent interactions with AI. A substantial majority (92; 71.9%) of respondents reported limited experience in AI model design (strongly disagree, 57 [44.5%]; disagree, 35 [27.3%] to the question, “I have experience in designing AI models, for example, a neural network.”). Most respondents (99; 77.3%) reported limited experience in AI product use, design, or development (strongly disagree, 60 [46.9%]; disagree, 39 [30.5%] to the question, “I have experience in the development of AI products.”). In answering the question, “In general, I know the unique facets of AI and humans and their potential roles in human-AI collaboration within healthcare,” a high proportion reported knowledge (agree, 50 [39.1%]; strongly agree, 16 [12.5%]), while 27.3% (35) indicated a lack of understanding by either disagreeing or strongly disagreeing with the statement.

Table 3 displays responses to Section 3, the general, objective survey questions regarding AI. When asked about the historical emergence of AI, fewer than half of the respondents (60; 46.9%) correctly identified the 1950s as the era when AI was first formally mentioned. A majority of respondents (93; 72.7%) correctly identified that humans and AI have distinct strengths and weaknesses. A substantial majority (97; 75.8%) accurately acknowledged that AI research is interdisciplinary and encompasses multiple technologies. When assessing awareness of AI-related risks, 61.7% (79) correctly identified “deep fakes rendering videos unattributable” as a significant concern. Other selected responses—such as digital assistants controlling self-driving cars (18; 14.1%) or AI image and voice generation affecting art and language (13 [10.2%] and 18 [14.1%], respectively)—reflect secondary or less widely accepted risks.

Understanding of supervised versus unsupervised learning varied, with only 37.5% (48) correctly identifying that supervised learning involves labeled datasets. A larger portion (53; 41.4%) chose "unsupervised learning may happen anytime. A strong majority (101; 78.9%) correctly identified that ML is a subset of AI while 13.3% (17) believed the inverse (that AI is a part of ML). The most correctly identified application where AI outperforms traditional methods was image recognition (71; 55.5%). Undefined processes (24;17.8%), annual report creation (20, 15.6%), and hardware space analysis (13; 10.2%) demonstrated a mix of respondents’ overestimation and underestimation of AI’s capabilities in real-world scenarios. In evaluating the understanding of stochasticity in AI models, 60.9% (78) correctly recognized that running the same input through an AI system could yield different results. Others held misconceptions, reporting beliefs that it never gives different results (17;13.3%), increases computing speed (29; 22.7%), or doubles computing time (4; 3.1%).

Table 4 displays the assessment of the technical understanding of AI systems and their regulatory, ethical, and computational dimensions. A set of technical objective questions was administered to 22 respondents of the 128 who stated they were involved in programming, creating, developing, or evaluating AI models. When asked to identify an objective that is not part of current AI regulation, 31.8% (7) gave the correct response that “Enforcing a no-bias policy to ban all potential biases that can arise from AI.” Some (10; 45.5%) incorrectly selected “Ensuring legal certainty to facilitate investment and innovation in AI.” Just over half of respondents (12; 54.5%) correctly identified the European Commission’s AI Act as a major regulatory initiative, while 36.4% (8) mistakenly attributed regulatory leadership to the United Nations. Most respondents (17; 77.3%) correctly identified that working with AI leads to a shift in tasks performed by humans. A high percentage of respondents (16; 72.7%) accurately identified diversity, bias, and transparency as central ethical concerns in AI development and deployment.

Responses were more varied on model interpretability. While 40.9% (9) correctly selected “Decision trees are more interpretable,” others believed they generate more predictions (5; 22.7%), are more implicit (5; 22.7%), or train faster (3; 13.6%). Equal proportions of respondents (10; 45.5%) selected “80% training, 20% testing” (correct) and a flawed option describing a “40/40/20” split, including a test-training mix. Half of the respondents (11; 50.0%) accurately identified benchmarking as not strictly necessary in every AI system development process. Other respondents incorrectly excluded essential steps such as data preprocessing (16; 72.7%) or training/learning (4; 18.2%). A clear majority (16; 72.7%) correctly identified the “User layer” as not a structural component of an ANN (Artificial Neural Network), demonstrating a solid grasp of ANN architecture. However, small proportions mistakenly excluded essential layers such as the input and hidden layers (3; 13.6% each).

Inferential Statistics

Chi-Square

Chi-square tests were used to examine associations between demographic characteristics and subjective survey responses, as well as between subjective questions and individual objective knowledge questions. Four chi-square tests showed significant demographic associations at p < 0.05. First, educational level was significantly associated with subjective AI experience, with higher educational attainment correlating to more AI tool interaction, likely due to the greater access to AI technologies or curricular exposure within higher education settings (V=0.266, p = 0.014).

Second, educational level is associated with an understanding of human–AI roles, with academic educator background influencing perceptions of AI’s healthcare collaborative function and academic educators reporting more frequent AI use (V=0.261, p = 0.02). Third, job type was significantly associated with AI product development involvement with a large effect size (V=0.792, p = 0.022). Fourth, academic/educator status was associated with frequent AI use (V=0.386, p < 0.001), and AI experience (V=0.274, p = 0.047), suggesting educators may have more exposure to AI technologies through teaching, research, or academic institutions. Post-hoc tests with Bonferroni corrections indicated no individual pairwise comparisons remained significant after correction, likely due to modest differences in proportions and limited subgroup sizes, which reduce the statistical power of Bonferroni corrections.

Additionally, chi-square tests between individual subjective survey questions and objective knowledge items showed specific patterns of association. For instance, frequent AI usage and correct identification of the historical origin of AI were strongly associated (V = 0.38, p < 0.001). Respondents who reported experience in designing AI models were significantly more likely to answer two key items correctly: the timeline of AI’s emergence (V = 0.273, p = 0.0486) and the difference between supervised and unsupervised learning (V = 0.275, p = 0.0460).

Experience in developing AI products was significantly linked to an improved understanding of supervised versus unsupervised learning paradigms (V = 0.284, p = 0.0352), reinforcing the idea that applied, hands-on development fosters comprehension of core AI concepts. When testing associations between subjective questions and technical AI knowledge the analysis revealed significant relationships between self-reported AI experience and respondents’ ability to correctly answer technical questions related to AI development and regulation. First, respondents who reported greater overall experience with AI were significantly more likely to correctly identify benchmarking as not being a key part of the AI development process. Additionally, a significant relationship was found between frequent AI usage and the European Commission’s AI Act as a major regulatory framework (V = 0.615, p = 0.0396).

Chi-Square Educators vs. Non-Educators

Table 5 presents Chi-square test results comparing educators with non-educators on general AI literacy questions, while Table 6 focuses on technical AI literacy. Given that educators represented the largest job category in our sample (19.2%) and reported significantly higher AI experience and usage, we separated the data for educators and non-educators to examine whether differences in AI literacy were statistically significant.

In the general AI literacy section, educators performed slightly better on some questions—such as human-AI relationship and AI research scope—but none of the differences reached statistical significance. All effect sizes were weak, suggesting that both groups demonstrated similar levels of general AI knowledge.

Similarly, the technical AI Chi Square tests showed no statistically significant differences and effect sizes were small, the largest being (V=0.185) for AI development process and training testing ratio (V=0.110), which may warrant further analysis. Notably, the sample size for this comparison was small (22), which limits the statistical power of the analysis.

ANOVA

ANOVA analyses found significant differences by country, professional associations, self-reported AI knowledge, and AI usage. Total objective knowledge scores, which reflect respondents’ conceptual understanding of AI, differed significantly by country for general AI knowledge (p < 0.005) and technical AI knowledge (p < 0.004). Professional association membership (p = 0.01) and educational level were significantly associated with general AI knowledge (p = 0.04). Tukey HSD post-hoc results revealed that respondents from Nigeria (p = 0.02, Cohen’s D = 0.008) and Saudi Arabia outperformed their US counterparts (p = 0.03, Cohen’s D = -0.090) with small and large effect sizes, respectively, indicating the effect sizes are negligible by conventional benchmarks.

Discussion

Previous studies on AI education were limited to clinical students and professionals and included subjective reporting.^1–3 This study assessed AI knowledge among HI professionals, offering insights to guide education, training, and professional development by identifying knowledge gaps. This knowledge is critical in addressing workforce shortages in digital health and in developing curricula to meet the needs of an AI-digitized world.¹⁸

Among our respondents, many HI professionals reported limited or no AI experience, and those having practical AI experience performed better on foundational AI knowledge questions, suggesting that experiential learning scaffolds AI literacy. Most respondents understood emerging AI-related threats; however, regardless of experiences with everyday AI tools, they struggled with AI modeling and product development. AI knowledge varied by demographics; country of employment or residence and professional association membership influenced familiarity with AI, likely due to differences in national digital health strategies, training, and access to professional resources. Job type significantly influenced the involvement in AI product development, with hands-on engagement varying across job roles such as academia, coding, and data analytics. AI experience increased with education level; educators, especially academics, reported the highest use, likely due to greater exposure in research settings. Respondents showed limited understanding of supervised and unsupervised learning, and the relationship between ML and AI’s stochastic models, but more familiarity with AI tools and human-AI interaction.

Comparison to Previous Research

Our results aligned with prior studies on AI knowledge and perceptions that found limited practical exposure to AI applications among healthcare professionals, with a majority reporting little or no exposure to AI applications in their work environment and organizational awareness of the need for employees with AI-related skills.^19–23 Our respondents struggled with AI models and the development and use of AI products due to a lack of hands-on experience with AI products and curricula development, and a low level of understanding of AI in general, which supports other studies.^24–27

In this study, responses related to the human-AI relationship varied. Over half recognized the distinct roles of humans and AI, while most identified their strengths and weaknesses and how these differences shift the task distributions. However, 27.4% lacked an understanding of this collaboration, which agrees with other research.²⁸ In contrast, a few respondents considered AI and ML to be the same, but most failed to identify the misconception that AI and ML are interchangeable terms.

In our study, most respondents correctly recognized diversity, bias, and transparency as key ethical issues in AI. In contrast, another study identified the misconception that AI is unbiased.²⁸ Similarly to the results from our study, a systematic review revealed that bias was the most frequently discussed ethical issue related to justice and fairness transparency²⁹; nurse interviews recognized the issue of transparency and privacy challenges and the risk of perpetuating biases and disparities with AI algorithms, which could result in unequal treatment or misdiagnosis.³⁰

Our study found that professional association membership correlated with higher AI knowledge, suggesting that access to resources and training increases AI literacy. This coincides with our suggestion for HI colleges and universities to provide faculty AI training and instruction on developing AI curricula, aligning with other research on integrating AI content into classroom instruction.³¹ Academic institutions should offer faculty development opportunities such as workshops and online certificate programs, and communities of practice to enhance AI literacy and pedagogical readiness.³² Effective integration of AI into the curriculum requires multidisciplinary collaboration, particularly between health informatics and computer science faculties, to co-design syllabi and develop relevant course content.³³

Limitations

This study had some limitations. First, we were unable to conduct a power analysis because there was uncertainty about the population of IFHIMA members. IFHIMA has membership numbers for individual member associations, but it does not report the total number of members. Therefore, the sample may not be a true representation of the HI population. However, we believe it is a good starting point for identifying their strengths and needs in AI. Second, a small subsample (22) involved in AI development was used to examine technical, ethical, computational, and regulatory understanding. For reasons stated above, this sample is also not a true representation of the population under study and not fully generalizable which could impact our results. However, our findings could provide a starting point for future research addressing AI knowledge gaps through targeted curriculum development and educator training. Third, the study had a high dropout rate, with 176 respondents starting the survey and only 128 completing it, resulting in 27% study attrition. This leads to non-response bias, which could impact our results and not accurately reflect on our target population. Fourth, we recognize the limitations of self-reported data and relied on respondents to answer in good faith that they were health informatics professionals. Finally, there is a potential sampling bias introduced by recruitment, which occurred through the IFHIMA membership list and members outreach through LinkedIn. This recruitment may have attracted respondents who are professionally employed, have higher digital literacy levels, and have HI expertise, which limited generalization to the HI population.

Conclusions

The survey analysis confirmed that while a considerable proportion of respondents reported some level of experience with AI systems, several reported limited or no such experience. Regardless of frequent experiences with everyday AI tools, many respondents struggled with AI models and the development and use of AI products. Lower educational levels reported lower experiences with AI, providing a group to target for additional education and training. This study identifies a major gap in AI knowledge and provides input for educators aiming to align educational programs with job market demand by increasing the AI knowledge content.

Author contributions

SF, DD, EL, VW, designed the study; SF, DD, EL, VW, MM, KL, JS researched the literature; SF, DD, EL contributed to the survey; SF, DD, EL, VW provided statistical advice; SF, EL, VW extracted and analyzed the data; SF, DD, EL, VW reviewed the analyses; SF, DD, EL contributed to the manuscript drafts and SF, DD, EL, VW, MM, KL, JS finalized the manuscript.

Disclosures

The authors have nothing to disclose.

Funding

The authors received no funding for this research.