The learner-adjusted assessment tool for endotracheal intubation considering examiners’ different expectations of competence

Kang, Hyun Young; Han, Kap Su; Lee, Sung Woo; Choi, Hyuk-Joong; Lim, Tae Ho; Kim, Chan Woong; Chang, Chul Ho; Kim, Su Jin

J Korean Soc Emerg Med > Volume 30(1); 2019 > Article

Original Article

Journal of The Korean Society of Emergency Medicine 2019;30(1): 100-110.

The learner-adjusted assessment tool for endotracheal intubation considering examiners’ different expectations of competence

¹Department of Emergency Medicine, Korea University College of Medicine, Seoul, Korea
²Department of Emergency Medicine, Hanyang University College of Medicine, Seoul, Korea
³Department of Emergency Medicine, Chung-Ang University College of Medicine, Seoul, Korea
⁴Department of Anesthesiology and Pain Medicine, Yonsei University College of Medicine, Seoul, Korea

Correspondence

Su Jin Kim ,Tel: 02-920-5408, Fax: 02-920-5407, Email: icarusksj@gmail.com,

Received: October 5, 2018; Revised: December 9, 2018 Accepted: December 11, 2018. Published online: February 28, 2019.

ABSTRACT

Objective:
Endotracheal intubation (ETI) is the most reliable way to manage the airway. Stepwise deliberate practice and mastery training is essential in maintaining and promoting the skill of intubation. This study was conducted to identify differences in examiners’ expectations regarding competent skill performance and to develop learner-adjusted assessment tools with appropriate levels according to student and resident learners based on the expectations and limited observation of performance by examiners.
Method:
This was an exploratory, psychometric study using a simple airway part task trainer. The draft ETI assessment tool from the literatures, previous tools, and the preliminary learner-adjusted assessment tool for students and residents were developed and analyzed. Knowledge-based and competence-based items for each learner group were identified based on experts’ expectations. The final learner-adjusted tools were refined through analyzing the content validity, internal consistency, and interrater reliability based on assessing the observed performance of 14 students and 12 residents by ten experts.
Results:
The preliminary student-adjusted assessment tool and resident-adjusted assessment tool had 12 items on the checklist with a ternary scoring system and a ternary scoring checklist including 15 items, and an overall Global Rating Scale. The final student-adjusted assessment tool was composed of a ternary scoring checklist including 9 items (total CVI, 86.6%; Cronbach’s α, 0.83; interrater reliability, 0.64). The resident-adjusted assessment, on the other hand, was also composed of a ternary scoring checklist including 12 items (total CVI, 86.4%; Cronbach’s α, 0.7; interrater reliability, 0.78), in addition to global rating scale including ‘rating of the overall process’.
Conclusion:
The experts had different expectations regarding the level of competence in each step according to learner groups with different levels of difficulty. Understanding the factors influencing assessments can provide a guide for teaching and objectively assessing to the examiner.

Key words: Assessment; Validity; Reliability; Endotracheal intubation; Training; Psychometrics