The National Association of Flight Instructors (NAFI) took part in a meeting in Oklahoma City April 20 with a number of aviation industry organizations and training-materials providers about the process of airman knowledge test development and administration. This meeting, while a regular, yearly event, focused on recent concerns the industry has expressed about changes to knowledge test banks for pilot candidates.
“While NAFI recognizes the FAA’s goal is to have a large number of potential test questions, the focus really needs to be on quality and relevance of questions,” said NAFI Executive Director Jason Blair. “The flight-training industry agrees that the goal of industry shouldn’t be to have the test bank made public again, and attendees likewise agreed that studying for rote memorization process isn’t the best method of knowledge transfer for student pilots. Instead, the issue at hand is ensuring that the knowledge tests are comprised of material referenced by FAA documents, including handbooks, advisory circulars, and other resources, that flight-training providers can direct their students to study.”
The Fundamentals of Instruction knowledge test has seen the greatest number of changes in content and resulting decrease in performance by new applicants, Blair said. This is likely due to the agency’s efforts to bring the test questions current with the 2008 version of the Aviation Instructors Handbook. The change was delayed until February 14, 2011, when a backlog of questions based on the updated handbook was integrated into the test bank. The FAA characterized it as “primarily a terminology” update to reference the new handbook, Blair said. “There has been some questions of the validity of this test, but we believe this statement to be correct: that the test questions that were implemented are based on this updated handbook,” he said.
During the meeting, the FAA’s AFS-630 staff detailed the process used to create and validate knowledge tests, including putting new proposed test questions into circulation for at least 120 days, during which time the questions do not count in an applicant’s score. That allows FAA staff to review these questions and determine if they are valid.
“One concern is that, even when a question is performing poorly, the same person who wrote the question determines if it should remain,” Blair said. “This is a potential area for improvement, and alternative review processes may be more appropriate than a judgment call by whoever wrote the question in the first place.”
In addition to that review for new questions, all test questions are reviewed for continued applicability and quality at least every four years by airman testing staff.
The FAA’s goal is to develop a process where it first develops handbooks, then practical test standard (PTS) documents, and finally knowledge tests referenced to both, so that it can to create a logical and robust training and testing process that creates safe pilots, Blair said. It has compiled between 11,000 and 12,000 questions in its knowledge test banks for all airman knowledge tests, and the agency’s long-standing goal is to build the test knowledge bank to more than 100,000 questions.
FAA staff and industry stakeholders agreed that the testing process has challenges, and all parties understand that for the testing process to improve, more modern testing technology may be needed, Blair said. The FAA indicated it is moving toward modular knowledge testing. “Each section of a test, such as weather, aerodynamics, navigation, or airspace, would need to be passed by an applicant in order for him to pass the overall test,” he said. “Currently, if an applicant got a 72% on the overall test, but missed all of the questions in a particular section, he would still pass the test. In a modular test environment, an applicant who scored 90% overall, but only scored 68% on one particular section, would not pass the overall test; instead, he would be required to retest on the sections he missed.”
This is something that NAFI will follow closely in the future since it represents a significant change in the testing process, Blair said. Likewise, forthcoming updates to PTS documents, including the private pilot, commercial pilot, and certificated flight instructor, and updates in the Weight and Balance handbook and the two helicopter-related handbooks should be of interest to the flight-instructor community, as well. NAFI will watch for these changes inform flight-training providers when they are completed, he added.
“The FAA staff — both from Oklahoma City and from Washington, DC headquarters — recognize the questions that have been raised, and they indicate their desire to work with the industry to make sure that tests continue to develop an improve the testing process and content,” Blair said. “We’re hopeful that the outcomes of working together will continue to allow applicants to be successful in their pilot certification process.”
NAFI continues to recommend that flight-training providers focus on knowledge-based training instead of concentrating on rote memorization, Blair said. The practical application of skills developed from a knowledge-based approach to training is the best method for applicants to be successful in their airman knowledge-test experiences.
“We’ll continue to work collaboratively with FAA in the evolution of knowledge tests,” Blair said. “We also encourage members, flight-training professionals, and those pursuing training to continue to provide their feedback to the FAA based on their experiences with knowledge testing.”