Soon after passing my ASQ Certified Quality Auditor (CQA) exam in 2006 I received an email from my now long-time friend Jeff Plodzien inviting me to participate in the CQA exam development process. This email launched a long relationship with ASQ and their exam certification process. From Jeff recruiting me to help with exam development and culminating with my tenure as Certification Exam Chair from 2017-2018, I was very involved with developing and maintaining the Certification Exam process. Now I’d like to peel back the curtain and give you a glimpse of the process.

As a certification body, ASQ maintains the requirements set forth in ISO/IEC 17024, ensuring that all certifications are impartial and free from conflict of interest, competency-based, job relevant, and based on valid and reliable assessment methods. To do this, the ASQ Certification Council (a group of ASQ member volunteers representing each exam as well as a Vice-Chair and Chair) has the responsibility for overseeing the process, reviewing and approving new certifications (or possibly sunsetting a certification), as well as numerous issues that may arise during the certification process.

The exam development process consists of a systemic, multistep process led by volunteer Subject Matter Experts (SMEs), psychometricians and ASQ staff as follows:

1. Job Task Analysis (JTA) – every 5-7 years ASQ conducts a JTA to define the knowledge, skills and abilities required for competent job performance. This can largely be influenced by the evolution of industry, technology and new business practices. Through SME workshops, industry surveys and market research, the new or existing Body of Knowledge (BoK) is developed or refined, ensuring that the certification will reflect current, real-world practices.
2. Item Development – based on the BoK the next step is Item Development and Review. Utilizing volunteer SMEs, exam items are developed or refined based on the changes to the BoK as well as a review of recent certification exam results. Exam items that perform poorly (i.e., exam takers not choosing the key / correct answer) are flagged for review as well as newly developed exam items.
3. Exam Review and Cut Score Determination – once a pool of questions has been validated and approved, a pool of 12-16 SMEs are asked to take the exam (yes, all 100 questions) and submit their answers which will be tabulated and discussed openly as to the exam team’s performance. Poor item performance will be evaluated to understand the root cause of the issue…was the item confusing, the answers not clear, or in exam lingo, “were the distractors too good.” These items will be reworked and then retested within a new SME pool to continue refinement before approval to be used in an exam. Statistical analysis will be performed across the history of an existing exam item, or generated with new items, to ensure that the question is fair, accurate and unbiased to a minimally qualified candidate sitting for the exam.
    Additionally, the Cut-Score (minimum passing score) for each pool of questions is established during this time. Again, a new pool of SMEs are tasked with taking the exam, mimicking the requirements of sitting for the exam (no internet, only approved references, and the time limit for the exam) to appropriately simulate the real-world process. The Cut-Score is then developed from these results utilizing a modified Angoff method, which is an internationally accepted standard-setting approach. This Cut-Score is developed by psychometric experts within ASQ and reviewed by the ASQ Certification Council before approval.
4. Validation and Data Integrity – a series of psychometric techniques are then utilized to support exam fairness and statistical validity, including Field Testing (new items embedded in live exams and evaluated without affecting scores), Item Analysis (items reviewed for difficulty, discrimination and performance against expected norms), and Equating (statistical linking to ensure consistency across different versions of exams). Exam results are then reviewed by ASQ Staff and a new pool of SMEs to evaluate item performance, rework items and continuously improve the process.

As one can imagine, maintaining the 19 ASQ certification exams involves hundreds of SME volunteers annually as well as a dedicated staff within ASQ to keep the exams current, accurate and fair. If you are interested in learning more or perhaps participating in the exam development and maintenance process, reach out to ASQ via ASQ.org or email [email protected] to start your journey.