Certification: Exam Development Committee
This policy is designed to outline the steps to prepare for the exam development committee meetings, the meeting procedures and desired work product to come out of the meetings.
The Certification Exam Development Committee Chair develops the agenda for the Item Writing Meeting. The Certification Program administrative staff advises the Chair on the agenda, prepares materials for the meeting, and follows through on any post meeting activities. The Certification Program administrative staff provides administrative support for the Certification Council and Exam Development Committee Meetings, and is responsible for making logistical arrangements for the meeting (e.g., hotel, catering, equipment, supplies, etc.) and for providing logistical and administrative support during and after the meeting.
The Exam Development Committee members, applicable staff, a high scorer and other applicable attendees are invited to the meeting to make up the Item Writing Subcommittee.
Meeting Arrangements and Materials
The Certification Program administrative staff makes hotel reservations for out-of-town participants, reserves meeting rooms, and arranges for meals and refreshments during the meeting. The Coordinator also arranges for needed equipment and supplies for the meeting, generally including:
- LED Projector and compatible computer
- Electronic access to test item bank;
- Printer and paper;
- Pads of paper, pens/pencils;
- Scissors, tape, post-it notes, paper clips, stapler, etc.
- Reference materials (copies of JPE, Giving Birth with Confidence etc.)
- Conflict of Interest and Confidentiality Agreements
- Item Construction Forms
A general information/reminder email is sent to Certification Council and Exam Development Committee members two months before the meeting. This serves to remind them of the meeting date, give them hotel information, and remind them to make travel arrangements, etc.
The Certification Program Director prepares the necessary materials so they can be ready prior to the meeting. The meeting agenda and any other relevant materials are put into a folder for each meeting attendee to be distributed at the meeting.
Prior to the start of the meeting the Certification Program administrative staff sees that the equipment is set up and in working order. Additionally, all subject matters experts, vendors and staff must return their signed confidentiality and conflict of interest agreements to the onsite staff person.
During this meeting, the psychometrician provides an orientation that highlights the types of items that the Council desires, and guidance for how to frame these items. In particular, emphasis is placed of writing items that assess application of knowledge rather than recall of knowledge. The exam blueprint or exam content outline is used to indicate where new items are most needed, and to structure several content classification codes for entry into the exam item database.
At this meeting the Committee applies a modified Angoff procedure to determine a minimally acceptable performance index for each item based on entry-level competence. Procedures for structuring these determinations is part of the psychometrician’s orientation described earlier and includes a discussion of the level of knowledge and skill represented by entry-level competence. As a pragmatic definition, entry-level is defined as representing the knowledge and ability of childbirth educators who are minimally qualified to sit for the examination, who would exhibit marginally effective performance as practitioners, and who would be expected to attain barely passing scores on the examination. These practitioners would be expected to increase their level of knowledge, ability, and effectiveness as they accrue additional professional experience.
The items are written by individual subject matter expert (SMEs) and then given to the psychometrician (on site at meeting) or approved staff person to review and input as a draft into the item bank. All items are then projected for the full item writing subcommittee to review, critique and revise as needed. If approved, an Angoff index that corresponds to the subcommittee’s perceived level of difficulty is applied and the item is made active in the item bank and organized by competency. All item writing members are required to sign a confidentiality agreement and all "scratch" paper is kept in the meeting room throughout the onsite meeting and shredded at the close of the meeting.
The Angoff technique is applied by the Examination Committee in assessing the difficulty of each item. The Committee’s determination is made by consensus when the item is approved and becomes active in the item bank. In order to guide the Committee members in their Angoff assessments, the psychometrician provides guidance (Table 2, cut score report page 3) that structures the assessments as a step-wise decision involving two related judgments. The first judgment is whether the item is easy, difficulty, or moderate. The second judgment is the degree of easiness or difficulty. For example, if an item is judged by Committee consensus to be easy, the Committee then determines whether the Angoff index should be 80 (easy) or 90 (very easy). If an item is judged by the Committee to be difficult, the Committee then determines whether the Angoff index should be 30, 40, or 50 based on the consensual judgment of how difficult. As in other applications of the modified Angoff technique, the Committee judgments are shown in 10-point increments. The purpose of these increments is to avoid the arbitrariness that can result from attempting to render finer distinctions. The Angoff scale also limits the highest and lowest indices to avoid expectations of perfection (i.e., 100%) in the upper end of the scale, and to avoid expectations of chance (i.e., <30%) in the lower end of the scale, as chance is the equivalent of guessing that results from candidates having little or no knowledge of an item’s content. As noted earlier, the Committee members are instructed to focus their Angoff ratings on entry-level knowledge and ability.
At the meeting, the administrative staff is charged with ensuring proper handling and storage of secure item and test information. Staff is responsible for collecting and filing all conflict of interest and confidentiality agreements. Additionally, staff is responsible for taking minutes/ detailed notes of the action items accomplished during the meeting. A record should be kept of the number of items that were revised and developed per competency.
Following the meeting the Certification Program Coordinator returns all equipment, routes meeting minutes to chair for comment, shreds printed copies of test items, scratch paper and sends an email of thanks from the Chair with approved minutes to the attendees.
Approved: January 1994
Updated: June 2002, June 2017