Associate Vice President, Psychometrics & Assessments

Job Description

Job Description

This is a hybrid position requiring 2 days per (Tuesday and Wednesday) in-person each week. Office is located in downtown Chicago, IL in the Streeterville/Mag Mile area.

JOB SUMMARY

Reporting to the VP, Certifying Board Services, the AVP of Psychometrics and Assessments is the organization’s lead Psychometric member that ensures all assessment practices meet the highest standard for rigor, fairness, legal defensibility, and compliance with all licensure, accreditation standards and regulatory requirements.

The AVP is responsible for the overall direction of Certifying Board Services' ("CBS") exam development and psychometrics activities and will partner with CBS staff and all AOA certifying boards to deliver valid, reliable, and legally defensible AOA osteopathic board certification exams and assessments for both initial board certification and recertification purposes.

The role may occasionally require evening and weekend work to accommodate the schedules of our physician board members and volunteers, as well as to facilitate meetings (both in-person and virtual) addressing essential exam development activities.

KEY RESPONSIBLITIES

• Lead the design, analysis, deployment, and validation of all certification assessments (e.g., written, oral, clinical, and other performance-based assessments) for Osteopathic Physicians across multiple specialties and sub-specialties., including all research, test development, publication, psychometric, and operational data analysis functions within CBS.

• Oversee all psychometric functions including test development, item analyses, equating, scoring, and standard setting, ensuring alignment with AERA/APA/NCME, NCCA, and ANSI/ISO 17024 guidelines and standards.

• Coordinate closely with both the AVP, Operations, and the AVP, Policy, to develop processes and procedures, oversee execution of key strategic initiatives, such as the launch of innovative testing processes and/or systems, and coordinate staff functions.

• Oversee and lead continuous quality improvement activities that optimize structure, staffing and workflows to ensure sustainable efficiencies and facilitate innovation adoption for current and future assessments.

• Understand, navigate, and communicate diplomatically within a multi-faceted environment while executing a complex system of inter-related activities.

• Interact seamlessly with AOA Physician and Executive Leadership, BOS Chair and board members, Senior Leadership Team, CBS, and AOA colleagues on all levels within the association, as well as all affiliates and other external partners or stakeholders when appropriate with advocating for and promulgating Psychometric best practices and AOA/CBS/BOS innovations, policy changes, and revisions/updates to any/all exam development/administration/scoring changes as they occur.

• Handle highly sensitive situations and confidential material with the utmost discretion.

• Evaluate a team of Psychometricians, test developers, test publishers, data analysts, and other assessment professionals by identifying internal high potential individuals and providing them with effective development experiences and assignments across operational, developmental, and research initiatives.

ESSENTIAL FUNCTIONS

Psychometrics

• Oversee the development and execution of exam blueprints (e.g., table of specification) development, standard setting studies, operational scoring and score reporting, technical reporting, data analysis and survey studies, and other activities in support of CBS assessment programs and products.

Test Development and Publication

• Oversee the development, production and processing of secure examinations and examination-related material within the department including item writer training, item development and revision, item banking processes, exam publication approvals, and associated vendor relationships.

Research

• Develop, deploy, and manage a comprehensive and sustainable CBS research agenda which positively and substantively improves the AOA board certification value proposition over time.
• Develop and/or oversee reliability and validity studies and other psychometric activities to support all AOA certifying boards.
• Manage the certification testing research presence of the AOA, identify opportunities and assemble and submit proposals and manuscripts for presentations at testing and measurement conferences or appropriate testing and medical education journals.

Security and Legal Defensibility

• Ensure compliance with all best practices and third-party accreditation standards with respect to exam security.
• Optimize and maintain legal defensibility of exam development, administration, scoring, and maintenance activities, as well as candidate and diplomate rights through complying with the American with Disabilities Act (ADA) and Equal Employment Opportunity Commission (EEOC), and other legal and regulatory requirements, as appropriate.

MINIMUM QUALIFICATIONS OR EQUIVALENTS

Education:

• Terminal degree required (Ph.D., ED.D or Psy.D) in educational measurement, psychological measurement, evaluation, assessment, organizational/industrial psychology, or a related field with an emphasis on psychometrics and/or data analytics.

Experience:

• A minimum of ten (10) years of experience in psychometrics, measurement, analytics, high-stakes healthcare certifications (preferably within a medical board or licensure context), or high-stakes testing required.
• Experience in the development and scoring of certification examinations and oversight of certification exam programs is required, experience in certification in the healthcare field is preferred

Licensure or Certification:

• ICE Certified Credentialing Professional (ICE-CCP) or ASAE Certified Association Executive (CAE) preferred.

Systems:

• High proficiency with computer programs and applications including MS Office and database management and data manipulation is required. High proficiency with statistical (e.g., SPSS, SAS, R) and psychometric software (e.g., Winsteps or IRTPRO) is required.

Communication:

• Ability to prioritize conflicting goals while negotiating in a persuasive manner during group decision making. This would generally occur during item review calls and the evaluation of the suitability of new or equated pass points to guide subject matter expert decisions that are congruent with psychometric and assessment best practices.
• Excellent verbal and written communication skills are required with a demonstrated ability to simplify complex information and craft messages for a wide range of audiences.

Special Skills:

Expert knowledge and track record of success with qualitative and/or quantitative data handling, knowledge of assessment standards, testing methodologies, data validation and analysis and reporting is required. This position requires the ability to interact tactfully and in a sensitive manner with professionals on all levels within and outside the AOA. Key competencies also include:

• Solid understanding of complex data structures and the ability to manage, manipulate and summarize small and large quantities of data
• In-Depth knowledge of statistical theory, IRT with an emphasis on Rasch models, certification testing, research methodologies, and applied measurement
• Self-starter, organized, versatile, capable of performing work independently with minimal direction
• Excellent problem-solving skills and ability to prioritize simultaneous, competing client activities
• Takes personal responsibility for service and quality of work delivered
• Excellent communication skills in both written and spoken English
• Strong interpersonal skills with the ability to act in team leader and team member roles, integrate competing ideas, and develop solutions
• Detail oriented, with strong time and project management skills

PHYSICAL, MENTAL DEMANDS/WORKING ENVIRONMENT

• This is a hybrid role requiring Tuesdays and Wednesdays in-person.
• Work may include weekday and weekend travel.
• Extended workday hours, including occasional evenings and weekends, may be necessary around major meetings, volunteer engagements, and other high-volume work periods.

Annual Salary Range: $165,000.00 - $175,000.00

The AOA offers the following benefits:

Medical, Dental, Vision, Life and AD&D, Long and Short-term Disability, FSA/HSA, Commuter Benefits, 401(k) with match

We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

American Osteopathic Association is unable to sponsor work visas at this time.