NCARB 2013 Annual Report

 

Chair: Steven B. Miller, AIA, NCARB

Examination

Committee

The Examination Committee coordinates the preparation of the Architect Registration Examination® (ARE®) for use by Member Boards. It reviews grading methods and procedures and recommends the standard for passing the examination. It also evaluates the effectiveness  of examination procedures and makes recommendations to the Board of Directors for any necessary changes.

Committee Members: Robert M. Calvani, AIA, NCARBNew Mexico Member Board Member David Cronrath, AIA, NCARB Charles F. Farrell, AIAWashington Member Board Member Joseph Gardner James R. Lev, AIA, NCARBIllinois Member Board Member Gary E. Demele, AIA, NCARB
Board Liaison Erica J. Brown, AIA, NCARB, StaffFY13 has been a year full of change and excitement for the Architect Registration Examination® (ARE®). Over the course of the year, the Examination Committee led the evaluation and implementation of several key decisions made by the NCARB Board of Directors in recent years.

 

Highlights/Year-End Summary

 

MIGRATION  TO  NEW  VENDOR  SYSTEMS

This migration involved the transfer of all exam content as well as all candidate data to the new exam vendor. Moving from our long-time test development consultant required a significant technological and quality control effort between the consultants. For the first time in the history of the ARE, all data for all candidates will now be accessible through one online system. This is a significant advancement in the service provided by NCARB and will provide more transparency and access to data for both candidates and Member Boards alike.

●  Content Management System (CMS):

○  A new online tool will provide better management of the item development process and insight into the item pool.

○  More than 20,000 exam items were moved to the new CMS.

○  Subcommittees used the CMS to complete all assignments this year.

●  Candidate Management:

○  Member Boards and candidates will have a new interface in FY14. The new My Examination system will provide more transparency and access to candidate data than ever before.

○  A revised score report design, integrated appointment scheduling, and automatic messaging will enhance the candidate experience moving forward.

○  The data for more than 90,000 candidates who have tested since the launch of the computer-based ARE in 1997 were migrated to the new test consultant. This was an unprecedented effort that included:

▪  Contact information

▪  Eligibility history

▪  ARE 3.1 to ARE 4.0 conversions

▪  Rolling Clock information

▪  Pass or Fail results for all exams administered

▪  The answer to every question answered by each candidate

○  All candidate information has been linked to the internal NCARB database through My NCARB.

 

PRACTICE ANALYSIS

The Examination Committee also provided leadership to the short- and long-term implementation planning related to the results of the 2012 NCARB Practice Analysis of Architecture. In FY13, the results were quickly integrated into the current ARE Test Specification and influenced the work of the ARE Research & Development Subcommittee.

 

In addition to managing all of the changing aspects of the ARE, the Examination Committee provided oversight and guidance to the subcommittees engaged and committed to the successful development and delivery of the current ARE.

 

ARE RESEARCH & DEVELOPMENT
(R&D) SUBCOMMITTEE

The ARE Research & Development Subcommittee studies the future of the ARE and will begin the development of ARE 5.0.

 

The FY13 R&D Subcommittee efforts were a culmination of the past five years of research—all leading to watershed recommendations. Investigations into examination delivery, security, item-type effectiveness, and other ideas came together to inform recommendations on the future of the ARE.

 

Summary of Charges:

●  The subcommittee gained greater insight into the examination configurations needed to better position NCARB and the ARE to be compatible with various delivery models in the future.

○  Investigated delivery of the ARE in an internet-based model, which allows for much greater flexibility in delivery time and location. Ever mindful of the security implications, psychometric validity, and fairness issues related to licensure testing, the subcommittee:

▪  determined that current technology does not fully support a standalone and internet-delivered ARE.

▪  interviewed four internet-delivery providers to gain insight into the challenges and opportunities of various internet delivery models.

▪  recommended that internet delivery be pursued in test centers and future proctored "pop-up" test sites, while keeping options open in the future for individually administered exams via the internet.

○ Completed an extensive review of the ARE's current item types, as well as other industry standard item types that could be recommended for inclusion in a future version of the ARE.

▪  New performance item types were determined to provide ample opportunities to measure the knowledge, skills, and tasks associated with the practice of architecture.

●  Content area investigations from previous years were combined with the knowledge gained regarding internet delivery and item types to formulate an ARE 5.0 recommendation.

 

ARE MULTIPLE-CHOICE SUBCOMMITTEE

The ARE Multiple-Choice Subcommittee is responsible for writing, editing, and pretesting the multiple-choice items used in the seven divisions of the ARE.

 

This was the first year the committee used the new online portal for form review and item development. The paper methods of the past have been replaced with a secure online portal that allows for controlled access to each subcommittee member's assignments. Being the first year of this online effort, potential improvements were noted throughout each assignment so processes and the portal can continue to be refined.

 

Summary of Charges:

●  Completed item-writing assignments; all newly writ- ten items were peer reviewed and refined for appropriateness and compliance with the test specification.

●  Reviewed the statistical performance of flagged items and provided appropriate direction to the test development consultant for each item.

●  Participated in an online review of all forms and items to be delivered to candidates during FY14.

●  Removed System International (metric) units and Canadian content from the ARE.

●  Reviewed and updated code related items to ICC 2009 and its associated references.

●  Updated all quantitative fill-in-the-blank items to include key ranges now supported in the online item portal.

 

ARE GRAPHICS DEVELOPMENT
AND GRAPHICS PRETEST SUBCOMMITTEES

The ARE Graphics Development Subcommittee is comprised of two graphics groups that develop the graphic vignette simulations that are included in all seven divisions of the ARE. The Graphic Pretest Subcommittee is responsible for pretesting all newly developed graphic vignettes prior to their release in test centers. The committee helps to ensure that all vignettes are solvable and consistent.

 

The Graphics Development and Graphics Pretest subcommittees were able to accomplish a significant amount of work this year that critically increases the vignette script and isomorph pool for the coming years.

 

Summary of Charges:

●  Created solutions and completed template checklists (Graphics Development) and graphic checklists (Pretest) for 60 different isomorphs across six different vignettes.

●  Made necessary changes to scripts currently in development.

●  Authored new scripts for the Building Layout, Interior Layout, Site Zoning, Accessibility/Ramp, and Stair Design vignettes.

●  Reviewed all isomorphs authored prior to this fiscal year to determine readiness to be released into the testing channel for FY14.

 

In addition to creating vignette solutions for the Graphics Development Subcommittee to use in their work, the Graphics Pretest Subcommittee also completed a series of solutions and final quality assurance checks for all Site Design isomorphs. This vignette is being released this year with an updated Program requirement—this update will provide more clarity to candidates, but maintains the same level of candidate assessment.

 

ARE GRAPHICS  GRADING  SUBCOMMITTEE

The Graphics Grading Subcommittee is responsible for the development and application of the grading criteria to be used on solutions to vignette scripts. The scoring engines and statistical performance of each script are continually monitored by a random sampling of actual candidate solutions throughout the year.

 

The work of the Graphics Grading Subcommittee shifted in FY13 to accommodate the decision to no longer develop the ARE Desktop and its supporting scoring engine. For the last few years, this subcommittee had been working on the development and evaluation of the proposed scoring engine. With that system being set aside, the subcommittee turned its focus to the existing software to ensure it will remain a viable system for the foreseeable future.

 

Summary of Charges:

●  Continued monitoring the vignette scoring software to ensure accurate assessment of candidate's solutions.

●  Reviewed the statistical analysis of 46 script families recommended for review and determined to turn off three script families and monitor another four script families.

●  Developed a proof of concept (POC) for two vignettes to propose a human scoring rubric that could be implemented should the ARE Desktop be implemented utilizing a distributed human scoring model.

○  The POC for each vignette was based on the current computer-based scoring of the vignette.

○  The POC included assessment criteria, assessment scales, scoring feature descriptions, a computer-based scoring to human scoring feature map, and scored solution examples.

●  Assisted the R&D Subcommittee by developing examples of a proposed performance item type. These examples helped to inform the R&D Subcommittee as it developed a recommended structure for ARE 5.0.

 

FY13 TEST SPECIFICATION TASK FORCE

The Test Specification Task Force is responsible for the update to the test specification for the ARE. The test specification will be updated within the current structural framework and be based on the findings of the 2012 Practice Analysis of Architecture. The task force is responsible for aligning the appropriate knowledge, skills, and tasks identified in the practice analysis into the revised structure of the ARE.

 

The FY13 Practice Analysis Test Specification Task Force was charged with implementing the results of the 2012 Practice Analysis of Architecture by refreshing the cur- rent ARE 4.0 Test Specification. This effort allows NCARB to more quickly align the ARE with the results of the most recent evaluation of the practice of architecture.

 

Summary of Charges:

●  Mapped the knowledge, skills, and tasks of the 2012 Practice Analysis to the current test specification.

●  Reviewed each content area of the examination and updated descriptions to better reflect the cognitive complexity required to practice architecture.

●  The measurement ability of the vignettes was factored into the overall weightings of the content areas, allowing a more rounded assessment of candidate performance.