Our Client Case Studies
​
Our Client Case Studies
​
CFA Institute
​
World Class Arena Limited has worked for the Chartered Financial Analysts Institute (CFA Institute) from 2016 to 2020, supporting their transition from paper-based examinations to computer-based. This involved design of the overall strategic approach, building the Business Case, documenting requirements for item authoring, item banking and exam delivery, supporting the procurement, contractual negotiations and implementation planning.
Background
​
There are three levels of CFA Program exams, administered to nearly 300,000 candidates across around 80 countries each year. In recent years, the paper-based forms of these exams have been provided on two days a year (one date each June and one each December), with the largest venues in 2018 accommodating close to 20,000 candidates.
​
The logistical challenges of supporting such a large paper-based exam operation across the world became one of the most important drivers for CFA Institute to look at alternative exam delivery options. There was also a wish to look at improving the candidate experience and candidate journey through the three levels of CFA Program exams, and to assess the potential to align exam content with work-place practices through the use of computer-delivered exams.
The Requirements
​
CFA Institute wanted a strategic approach to assess current pressures on their exams operations and to evaluate future options. The organisation wanted an approach that could thoroughly examine the Case for Change - including risks, costs and timescales - and which could provide for strong strategic control of decision-making. It was clear that migration to CBT was not a foregone conclusion. CFA Institute also sought guidance in approaching the market-place of vendors, to understand what solutions are available for item-authoring, item-banking, form creation and exam delivery. CFA Institute wanted solutions that would enable them to retain control of the candidate experience, of private data and of secure IP. Finally, CFA Institute sought support in organising all of this work into a structured approach, within a controlled project management environment.
The 5 stages were designed to cover the full potential cycle of migration to CBT, from early ideas through to operational service.
​
Stage One was a research and investigative phase; it identified and assessed future business models, their benefits and capabilities required. The stage also included stakeholder analysis and development of a cost and revenue model. An Outline Business Case was provided to CFA Institute's leadership team and board.
​
In Stage Two a full risk analysis of CBT was undertaken and a detailed Statement of Requirements built. A plan for the procurement was developed, supported by industry and product analysis, as well as completing a Request for Information with vendors. The vendor evaluation approach and criteria were developed, too.
​
Stage Three comprised the full procurement phase, including negotiation of contractual terms. This included achieving the outcomes and contractual protections important to CFA Institute. The first Statements of Work were developed and agreed.
​
The program is currently (2019) beginning Stage Four, which includes all build activity, testing and an assessment of Readiness for Service.
Our Solution
​
CFA Institute created a dedicated internal team with responsibility to lead the analysis and modelling. The team was named the Future of Exams Delivery, or FED, team.
​
A multi-year, 5-stage approach, based on the OGC framework developed by the UK government, was adopted. This approach provided strategic control by putting in place executive and board-level re-assessments of the Business Case at critical junctures, including at the end of each stage of the programme.
The Result
​
World Class Arena Limited (WCAL) has supported the CFA Institute FED Leadership Team throughout the CBT project. This role has involved helping to design the overall strategic approach, working with the SMEs and business experts at CFA Institute, creating drafts of documents, and helping lead the evaluation of vendor's proposals.
​
During the procurement in 2017 and 2018, decisions were made to stagger the launch of CFA Program exams into the CBT environment. The CFA Level I Program exam will be the first to be launched, once the Implementation Phase activities are complete and upon successful completion of a Readiness-for-Service review.
​
The following are some of the notable successes to date.
​
Procurement
​
The CFA Institute procurement process was completed in a little over a year. This included the following activities, designed to support the selection of vendor(s): ​
​
-
Gauging initial vendor experience and interest through a Request for Information (RFI) questionnaire sent to potentially interested CBT vendors.
-
Provision of advice to vendors regarding the overall approach to evaluation (see text box below).
-
Designing and executing a vendor evaluation process that included:
-
Design of a vendor scoring matrix;
-
Scoring of vendor’s proposals;
-
Site visits;
-
Background due diligence;
-
Interviews and ‘discovery’ workshops;
-
Provision of a vendor(s) recommendation to the leadership team and governors.
-
-
Finalising the Master Services Agreement (MSA) and initial Statements of Work before the start of work. This included agreement on Critical Performance Indicators (CPIs) and Key Performance Indicators (KPIs).
​
Vendors Appointed
​
As the last step in the competitive phase of the procurement, vendors were invited to present their Best And Final Offers (BAFO). Following this, the evaluation of vendor's proposals was completed and decisions made to appoint Prometric (for item-banking software, exam delivery and exam centers).
​
Item Banking
During the procurement, a decision was made to move away from the item banking software that CFA Institute had been using for a few years. As a consequence, one of the first tasks for the FED Program in Stage 4 was to complete a successful migration of the item bank to the new software. Set-up of the item banking software also included configuring the new software to support the item-types and item authoring processes required by CFA Institute and to commence training of around 200 authors.
Implementation Phase commenced
​
The FED Implementation Team at CFA Institute has been created, and all contributing projects have been defined and stood-up, as required. The FED Team has been modelled broadly along the lines of a traditional project management PMO.
