Client Login

Program Evaluation

Our services are conducted within the Treasury Board Guidelines for program evaluation, which are a foundation for every evaluation project in which we participate.

Evaluation and performance measurement are part of the modern management agenda of the federal government, as expressed through the thrust towards Results-Based Management and Accountability Frameworks (RMAF), the Modern Comptrollership initiative and the Service Improvement Initiative. More recently, they are seen as key parts of Treasury Board Secretariat’s Management Accountability Framework (MAF) and Program Activity Architecture (PAA), with the focus upon client focus, results, measuring performance, and resource reallocation.

Evaluation and performance measurement contribute to the achievement of Results for Canadians. In this context, rigorous evaluation against performance measures and targets becomes an integral part of ongoing management of programs, in terms of planning, execution, and reporting. It is important for both external and internal stakeholder audiences, including Parliament, citizens, senior departmental management, central agencies, clients, partners and program managers/staff. A formal program evaluation, as well as an approved RMAF, has also become requirements for the approval of funding for programs involving grants and contributions.

The Program at Objective Level structure (POL) implemented for Program for Energy Research and Development (PERD) is intended, through its planning and reporting activities, to link program level activities, outputs and outcomes (with associated performance measures and targets) to the Departmental and governmental goals and priorities. These then form a basis for interim (mid-course adjustment if necessary) and final evaluations of programs attached to the various POL’s.

The importance of such frameworks to decision-making, that the frameworks properly reflect the programs and that they are used appropriately, are stressed by the questions raised in any evaluation and their focus upon relevance, success, design and delivery, and the application of Results-based Management.

We have provided leadership in the areas of performance measurement, service standards and measurement of client satisfaction. 

Our Approach to Program Evaluation

In performing a program evaluation, we typically propose the following approach which is then customized to specifics in the Statement of Work.

  1. Profile
    Initial review of program/policy documents and orientation including the original Treasury Board submission, business plan(s) and program promotional material.
    At this initial stage we also propose a briefing session with program management staff.
     
  2. Logic Model
    The Methodology Report includes a draft logic model for the program. The draft logic model will be developed based upon the initial review of documents and in a working session with the Program Manager and selected staff.
     
  3. Identify Evaluation Questions
    The Methodology Report includes the final evaluation issues and interview guide(s) related to them. The interview guide(s) will include questions related to each of the evaluation questions.

  4. Data Requirements and Collection Strategy

    4.1   Interview Questions
    In formulating the interview questions, as well as in our other data collection activities, the notions of incrementality (i.e., what was achieved by the program that would not have been achieved in its absence) and attribution (i.e., who has achieved the benefit / incurred the cost) will be important to capture. Multiple lines of inquiry (e.g., document review, interviews with different stakeholder groups with different views / relationships with the program, surveys, case studies, comparison with other jurisdictions) are used to ensure balance in information collection.

    Interviews with key players and stakeholders including individuals representing program staff and management, participating communities and private sector partners, academic participants, and project-level management.

    Interviews are in the official language of the respondent’s choice. Interviews with respondents in the National Capital Region and its near proximity are face-to-face or by telephone, as preferred by the respondent, while those with respondents in other parts of Canada could be by telephone or in person depending upon project requirements.

    4.2  Surveys 
    Surveys are designed to collect information from respondents in a form that is useful for further analysis. The survey design considers who (the respondents – population, sample), what (the questions to be asked), how (survey approach – e.g., mail-out, telephone, in-person, Internet-based), when (timing) and other factors. Before the survey is administered, approval, as required by government policy, is obtained from Public Works and Government Services of Canada (PWGSC).

    It should be noted that if the survey is to go to clients to measure their satisfaction with program services, then the use of the Common Measurement Tool (CMT) is recommended. The CMT, developed as part of the Service Improvement Initiative (SII), provides a government-wide standard and core set of questions for inter-program comparisons. Reporting on the results of the SII and improvements in client satisfaction are also part of each Deputy Minister’s reporting requirement.

    4.3  Case Studies
    Preparation of case studies for a sample of current projects to acquire a more detailed knowledge of activities and outcomes at the project level.
    A selection of projects is chosen for case studies that cumulatively cover the activities, outputs and outcomes documented in the logic model.

    4.4 Data Collection Report/Presentation
    The Data Collection Report / Presentation summarizes the data collected from the file review, interviews and case studies. The report is presented to the Project Authority and project governance structure.
     
  1. Reporting Strategy
    The Draft Evaluation Report presents the results of the evaluation, conclusions, lessons learned and recommendations. The report is presented to the Project Authority.

    The Final Evaluation Report incorporates input received via Audit and Evaluation Branch (AEB). It is prepared in accordance with AEB’s guidelines. 

    We allocate resources to each individual project to provide the best “package” to our clients. This means the right mix of categories (e.g., Project Manager, Senior Evaluator, etc.), the right numbers in each category and the right language profile for our team.