Exploring New Ways to Audit Training, Build Capacity, Address Sustainability

by Aamir Fayyaz, Director, SAI Pakistan, Office of the Deputy Auditor General (Central), Lahore

Capacity building is critical for all dynamic organizations, both public and private. Training, an essential tool to systematically build capacity, also helps achieve desired organizational objectives. Public sector training programs are generally funded through national budgetary allocations and tend to be subject to audit by Supreme Audit Institutions (SAIs), the nature of which is determined by the SAI mandate that, typically, covers regularity and performance auditing.¹

Often, training organizations are viewed as ordinary public sector establishments, and, accordingly, the audit approach tends to remain limited to compliance (C). This approach is limiting, as it does not address the issue of effectiveness. Performance audits seek to collect evidence on (1) Economy, (2) Efficiency and (3) Effectiveness (3Es) but has no bearing on assessing an organization’s compliance.

Objectives
Uniting compliance and performance audit objectives into one audit assignment is covered in ISSAI 100 relating to the fundamental principles of public sector auditing.² This integrative approach of uniting Compliance, Economy, Efficiency and Effectiveness (C+3Es) can (1) create a broad-based audit, (2) provide multi-dimensional reports to stakeholders and (3) lead to SAI efficient resource utilization.

Applying a C+3Es method when auditing training institutions is of particular importance, as it will comprehensively review adherence to regulatory guidance, as well as identify gaps in performance that affect the institution’s ability to develop capacity.

Assumptions
To measure performance and return on investment in training personnel, several assumptions are used regarding an employee’s pay, cost for training, and how the training establishment distributes learning modules.

An estimated $50,000 in government funding was spent throughout the employee’s career on training, equal to 10% of the total pay received throughout the employee’s career (30-35 years of service with career total emoluments of $1.5 million).³

Training time, when aggregated, equals three years (spanning entire career).

Because this figure represents a significant investment, closer scrutiny is required from multiple angles by all stakeholders, including the employee (as training recipient).

It is assumed the training establishment imparts three 12-week training courses annually, each course having an average enrollment of 40-50 trainees. The entity is 100% funded through public budgetary allocations divided into two main areas: employee-related expenses and organizational operations.

Performing the Audit
The first question the audit team should ask is, “What was the institution’s budget for the last three years?” This allows for a stronger understanding of operations and associated risks. The audit team should also examine and analyze historical financial plans of two other academies to better compare budget adequacy and training costs.

Collecting evidence on what (and how) provisions are procured is essential to uncover material instances of operational waste and inefficiency.

The audit should also include interviewing a relatively large representative sample of trainees (both current and past), as well as immediate supervisors. This method will help corroborate audit findings and provide a holistic approach to conclusions.

The hallmark of a C+3Es audit of a training institution is assessing training’s soft side, including the study of course curricula. In doing so, the audit team should focus on areas, such as:

• Existence of public sector training policy and the degree of training program compliance⁴;
• Traditional stimuli for program revisions and the time it takes for revision completion;
• Formal training needs assessments through stakeholder identification and consultation; and
• Curriculum inclusion of generalized and specific modules (depending upon the training needs assessment).

Training programs having multiple components require auditors to make informed comments on the significance assigned to various parts, as well as adherence to time and cost scheduling.

Clarifying these points is necessary in identifying material gaps, but auditors should not expect resource utilization to be proportionate to assigned content weighting.

For example, auditors may discover that while component x was given 30% weight in module A, the actual cost incurred (in terms of number of hours) remained at 5% total training hours.

Conversely, auditors may note, for instance, that study hours somehow consumed a significant portion of the training budget. Though this data may have no documented relevance to training objectives, reporting such matters are important as they provide a record, and may better explain, patterns of deviations that have actual (and potential) impacts on achieving training objectives.

These reports are expected to make the training establishment more responsible and accountable for its resources, which directly influences the organization’s commitment level to deliverables enshrined in Sustainable Development Goal (SDG) 16.

Other factors, such as evaluation criteria used by faculty to rank trainees at regular intervals; core faculty and critical support staff availability; and overall gender makeup⁵ should also be investigated, as these aspects would specifically address SDG targets relating to qualified faculty in educational institutions⁶, which impacts capacity development quality and effectiveness.

The proportion of in-house to outsourced faculty factors to consider include the historical context when sanctioned strength was determined; the training establishment’s mandate and clientele at the time; and relevance in the wake of present expectations.

Similarly, prevalent pay scales must be understood, comparisons drawn with similar institutions and evidence collected to highlight potential non-compliance to “equal pay for work of equal value” targets⁷ as outlined in SDG 8.

Ascertaining the adequate availability of training establishments that incorporate special needs facilities and functions help determine adherence to internationally acceptable accessibility standards.⁸ In addition to using this data to assess compliance, these statistics allow for correlating accessibility to productivity and analyzing actual and potential impact on achieving training objectives.

A C+3Es audit approach requires assessing training costs—per individual course, participants in each course, and that of the training establishment. This data could then be linked to criteria on the training program’s economy, efficiency and compliance levels.

Of note, calculating training costs can be an intricate process due to a multitude of variables, such as the number of courses, time periods, and amount of trainees in each course. Likewise, the relative importance of various trainings may also create complexities regarding the generic notion of training costs.

While the aspects stressed in this article are not exhaustive, the goal is to gain an appreciation that a compliance ONLY audit may prove inadequate to grasp the diversity of certain public sector establishments. Therefore, augmenting a compliance audit with the 3Es may provide immense learning opportunities to the SAI and, in the process, raise reporting levels and usefulness for stakeholders.

References
¹SAI Pakistan has statutory mandate to perform all the three types of auditing viz, financial, compliance and performance.
²ISSAI 100/23; available at http://www.issai.org/en_us/site-issai/issai-framework/3-fundamental-auditing-priciples.htm.
³These are rough estimates and are not the result of any research on the subject. The figures are based upon pay regimes generally applicable on such employees. In reality, the total emoluments received may depend upon the pay package adopted, such as a purely monetized regime. The purpose of these figures is to contextualize the importance of expenditure on capacity building with reference to the total benefits received.
⁴Training Program (TP) here denotes and includes all curricular and extracurricular activities, scheduling of the TP, matching of available resources with the tasks, etc.
⁵Goal 5, Target 5.5.2, available at sustainabledevelopment.un.org/sdg5 (accessed on 9.30.2017).
⁶Goal 4, Targets 4.A and 4.C, available at sustainabledevelopment.un.org/sdg4 (accessed on 9.30.2017).
⁷Goal 8,Target 8.5, available at sustainabledevelopment.un.org/sdg8 (accessed on 9.30.2017).
⁸Goal 4, Target 4.A.1, available at sustainabledevelopment.un.org/sdg4, Target 6.2.1, available at sustainabledevelopment.un.org/sdg6 (accessed on 9.30.2017).