Chapter 1: Introduction to the Study
Introduction
Beginning in the 1970s, the number of public sector audit organizations around
the world that have powers to conduct performance audits increased significantly (The
International Organization of Supreme Audit Institutions Development Initiative (IDI),
2018; The International Organization of Supreme Audit Institutions (INTOSAI), 2004,
2013a; Torres, Yetano, & Pina, 2019). These performance audits are conducted to (a)
increase the accountability of governments by providing them and the general public with
an independent and objective opinion about meeting the principals of economy,
efficiency, and effectiveness by government entities; and (b) improve the public sector
performance by making recommendations (INTOSAI, 2019).
While the performance auditing phenomenon has received considerable attention
from both the academic community and scholars-practitioners, the demand for
performance audits and their impact on public administration are poorly studied
(Desmedt, Morin, Pattyn, & Brans, 2017; Hay & Cordery, 2018; Reichborn-Kjennerud &
Johnsen, 2018). This identified information gap in the literature was critical, especially
for Kazakhstan where the performance auditing is a novelty. By exploring the
Kazakhstani case in terms of demand for and impacts of performance audits on public
administration, I incrementally addressed and attempted to reduce this information gap.
My study might be helpful to auditors who are mastering a new profession, i.e.,
performance auditors, and their organizations. Additionally, governments and their
entities that are subject to performance audits to include parliamentarians and legislative
bodies who use performance audit report in policymaking may benefit from a detailed
look at performance auditing processes and outcomes in order to evaluate public policy
effectiveness.
Background
Contemporary public sector audit organizations are empowered with mandates to
conduct performance audits (IDI, 2018; INTOSAI, 2004, 2013a; Torres et al., 2019).
Before using the performance audits, public auditors conducted traditional compliance
and financial audits, i.e., their objectives were an examination of compliance of auditees’
operations with the laws and regulations and conduction of audits of financial statements
of government entities and other auditees, such as government-owned enterprises or
private companies that deliver public services (Glynn 1995; Pollitt, 2003). Performance
auditors’ objectives are to determine whether governmental initiatives (programs,
projects, and other undertakings) or auditees’ activity are in line with the principles of
economy, efficiency, and effectiveness (INTOSAI, 2019, p. 8).
The following two key trends are associated with the development of performance
auditing. Firstly, performance auditing is a widespread practice used within the public
sector in many countries; as a rule, performance audits conducted by public sector audit
organizations called supreme audit institutions (SAIs). Issuance of performance auditing
standards by the International Organization of Supreme Audit Institutions (INTOSAI)
supports this position statement given that more than 190 countries have national and
regional organizations that consists of SAIs whose purpose is to undertake performance
audits (INTOSAI, 2018). Secondly, performance audits replace traditional compliance
and financial audits from SAIs’ audit portfolios. This trend means that today’s SAIs spend
more resources to conduct performance audits. For instance, in the United States, Canada,
United Kingdom, Sweden, and other counties, which are pioneers and leaders in
performance auditing, SAIs increase their resources allocated for performance audits
(INTOSAI, 2004, 2013a; IDI, 2018). Hay and Cordery (2018) offered that performance
audits are in high demand. Despite the widely accepted objective of performance
auditing, i.e., contribution to accountability, efficiency, and effectiveness of executives’
programs and operations, and more than 50 years of historical experience, this question is
still topical (Hay & Cordery, 2018; INTOSAI, 2013).
Several studies have illustrated the relation between the performance auditing
emergence and the 1970s reforms in public administration coined new public
management (Azuma 2003, 2005; English & Skaerbaek, 2007; Lapsley, 1999; Power,
1996). Countries’ experiences in adopting the performance audit have also received
considerable attention from scholars (Flesher, Samson, & Previts, 2003; Grönlund,
Svärdsten, & Öhman, 2011; Hossain, 2010; Jacobs, 1998; Morin, 2001; Morin & Hazgui,
2016; Reichborn-Kjennerud, 2013a; Skaerbaek, 2009; Weihrich, 2018; Yamamoto &
Waranabe, 1989). For instance, based on experience of Australian SAI, Hossain (2010)
concluded that the development of performance audit is related to changes within the
public administration system and the increasing need for accountability and responsibility
of governments in dealing with the taxpayers’ money. The problems associated with
performance auditing were also subject to more recent studies. In particular, according to
Morin and Hazgui (2016) and Hossain (2010), governments resisted to expanding public
auditors’ powers by providing them with mandates to conduct performance audits of
governmental programs. Funnell, Wade, and Jupe (2016) and Morin (2008) offered
critiques of performance auditing in terms of ambitiousness of performance audits’
objectives and limitations associated with SAIs’ capacity to objectively evaluate the
effectiveness of auditees’ activity and found them to be reasonable. Critics of
performance auditing coupled with the isomorphism phenomena, i.e., applying reforms or
practice because of their popularity rather than their reasonability, strengthens the
topicality of questions regarding the demand for performance audits. These questions are
at issue due to the scarce availability of systematic empirical evidence on the impact of
performance auditing on public administration (Desmedt et al., 2017;
ReichbornKjennerud & Johnsen, 2018).
There have been several influential studies that focused on the impact of
performance audits on public administration systems (Alwardat, Benamraoui, & Rieple,
2015; Desmedt et al., 2017; Funnell et al., 2016; Morin, 2001, 2008;
ReichbornKjennerud, 2013b; 2014a; Reichborn-Kjennerud & Johnsen, 2018; Reichborn-
Kjennerud & Vabo, 2017; Torres et al., 2019). According to Morin (2001, 2008),
performance auditors’ work leads to changes for auditees. Interestingly, the extent of
performance audits’ impacts may depend on the perceived roles of performance auditors,
such as controllers or watchdogs and assistants or modernizers (Lapsley & Pong, 2000;
Morin & Hazgui, 2016). However, in all cases, it is difficult or even impossible to
objectively define and measure the effects of performance audits on improving the public
sector performance (Funnell & Wade, 2012; Morin, 2008; Pollitt, 2003). The absence of
an indicator that is acceptable to all within the public sector (like a profit within the
private sector) explains this limitation of performance auditing (Lapsley & Pong, 2000).
However, performance auditors’ work is in demand, and the number of users of
performance audit reports, including parliamentarians, has been increasing (Torres et al.,
2019). In particular, Funnell et al. (2016) offered that performance audits may also be
used to legitimize earlier political decisions or to promote democratization by public
officials. For instance, in some European countries, policymakers used arguments on
improving the quality of political and democratic processes as the rationale for
empowering public sector audit organizations with mandates to conduct performance
audits (Tillema & Bogt, 2010). Approaches and practices of performance audit
significantly differed depending on country specifics, including the organizational styles
of their public administration structures (Glynn, 1995; Hossain, 2010; Morin, 2016;
Torres et al., 2019). As such, Hay and Cordery (2018) urged exploring the causes of
demand for performance audits, while Reichborn-Kjennerud and Johnsen (2018)
reasonably called for more empirical studies on impact of performance audits.
Problem Statement
The most recent studies on exploring the impact of performance auditing on
public administration focused on the cases of developed countries, while the developing
countries’ perspectives were fragmentarily explored within a few studies (Loke, Ismail, &
Hamid, 2016). Despite legitimization of performance audits, i.e., empowering the public
sector audit organizations with mandates to conduct performance audits by law, and
stable increase in the number of performance audits conducted by the Accounts
Committee for Control over Execution of the Republican Budget (the Accounts
Committee), the central public sector audit organization, and Revision Commissions, the
local public sector audit organizations, there were no empirical studies on demand for and
impacts of performance auditing on public administration in Kazakhstan. A few applied
pieces of research on international experience in performance auditing were conducted by
the Accounts Committee to develop standards and establish the methodological base for
performance auditing (The Centre for Financial Violations Research (CFVR), 2014, 2016,
2017). As a consequence, it was unknown why performance audits are in demand in
Kazakhstan and how they influence the national public administration system.
This gap was not surprising, since, unlike the Anglo-American countries or
countries of Continental Europe, the use of performance auditing in Kazakhstan started in
2002 from two pilot audits conducted by the Accounts Committee and the intensive use
of performance audits by both the Accounts Committee and Revision Commissions
started only in 2015 (The Information System of Legal Acts of the Republic of
Kazakhstan (ISLARK), 2015). The problem was that a lack of empirical evidence about
the usefulness or, more precisely, the positive impact of performance audits on public
administration, challenges the credibility of performance auditors’ works and their
organizations’ activity in general (Funnell et al., 2016; Reichborn-Kjennerud & Johnsen,
2018). This problem increased risks of missed opportunities of public sector audit
organizations in Kazakhstan in the government’s efforts to increase efficiency and
effectiveness of the public sector performance.
A gap, therefore, existed for scholars-practitioners in understanding the factors
that might explain the demand for performance auditing in Kazakhstan and it was
unknown the perceived effects of performance audits on Kazakhstani public
administration. This gap was strengthened by recent scholars who rightly stated that
performance auditing and its impacts vary depending on the development of a nation and
styles of public administration (Desmedt et al., 2017; Loke et al., 2016; Torres et al.,
2019). Hay and Cordery (2018) reasonably suggested a more extensive examination of
the demand for performance audits within different settings and highlighted the
usefulness of new explanations for performance auditing. In this regard, the case of
Kazakhstan was unique.
There are four aspects that separate the Kazakhstani case from well-studied cases
of developed and some developing countries. Unlike the pioneers and leaders in
performance auditing, such as the United States, Canada, United Kingdom, Sweden, and
Australia that started to use performance audits in the 1970s or even earlier, Kazakhstan
has been using the performance audits for only a few years. Kazakhstani public sector
audit organizations including the Accounts Committee and Revision Commissions
referred to the Board Models of SAIs that is significantly different from Westminster or
Napoleonic models of SAIs (ISLARK, 2015). Unlike developed countries, in
Kazakhstan, neither central nor local governments resisted the expanding public auditors’
mandates from conduction of traditional to performance audits. Finally, Kazakhstan is a
young nation that started to establish its public administration system after the breakup of
the Soviet Union in 1991 and, therefore, Kazakhstani public administration system
referred to recently emerged styles (Adnan & Fatima, 2015; Knox, 2008; Riboud, 2015).
These aspects exacerbated the problem of a lack of information regarding the impact of
performance audits on Kazakhstani public administration.
Purpose Statement
The purpose of this qualitative study was to explore the factors that might explain
the demand for performance auditing in Kazakhstan and the perceived effects of
performance audits on Kazakhstani public administration. To achieve this purpose, in
depth understanding of the reasons for conduction more performance audits and these
audits’ contributions to changes in public administration through the eyes of those
individuals performing or requesting audits was needed. In this study, I focused on
exploring the lived experiences and perceptions of performance auditors, managers of
auditees, and parliamentarians related to (a) the influential factors that may explain
recently existing demand for performance auditing in Kazakhstan, and (b) impacts of
performance audits on auditors, managers of auditees, parliamentarians, their
organizations, and public administration in general.
Exploring the perceptions of public sector audit organization representatives and
executive and legislative bodies was critical in my quest for a better understanding of this
studied phenomena. Furthermore, exploring the causes of performance audit demand and
audit reporting’s influence on Kazakhstan’s public administration will contribute to
greater understanding for other newly emerged public administration styles that differ
from a Western style of government.
Research Question
The following central research question guided my study:
Research Question (RQ): What are the lived experiences and perceptions of
performance auditors, managers of auditees, and parliamentarians, the key participants
and users of performance audits, regarding the demand for and impacts of these audits on
Kazakhstan’s public administration?
Theoretical Framework
The theoretical framework for my study was a combination of Hood’s new public
management theory (NPM) and Waterman and Meier’s principal-agent model (PAM;
Hood, 1991; Waterman & Meier, 1998). As previously described, performance auditing
emerged due to the 1970 reforms called new public management and performance audits
are used as an instrument to implement these reforms aimed at promoting improvements,
increasing accountability, and achieving the desired goals and results in the public sector.
In line with an ideology of new public management, transferring from bureaucratic style
to customer-oriented and accountable management style grounded in the private sector
leads to an improved public sector performance (Hood, 1995; Kapucu, 2006). NPM
implies replacing process-oriented approaches with result-oriented techniques to increase
the efficiency and effectiveness within the public sector (Bao, Wang, Larsen, & Morgan,
2012; Hood, 1995; Kapucu, 2006). This formula of improvements works in today’s
realities (Bao et al., 2012; Esposito, Ferlie, & Gaeta, 2018; Reiter & Klenk, 2019;
Verbeeten & Spekle, 2015).
Expression of an independent and objective opinion by performance auditors
regarding the efficiency and effectiveness of government initiatives is a tool for
implementing the accountability principle, while performance auditors’ recommendations
are a means to promote improvements. Logically, introducing the performance auditing,
i.e., a public auditors’ professional practice used to establish whether the public sector
organizations operate with due regard to efficiency and effectiveness, directly relates to
NPM (Azuma 2003, 2005; English, 2003; Funnell, 2015; Reichborn-Kjennerud, 2013a).
Thus, I used NPM as the theoretical lens for identification and interpretation factors that
might explain the phenomenon of demand in performance audits in Kazakhstan.
Further, performance audit is a process of trilateral interaction between auditors,
managers of auditees, and parliamentarians who use performance audit reports to make
political decisions (Morin, 2008). The participants of a performance auditing process are
three parties with a different status, mandates, and responsibilities. This construction of
interacting between auditors, managers of auditees, and parliamentarians is embeddable
in the principal-agent model. PAM, in the context of executing the accountability
principle, implies dividing the participants of a performance auditing process to principals
or, policy makers, and agents or, policy implementers (Barzelay, 2001; Funnell et al.,
2016; Morin, 2008; Reichborn-Kjennerud, 2013a).
With reference to performance auditing, PAM is the construction of three-level
relations between principals and agents, where, as it is demonstrated in Figure 1,
performance auditors are principals for managers of auditees and agents for
parliamentarians, parliamentarians are principals for both performance auditors and
managers of auditees, and finally, managers of auditees are agents for both
parliamentarians and performance auditors (Barzelay, 2001; Reichborn-Kjennerud,
2013a).
Figure 1. The principal-agent model
Note: Developed based on Barzelay (2001) and Reichborn-Kjennerud (2013a).
I used PAM as a lens to interpret rules, approaches, and practices of interactions
between the participants of a performance auditing process. Barzelay (2001) argued that
PAM in the context of NPM takes on a specific feature; both principals and agents have
agreements on improving public sector performance. This aspect was the rationale to use
two frameworks in a combined manner since it provided a foundation for the
interpretations of perceptions of auditors, managers of auditees, and parliamentarians
regarding the usefulness of performance audits and their impact on public administration
in Kazakhstan.
Nature of the Study
To address the purpose of my study, I used a qualitative method with a
phenomenological approach.
PRINCIPALS
AGENTS
AGENTS
parliamentarians
managers of auditees
performance auditors
The Qualitative Method
Based on the comparison of application requirements, advantages and limitations
of qualitative, quantitative, and mixed methods in researching, I selected the qualitative
method as the most suitable to explore causes of demand for performance audits in
Kazakhstan and their impact on Kazakhstani public administration. Qualitative inquiries
are used to examine a phenomenon that impacts individuals or groups in a specific setting
(Frankfort-Nachmias & Nachmias, 2008; Maxwell, 2013; Patton, 2015). The selected
method implies communications with study participants and observations to gather rich
and first-hand data in a particular cultural and social context (Mills & Birks, 2017).
Patton (2015) stated, through applying the qualitative method, a researcher may interpret
meaning-making processes. Thus, direct access to the study participants who are
experienced participants of a performance auditing process, i.e., auditors who conduct
performance audits as well as managers of auditees and parliamentarians who use
performance audit reports, contributed to greater information and understanding of the
recent phenomenon of demand for performance audits in Kazakhstan and to determine
whether these audits are impacting public administration approach and effectiveness. I
used a comparative analysis to determine whether the case of Kazakhstan differs from the
well-studied cases of developed countries.
The Phenomenological Approach
According to Moustakas (1994), the phenomenological approach implies the
obtaining of comprehensive descriptions of study participants’ experience to analyze and
portray the essence of their experience. Within phenomenological studies, a researcher
collects data through in-depth interviews with people who have directly experienced the
studied phenomenon and then he or she transfers their experiences into consciousness
(Moustakas, 1994; Patton, 2015). These applied aspects of phenomenological inquiries
and lack of empirical evidence regarding the impact of performance audits (Desmedt et
al., 2017; Reichborn-Kjennerud & Johnsen, 2018) were the key arguments to use an
empirical phenomenological approach in my research.
To comprehensively explore the performance auditing as a widespread
phenomenon and a means of influence for policy-makers and policy-implementers, I
gathered, learned, and analyzed information shared by the study participants selected
about their experience. Therefore, I used in-depth phenomenological interviews to
explore opinions, lived experiences, and perceptions of auditors, managers of auditees,
and parliamentarians associated with performance audit use within Kazakhstani public
administration.
O’Sullivan, Rassel, and Berner (2008) offered that the most challenging aspect of
a selected research method is finding individuals who have experienced a particular
phenomenon of interest who are willing to participate in the research inquiry. Thus, I used
the group characteristics sampling and single-significant-case sampling methods to
identify my research participants with additional participant encouragement through snow
ball sampling. I used member checking and data triangulation to verify my primary data
prior to and as a foundation of thematic analysis (see Koelsch, 2013; Lietz, Langer,
& Furman, 2006; Patton, 2015; Ravitch & Carl, 2016, for more).
Generally, the studied phenomena affect the public and private sectors’
organizations that deliver public service, taxpayers, and recipients of public services. I
focused on the impact of performance audits on public administration to include
participants of a performance auditing process; hence, the central target population
consisted of:
▪ public auditors who served in 18 public sector audit organizations in
Kazakhstan that empowered with the mandates to conduct performance audits
including the Accounts Committee and 17 Revision Commissions;
▪ managers of auditees, i.e., central and local governments’ entities;
▪ parliamentarians who are members of the Parliament of the Republic of
Kazakhstan (the central legislative body) and parliamentarians who are
members of the Maslikhats (the local legislative bodies).
In order to reach a threshold of data saturation, I interviewed 14 government
officials whose primary jobs include governmental budgeting and managing public funds,
auditing, and higher-level parliamentary commissioners responsible for auditing
outcomes.
Definitions
The following key terms were used within the study.
Auditee: An organization that is subject to audit. According to the Act of the
Republic of Kazakhstan “On Public Auditing and Financial Control” adopted in 2015
(the 2015 Public Auditing Act), auditees are government bodies and entities,
organizations of the quasi-public sector and the recipients of budget funds (ISLARK,
2015, para. 1).
Auditor: An employee of an audit organization who due to his or her official
duties participates in audits. According to the 2015 Public Auditing Act, a public auditor
is a public servant who conduct public audit and financial control and has the public
auditor certificate (ISLARK, 2015, para. 1).
Demand for performance auditing: Conditions or reasons that cause the need for
performance audits.
Economy: A principle of governance and managing the public sources that used as
an evaluation criterion for performance auditing objectives. “The principle of economy
means minimising the costs of resources. The resources used should be available in due
time, in and of appropriate quantity and quality and at the best price” (INTOSAI, 2019, p.
9).
Effectiveness: A principle of governance and managing the public sources that
used as an evaluation criterion for performance auditing objectives. “The principle of
effectiveness concerns meeting the objectives set and achieving the intended results”
(INTOSAI, 2019, p. 9).
Efficiency: A principle of governance and managing the public sources that used
as an evaluation criterion for performance auditing objectives. “The principle of
efficiency means getting the most from the available resources. It is concerned with the
relationship between resources employed and outputs delivered in terms of quantity,
quality and timing” (INTOSAI, 2019, p. 9).
Impact of performance audits: A set of direct and indirect, desired and undesired
effects that SAIs, local public sector audit organizations, and their performance auditors
have on auditees, audited areas of public administration, and the public sector in general
(Lonsdale, 2000; Morin, 2001).
Local revision commission: A public sector audit organization that is responsible
for fulfilling external local audit functions established by local laws or by a law-making
body at local level. According to the 2015 Public Auditing Act, the Revision
Commissions are external public audit organizations that have the power to conduct
performance audits at the local government level (ISLARK, 2015, para. 13).
Performance audit (within the public sector): A legitimate professional activity of
independent public sector audit organizations on examination of whether an audited area
of public administration (reforms, programs, policies, and projects) or an auditee (its
operations, systems, and activities) meet the generally-accepted principles of economy,
efficiency, effectiveness, and other criteria of good governance. Performance audits are
carried out to promote improvements in public administration through providing the
managers of auditees, executives, parliamentarians, and the general public with new
insights about the audited area and auditee and recommendations on improvements, if
necessary (Alwardat et al., 2015; Desmedt et al., 2017; Funnell & Wade, 2012;
INTOSAI, 2013b; Morin, 2016; Reichborn-Kjennerud & Johnsen, 2018).
Supreme audit institution: “A supreme audit institution (SAI), or national audit
institution, fulfils the independent and technical public sector external audit function that
is typically established within a country’s constitution or by the supreme law-making
body” (OECD, 2016, p. 20). According to the 2015 Public Auditing Act, the Accounts
Committee is SAI of Kazakhstan that has the power to conduct performance audits at the
central government level (ISLARK, 2015, para. 12).
Public administration: A scholarly area of enquiry and research and a field of
practice associated with the formal legal and procedural governance of society (Fenwick
& McMillan, 2014).
Assumptions
There were several assumptions associated with my study. These assumptions
related to the studied phenomena, selected theoretical framework and methodology,
research participants, and expected results of my research.
Despite the examples of resistance to adoption and development of performance
auditing, conduction of performance audits has been established as a legal mandate of
public sector organizations in many developed and developing countries (Hossain, 2010;
INTOSAI, 2018; Torres et al., 2019). Based on this historical experience and long-term
priorities of Kazakhstani public policy on increasing the effectiveness of public
administration to join the group of the most 30 developed countries of the world
(ISLARK, 2018), I assumed that the performance auditing as a legal mandate of the
Accounts Committee and Revision Commissions in Kazakhstan will not change
significantly until my research is complete.
I used the new public management theory, which stimulated the extensive use of
performance audits in many countries including Kazakhstan, and the principal-agent
model, that may explain in-depth nature of relations between participants of performance
audits, to exhaustively explore the studied phenomena. I applied the phenomenological
approach to examine the lived experiences of those actors involved in performance audits
within Kazakhstan to answer my posed research question.
Further, I used group characteristics sampling and single-significant-case
sampling methods to select and gain access to the study participants who are in line with
the selection criteria including being experienced in performance auditing. My intention
was be able to establish a trusting and inspiring atmosphere during the communications
with my study participants by application of the selected techniques on gathering data,
including an informed consent process. All study participants are public servants and,
thus, they refer to a specific cohort of employees who, due to their job positions and
official duties, should be committed to principles of good governance (Frederickson,
1982; Frederickson & David, 1985). Therefore, I assumed that the study participants will
share honest and valuable information based on their professional experiences. I also
assumed there will be agreement of my study participants to take part in my research (i.e.,
the study participants’ informed decision) and their willingness to take an active position
of interest in possible improvements in Kazakhstani public administration will be an
encouraging element in their affirmative participation decision. I also assumed that
documents to be reviewed during my research, such as performance audit reports of the
Accounts Committee and Revision Commissions, are properly prepared and consist of
accurate data.
Last but not least, I assumed that the practice of using performance audits by
public sector audit organizations, auditees, and legislative bodies in Kazakhstan will be
positively affected by the results of my proposed study. I expected that, based on my
research results, I will be able to formulate concrete theoretical and practical implications
regarding the demand for and impacts of performance auditing on Kazakhstani public
administration and associated positive socials changes.
Scope and Delimitation
In this study, I focused on the demand for performance audits conducted by
external public sector audit organizations in Kazakhstan. Using the performance audits or
their certain instrumentations by internal auditors or authorized government bodies within
the public sector (for instance, the Ministry of National Economy of the Republic of
Kazakhstan has powers to evaluate the effectiveness of government entities’ activity), and
organizations of the quasi-public and private sectors was out of the scope of my research.
Generally, the performance auditing influences all areas of human being, directly
or indirectly affecting the interests of all citizens in Kazakhstan. However, to effectively
conduct my research, I focused on the impact of performance audits on public
administration system or, in detail, on public sector audit organizations, auditees, and
legislative bodies. I explored the perceived effects primarily relying on experience and
opinion of participants of performance auditing, such as performance auditors, managers
of auditees, and parliamentarians. Thus, my research confined to specific groups of users
of performance audits in Kazakhstan.
The list of my research participants consisted of representatives of the Accounts
Committee and the Parliament of Kazakhstan, i.e., single organizations of their kind, as
well as representatives of one from 17 Revision Commissions and one from 17 local
legislative bodies. I selected the Revision Commission on Nur-Sultan city and the
Maslikhat on Nur-Sultan city (the local legislative body) applying the following selection
criteria:
▪ performance auditing scope shaped by the size of the economy and allocated
budget funds within the defined region;
▪ study participants availability (unlike other regions, there are minimum business-
trips of employees of entities in Nur-Sultan city due to their location and hosting
both governmental and parliaments’ events in the capital city). The list of the
research participants also consisted of representatives of two from 32 central
government organizations and two from more than 400 local executive
organizations.
The scope of my research was also delimitated by a specified timeframe. Despite
the first application of performance audits in Kazakhstan dated back to 2002, I focused on
performance auditing practice specifically between 2016 and 2019 (i.e., selection of the
study participants experienced in performance audits conducted during that timeframe;
reviewing performance audit reports of the Accounts Committee and Revision
Commissions and other related documents issued in 2016–2020). The rationale for this
delimitation was that the rules, methodology, and practice associated with performance
auditing significantly differed until the adoption of the 2015 Public Auditing Act.
Limitations
The total number of my research participants was 14, with a distribution across
five public auditors, five managers of auditees, and four parliamentarians. It was
anticipated that this sample size and distribution was sufficient to conduct my qualitative
phenomenological study to the point of content saturation; however, there were some
limitations associated with the generalizability of research findings because of a small
sample (Frankfort-Nachmias & Nachmias, 2008; Patton, 2015). Specifically, auditors
who participated in performance audits before 2015, auditors from 16 Revision
Commissions, parliamentarians from 16 local legislative bodies, auditees’ managers and
parliamentarians without experience on being involved in performance audits, and
managers of more than 400 central and local executive organizations were out of the
defined sample. Whereas performance audits differ depending on the level of public
management (central or local) and environmental (organizational and cultural) aspects
(Glynn, 1995; Hossain, 2010; Lonsdale, 2008; Morin, 2016), experiences of my study
participants may significantly differ from the other non-selected participants involved in
their unique performance audits in Kazakhstan. I managed this limitation by applying
secondary data and proposing related areas for future research extension.
Given that all study participants were public servants there were risks of the lack
of availability for interviews or follow up communication for content clarity and member
checking of transcribed data. To minimize these risks, I prepared a reserve list of the
study participants and agreed on the intensity of our communications in advance.
Phenomenological interviewing is a complex process of the external intervention
for the study participants (Bevan, 2014; Patton, 2015). Interviewees may experience
difficulties, especially in situations when they need to share unsuccessful or negative
experiences. Or, due to personal or political reasons, my study participants may not
disclose information that is important to the study. Additionally, my study was associated
with a potential for researcher bias given that I have more than 18 years of experience in
the public auditing area, including serving as both employee and manager of the Accounts
Committee, advisor of the Accounts Committee and Revision Commissions, trainer of
Kazakhstani public auditors’ certification program, regional public auditor advisor in
Central Asian republics, member of working groups and consultative committees. As
such, my insight into data interpretation may carry an interpretive cloud that would not
otherwise be found in an external individual researcher. I am no longer an employee of
any Kazakhstan public audit organization nor any government entities; therefore, I no
longer have official powers to influence the research participants participation. I hoped to
use my knowledge of this specific public administration in order to gain sufficient
understanding that I needed manage and limit any associated power distance between
myself and my research participants (see Patton, 2015, for more).
There was one more limitation associated with the research and research
participants biases. A phenomenological inquiry implies using interpretations
(Moustakas, 1994). I was dealing with the following four chains of interpretations: (a) the
research participants’ interpretations associated the studied phenomena; (b) my
interpretation of the research participants’ vision, experiences, and perceptions; (c) my
interpretations associated the studied phenomena (i.e., a researcher’s active and prolonged
engagement into the study); and (d) my interpretation of documentary data. As such, my
qualitative phenomenological study may be associated with the subjectivism
(Mills & Birks, 2017; Moustakas, 1994).
Significance
What was not addressed in the literature is why performance auditing is in
demand in Kazakhstan and what are the perceived effects of performance audits on
Kazakhstani public administration. Hay and Cordery (2018) argued that, whereas
performance audits are used by many countries to improve the governance within the
public sector, extensive studies on demand for performance auditing are needed. Many
scholarly works conducted beginning in the 1970s evidenced that performance audits
differ depending on the country where there used (Glynn, 1995; Hossain, 2010;
Reichborn-Kjennerud, 2013b; Morin, 2008, 2016). Thus, available researches relating to
the impact of performance auditing on auditees and public administration, in general, may
not be applicable for countries uncovered by these researches (Reichborn-Kjennerud &
Johnsen, 2018). My study, therefore, was significant in addressing the mentioned gap
since it was the first-time study aimed at exploring the case of Kazakhstan. As
performance auditing is one of the widely recognized mechanisms for examining public
policy formulation, implementation, and evaluation (The Organization for Economic
Cooperation and Development (OECD), 2016), my study seeks to capitalize and expand
on this phenomenon by exploring the impact of performance audits specific to
Kazakhstani public administration, a unique experience.
For today’s Kazakhstani public administration system, the performance auditing is
a novelty. The intensive use of performance audits stimulated by the adoption of the
Conception on Introduction of Public Auditing in Kazakhstan, known as the 2013
Conception (ISLARK, 2013), the 2015 Public Auditing Act, and development of
methodology. This trend preceded by pilot efforts of the Accounts Committee and
fragmental practices of the Accounts Committee and Revision Commissions on
incorporation audit questions on efficiency evaluation of auditees’ operations into the
traditional compliance audit programs. Unlike developed countries with more than 50
years of performance auditing experience, performance auditing in Kazakhstan has been
conducted only recently. For countries like Kazakhstan (e.g., other Post-Soviet or newly
emerged countries) where the performance auditing is an innovative practice, examining
the impact of performance audits is especially relevant for capacity building in this
emerging territory of public administration audit organizations.
It was the first case of exploring the perceptions of (a) auditors as principals and
sources of influence, (b) auditees as agents and targets of influence, and (c)
parliamentarians as principals for both auditors and auditees (see Morin, 2001, 2008;
Reichborn-Kjennerud, 2014b, for more) within performance auditing processes in
Kazakhstan. Understanding the perceived impacts of performance audits in Kazakhstan
and whether the experience of developed countries in performance auditing, such as the
performance audits’ contributions to better changes in auditees’ performance or to more
accountable and efficient governance within the public sector (Desmedt et al., 2017;
Funnell, 2015; Morin, 2008, 2014, 2016; Reichborn-Kjennerud & Johnsen, 2018; Torres
et al., 2019) are truth or applicable for Kazakhstan’s case may inform national auditors
about well-tested approaches on performance auditing and help them to identify new
strategies to achieve positive impacts on public administration through their audits.
Findings related to factors that might explain the demand for performance audits
may allow the public sector audit organizations to take them into account in the course of
their operational activities. Thus, defining new information about the demand for
performance auditing and its possible impacts on Kazakhstani public administration may
help both the Accounts Committee and Revision Commissions to establish proper
policies, strategies, and methodologies associated with their audits. The findings of my
study may also help auditees and legislative bodies to establish policies on the
incorporation of the performance auditing into their organizations’ strategies stimulating
the credible performance audits (see Funnell & Wade, 2012, for more); as a result,
contributing to sustainable national fiscal responsibility and effectiveness.
My study also may contribute to positive social changes. Firstly, the results of the
study have the potential to influence national auditors’ perspectives through exploring
their experiences in mastering a new and challengeable profession, i.e., performance
auditor. Secondly, performance auditing is a novelty for managers of auditees and
parliamentarians in Kazakhstan, i.e., the persons who can use the performance auditing as
an opportunity for learning (Reichborn-Kjennerud, 2013a). Therefore, my study may also
influence their perspectives by providing them with new insights relating to usefulness or
limitations of using performance audits to implement their daily duties.
Summary
Performance auditing is the most demanded type of public sector audit
organizations’ activity that have emerged in response to increasing the needs in the more
accountable and effective management of the public resources and taxpayers’ money.
Exploring the causes of demand for performance audits and their impact on public
administration is a topical and perspective objective for both scholars and practitioners.
This objective is a critical especially for Kazakhstan where today, like in developed
countries several decades ago, the performance auditing has become a widespread
practice; however, there were no empirical studies on the demand for and impact of
performance audits on public administration in Kazakhstan. The lack of knowledge about
the studied phenomenon increased risks of missed opportunities of the Accounts
Committee, i.e., SAI of Kazakhstan, and local revision commissions to increase
accountability of governments and improve the public sector performance through their
audit works.
In Chapter 1, I presented the rationale for my study aimed at exploring the causes
of demand for performance auditing in Kazakhstan and the perceived effects that
performance audits have on Kazakhstani public administration. I described my vision on
combined applying the new public management theory and the principal-agent model as
the theoretical lens to guide an entire research process. I also rationalized applying the
qualitative method with the phenomenological approach as the most suitable research
design to do my study. The key term definitions, assumptions, scope and delimitations,
and limitations associated with the proposed research are provided as well in this chapter.
In Chapter 2, based on the literature review results, including the
conceptualization of performance audits, describing the emergence and development of
performance auditing practice around the world and in Kazakhstan in particular, I
formulated the identified gap in the literature associated with the studied phenomena.
Chapter 2 also includes descriptions of the theoretical framework of my study. In Chapter
3, I rationalized the selected research design and approach, namely – a qualitative
phenomenological study. Descriptions regarding the selection of research participants, my
roles as the researcher, procedures on data collection and analysis, and measures on
managing the issues of trustworthiness and ethical issues are included in Chapter 3 as
well.
Chapter 2: Literature Review
Introduction
In this chapter, I present the results of reviewing and analyzing the literature about
the performance auditing phenomenon, and its emergence and impact on public
administration. I proposed a modified definition of a performance audit consistent with
recent theoretical and practical developments. Performance audits are carried out by
public sector audit organizations including SAIs, joined within INTOSAI, and local audit
organizations. I identified the current triggers for performance auditing based on these
organizations and INTOSAI’s approaches to contribute to good governance. I also
described four ways, i.e., instrumental, cognitive (conceptual), interactive, and strategic,
and three levels, i.e., micro-, meso-, and macrolevels, of targeting public administration
through performance audits.
In my research, I focused on the case of Kazakhstan, a young Central Asian state,
which gained independence after the 1991 collapse of the Union of Soviet Socialist
Republics (USSR). I explored why performance audits are in demand in Kazakhstan and
how auditors, managers of auditees, and parliamentarians perceive these audits’ impact on
Kazakhstani historical and present public administration approaches. Despite the short
historical period, establishing and developing the public administration system in
Kazakhstan are complicated processes and subjects for separate studies. Therefore,
instead of detailed descriptions associated with these processes, I focused on the key
milestones and characteristics of Kazakhstani public administration, which are significant
for understanding the studied phenomena, as well as focusing on performance audits
conducted by SAI and the local audit organizations.
Literature Search Strategy
The main literature sources of my study were peer-reviewed articles, theoretical
texts, dissertations, books, international professional organizations’ research and policy
papers specific to the topic, performance and audit reports of public sector audit
organizations. To find the literature on performance auditing, focusing on performance
audits’ impact on public administration, I used databases, such as SAGE Journals
(formerly, SAGE Premier), Political Science Complete, Taylor and Francis Online,
Public Administration Abstracts, ProQuest Dissertations & Theses, ScienceDirect, and
EBSCO Host. I also referenced the following journals: Critical Perspectives on
Accounting, Evaluation, Financial Accountability & Management, Government Auditing
Review, International Journal of Auditing, International Public Auditing Journal,
Managerial Auditing Journal.
I used the reference lists of recent publications, Google Scholar, and the
Find@Walden Tool of the Walden Library to find specific articles, theoretical texts, and
books. I used official websites of international organizations including INTOSAI, internet
sources of INTOSAI’s members including SAI of Kazakhstan, and websites of
Kazakhstani executive bodies to find official papers related to performance auditing.
In the beginning of my search, I used dates that ranged from 2015 to 2020, to
examine the literature on recent developments in an area of performance auditing.
Thereafter I did not limit the literature search by dates of publication, in order to make
appropriate historical references and find original theoretical works tied to the topic. I
limited the search for laws and regulations, audit and performance reports of public sector
audit organizations in Kazakhstan, and sources about Kazakhstani public administration
to dates between 2015–2019. Then, I expanded the search for that sources to dates
between 1991–2020 to analyze the perspectives of public administration in Kazakhstan’s
contemporary history and introducing the performance audits.
The keyword search terms included: performance auditing, performance audit,
value for money audit, public audit, public sector audit organizations, supreme audit
institutions, public auditors, auditees, public administration, impact of performance
audits on public administration, public administration in Kazakhstan, and performance
auditing in Kazakhstan. Referenced sources were in the English, Kazakh, and Russian
languages. None of the searches revealed empirical studies regarding the demand for and
impact of performance audits on public administration in Kazakhstan. To describe the
theoretical framework, I used the key search terms: new public management,
principalagent model, new public management and performance auditing, and principal-
agent model in the public sector. I also used SAGE Research Methods Online to find
sources regarding the selected qualitative phenomenological research method.
Conceptualizing Performance Audits
The 1970s reforms of public administration, entitled new public management,
were aimed at adopting managerial techniques of private companies within the public
sector and determined new starting points for scholar-practitioner led performance
auditing (Funnell, 2004a, 2004b, 2015; Reichborn-Kjennerud, 2013a). The extensive use
of performance audits, as these audits are interpreted today, started in the 1970s (Glynn,
1985; Levy, 1996; Pollitt, 2003); however, the new public management initiatives likely
classify the conceptualization of performance audits as only partially complete.
Performance auditing is not a new phenomenon (Flesher et al., 2003; Glynn,
1985). Mandates of auditors expanded from traditional examinations of auditee’s
operations to efficiency evaluations even before the appearance of managerial trends
within the public sector. One of the pioneers in using performance audits, or more
precisely predecessors of performance audits, is the United States (Flesher et al., 2003).
The predecessors of performance auditing in the United States are operational or
management audits. Operational auditing is a systematic nonfinancial examination of an
entity’s operations conducted for improvements (Flesher & Zarzeski, 2002), and it is
interpreted as a synonym of Dittenhofer’s (1971) performance auditing, incorporating an
improved degree of accuracy. In historical chronicles the concept of operational auditing
dates from the 12th century and starting from the 1940s operational audits have been used
by private sector internal audits with advancement into the public sector (Flesher et al.,
2003).
Another unique feature of the United States’ case is that within the public sector,
performance audits are firstly legitimated and used by states, i.e. at local levels. For
instance, beginning in 1963, public auditors of the state of Michigan were impowered to
conduct performance audits (Dittenhofer, 1971). It required a further 10 years before the
United States General Accounting Office’s (presently, the United States Government
Accountability Office [GAO]) the federal level audit organization, was granted the same
powers. In line with the 1972 GAO’s standards, public auditing relates to examinations of
both financial operations and economy and efficiency in achieving the purposes of
governmental programs (GAO as cited by Flesher & Zarzeski, 2002).
Along with the United States, Canada contributed to the modern concept of the
performance audit. Initially, performance audits in Canada were named comprehensive
audits (Flesher & Zarzeski, 2002; Glynn, 1985). Unlike the operational auditing in the
United States, comprehensive auditing emerged within Canada’s public sector and
specifically aimed to establish whether the public sector organizations have operated with
due regard to economy, efficiency, and effectiveness (Glynn, 1985).
The United Kingdom also significantly contributed to making performance
auditing a self-sufficient concept and discipline. Performance audits in the United
Kingdom, called value for money audits, first debuted in the 1880s (Glynn, 1985; Lapsley
& Pong, 2000). Like the United States, but unlike Canada, the United Kingdom’s
performance audits were aimed specifically to the examination of public fund use with
due regard to economy and efficiency. According to Glynn (1985), value for money
audits contribute to realizing the rights of taxpayers to receive efficient public services at
minimal costs as a part and condition of the country’s social contract. Thus, the key
feature of value for money audits is its strict focus on the needs and expectations of, and
by, public services; the National Health Service being an illustrative example.
Australia, New Zealand, some European and Asian countries also started use of
performance audits in the 1970–1980s, adding country features into the concept
(Grönlund et al., 2011; Hossain, 2010; Jacobs, 1998; Yamamoto & Waranabe, 1989).
Nevertheless, historical and international perspectives are not the sole determinants of
performance audits. The methodology drives changes in the interpretation of performance
audits including objectives, scopes, and methods of these audits (Alwardat et al., 2015;
Lonsdale, 2000). Kells and Hodge (2011) offered that intellectual and technological
developments also shape the concept of performance auditing.
A primary objective of performance audits is to establish whether the executes
have functioned with due regard to: a) economy, commonly referred by audit
professionals as 1st E, which means using minimum public funds to produce the required
volume of outputs with an acceptable level of quality; b) efficiency, referred as 2nd E,
which means getting the most from the available resources; and c) effectiveness, referred
as 3rd E, which means which means meeting the objectives set and achieving the
intended results (INTOSAI, 2019). According to Barrett (2010, 2011a), the scope of
performance audits can be confined to an audit of one or more selected functions of
auditees, management of governmental programs and other undertakings. The list of
methods of performance auditing consists of documentary reviews, questionnaires,
surveys, site visits, focus groups, expert panels, and international comparisons (Tillema &
Bogt, 2010).
Performance auditing means influencing auditees and other users of performance
audit reports to add value in terms of better governance within the public sector. This
view is shared by many scholars, including those who stressed that today’s performance
audits are conducted with both performance improvement and transparency goals
(Alwardat et al., 2015; Barrett, 2011b, 2012; Funnell, 2016; Hossain, 2010; Morin, 2016;
Reichborn-Kjennerud, 2013a, 2013b). In turn, the adding of value through performance
audits is possibly subject to both public sector audits organizations and auditors’
independence (Funnell et al., 2016; Morin, 2016). Thus, the concept of performance audit
has significantly evolved. As described in Chapter 1, I used the following definition of
performance audit developed based on recent interpretations (see Alwardat et al., 2015;
Desmedt et al., 2017; Funnell & Wade, 2012; Morin, 2016; Reichborn-Kjennerud &
Johnsen, 2018, for more) and INTOSAI’s definition as well. Performance audit is a
legitimate professional activity on examination of whether an audited area of public
administration or an auditee’s operations, systems, and activities meet the principles of
economy, efficiency, effectiveness, and other criteria of good governance. Performance
audits are carried out by public sector audit organizations to promote improvements in
public administration (INTOSAI, 2019, p. 8).
Performance Audit Institutions
Performance audits are carried out by audit organizations labeled SAIs, such as
the United States Government Accountability Office, the Office of Auditor General in
Canada, the National Audit Offices (NAO) in the United Kingdom, other European
countries, Australia, New Zealand (INTOSAI, 2004, 2013a), and the Accounts
Committee in Kazakhstan (ISLARK, 2013, 2015). In some countries, including
Kazakhstan, local public sector audit organizations also have the power to conduct
performance audits (ISLARK, 2015; Torres et al., 2019). Internal auditors and
governments’ evaluation agencies may also conduct performance audits or use certain
techniques of performance auditing to evaluate (Barton, Aibinu, & Oliveros, 2019;
Kweun, Wheeler, & Gifford, 2018). In my study, I focused on SAIs and local public
sector audit organizations’ perspectives since they are today's leaders in the use and
development of performance audits (see Torres et al., 2019, for more). According to Tara
and Gherai (2014), the appearance of SAIs or, more precisely, their predecessors, dates
from 14th century; as demonstrated in Figure 2, modern SAIs were established in 18th–
20th centuries, and today they operate in more than 190 countries.
Figure 2. The chronological map of appearance of SAIs
Note: Developed based on INTOSAI (n.d.).
Scholar-practitioners call SAIs watchdogs that oversee public finances (Jantz,
Reichborn-Kjennerud, & Vrangbaek, 2015; Morin & Hazgui, 2016; Norton & Smith,
2008). SAIs may operate as external audit bodies at central and local government levels
or as the only national SAIs with local branches (INTOSAI, n.d.).
Table 1 illustrates three models of SAIs.
Table 1
The Models of SAIs
Court (Napoleonic) Model
Westminster Model
Collegial (Board) Model
The key characteristics
Public sector audit
organizations granted with
judicial powers
Public sector audit
organizations leaded by
Auditors General
Public sector audit
organizations governed by a
collegial body which is
leaded by chairpersons
Countries
Brazil, Belgium, El Salvador,
France, French-speaking
Africa and Asia, Greece,
Italy, Portugal, Spain, Turkey
Chile, Colombia, Cyprus,
Denmark, Estonia, Finland,
Hungary, Ireland, Latvia,
Lithuania, Malta, Mexico,
Peru, Poland, South Africa,
United Kingdom
Argentina, Czech Republic,
European court of audit,
Germany, Luxembourg,
Nicaragua, Netherlands,
Slovak Republic, Indonesia,
Korea, Japan
Note: Developed based on Tara & Gherai (2015), INTOSAI (2004, 2013).
SAIs differ by status, mandates, accountability approaches, and resources; their
distinctive features depend on the state order, public administration structure, and
separation of powers in nations where they operate. For instance, some SAIs conduct
audits in all organizations, whereas others’ powers are limited to audits of government
entities (INTOSAI, 2013). Thus, the classifying of SAIs by Napoleonic, Westminster, and
Board models may be useful in understanding their place within the public
administration, but they are typically symbolic in nature. Most SAIs are independent of
the executes, accountable for non-executive elected institutions or officials, and conduct
performance audits (INTOSAI, 2013, 2018). As an independent public institution, SAIs
play an important role in public administration in democratic societies (OECD, 2016).
The expectations regarding SAIs’ roles as contributors to democratic
accountability is strengthened as their power expanded from traditional compliance and
financial audits to performance audits (Funnell, 2015; Morin, 2016). The 2011 Resolution
of the 66th United Nations (UN) General Assembly entitled “Promoting the Efficiency,
Accountability, Effectiveness and Transparency of Public Administration by
Strengthening Supreme Audit Institutions” supports this argument (UN, 2011). This and
other global initiatives, such as promoting the principles of good governance, citizen
participation in government auditing, and Sustainable Development Goals (SDG), have
been implemented by SAIs’ through their performance auditing functions (Baimyrzaeva
& Kose, 2014; OECD, 2015, 2016).
Like SAIs, local public sector audit organizations differ by status, powers,
requirements to their accountability, organizational capacities, and experiences in using
performance audit formats (English, 2003; Rosa, Morote, & Colomina, 2014; Tillema &
Bogt, 2016; Torres et al., 2019). Typically, SAI and local audit organizations’ powers on
performance auditing are distributed depending on the separation of powers between
central and local governments (Torres et al., 2019). However, there are no significant
differences in implementing performance audits by SAIs or local audit organizations.
This similarity is due to the application of common auditing standards and regulations,
sharing information and cooperation between SAIs and local audit organizations. For
example, in the United States all public sector audit organizations typically follow the
requirements of the Government Auditing Standards issued by GAO (USGAO, 2018). To
not follow these industry-expected standards would result in an organization being
subjected to added scrutiny whether justified or not.
Current Triggers for Performance Auditing
Identification of triggers for performance auditing contributes to a better
understanding of why performance audits are in demand. Current triggers for
performance auditing, such as exogenous and endogenous circumstances that rationalize
SAIs’ audit practice, typically yield more performance audits by volume. As outlined in
previous paragraphs, performance auditing enshrined as SAIs’ statutory power
consequent to the transformation of public administration, may be an initial trigger for a
widespread using of performance audits. This trigger has become classical, yet it remains
a standard rationalization term of performance auditing. Similar to the 1970s, today’s
performance audits are initiated and carried out in response to community and political
concerns regarding the effective administration of public affairs and delivering
anticipated public services (Hossain, 2010).
But there are also other exogenous circumstances that stimulate the initiation of
performance audits. Torres et al. (2019) argued that performance audits received more
attention after the 2008 Global Financial Crises. The crises trigger performance audits
because of increasing the risks of ineffective implementation of governmental tasks
including anti-crisis measures. Governments use privatization or public-private
partnership (PPP) mechanisms to cover the budget deficit to be able to deliver public
services under the tight financial constraints (OECD, 2012). Both privatization and PPP,
according to the fundamental principles of performance auditing, should be determined as
high-risk areas (INTOSAI, 1998, 2004, 2007). Thus, SAIs with well-developed
performance auditing methodology and practices respond to these governmental anticrisis
measures by carrying out performance audits of PPP programs and projects. Beginning in
the 2010s, many SAIs have included performance audits of privatization and PPP in their
audit portfolios (Barrett, 2011a; The Canadian Audit and Accountability Foundation
(CAAF), 2015; The European Court of Auditors (ECA), 2018). There are also other
external circumstances that lead to more performance audits in an area of public financial
administration when ministries of finance or government entities with the same powers
are core auditees who, in turn, are visited by performance auditors as a matter of public
policy, law, or statutory regulation.
Structural problems inherent in different public administration areas also increase
the number of performance audits. The list of these problems includes imbalances of
economy, structural flaws in social and health care systems, and low productivity
(Ahrend, Saia, & Schwellnus, 2017; Hugh-Jones, 2012). Unlike cyclical financial crises,
structural problems are systematic, and long-running measures including executives’
programs are required to address these problems. Executives’ programs are often
subjected to enhanced performance audits both in content and frequency (INTOSAI,
2016a). For instance, GAO has issued more than 300 performance audit reports and
related papers regarding the implementation of the Medicare and Medicaid Programs in
the last five years likely in relation to public scrutiny and need for information
transparency with the implementation of the Affordable Care Act in 2010 (USGAO,
n.d.). Innovations used to solve the structural problems also trigger new performance
audits. In particular, utilizing the new information and communication technologies by
governmental organizations lead to conduction more performance audits in the IT area.
For instance, NAO issued more than 100 performance audit reports and related papers
regarding the communications and digital government (UKNAO, n.d.).
Natural and anthropogenic disasters trigger performance audits as well. In the
language of the International Standards of Supreme Audit Institutions (ISSAIs), these
audits are called performance audits with environmental perspectives (INTOSAI, 2016b).
Environmental performance audits are conducted by SAIs to establish whether the
executives have functioned with due regard to the economy (i.e., 1st E), efficiency (2nd
E), and effectiveness (3rd E), as well with due regard to the environment ([4th E];
Leeuwen, 2004; Weihrich, 2018). According to INTOSAI’s Working Group on
Environmental Auditing ([WGEA]; 2015), the number of SAIs that conduct
environmental performance audits increases almost every year with specific focus on
topical environmental issues, such as climate change, environmental pollution, and
nonrenewable energy use. One of the explanations for the intensive use of performance
audits is the general public’s concerns about the negative impact of the environmental
issue on public health. Thus, natural and anthropogenic disasters trigger more
performance audits in an environmental area when ministries of healthcare and other
government entities, such as SAIs and local audit organizations’ clients, realize and fulfil
their executive functions.
Further, global environmental challenges may trigger international performance
audits with environmental perspectives conducted by SAIs collaboratively (WGEA,
2012, 2015). As for today, the scope of environmental performance audits has been
expanded to auditing governments’ preparedness to efficiently and effectively implement
SDG (UN & INTOSAI, 2018). Like SDG, other global initiatives are in the list of
exogenous circumstances that lead to intensive use of performance auditing. In my
research, I focused on current triggers for performance audits in areas when the Ministry
of Finance of the Republic of Kazakhstan and the Ministry of Healthcare of the Republic
of Kazakhstan are core executive authorities.
Targeting Public Administration through Performance Audits
SAIs conduct performance audits “to promote economical, effective and efficient
governance. It also contributes to accountability and transparency” (INTOSAI, 2019, p.
8). In pursuant of this objective, SAIs, as well as local public sector audit organizations,
impact public administration through their performance audits (Desmedt et al., 2017;
Morin, 2016; Reichborn-Kjennerud & Vabo, 2017; Torres et al., 2019). Impacting the
public administration through performance audits implies contribution to positive
changes in auditees’ activity or an audited area of public administration and improving
the public sector performance in general (Lonsdale, 2000; Morin, 2001).
Instrumental, Conceptual (Cognitive), Interactive, and Strategic Influences
There are several categories of impacts of performance auditing on public
administration. These categories differ depending on the use of performance audits and
levels of impacting the public administration. Performance auditing implies conduction of
evaluation and, therefore, categories of evaluation use are applicable for performance
audits (Lonsdale, Wilkins, & Ling, 2011). Lonsdale et al. (2011) identified instrumental,
conceptual, political-legitimizing, and tactical use of performance audits (pp. 180-181).
The instrumental use of performance auditing is a linear process converting
evaluation to knowledge and, then, converting knowledge to policy (Lonsdale et al.,
2011). This type of evaluation use means applying knowledge for actions, such as
decision-making and problem-solving (Alkin & King, 2016). Based on knowledge
acquired in the course of an audit, performance auditors provide managers of auditees and
other concerned parties with recommendation on the elimination of the revealed
shortcomings and further improvements. Thus, giving recommendations by auditors is the
central tool of instrumental use of performance audits (Lonsdale et al., 2011; Morin,
2008). Depending on the context of recommendations performance auditors impact
decision-making and contribute to problem-solving (Desmedt et al., 2017).
The conceptual use (impact) of performance audits means embedding knowledge
through different channels that leads to change in mental or intellectual frames (Lonsdale
et al., 2011). In the course of evaluations, knowledge is used to influence policymakers’
thinking (Alkin & King, 2017). Or, in other words, evaluators (performance auditors)
have a cognitive impact on auditees’ managers and other officials with power in policy
making within the audited area of public administration. The conceptual (cognitive)
impact is manifested through both visible and invisible changes, challenging the
identification and measurement of that impact (Desmedt et al., 2017; Lonsdale et al.,
2011).
Policymakers constructively respond to evaluations when evaluators provide them
with a new vision regarding the existed problems, or ‘enlighten’ them, and suggest new
perspectives on solving these problems (Weiss as cited by Alkin & King, 2016). In the
language of performance auditing, auditees’ managers constructively respond to
performance audits when performance auditors provide them with new information (or
previously misunderstood information) about the revealed shortcomings in effectively
managing the auditee’s affairs.
The interactive use explained by the fact that performance audits are not the sole
influence factor since other participants or sources of information join the influence
process as well (Lonsdale et al., 2011). In that case, knowledge used in conjunction with
personal insights and experiences of decision makers and communicated information
(Alkin & King, 2016). Users of evaluation verify the evaluation findings and, therefore,
the interactive use contribute to the evaluation credibility and informed decision-making
(Alkin & King, 2016). Auditees have rights to react on performance auditors’ opinions
and, in turn, performance auditors have obligations on responding to auditees’ reactions
(Alwardat et al., 2015; INTOSAI, 2019). Other concerned parties, from public officials to
citizens, may also join this communication process (Lonsdale et al., 2011).
According to Lonsdale et al. (2011), the political-legitimizing use of performance
audits implies the engagement of mass media and parliamentarians in performance
auditing that contributes to political debates, while the tactical use implies impacting the
decision making by auditees. The political-legitimizing use relates to using the evaluation
findings to rationalize earlier decisions and legitimize reforms (Alkin & King, 2016,
2017; Breidahl, Gjelstrup, Hansen, & Hansen, 2017). This type of use is close to the
symbolic use of evaluation. According to Alkin and King (2016), the symbolic use means
using the evaluation findings to support a political position or rationalize previous
decisions. Further, SAIs may plan and conduct their performance audits with intends to
legitimize policy; the political-legitimizing use occurs because of the nature of audited
issues or without SAIs’ political intentions (Funnell, 2015; Morin, 2016). SAIs’
performance audits may influence decisions regarding the prolongation, or opposite,
cancelation of governmental programs. GAO’s practice on annual performance audits
conducted to reduce fragmentation, overlap, and duplication of federal programs
(USGAO, n.d.) is one of the examples of the political-legitimizing use of performance
audits by SAIs.
In my study, I applied the strategic use of performance audit, which is the
combination of legitimize, symbolic, and tactical uses, as measuring units of analysis.
The strategic use of performance auditing means using the audit findings to contribute to
political debates and, as a result, to promote reforms, which are legitimate and rational
from the perspectives of good governance (Desmedt et al., 2017, INTOSAI, 2019).
Impacting the Micro, Meso, and Macro Levels
According to Lonsdale et al. (2011), performance auditing impacts the microlevel
(impacting individual performance audit itself), mesolevel (impacting audit organizations
and auditees), and macrolevel (impacting the public sector) of government and the wider
concept of public policy. Methodological aspects explain the impact of performance
audits at the microlevel. Unlike traditional compliance and financial audits, performance
audit is not a standardized process and, therefore, it implies frequent exercising
performance auditors’ judgments (Funnell et al., 2016). Each individual performance
audit is unique. Unlike compliance and financial auditors, performance auditors more
frequently are dealing with new knowledge regarding both the audited area of public
administration and implementation of audit procedures. Therefore, methods and findings
of a performance audit in tandem with communications between auditors and auditees
shape the individual performance audit (Lonsdale et al., 2011).
As SAIs and auditees gain experience on being involved in performance auditing,
they change and advance their internal policies and processes. These organizational
changes are examples of impacting the mesolevel related specifically to a performance
auditors’ interventions (Desmedt et al., 2017). From one side, SAIs’ reputations, powers,
and accountability, and from other side, auditees’ policy on knowledge management,
familiarity with performance auditing and attitude toward knowledge facilitate impacts of
performance audits at the mesolevel (Lonsdale et al., 2011).
Impacting the macrolevel implies changes of public administration in general.
Individual performance audits and SAIs’ practice on using the findings from all
individual performance audits to present a holistic (comprehensive) view regarding the
audited issues facilitate changes in the public sector (OECD, 2012). According to
Lonsdale et al. (2011), obligations to assess policy on a regular base and attitude toward
knowledge facilitate impacts of performance audits at the macrolevel.
Performance audits are not the sole factor that influences public administration; it
is difficult to single out performance audit’s influence among other influential factors
including changes in public policies initiated by the executes themselves. Unlike
impacting the micro- and mesolevel, impacting the macrolevel is more complex in terms
of its identification and measuring (Lonsdale et al., 2011). Integrated application of
instrumental, conceptual (cognitive), interactive, and strategic uses of performance audits
facilitates measuring their impact at macrolevel. Today, this approach is fragmentarily
used by some SAIs and realized by developing SAI’s Performance Evaluation
Framework (INTOSAI, 2016c).
Performance Auditing in Kazakhstan
Historical Perspectives of Kazakhstani Public Administration
Kazakhstan is a young state located in Central Asia with a unique history of
nation-building starting in 1991 with the dissolution of USSR. Establishing the public
administration system of a new sovereign state was complicated by the negative
consequences of the economic crisis inherited from the Soviet Union (Nazarbayev, 1992,
1994, 2017). At the initial stage, economic reforms were the highest priority opposite to
political and public management reforms (Knox, 2008; Nazarbayev, 2017).
The economic reforms aimed at implementation of a free market economy and
supported by abundant natural resources, such as oil, gas, uranium, copper, zinc, and
other minerals that are in demand on world markets, resulted in positive social changes
but ones that needed management to avoid exploitation (OECD, 2014, 2017a, 2017b;
Vakulchuk, 2016). Today, national government has gained experience in economic
reforms and these strengthened national institutions aid in smoothing any global financial
crisis impacts in which Kazakhstan now stands alone as a sovereign state.
As for today, Kazakhstan is categorically identified as a faster-growing economy
that demonstrates good results in macroeconomic data (Nazarbayev, 2017; OECD, 2017a;
Vakulchuk, 2016). Figure 3 illustrates the growth of the Kazakhstani economy resulting
expanded capacities of the government to deliver more public services and increase
citizens’ incomes, thus improving the quality of citizen’s lives (Nazarbayev, 2017;
OECD, 2014, 2017a).
Figure 3. The dynamic of GDP and GDP per capita Kazakhstan in 1991–2019
1 056.9
3 707.4
6
17 007.
68 639.5
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
2013
2015
2017
2019
GPD per capita (mln. tenge)
GDP (trln. tenge)
Note: Developed based on SCMNERK (n.d.).
The Declaration of state sovereignty (1990), Law on state independence (1991),
and adopting the national Constitution (1993) were the first legal acts of the new state
specific to planned and implemented political changes (Kemel & Shaikenova, 2015;
Shomanov et al., 2005). Ideas incorporated into the first Constitution were implemented
partially because of confrontations and resistance to political transformations
(Nazarbayev, 2017). Constitutional reforms, adopted in 2015, serve as the fundamental
principles and framework for today’s public government. Accordingly, “Kazakhstan
proclaims itself as a democratic, secular, legal and social state whose highest values are a
person, his life, rights, and freedoms” (Constitution, 1995, para. 1). In 1995, Kazakhstan
declared itself as a unitary state using a presidential form of government (Constitution,
1995).
During the early days of Kazakhstan’s political transformations changes in power
distribution between executive, legislative and judicial bodies, and dividing the expert
community into supporters and critics were primary political and social activities (Knox,
2008; Mesquita, 2016; Nazarbayev, 2017; Shomanov et al., 2005). Key Presidential
milestones occurred: (a) the President was vested with the highest administrative and
executive powers in 1990; (b) the post of the President became elected beginning in
1991; (c) till 2017 the Presidents’ powers have been expanded; and (d) part of the
Presidents’ powers were delegated to the Government in 2017 (ISLARK, 1995a;
Nazarbayev, 2017; Shomanov et al., 2005). The Presidential Administration is the
principal state entity with coordination and control functions at both the central and local
governmental levels (OECD, 2017a).
Coupled with Presidential power is the Parliament. The Parliament of Kazakhstan
is a representative body with legislative functions; it consists of the higher and lower
houses headed by chairpersons (ISLARK, 1995b). As the new government matured this
legislative body was gradually granted more power, including control and oversight
functions in fiscal administration (OECD, 2014, 2017a). The Government of Kazakhstan
(Cabinet of Ministries) directed by the Premier-Minister (ISLARK, 1995c) serve as the
supreme executive and administration bodies. The Government is a collegial body
accountable to the President and Parliament (ISLARK, 1995c). Political reforms between
1991–2018 resulted in changes in the structure of the Government, its powers and
functions; its members, i.e., ministries and agencies, have been granted more autonomy
for the last two decades (OECD, 2014, 2017a).
The same changes were implemented in terms of distribution powers between
central and local governments; it was initially prioritized establishing a vertical structure
of public administration that implies strengthening the central government and, then,
initiated decentralization that implies gradual delegation central government’s powers to
local (Bhuiyan, 2010; OECD, 2017b; Shomanov et al., 2005). Today, Kazakhstan is
divided into 14 oblasts (regions) and three cities with special status (Nur-Sultan, the
current capital renamed Astana; Almaty, the former capital; Shymkent, the second largest
city in the country by population). Local governments are directed by Akims (mayors)
who are appointed by the President and accountable to the President and Maslikhats, i.e.,
local legislative bodies (ISLARK, 2015).
Reforming public management was defined as a priority state policy in
Kazakhstan (Knox, 2008). Applying a state strategy approach and ‘importing’ NPM’s
ideas are two core aspects of reforming the Kazakhstani public administration. The state
strategy approach implies the implementation of country-development strategies which
are the highest priority papers within a hierarchy of strategic documents in Kazakhstan
(ISLARK, 2017a, 2017b). As for today, the central strategic goal of Kazakhstan is to join
the rank of the top 30 developed economies by 2050 (ISLARK, 2018). This ambiguous
and challengeable vision of the country-development outlined long-term plans on further
achievements in public governance consistent with NPM’s ideas (Linn, 2014;
Nazarbayev, 2017; Vakulchuk, 2016). NPM’s ideas, such as adopting managerial
techniques by government organizations, client- and result-oriented public service
delivery, granting executives with more autonomy, and changing civil servants’ roles and
responsibilities, shaped the public administration system of Kazakhstan (Janenova &
Knox, 2017; Oleinik, Yermekov, & Kuatbekov, 2015; Vakulchuk, 2016). According to
Oleinik et al. (2015), there are some gaps between declared objectives and achieved
results in improving Kazakhstani public administration system. Generally, the political
and public administration reforms in Kazakhstan are associated with both disadvantages
and advantages leading to related changes of the worldwide governance indicators for
Kazakhstan (OECD, 2017a).
As demonstrated in Figure 4, despite the linear trend on the Government
Effectiveness Indicator, improvements in this area of public governance are not stable.
There are several explanations for this dynamic including changes in the performance and
accountability systems. The accountability for the executives’ results were realized through
(a) new approaches to evaluation of government organizations’ performance; (b)
parliamentarians’ function on control and oversight the public funds; (c) open government
initiatives; and (d) the public auditing (OECD, 2014, 2017a, 2017b). According to OECD
(2014, 2017a), public auditing is one of the weak areas in reforming the accountability
system in Kazakhstan due to the lack of implementation of performance audits.
Figure 4. Worldwide governance indicators for Kazakhstan in 1996–2018
Note: Developed based on WB (n.d.).
Implementation of Performance Audits
Performance audits within Kazakhstan’s public sector are carried out by the
Accounts Committee and Revision Commissions. The Accounts Committee was
established in 1996 as a collegial body that consists of eight members and a chairman. It
is accountable to the President. It submits its annual report, which is the conclusion (audit
opinion) on the Government’s report on execution of the republican budget, to Parliament
for approval (ISLARK, 2002a). Revision Commissions were created in line with the
Decree of the President “On Improvement of Bodies of External State Financial Control
in Regions in 2011” (ISLARK, 2011). New commissions were created to replace the
Revision Commissions of Maslikhats that implemented local parliamentarians’ tasks
without the status of an independent body. Revision Commissions are accountable to
Maslikhats; they also submit their audit reports to SAI for coordination purposes, e.g., to
eliminate duplications in audits (ISLARK, 2015).
The starting point for implementation of performance audits was the adoption of
the 2013 Conception (ISLARK, 2013). The overarching concept aim was to replace the
system of financial control by the public auditing system consistent with international
standards and auditing practices (ISLARK, 2013). Following to the 2013 Conception, the
2015 Public Auditing Act provided the Accounts Committee and Revisions Commissions
with powers to conduct performance audits (ISLARK, 2015). Beginning in 2016,
performance auditing, implying analysis and evaluation, conducted by external audit
bodies to establish whether the executives operate with due regard to 3E, was
implemented and label as legitimate in Kazakhstan.
Like in the cases of developed countries, there were some predecessors of
performance auditing in Kazakhstan. For instance, the Accounts Committee and Revision
Commissions of Maslikhats conducted efficiency controls (ISLARK, 2002b); before
2013, it was not accepted to use the term of audit in laws, regulations, and other official
documents within the public sector. However, as it is illustrated in Table 2, similarities in
interpretations of efficiency control and performance audits became visible beginning in
2002.
Table 2
Interpretations of Predecessors of Performance Auditing in Kazakhstan
2002 Law on Control
over Execution of the
Republican Budget
2004 Budget Code
2008 Budget Code
Efficiency control is an
examination of
execution of a budget
program by controlled
(evaluated) entity in
line with a passport of
the budget program.
Efficiency control is a
checking the implementation
of actions within the
scheduled timeframe and
examination of achieving the
expected results and
indicators after the
implementation of a budget
program applying
evaluations of economy and
productivity.
Efficiency control is an examination
(conducted based on compliance control
and control over the financial
statements) of achieving by government
organizations outputs and outcomes
identified in their strategic plans, budget
programs, as well as a complex
evaluation of impacts of government
organizations and enterprises’ activity on
the economy, social sphere or a certain
area of public management.
Note: Developed based on ISLARK (2002b, 2004, 2008).
Use of efficiency controls by the Accounts Committee and Revision Commissions
of Maslikhats were scarce; these functions did not become entrenched as a standard
government practice since they are carried out within compliance controls (The Accounts
Committee for Control over Execution of the Republican Budget (ACCERB), 2003,
2004, 2013, 2014). SAI conducted two pilot efficiency controls in healthcare and
educations systems, however these experiences were far from the expectations of
performance auditing, e.g., SAI did not use 3E categories to make its conclusions
regarding the evaluated areas of public administration (ACCERB, 2004).
As for today, both SAI and Revision Commissions conduct performance audits by
different areas of public administration, including:
▪ budgeting and tax administration;
▪ implementing documents of the System of State Planning including state
strategies, governmental programs, strategies of government entities, and
development-plans of state-owned companies;
▪ public debt;
▪ public procurement;
▪ environmental protection;
▪ information technologies (Goryainov, 2015; ISLARK, 2015; Koszhanov,
2010; Oksikbayev, 2010; Zeinelgabdin, 2015).
The performance auditing procedural standards and methodological guidelines
were adopted by SAI to support performance auditing practices (ISLARK, 2016a,
2016b).
Literature Review and Gaps
There is an extensive body of research work aimed at exploring the impact of
performance auditing on public administration (Alwardat et al., 2015; Desmedt et al.,
2017; Funnell et al., 2016; Morin, 2001, 2004, 2008, 2014; Reichborn-Kjennerud, 2013b,
2014a; Reichborn-Kjennerud & Johnsen, 2018; Reichborn-Kjennerud & Vabo, 2017;
Torres et al., 2019). The above cited researches are examples of investigation of the
impact of performance audits by exploring the experiences of managers of auditees and
auditors. In the opinion of auditees’, performance audits positively influence their
organizations subject to good relations with auditors, placing auditors’ recommendations
among auditees’ priorities, and will of auditees’ leaders for changes (Alwardat et al.,
2015; Morin, 2008; Torres et al., 2019). While auditors perceive performance audits as
effective in terms of contribution to better governance, managers of auditees may contest
some auditors’ approaches, including their interventions to policy making, criticize the
quality and fairness of audit reports, and question auditors’ competencies (Alwardat et al.,
2015; Desmedt et al., 2017; Reichborn-Kjennerud, 2013b; Reichborn-Kjennerud &
Johnsen, 2018; Reichborn-Kjennerud & Vabo, 2017).
A few studies were conducted to learn parliamentarians’ views partially exploring
their attitude toward performance auditing where they are the key users of audit reports
(Funnell et al., 2016; Morin, 2016; Reichborn-Kjennerud, 2014b). Funnell et al. (2016)
argued that parliamentarians may use performance audit reports to enhance their positions
as elected officials with an enhanced interest in auditors’ findings that questioned the
effectiveness of executives’ actions. Loke et al. (2016) offered that recent studies in the
area of public and government audit are focused on countries used
Westminster or Napoleonic models of SAIs’, with fragmented audit research being
conducted on developing countries, of which Kazakhstan is classified.
There were no empirical studies on the demand for and impacts of performance
auditing on public administration in Kazakhstan. The Accounts Committee’s research
center conducted a few applied studies with the primary objective to establish the
methodological base for performance auditing practice (CFVR, 2014, 2016, 2017). As
mentioned in Chapter 1, the identified gap was not surprising. Unlike developed
countries, the use of performance auditing in Kazakhstan began as recent as 2002 with
two pilot audits conducted by the Accounts Committee. More robust use of performance
auditing by the Accounts Committee and Revision Commissions began in 2015
(ISLARK, 2015). Some research indicates a lack of empirical evidence about the
usefulness of performance audits, which threatens the credibility of performance auditors’
works (Funnell et al., 2016; Reichborn-Kjennerud & Johnsen, 2018). It, in turn, increased
risks of missed opportunities of public sector audit organizations in Kazakhstan whose
purpose is to promote efficient and effective public sector programs and operations using
validated audit outcomes.
A practice-related gap, therefore, existed for both scholars and practitioners in
understanding the reasons and causes of the high demand for performance auditing
observed in Kazakhstan beginning in 2016; the perceived effects of performance audits
on Kazakhstani public administration remained unknown. Whereas the impact of and, as
a consequence, demand for performance audits significantly vary depending on the level
of nations’ development, traditions, and changes of national public administration
systems (Desmedt et al., 2017; Loke et al., 2016; Torres et al., 2019), more extensive and
empirical studies are needed regarding the performance auditing phenomena, demand for
and usefulness of performance audits from the public sector perspectives (Hay &
Cordery, 2018). The identified information gap was especially topical for Kazakhstan as
a newly emerged country that has been adopting the performance auditing practice at a
relatively early stage of its nation-building and establishing a sovereign public
administration system.
Theoretical Foundation
New Public Management Theory
New public management is the 1970s trend of reforming the public sector
(Barzelay, 2001; Kapucu, 2006) and renovation of the 1830s vision on changing the
administration of public affairs (Bowrey, Hui, & Smark, 2017). It is established as a
selfsufficient theory in a public administration field by bridging the ideology of new
institutional economics with ideas on the applicability of business-like managerial
approaches within the public sector (Hood, 1991).
Hood (1991) derived the following seven doctrinal components of new public
management: (a) “hands-on professional management” in the public sector; (b) explicit
standards and measures of performance; (c) greater emphasis on output controls; (d) shift
to disaggregation of units in the public sector; (e) shift to greater competition in public
sector; (f) stress on private sector styles of management practice; and (g) stress on greater
discipline and parsimony in resource use (pp. 4-5). NPM implies the establishment of
new standards of performance in the public sector that might be achieved through using
skills mastered by professional managers within the private sector, who increase the
effectiveness of public sector organizations because of granting an autonomy, greater
control over the outputs, and working in competitive environments. Hood’s theory
stimulated the transformation of public administration systems through replacing
processoriented approaches by result-oriented techniques (Bao et al., 2012; Kapucu,
2006).
Despite the criticism of NPM, Hammerschmid, Van de Walle, Andrews, and Mostafa
(2018) offered that managerial reforms positively impact public administration supporting
the actuality of Hood’s theory.
Hood’s (1991) theoretical elaborations were used to explore the cause of demand
for performance audits conducted in Kazakhstan to establish whether the executives
operate with due regard to 3E categories, i.e., economy, efficiency, and effectiveness
(INTOSAI, 2019). These categories are in line with NPM’s core values including the
matching of resources to tasks for given goals (Hood, 1991). Further, NPM implies not
only removing differences between the private and public sector but also emphasizing
accountability over the results (Hood, 1995).
Principal-Agent Model
The principal-agent model theory describes who is accountable to whom
(Maggetti & Papadopoulos, 2018) and is used to construct various principal-agent models
(Gerber & Teske, 2000). PAM implies the identification of actors within the
accountability process who play the role of principals or agents. Since agents operate in
the interests of principals, principals define and structure incentives for agents (Maggetti
& Papadopoulos, 2018).
Waterman and Meier’s PAM is an extension of traditional principal-agent models
(Waterman & Meier, 1998). Firstly, Waterman and Meier’s model illustrates that goals of
principals and agents may conflict; secondly, agents may have more information than
principals creating an information asymmetry between them; thirdly, multiple principals
and agents may engage in one relationship chain (Waterman & Meier, 1998). Waterman
and Meier (1998) stressed that conflict of goals and information asymmetry is one of the
possible combinations that describe the relations between principals and agents. The
authors admitted that principals and agents may have the same level of access to
information or agents may have more information than principals or viсe versa
(Waterman & Meier, 1998). Similarly, there are various combinations of situations with
conflict of goals – from full agreements to full contradictions of principals and agents’
goals (Waterman & Meier, 1998).
Figure 5 illustrates the main advantage of Waterman and Meier’s PAM, i.e., a
combination of assumptions regarding the conflict of goals and information asymmetry in
dynamics.
Agent’s Information Level
LITTLE MUCH
Principals have information, agents
do not: Limted agents' resistance
Both princiapal and agent have
information: Cooperation principal-
agent relations
Princiapal and agent lack
information: No productive goal
conflicts
Agent has information advantage
over prindpal: Classical goal
conflicts
Figure 5. Combining information and goal
Note: Developed based on Waterman & Meier (1998).
NPM and PAM Integration in the Context of Performance Auditing
Hood (1991, p. 5) interpreted the origin of NPM as “as a marriage of two different
streams of ideas” including (a) the institutional economics created on development of
public choice, transactions cost theory, principal-agent theory, theory of bureaucracy, and
(b) “a set of successive waves of business-type ‘managerialism’ in the public sector.”
Barzelay (2001) strengthened the linkage between the new public management theory and
the principal-agent theory, arguing that principals and agents have agreements on
improving public sector performance in a new public management environment.
According to Morin (2001, 2003), performance auditors try to influence managers
of auditees or, in the language of PAM, they structure intensives for auditees.
Conversely, performance auditors conduct their audit work in the interests of citizens
(The United Nations Department of Economic and Social Affairs (UNDESA), 2013);
they are accountable to elected parliaments or presidents. Thus, referring to the
performance auditing, PAM is the construction of three-level relations between principal
and agents, where performance auditors are principals for managers of auditees and
agents for parliamentarians, parliamentarians are principals for both performance auditors
and managers of auditees, and managers of auditees are agents for both parliamentarians
and performance auditors (Barzelay, 2001; Reichborn-Kjennerud, 2013a; Waterman &
Meier, 1998).
Further, two key assumptions of PAM, i.e., conflict of goals and information
asymmetry between principals and agents, are fair to performance auditing chain
relations. For instance, goals of auditors who are principals and auditees who are agents
may conflict (Morin 2001, 2003, 2008). Auditees may have more information that
auditors. Both conflicts of goals and asymmetry information take place in the following
chains: managers of auditees and parliamentarians, parliamentarians and auditors (see
Funnell, 2015; Morin, 2016, for more). Underpinned to the language of PAM,
performance auditing is a chain of relations between auditors, managers of auditees, and
parliamentarians, i.e., multiple principals and agents. As demonstrated in Figure 6,
embedding these relations into PAM implies the fact that one actor (e.g., auditors) play
both principal and agent’s role.
Principals have
information, agents do
not: Limted agents'
resistance
Both princiapal and Principals have agent
have information, agents do
information: not: Limted agents'
Cooperation principal- resistance
agent relations
Both princiapal and
agent have information:
Cooperation
principalagent
relations
Princiapal and agent
lack information: No
productive goal
conflicts
Agent has information
advantage over
prindpal: Classical goal
conflicts
Princiapal and agent
lack information: No
productive goal
conflicts
Agent has information
advantage over
prindpal: Classical
goal conflicts
Principals have
information, agents do
not: Limted agents'
resistance
Both princiapal and
agent have information:
Cooperation
principalagent
relations
Princiapal and agent
lack information: No
productive goal
conflicts
Agent has information
advantage over
prindpal: Classical
goal conflicts
NPM’s principles realization [3E]: adding
value through auditing
Figure 6. Combining information and goal on PAM
Note: Developed based on Funnell (2015; 2016); Hood (1991), Morin (2001; 2003); Waterman & Meier
(1998).
The key integrators between NPM and PAM are 3Е categories, i.e., economy,
efficiency, and effectiveness. They are generally-accepted principles of good governance
declared by parliamentarians and other elected officials (INTOSAI, 2019; OECD, 2016).
Performance auditors’ opinion regarding the addressing of 3E’s requirements by auditees
is important information derived within conflict goals and asymmetry of the information
NPM’s principles realization
[
E
3
]
:
Auditors’ opinion on
auditees’ performance
NPM’s principles realization
[
3
E
]
:
result
-
oriented
management
parliamentarians
PRINCIPALS
AGENTS
auditees
AGENTS
AGENTS
performance auditors
PRINCIPALS
environment. Thus, integration of NPM with PAM provided a foundation for the
interpretations of perceptions of auditors, managers of auditees, and parliamentarians
regarding the usefulness of performance audits and their impact on public administration
in Kazakhstan.
Summary
The performance auditing became an area of interest for scholar-practitioners
almost 50 years ago. It is a power of independent public sector audit organizations on
examination of the effectiveness of executives’ activity; performance auditors’ findings
(or, more precisely, their opinion regarding the following to principles of economy,
efficiency, and effectiveness by the executives) is publicly available information. Based
on the literature review, I determined changes in public administration, global financial
crises, structural problems, environmental issues, other national and global challenges as
triggers for performance auditing. Herewith, the impacts of performance audits differ by
means and targeted levels of public administration.
The perspectives of developed countries in terms of impacting the public
administration through performance audits have received considerable attention, while
there is a limited number of studies on exploring the perspectives of developing countries
including Kazakhstan. The case of Kazakhstan is unique since, unlike developed
countries, Kazakhstan has been applying performance audits during the last few years.
Establishing this practice coincided with modernization of the public administration
system in Kazakhstan. I described the historical perspectives and current features of the
Kazakhstani public administration system focusing on the implementation of
performance audits within the public sector. I concluded that gap existed in understanding
the factors that might explain the demand for performance auditing in
Kazakhstan and it was unknown what are the perceived effects of performance audits on
Kazakhstani public administration. In this Chapter, I also described the approach on the
combination of the new public management theory and the principal-agent model used as
the theoretical framework to contribute to filling the identified gap in the literature.
Chapter 3: Research Method
Introduction
The purpose of my research was to explore the perceived demand for and impacts
of performance auditing on Kazakhstani public administration. The lived experiences of
participants of a performance auditing process, i.e., auditors, managers of auditees, and
parliamentarians, and their associated perceptions were the essential information that I
used to achieve my research purpose. I applied a qualitative research design with the
phenomenological approach as the most suitable to explore the studied phenomena
through learning the perceptions of individuals experienced in performance auditing (see
Moustakas, 1994).
I used semistructured phenomenological interviews to collect the primary data and
documentary reviews to collect the secondary data. I selected interviewees, i.e., the
research participants, by applying group characteristics sampling and single significant
case sampling strategies as the most effective to get access to informant rich cases
(Patton, 2015). I interviewed 14 individuals experienced in performance auditing to
achieve data saturation.
My role as the researcher varied depending on the specification of research
procedures and fieldworks. I was responsible for conducting my study in a way that
ensured achieving the research purpose efficiently and for compliance with the
Institutional Review Board’s (IRB) principal and requirements on the conduction of an
ethical research (IRB, n.d.). I ensured the trustworthiness of my research by establishing
credibility, transferability, dependability, and confirmability (see Lincoln & Guba, 1985,
for more). I took both formal and practical measures to manage ethical issues, including
the protection of research participants.
Rationalizing the Research Design and Methodology
Design: Qualitative Study
To explore the causes of demand for performance audits in Kazakhstan and their
impact on Kazakhstani public administration, I answered the following question: What
are the lived experiences and perceptions of performance auditors, managers of auditees,
and parliamentarians, the key participants and users of performance audits, regarding the
demand for and impacts of these audits on Kazakhstan’s public administration?
I answered this question by conducting semistructured interviews through which I
identified (a) the factors contributed to making decisions on the conduction of more
performance audits by the key participants of a performance auditing process, namely:
auditors, managers of auditees, and parliamentarians, and (b) how they perceive the
impact of performance audits on their perspectives, organizations, and public
administration in general.
This qualitative research design was the most suitable for my research because it
involves learning about peoples’ lives and experiences through their stories
(FrankfortNachmias & Nachmias, 2008; Patton, 2015). The qualitative method is
especially valuable in social sciences, including the public policy and administration
discipline, because it provides scholars with new perspectives regarding the research
problem (Mills & Birks, 2017). The selection of qualitative research design was also
justified because qualitative inquiries close the gap between scholars and practitioners
(see Ospina, Esteve,
& Lee, 2018, for more).
Unlike quantitative studies, in which researchers test existing theories and
determine best practices through the predictable hypotheses about relationships between
associated variables, qualitative studies produce new patterns and themes based on
indepth learning of real cases of people’s lives, which are not subject to predictions
(Patton, 2015). The subject of qualitative studies is multiple realities shaped by the lived
experience of study participants, while the subject of quantitative studies is objective
realities based on statistical abstractions (Brower, Abolafia, & Carr, 2000). The lack of
necessary data for developing variables was a barrier to the application of the quantitative
method in my research. The novelty of performance auditing practice in Kazakhstan is
the cause of the lack of data, and applying both quantitative and mixed methods to
explore the performance auditing phenomenon is a viable option for future studies subject
to further developments in performance auditing practice and accumulating sufficient
quantifiable database (Baskarada & Koronios, 2018).
Qualitative research designs may involve a large number of elements of data for
interpretations of observed changes; these data are accessible through interviews and
observations made in a naturalistic context (Taguchi, 2018).
Methodology: Phenomenological Approach
Moustakas (1994) defined perceptions as the central source of knowledge in
phenomenology and justified the application of the phenomenological approach as a
means to portray the essence of individuals’ experience. Applying this approach leads to
the generation of new knowledge about a specific phenomenon from learning about the
world of individuals (Husserl as cited by Moustakas, 1994). Agreeing with these
arguments of Clark Moustakas and Edmund Husserl, the founders of the phenomenology
(Patton, 2015), I used the phenomenological approach that is one of the most influential
approaches of qualitative inquiries (Jamali, 2018).
The central phenomena of my study were the perceived factors of demand for
performance audits and the perceived impacts of these audits on public administration.
The application of phenomenological approach makes possible the transformation of an
individual or groups’ experience into the meaning; it is achieved through exploring and
understanding the perceptions, descriptions, judgments, and other means of
meaningmaking by the research participants (Patton, 2015). I explored and learned the
perceptions of three different groups of individuals who are participants of performance
audits. Participation in performance audits means being experienced in the phenomena
under study or, in other words, to be in line with the characteristics of the source of
information in phenomenology (Moustakas, 1994).
There are other well-tested approaches to qualitative studies, such as ethnography,
autoethnography, grounded theory, and case studies (Abutabenjeh & Jaradat, 2018;
Moustakas, 1994; Patton, 2015). However, unlike these approaches, phenomenology was
the most appropriate for my research since portraying the essence of experience of
auditors (who initiate and conduct performance audits), managers of auditees (who are
subjects for performance audits), and parliamentarians (who are users of performance
audits findings) allowed me to understand why performance audits are in demand as for
today and whether and how they [performance audits] impact public administration in
Kazakhstan.
By exclusion of other methodological approaches because of their limitations, I
concluded that the phenomenological approach was the best for my study. Unlike
ethnography, in which activities of a specific group with shared values or cultural
specifics are explored through extensive fieldwork with prolonged participant
observation, phenomenology is used to explore the lived experience of diversified groups
(Moustakas, 1994; Park & Park, 2016). In case studies, the researcher focuses on an
individual case that is limited by boundaries, narrowing the opportunities to
comprehensively understand the phenomenon under study, whereas the
phenomenological approach is not associated with that limitation (Creswell, 2013; Park &
Park, 2016; Patton 2015). Both ethnography and case studies are focused on particular
activities or processes, while the phenomenology is focused on the study participants’
experiences scoped by the research problem (Creswell, 2013; Moustakas, 1994; Park &
Park, 2016; Patton 2015). My research goal was not to generate a new theory; therefore, I
did not use grounded theory.
Research Participants and Researcher
Target Population and Selection of Participants
The population consists of all individuals, organizations, cases, and other units of
analysis or data used in researching (Frankfort-Nachmias & Nachmias, 2008).
Conceptually, individuals and organizations create the demand for performance audits.
The impact of performance audits implies influence on the actions of individuals and
organizations alike (Hay & Cordery, 2018; Lonsdale et al., 2011; Morin, 2001, 2008;
Reichborn-Kjennerud, 2014a). Thus, in the context of my research, the general population
consists of all individuals and organizations that need performance audits and are subject
to performance audits.
The target population consists of individuals and organizations depending on their
relation to the research problem and scope (Frankfort-Nachmias & Nachmias, 2008). To
identify the target population of my study, I defined organizations that belong to the
public sector and are directly associated with the performance auditing. I then determined
the target groups of individuals who belong to these organizations and participate in
performance audits.
Performance auditing is a practice of public sector audit organizations, such as the
Accounts Committee and Revision Commissions of 17 regions in Kazakhstan (ISLARK,
2015). Among 17 Revision Commissions I selected Revision Commission on Nir-Sultan
city since, unlike other commissions, it was a significant case in terms of increasing
efficiency in using associated budget funds (see INTOSAI, 2019, for more). The
Accounts Committee and Revision Commission on Nir-Sultan city belong to the first
cluster of the target population. The target group of individuals consisted of all public
auditors who belong to both managing and executive staffs of these public sector audit
organizations.
Auditees may initiate performance audits, and they are subject to performance
audits conducted by the Accounts Committee and Revision Commissions (ISLARK,
2015). According to the 2015 Public Auditing Act, government entities, quasi-public
organizations (i.e., state-owned companies or organizations with state participation), and
any other organizations that receive budget funds are subjects for performance audits
(ISLARK, 2015). In my research, I focused on government entities and, hence, all central
and local government entities belong to the second cluster of the target population. This
cluster consisted of government entities responsible for financial and healthcare affairs
(i.e., the Ministry of Finance of the Republic of Kazakhstan and financial departments of
local governments, and the Ministry of Healthcare of the Republic of Kazakhstan and
healthcare departments of local governments). I selected these organizations due to their
significance in terms of increasing efficiency in using associated budget funds and adding
value through performance audits conducted by the Accounts Committee and Revision
Commissions. The target group of individuals consisted of all managers of auditees, i.e.,
government officials who are responsible for interaction with external auditors.
Parliamentarians may also initiate or, more precisely, request conduction of
performance audits, and they are users of all audit reports (ISLARK, 1995b). The
legislative bodies may contribute to the implementation of performance auditors’
recommendations addressed to auditees (Funnell et al., 2016; Morin, 2016). Therefore,
the Parliament of the Republic of Kazakhstan and 17 Maslikhats, local legislative bodies,
belong to the third cluster of the target population. Unlike employees of public sector
audit organizations and managers of auditees, all parliamentarians, i.e., members of the
Parliament of the Republic of Kazakhstan and Maslikhats, belong to the third target group
of individuals since they empowered with the same mandates. Thereby, as demonstrated
in Table 3, the target population consisted of three clusters of organizations and three
target groups of individuals who are participants of performance audits.
Table 3
The General and Target Population
The general
population
The target population
All individuals
and organizations
Organizations
Individuals
1st cluster
Accounts Committee 1st target
Revision Commissions group
Public auditors
that need in Central
performance 2nd cluster government entities 2nd target Managers of auditees
audits and are Local group
subject of government entities
performance audits rd cluster Parliament Maslikhats 3grouprd target
Parliamentarians 3
Based on the identified target population, I determined a sample, i.e., a subset of
the target population that is subject to further in-depth analysis (see Frankfort-Nachmias
& Nachmias, 2008, for more). Sampling is an essential procedure of all researches
regardless of the selected research design because it is impossible or inadvisable to
analyze the general population to generate new patterns and themes or to test the existing
theories (Frankfort-Nachmias & Nachmias, 2008). However, sample design and sample
size significantly vary depending on research design (Park & Park, 2016; Patton 2015)
since the sampling implies taking into account research problem, purpose, and nature
(Blaikie, 2018).
I used purposeful sampling strategies including group characteristic sampling
(GCS) and single sufficient case sampling (SSCS) strategies as the most suitable for
conducting a phenomenological study (see Mills & Birks, 2017; Patton, 2015, for more).
“The logic and power of purposeful sampling lies in selection information-rich cases for
in-depth study” (Patton, 2015, p. 264). The information-rich cases in my research are
individuals experienced in performance auditing since my research aimed at exploring the
perceived demand for and impact of performance audits. Patton (2015) stated, applying
the purposeful sampling means learning information shared by individuals who
“deal about issues of central importance to the purpose of inquiry” (p. 264).
I used an algorithm for designing the sample similar to the algorithm of
identification of the target population. I have defined three groups of the research
participants applying GCS with a shared characteristic, namely – belonging to a specific
organization. This strategy allowed me to reveal and illuminate important group patterns
(see Patton, 2015, for more). This is an important advantage in the context of my
research. As mentioned in previous chapters, the performance auditing is a process of
interaction between three groups (i.e., the group of auditors, group of managers of
auditees, and group of parliamentarians). Therefore, group patterns were critical in
obtaining a better understanding of the studied phenomena.
I selected the study participants applying SSCS. According to Patton (2015), a
single significant case is an individual or another unit of analysis that provides a rich
understanding of the studied phenomena. So, the single significant cases in my research
were individuals who met the following characteristics: (a) they are high-impact cases, or
they may influence policies and practices related to the performance auditing; and (b)
they are crucial cases or it means that they share information similar to the information
that might be shared by other representatives of the target population (Patton, 2015).
I used the following criteria to select and recruit the research participants:
1. general criteria for all participants –
▪ being experienced in performance audits conducted in 2006–2019;
▪ willingness and consent to participate in the study;
2. specific criteria for auditors – ▪ being certified public auditors;
▪ hold managing position or belong to audit departments of the Accounts
Committee and Revision Commission on Nur-Sultan city;
▪ participate in planning and conduction of performance audits;
▪ being experienced in making strategic decisions related to performance
auditing practice in their organizations;
3. specific criteria for managers of auditees –
▪ hold managing position in auditee and being empowered with
mandates to coordinate financial, economic or strategic affairs in their
organizations;
▪ being experienced in interactions with external auditors;
▪ empowered with mandates to initiate performance audits;
4. specific criteria for parliamentarians –
▪ being a member of parliamentarian committees empowered with
mandates in area of financial, economic, and budget affairs;
▪ being experienced in interactions with external auditors;
▪ being experienced in participation of parliamentarian committees’
meeting on consideration performance audit reports.
There is no singular formula on defining the sample size for qualitative inquiries
(Patton, 2015). A researcher may define the sample size methodologically or based on
previous studies (Dworkin as cited by Sim, Saunders, Waterfield, & Kingstone, 2018).
Proper sampling leads to data saturation in case of relying on information power
(Malterud, Siersma, & Guassora, 2016). Malterud et al. (2016) suggested the following
relationships: “The larger information power the sample holds, the lower N is needed, and
vice versa” (p. 1754). Well-tested or recommended sample sizes are ranged from 2 till 60;
the most often used sizes are ranged from 10 till 20 (Blaikie, 2018; Guest, Bunce, &
Johnson, 2006; Guest, Namey, & McKenna, 2017; Sim et al., 2018). According Guest et
al. (2006), within phenomenological studies, the data saturation achieved in the case of
conduction 12 interviews, it means that all themes emerge as a result of the interpretation
of responses of 12 interviewees. Sim et al. (2018) stated that preliminarily defining the
sample size is problematic, especially in interpretive models of qualitative studies.
Table 4 illustrates how I defined the sample size as at least 12.
Table 4
The Sample Design and Size
Target groups of individuals Sample: a-priory size
level of public administration
Total central
local
1 Auditors
2
2
4
2 Managers of auditees
2
2
4
3 Parliamentarians
2
2
4
Total
6
6
12
I conducted interviews with 14 participants of performance auditing selected by applying
GCS and SSCS and identified 90% of all patterns and themes associated with the studied
phenomena (see Guest et al., 2017; Malterud et al., 2016; Sim et al., 2018, for more).
Researcher’s Role and Participants’ Values
A researcher plays multiple roles shaped by the research problem, purpose,
setting, and design (Patton, 2015). My key role as the researcher was to project and
implement my study by applying approaches and techniques that ensure achieving the
research purpose. Researcher implements a set of consistent and logically linked
procedures, including a delineation of research focus, development of understandable
research questions, quiring individuals to gather data, and data analysis (von Eckartsberg
as cited by Moustakas, 1994). In the course of fieldworks, a qualitative researcher’s roles
vary from participant-observer to spectator-observer (Patton, 2015). Whereas the purpose
of my phenomenological study was to explore the perceptions of study participants
regarding the increasing demand for performance audits and their impact on the national
system of public administration, I acted as a participant-observer.
In communications with the study participants and reviewing documents, I learned
to and implemented the role of both investigator and interpreter. As the investigator, I
searched for in-depth information constructing relations with the study participants in a
way that avoids threats and risks of negative consequences that might emerge because of
my access to their experience in performance auditing (Alase, 2017). According to
Moustakas (1994), the central objective of a researcher-phenomenologist is to interpret
the lived experience of individuals; it is achievable through contacting the people
experienced in the phenomena (Van Manen, 2014). So, as the interpreter, I made
meanings associated with the studied phenomena relying on the study participants’ stories
and maintain an acceptable level of distance from them (Moustakas, 1994).
As the researcher I also:
▪ collected data as an instrument (Janesick, 2011; Patton, 2015);
▪ conducted negotiations with the study participants (Rahiem et al., 2016);
▪ conducted interviews mastering skills on asking open-ended and probing
questions (Patton, 2015);
▪ conducted observations mastering skills on interpretation of body language
and facial expressions (Bevan, 2014; Patton, 2015);
▪ implemented analytical procedures arguing the research findings (Patton,
2015);
▪ created a phenomenological text as a writer-phenomenologist (Fletcher,
2017).
While the researcher is a data collection instrumentation, the study participants are
co-researchers (Janesick, 2011; Patton, 2015). Unlike researchers, study participants are
principal owners of information important to the study. The results of phenomenological
studies based on the lived experience of study participants, their values, ideas, and views;
they shape their perceptions related to the phenomena under study (Moustakas, 1994).
Therefore, conduction of quality phenomenological study requires understanding and
illuminating the participants’ values (Brower et al. 2000;
Fletcher, 2016). I learned and illuminated the participants’ values to better understand the
essence of their lived experience avoiding judgmental attitude toward them (Karagiozis,
2018). Understanding the participants’ values allowed me, in turn, to make valuable their
participation in my study building trust with them. As Alase (2017), Janesick (2015),
Moustakas (1994), and Patton (2015) stated, establishing a trust and rapport with the
participants contributes to obtaining needed data.
Data Sources, Collection, and Analysis
Primary and Secondary Sources
Using multiple sources contributes to quality qualitative studies (Patton, 2015). I
used interviews and documentary reviews to collect data needed for my research work.
The lived experience of study participants is essential information in phenomenological
studies. The empirical phenomenological approach implies returning to experience
allowing “the phenomena speak for themselves” (Giorgi as cited by Moustakas, 1994, p.
11). By conduction of 14 interviews I collected the primary data that describe the
perceptions of the study participants regarding the (a) causes of demand for performance
audits within the public sector in Kazakhstan and (b) impact of these audits on
participants’ perspectives, their organizations, and public administration in general.
I logically accumulated and structurally organized the interview data, including
my notes on observations, in compliance with IRB’s requirement. I started to collect these
primary data after getting IRB’s approval (IRB, n.d.; Janesick, 2011; Patton, 2015). To
corroborate the interview data, I used the secondary data collected by documentary
reviews. I did it for data saturation, triangulation and for ensuring the credibility of my
study (Patton, 2015).
The list of documents to be reviewed included:
▪ annual reports and performance audit reports issued by the Accounts
Committee and Revision Commissions in 2016–2020;
▪ press releases on performance audits’ results issued by the Accounts
Committee and Revision Commissions in 2016–2020;
▪ orders of the Parliament of the Republic of Kazakhstan and Maslikhats related
to approval of annual reports issued by the Accounts Committee and Revision
Commissions in 2016–2020;
▪ press releases on parliamentarians’ meetings on considering annual reports
issued by the Accounts Committee and Revision Commissions in 2016–2020.
Permission was not needed to use the secondary data since all these data are
publicly available.
Instrumentation for Data Collection
In the course of my study, I conducted 14 phenomenological interviews to find out
why, from the point of view of the study participants, performance audits often used as
for today and whether these audits impact on national system of public administration in
Kazakhstan. I recruited the study participants by contacting them using the publicly
available contact information, such as phone numbers and email addresses, and invited
them to participate in the proposed study by sending them invitation letters (see Appendix
A). I used a snowball approach, i.e., asking my study participants to recommend other
persons who meet the selection criteria and may consider the invitation to take part in the
study. The study participants were not be asked to provide non-public contact information
for their co-workers nor were they be asked to distribute the invitation to their co-workers
at work.
The main objective of all planned interviews is to capture the lived experience of
participants in the performance auditing process (Van Manen as cited by Patton, 2015, p.
433). I used this approach to shed some light on the research problem and find out
previously unknown or undisclosed information about the performance auditing practice
that is a novelty for Kazakhstan. I conducted semistructured face-to-face and phone
interviews with open-ended questions since they are best suited for phenomenological
studies (Bevan, 2014; Giorgi, 1997; Merriam, 2009; Mills & Birk, 2017; Moustakas,
1994; Patton, 2015; Seidman, 2006; Van Manen, 2014). Structuring the interview means
using a set of shared questions, and it allows to focus on research purpose guaranteeing
alignment the research questions with the interview questions (Bevan, 2014; Merriam,
2009; Seidman, 2006).
Seidman (2006) suggested consequently use questions related to life histories,
reconstructing the experience, and reflection on the meaning of an experience. Bevan
(2014) suggested to structure interviews by using questions needed for contextualization
(e.g., to explore bibliographic data), apprehending (i.e., focusing on the phenomenon
under study), and clarifying (i.e., clarifications related to specific elements participants’
experience). To ensure a correct understanding of the information shared by participants,
researchers use probing questions (Patton, 2015). Pessoa, Harper, Santos, and Gracino
(2019) proposed to use a wide range of interview questions, including confirmatory
questions, questions to organize chronological events, and questions to explain
contradictions. I used three sets of questions that consisted of (a) general questions for all
participants and (b) specific questions for a particular group of participants. I asked two
types of questions. I used broad and open-ended questions directly related to the research
questions to gain access to information relevant to the purpose of my research. I then used
additional questions to ensure accurate interpretation of the essence of participants
experience in performance auditing.
I developed the interview questions aligned with the central research question. To
effectively collect data during the interviews, I developed the Interview Guide, i.e., the
first instrumentation (Patton, 2015; see Appendix B). My interview questions based on
the literature, including the works of Desmedt et al. (2017), Funnel (2015), Morin (2001,
2008, 2014, 2016), Reichborn-Kjennerud and Johnsen (2018). These authors shared the
view that exploring the impact of performance audits on public administration through
learning reactions of managers of auditees and parliamentarians and auditors’ perceptions
is advisable. Since the impact of performance audits manifests at different levels and vary
depending on organizational specifics (Lonsdale, 2000), I also took into account the
Supreme Audit Institutions Performance Measurement Framework (SAI PMF) issued by
INTOSAI (INTOSAI, 2016c). SAI PMF consists of performance measurement indicators,
including the indicators that measure the effectiveness of performance audits (see
Appendixes C and D). Finally, since the demand for performance audits have not been
studied (Hay & Cordery, 2018), I developed a related set of questions relying on my
experience in developing national performance audit standards and methodological
guidelines, in planning and conducting performance audits, and in teaching the
performance auditing.
I conducted ten interviews in face-to-face mode and four phone interviews for 8
weeks. I agreed with all interviewees the mode, place, and duration of the interview in
advance. To effectively prepare and conduct all interviews and follow-up activities. Table
5 illustrates how I used the Interview Organization Matrix, i.e., the second
instrumentation in my interview approach. Along with the matrix, I used the Interview
Protocol, i.e., the third instrumentation, to conduct all interviews within a scheduled
timeframe (Janesick, 2011; Patton, 2015; see Appendix E).
Table 5
The Interview Organization Matrix
Interviewee’s
code
Sending
invitation
letter
plan
fact
Signing
informed
consent
plan
fact
Interviewing
Date
Location
Mode
Duration
F
O P
plan
fact
Auditor-1 + + TBD TBD + TBD 40-
Auditor-2 + + TBD TBD + TBD 60
Auditor-3 + + TBD TBD + TBD min
Auditor-4 + + TBD TBD + TBD
Manager-1 + + TBD TBD + TBD 40-
Manager-2 + + TBD TBD + TBD 60
Manager-3 + + TBD TBD + TBD min
Manager-4 + + TBD TBD + TBD
Parliam-1 + + TBD TBD + TBD 40-
Parliam-2 + + TBD TBD + TBD 60
Parliam-3 + + TBD TBD + TBD min
Parliam-4 + + TBD TBD + TBD
I also made notes in the Interview Journal, i.e., the fourth instrumentation, to
accurately interpret body language and facial emotions that might add insight to the
results of verbal communications (Roulston, 2018). I used this journal to note issues or
unexpected situations that may emerge during interviews as well (see Appendix F). I used
two digital audio recorders and my personal computer with audio-recording function.
Strategies and Techniques for Data Analysis
Properly collected and organized raw data are crucial in analyzing qualitative data
(Patton, 2015). Figure 7 illustrates how I grouped all collected data by sources and in
chronological sequence; it allowed me to effectively implement further procedures on
triangulation to ensure the validity and reliability of data.
Note: Developed based on Patton (2015).
According to Moustakas (1994) and Van Manen (2014), to properly analyze data
within phenomenological studies, researchers need to bracket their knowledge about the
phenomena under study. Bracketing means for researchers being away from their prior
knowledge and experience and, therefore, it allows them to understand the phenomena
with a new attitude (Mills & Birks, 2017). I used bracketing to analyze interview data to
explore perceptions of the study participants about the reasons to use more performance
audits and effects of these audits on public administration in Kazakhstan from the scratch;
Figure 7.
Data collection and organization
TRIANGULATION
DOCUMENTARY
REVIEW
reviewing
preparing
making
notes
INTERVIEW
intervieweing
writing
trascipts
developing
questions
discrepancies
checking
i.e., by bracketing my experience and prior knowledge about the performance auditing
phenomena.
The bracketing approach is associated with limitations (e.g., in specific areas of
human experiences); it is difficult or even impossible to analyze, examine, and interpret
gathered information without prior knowledge (Creswell, 2013; Mills & Birks, 2017). To
manage these limitations of bracketing, I used inductive and deductive methods to
analyze primary and secondary data. I used these methods depending on each stage of
working with data to be analyzed. I applied the deductive approach to generate new or
previously undisclosed explanations about the demand for performance audits and
whether these audits influence auditors, managers of auditees, and parliamentarians and
change their organizations. These results were preceded by coding or, in other words,
identification and labeling pieces of information significant to understand the studied
phenomena (Ravitch & Carl, 2016). I used two-cycle coding applying the content and
thematic analyses, as demonstrated in Figure 8.
Figure 8. Data analysis algorithm
Data Sources
Raw Materials
1
st cycle coding
2
nd cycle coding
Generating themes
Interview
Transcripts
...
...
Constructing the meanings
Documentaru review
Notes
analyzing consistency
analyzing consistency
analyzing consistency
Note: Developed based on Patton (2015).
During 1st cycle, I used descriptive and concepts codes, and during the 2nd cycle
I used patterns and themes codes derived from textual and structural descriptions; both
1st and 2nd cycles codes were aligned with the theoretical framework and the selected
strategies of sampling (Miles, Huberman, & Saldana, 2014; Patton, 2015). I used the
thematic analysis to identify and group the significant statements derived from the
interview transcripts to themes, i.e., meaning unites; to compare the themes derived from
my prior knowledge and previous experience with the themes derived from the
participants’ experience; to portray the essence of the study participants experience
through associated themes and patterns (Moustakas, 1994; Van Manen, 2014).
I derived patterns and themes ensuring their logical consistency with the research
question (see Miles et al., 2014; Patton, 2015, for more). It allowed me to identify and
systematize influential factors that might explain the reasons for increasing the number of
performance audits and shed some light on their impact by the evaluation categories, such
as positive, negative or neutral impacts. I applied visual tools to demonstrate the results of
data analyses, including the charts presented in Figure 9.
Digital tools contribute to qualitative study (Paulus, Jackson, & Davidson, 2017).
I utilized the software program, NVivo 12 Plus, to organize and analyze qualitative
information collected in text formats, tables, diagrams, charts, and audio-records. This
program allows to use of symbolic systems and visualize the results of coding and
developing patterns (Creswell, 2013; Oliveira, Bitencourt, Santos, & Teixeira, 2016;
Paulus et al., 2017). I selected NVivo 12 Plus due to also its capabilities on avoiding
mistakes and gaps in analyzing the primary data (Patton, 2015).
Figure 9. Data Analysis Results
Note: Developed based on Patton (2015).
Managing Issues of Trustworthiness
Trustworthiness is critical for phenomenological studies since the findings of these
studies shaped by researchers’ judgments (Frankfort-Nachmias & Nachmias, 2008;
Lincoln & Guba, 1985; Patton, 2015). Unlike quantitative studies, when formulas used to
ensure the validity and reliability of research findings, the validity and reliability of
qualitative studies ensured when the credibility, transferability, dependability, and
confirmability, i.e., criteria of trustworthiness, are established (Lincoln & Guba, 1985).
themes
themes
themes
themes
themes
themes
themes
themes
themes
themes
themes
themes
themes
themes
Demand
for
performance
audits
Pattern 1
Pattern 3
...
Pattren 4
Pattern 2
Impact of
performance
audits
Pattern 1
Pattern 3
...
Pattren 4
Pattern 2
Credibility and Transferability
Credibility refers to the internal validity, whereas transferability relates to the
external validity of researchers’ work (Patton, 2015; Shenton, 2004). Establishing
credibility means a maximum approximation of research results to reality. In the context
of my research, I ensured the credibility by interpretation of the study participants’
perceptions in a way that leads to exploring the essence of their experience in performance
auditing. I focused on my research problem, asking the participants clear questions that are
related to the studied phenomena, avoiding the pressure on them (see Lietz et al., 2006;
Shenton, 2004, for more). I interpreted the perceptions of auditors, managers of auditees,
and parliamentarians accurately, and I did not change an “emerging picture” because of
my perspectives or beliefs.
Transferability implies possibilities on transferring the research findings and
results into the reality within different settings (Lincoln & Guba, 1985; Shenton, 2004).
The performance auditing is a novelty for Kazakhstan and it becomes widespread
practice; for this reason, transferability of my research work is critical. It may contribute
to reducing the gap in the knowledge about the performance auditing phenomena. The
research findings and results are transferable when they acceptable for individuals or
organizations uncovered by research (e.g., 16 local public sector audit organizations in
Kazakhstan or public sector audit organizations in other countries). I made my research
useful for scholars and practitioners by applying the comparative techniques and detailed
documentation of my fieldworks.
Dependability and Confirmability
According to Lincoln and Guba (1985), dependability established when all
planned and required tasks on doing research completed. As an indicator to measure the
dependability, I used the following consideration: Repeated data collection and analysis
will lead to results that are similar to the initial results (Shenton, 2004). To ensure the
dependability, I used an audit trail. It means that I made detailed documentation at each
stage of my research work – starting from the justification of my research till
interpretation the research findings and dissemination of the research results.
Confirmability established in case of maintaining neutrality and being objective
throughout an entire research process (Lincoln & Guba, 1985; Patton, 2015). To be deal
with that task, I relied on reliable and tested evidence focusing on my research purpose
and the study participants’ perspectives. I established trust relationships with the research
participant and complete checking procedures, including compliance with IRB’s
requirements on the conduction of ethical research (see Birt et al., 2016; IRB, n.d.;
Patton, 2015, for more).
In sum, I used triangulation to address the issues of trustworthiness in general and
its criterions in particular (see Patton, 2015, for more). One of the well-tested strategies is
to use multiple sources of information and, therefore, I used interviews and documentary
reviews (Patton, 2015). I used triangulation of qualitative sources that implies checking
the compliance of data among different sources (Patton, 2015). In that case, incompliance
is not a sign of incredible, untransferable, undependable, or unconfirmable research work.
As Moustakas (1994) argued, phenomenological studies are associated with
inconsistencies in the collected data. So, my objective was to understand and correctly
interpret identified inconsistencies (in case of their detection) since they may provide
opportunities for deeper insights (Patton, 2015).
Managing Ethical Issues
Researcher’s Biases and Protection of Participants
Unmanaged bias of a researcher may lead to distortion of research results and, as a
consequence, threaten the reliability of research (Frankfort-Nachmias & Nachmias,
2008). Biases caused because of different reasons, such as previous experience and
awareness of a researcher about the problem under study or a researcher’s commitment to
his or her prospects (Patton, 2015). Frankfort-Nachmias and Nachmias (2008) argued that
biases manifest in the course of interviewing through verbal cues and nonverbal
communications. Thus, it needs to identify risks, sources, and manifestations of biases to
avoid them.
My experience in an area of public auditing, i.e., serving in the Accounts
Committee and working as both independent consultant and trainer, or my involvement
into the development of national standards and guidelines on performance auditing is a
possible source of bias. I managed all my actions, ensuring their compliance with IRB’s
requirement on the conduction of quality and unbiased research work (IRB, n.d.). I
exercised application judgments without any pressure on my research participants. My
objective was to interpret the words (stories) of research participants in the meanings that
are laid in these words (of the story) by the participants (Moustakas, 1994).
I am committed to the performance auditing ideology, but this commitment did
not influence the means I used to gather and interpret both the raw and secondary data. I
maintained an unbiased and neutral attitude throughout an entire process of my study,
since, as Patton (2015) stated, “the investigator’s commitment is to understand the world
as it unfolds, be true to complexities and multiple perspectives as they emerge, and be
balanced in reporting both confirming and disconfirming evidence with regard to any
conclusions offered” (p. 58). Unbiased researching allowed me to communicate with the
research participants effectively; my reciprocal and respectful attitude toward them
provided the necessary access to information about the phenomenon under my study (see
Riese, 2018, for more).
Unbiased and (as a result) ethical researching implies protection of the research
participants that was my responsibility as the researcher. The protection of research
participant is important since it is essential in realizing their rights to freedom and
selfdetermination (Frankfort-Nachmias & Nachmias, 2008). So, throughout my study
including the fieldwork stage (i.e., interviewing, documentary reviewing), I took
measures to prevent undesirable and negative consequences for my participants as a result
of their participation in my study.
I started direct communications with the research participants after getting the
Walden University IRB approval (12-16-19-0425017) on December 16, 2019. I
completed the National Institutes of Health online training entitled “Protecting Human
Research Participants”. In addition to these preventive measures, I took appropriate
measures to protect personal data and identifiers of my research participants. I agreed the
place of all interviews with the research participants. All potential study participants were
employees of the public sector organizations including government entities; the identity
of organizations was masked (i.e., audit organizations, audited government entities, and
legislative bodies); participants were de-identified and reported using their assigned
participant code to ensure confidentiality.
During documentation of my fieldworks (e.g., interview transcripts and journal
notes) I deidentified and separately coded personal data and any other identifiers of the
study participants. Cross-referencing of the participants-to-codes was kept secured.
Research data are stored on an encrypted thumb drive and will remain stored for a period
of 5 years and then the drive will be re-formatted. Paper records are stored for a period of
5 years and will be destroyed by shredding.
I conducted interviews in Kazakh (the state language in Kazakhstan) and Russian
(the official language in Kazakhstan), and I translated significant statements of all
interviews into English. I received an official confirmation regarding the accuracy of
translation from a translation agency; before providing the interview transcripts, I signed
nondisclosure agreement with the translation agency. The interview transcripts did not
include personal data in line with the Interview Protocol (see Appendix E). I started audio
recording after the introduction section; demographic data were coded.
Informed Consent
To protect my research participants, I used informed consent. It is one of the
essential procedures in researching, and it implies decision-making by the study
participants regarding their participation in research being aware of the purpose and
nature of research (Diener & Crandal as cited by Frankfort-Nachmias & Nachmias,
2008). Timeliness is critical in using informed consent. The informed consent form was
submitted to my study participants before interviewing (for review and asking questions,
in case of need). According to the Interview Protocol (see Appendix E), the study
participants were asked to sign the informed consent form before or at the time of
interviews. Signature or verbal verifications took place prior to starting interviews. It
allowed me to guarantee that the participation in my research is the result of a voluntary
and informed decision made by the research participants.
The informed consent consists of information about the research purpose and
nature, risks and benefits for research participants, and means of ensuring confidentiality
(Frankfort-Nachmias & Nachmias, 2008). One of the key objectives of using informed
consent is to inform about voluntary participation in the study (Brear, 2018a;
FrankfortNachmias & Nachmias, 2008). It means that the research participants will get
information about their rights to voluntarily participate in the research and withdraw from
the research at any time; their decision will not entail sanctions (Brear, 2018b;
FrankfortNachmias & Nachmias, 2008). I developed the informed consent sample that
includes all these important points. Using the developed sample of the informed consent
allowed me to build trust with the research participants (see Mockler, 2014, for more).
Additional Ethical Procedures
The ethical issues arise from the research problem itself, data collection methods,
and research participants’ positions (Frankfort-Nachmias & Nachmias, 2008). So, ethical
aspects were considered and managed at all stages of my study.
The research participants who are senior-level public officials may express an
ambiguous or biased opinion during interviews. For instance, the participants who are
senior-level managers of auditees may express a vague opinion because of their biased
attitude to the identified research problem; or, public auditors who lead SAI’s
organizational units may express an ambiguous opinion because of potential threats to
their career. The participants may also experience psychological stress since qualitative
interviews imply access to detailed information about their lives (Wiles & Boddy, 2013),
but this sharing was not expected to be of any greater risk than the participant’s normal
working environment. Along with the mentioned procedural measures, researchers use
measures on addressing ethical issues in practice (Guillemin & Gillam, 2004). To ensure
proper managing all ethical issues, I followed IRB’s principles and regulations on the
conduction of ethical studies (IRB, n.d.).
Summary
A qualitative method with the phenomenological approach was the most suitable
research design to explore the factors that might explain the increasing number of
performance audits conducted by public sector audits organizations in Kazakhstan and the
perceived impacts of these audits on Kazakhstani public administration.
In Chapter 3, I rationalized the selected research design in detail. I described
strategies, techniques, and algorithms used to identify the general and target population,
sample design and size, and to select research participants. I described the varied nature
of my role as the researcher and my responsibilities as well.
To ensure the validity and reliability of my study, I used multiple sources of
information. Thus, in Chapter 3, I listed the central sources of information and explained
instrumentations that I used to collect both primary and secondary data, including
interviews and documentary reviews. Chapter 3 consists of visualized information about
strategies for data analysis (e.g., the deriving of patterns and themes applying the content
and thematic analyses).
In this chapter, I also detailed measure on establishing the credibility,
transferability, dependability, and confirmability of my research work, including the
triangulation procedures. Risks associated with researcher biases and ethical issues are
also listed in this chapter. Thus, I detailed measures on protection of my research
participants (e.g., utilizing the informant content process that I used in the course of
gathering and analyzing the interview data).
Chapter 4: Results of the Study
Introduction
The purpose of my study was to explore the factors that might explain the demand
for performance auditing in Kazakhstan and the perceived effects of performance audits
on Kazakhstani public administration. I used a phenomenological approach to answer the
following research question: What are the lived experiences and perceptions of
performance auditors, managers of auditees, and parliamentarians, the key participants
and users of performance audits, regarding the demand for and impacts of these audits on
Kazakhstan’s public administration?
Through using both phenomenological interviews and documentary reviews, I
figured out some real causes for expanding the performance auditing practice or factors
that explained why these audits are highly demanded in Kazakhstan. I also tried to
explore the perceived effects of performance audits on public administration from the
perspectives of auditors, managers of auditees, and parliamentarians, who are directly
involved in processes of performance audits and use the results of these audits in their
professional and legislative activities. Whereas performance auditing is a Western
practice adopted by Kazakhstani public audit organizations, I tried to understand whether
the impacts of performance audits on public administrations in Kazakhstan and
wellstudied cases of western countries differ.
This chapter consists of detailed descriptions of implementing the field stage of
my study, including the research setting, demographics, and collection and analysis of
phenomenological data.
Setting
I conducted my research in Nur-Sultan city, the capital of Kazakhstan. As planned,
I primarily focused on the practice of performance audits conducted in 2016– 2019 by
external public audit organizations at central and local levels of public administration.
During that timeframe, auditors from 18 public audit organizations, managers of more
than 1,600 government entities, public and quasi-public organizations, and
parliamentarians from 18 legislative bodies were involved in the process of performance
auditing. From that target population, as described in Chapter 3, I defined the potential
study participants by applying group characteristics and single sufficient case sampling
methods to collect primary data. Fourteen participants and users of performance audits,
including five performance auditors, five managers of auditees, and four parliamentarians
participated in my study.
I interviewed the study participants in January and February 2020. Generally, that
time was a favorable period for interviews since it did not coincide with the planning or
reporting period of both public audit organizations and auditees and with the time of
beginning or finishing parliamentarians’ terms. There were no significant organizational
changes in public audit organizations, auditees, and legislative bodies as well as
unscheduled sessions of legislative bodies or large-scale government events during the
interview time or immediately preceding. However, due to weather conditions in
NurSultan city, a state of emergency was declared starting from January 27, 2020 limiting
travel. Consequently, and to not endanger my study participants or myself, I conducted
four interviews by phone. This plan alteration was not a deviation from my original data
collection design as I incorporated face-to-face interviews as well as phone or online
technology modes initially depending on the study participants’ availability. As such, no
organizational or personal conditions impacted my study participants which may have
contributed adversely to study interpretation.
The study participants defined the dates, time, and locations for face-to-face
interviews and similarly their preferred timing for phone interviews. All 14 participants
participated voluntarily and consent to participate was obtained. All documents reviewed
and analyzed during the field stage of my research work were collected from open access
sources.
Demographics
Whereas the lived experiences of auditors, managers of auditees, and
parliamentarians in the studied phenomena are the key source of meaningful information,
I defined the experience in performance auditing as the general inclusion criteria for all
study participants. All participants met that criteria and their professional experience was
9.6 years per a participant on average, including 55 years of experience in performance
audits conducted in 2016–2019, or 3.9 years per a participant on average. As
demonstrated in Table 6, the study participants also met specific inclusion criteria applied
to them depending on their roles in the process of performance auditing.
All five auditors are certified external public auditors. Three auditors hold
leadership positions, i.e., they are senior-level managers in their organizations; two
auditors are members of public audit organizations’ departments responsible principally
for conducting the performance audits. All auditors participated in both planning and
conducting performance audits as well as making strategic decisions associated with the
performance auditing practice. For instance, they were responsible for selecting the areas
of public administration or organizations to be audited. Additionally, they made
recommendations based on performance audits’ results and developed the performance
auditing rules and methodologies.
Table 6
Meeting Inclusion Criteria by the Participants
Participants’ Participants’ role in Meeting Professional work Meeting
Codes performance the general experience the
auditing criteria in total in performance
specific (years) auditing practice
criteria in 2016-2019 (years)
AC01
Auditor
Met
17
4
Met
AC02
Auditor
Met
4
4
Met
AL03
Auditor
Met
15
4
Met
AL04
Auditor
Met
7
4
Met
AC/L05
Auditor
Met
8
4
Met
PC06
Parliamentarian
Met
8
4
Met
PC07
Parliamentarian
Met
16
4
Met
PL08
Parliamentarian
Met
18
4
Met
PL09
Parliamentarian
Met
8
4
Met
MC10
Manager of auditee
Met
8
4
Met
MC11
Manager of auditee
Met
12
4
Met
ML12
Manager of auditee
Met
6
4
Met
ML13
Manager of auditee
Met
3
3
Met
ML/C14
Manager of auditee
Met
4
4
Met
Two parliamentarians are members of the Parliament of the Republic of
Kazakhstan, and two parliamentarians are members of the Maslikhat of Nur-Sultan city.
All four parliamentarians are members of committees of the central and local legislative
bodies principally responsible for coordinating economic, finance, and budget affairs.
These four parliamentarians where highly experienced with external auditor interactions
and legislative body presentations of performance audit results to include public hearings.
All five managers of auditees are senior-level managers in their organizations. They are
highly experienced with external auditors. These managers possessed legitimate powers
to request performance audits; however, none stated that they had ever initiated
performance audits to be conducted in their organizations.
To obtain a holistic view of the studied phenomena, I included in the list of
participants auditors, managers of auditees, and parliamentarians who work in
organizations that belong to the central and local levels of public administration. As
illustrated in Figure 10, one auditor and one manager of auditees had experience in both
levels of public administration, thus providing a good opportunity to explore differences
in their lived experiences depending on working in central or local organizations.
My study participant varied by age, gender, ethnicity, and religion; however, I did
not use these demographic characteristics to evaluate any significance in terms of the
phenomena under study. The study participants are qualified in their professional areas
and well-experienced in performance auditing; that fact allowed me to gather rich
information and met data saturation requirements.
Figure 10. The study participants distribution by groups
Data Collection
I started my data collection stage based on the IRB’s approval dated December 16,
2019. I conducted 14 phenomenological interviews to collect primary data. Additionally, I
reviewed more than 200 documents posted on official public sites of public audit
organizations, government entities, and quasi-public organizations all having been
subjected to performance audits in 2016–2019. Documents were also reviewed from
public websites of legislative bodies within the same time frame. For comparative
Auditors [AC]
;
2
Auditors [AC/L]
;
1
Auditors [AL]
;
2
Managers [MC]
;
2
Managers [ML/C]
;
1
Managers [ML]
;
2
Parliamentarians [PC]
;
2
Parliamentarians [PL]
;
2
analysis, I also reviewed reports on 2016–2019 performance audits posted on official
public web sites of supreme audit institutions of the United States, United Kingdom,
Australia, and Canada.
Participants Recruiting
I prepared the list of potential study participants using organizational charts and
contact information posted on the public-facing websites of the following organizations:
the Accounts Committee, Revision Commission on Nur-Sultan city, Ministry of Finance,
Department of Finance on Nur-Sultan city, Ministry of Healthcare, Department of
Healthcare on Nur-Sultan city, Parliament of the Republic of Kazakhstan, and Maslikhat
on Nur-Sultan city. I recruited the study participants using different strategies.
I called four auditors for 2 days and invited them to take part in my study. During
first calls, I explained to them (a) the purpose of my research, (b) conditions on their
participation such as voluntariness and confidentiality, (c) reasons for their selection, and
(d) their contributions to the research. Three auditors accepted the invitation, and we
agreed on the date, time, and place for their interview. One auditor preferred to meet
firstly. We met the next day after my call and, during our meeting, I explained the purpose
of my study in detail. The auditor was interested in my study, actively asked questions,
shared some examples from performance auditing practice, and expressed the view in
favor of conducting more performance audits. However, I did not use that information at
the next stages of my study since the potential study participants declined my invitation.
For those who declined the invitation to participate, I did not solicit explanations as to
why they declined. Some potential participants explained the reasons for their decision to
not participate, while others did not. I called the fifth auditor who asked me to call at a
later time. To be in line with my data collection plan, I moved on to the sixth auditor, who
accepted the invitation. The fifth auditor also accepted the invitation during my second
call. Five auditors ultimately participated in the study.
I sent invitation letters to four parliamentarians, and I called them as follow-up to
those letters across two days. Parliamentarians were informed about the purpose of my
research and their rights as participants. Like in the case of auditors, I explained to
parliamentarians why I selected them as potential participants and how I will ensure the
confidentiality of their participation. Two parliamentarians accepted the invitation, and
we agreed on the date, time, and place for the interview. One parliamentarian did not
reply, and one parliamentarian declined the invitation due to a busy schedule. Two
interviewed parliamentarians referred me to their equally qualified colleagues from the
deputy corps who might accept participation in the study. Through these snowball
methods, two parliamentarians were recruited. All parliamentarians were contacted using
publicly available, open-access contact information. In all, four parliamentarians agreed
to participated in the study.
Unlike the first and second groups, more time was needed to recruit and complete
interviews with managers of auditees. To obtain the participation of at least four managers
of auditees, as described in Chapter 3, I communicated with eight potential participants
from the third group. The first potential manager of the auditee accepted the invitation
during my first call. Two managers declined the invitation since they were not interested
in the topic of my research. Two managers agreed to participation but then asked to
reschedule interviews (twice and every time for a week). Due to these postponing
behaviors of two potential interviewees, I called three additional managers of auditees, all
of which accepted my participation invitation. One of the two managers who postponed
interviews ultimately declined study participation without explanation. Five managers of
auditees ultimately participated in my study.
Implementing the Interview Matrix
I conducted 14 interviews from January 8 to February 28, 2020. The total time of
interviews was approximately 12 hours (51.1 minutes per an interviewee on average),
segmented as 2.4 hours explaining provisions of the Informed Consent Form and
collecting demographic data and 9.6 hours for interview questions directly related to the
studied phenomena, as it is demonstrated in Table 7.
Table 7
Completed Interviews Matrix
Participants’
Codes
Being consent:
S – signed and
V – verbal
acknowledgments
Date
Mode
Place
Duration
(minutes)
AC01
S
01.08.2020
Face-to-face
Café
65
AC02
S
01.08.2020
Face-to-face
Café
70
AL03
S
01.09.2020
Face-to-face
Café
50
AL04
S
01.10.2020
Face-to-face
Business center
45
AC/L05
V
01.15.2020
Face-to-face
Café
65
PC06
V
01.16.2020
Face-to-face
Personal office
50
PC07
V
01.17.2020
Face-to-face
Personal office
40
PL08
V
01.20.2020
Face-to-face
Personal office
40
PL09
V
01.23.2020
Face-to-face
Personal office
60
MC10
V
02.11.2020
Phone
-
40
MC11
V
Phone
-
45
ML12
V
Phone
-
55
ML13
V
Phone
-
40
ML/C14
V
Face-to-face
Business center
50
As illustrated in Table 7, I conducted ten interviews using a face-to-face
technique. These interviews were conducted in private offices and private areas of cafes
and business-centers. Four interviews were required to be conducted by phone. Four
study participants signed the Informed Consent Form, and ten study participants verbally
acknowledged that they agreed to the provisions of the Informed Consent Form but
declined to provide signatures.
Twelve interviewees did not express objections to be audio recorded. Two
interviewees declined to be audio recorded but did accept that I could making detailed
notes that were then member-checked for validity after the interviews were conducted.
According to the Interview Protocol, I started audio recording after the introduction and
demographic sections; this strategy contributed to maintaining trust and maintaining
confidentiality in my communications with each study participant.
Personal data and other participant identifiers were coded using letters and figures.
I modified originally planned participants’ codes in order to incorporate needed changes
regarding additional information collection that was not anticipated with my initial study
design. For instance, I discovered that a few participants were experienced in
performance audits conducted by both the central and local public audit organizations. To
mask participants, I used letters A, P, and M depending on their roles in the process of
performance auditing, i.e., auditors, managers of auditees, and parliamentarians. The
letters C and L were applied depending on where the participant and their organizations
resided in the government’s public administration structure; central (C) or local levels
(L). I applied the code C/L for masking participants who worked at central and then at
local organizations, and the code L/C for masking participants who worked at local and
then at central organizations. Finally, I used figures from 01 to 14 as the serial number of
participants. Thus, instead of the originally planned codes, such as Auditor – 1, Parliam –
1, Manager – 1, I used modified coding, such as AC01, PC06, and MC10. For instance:
▪ AC01 means that the study participant is an auditor who worked at central
public audit organization, the first participant;
▪ PL08 means that the study participant is parliamentarian who worked at the
local legislative body, the eighth participant;
▪ ML/C14 means that the study participant is the manager of an auditee who
worked at local and then at central organization, the 14th participant.
The study participants selected their preferred language for interviewing. I
conducted six interviews in the Kazakh and Russian languages combined and eight
interviews in the Russian language alone. I wrote 12 interview transcripts and two
interview notes, and I printed them in Microsoft Word. I translated the significant
statements and results of manual coding into the English language. As described in
Chapter 3, my translations were verified by a translation agency according to our signed
confidential agreement.
I used the Interview Guide and Interview Protocol during all interviews,
regardless of their mode. To collect rich data, I changed the sequence of interview
questions and asked additional questions, depending on participants’ responses. I also
asked some questions with the member checking objectives.
The study participants were active and expressed readiness for further
communications in case of need. I reviewed 12 transcripts and concluded that
interviewees clearly and exhaustively responded to all questions. Notes from two
interviews were reviewed and accepted by respective participants during our meetings.
I made notes in my interview journal using symbols to mark behavioral aspects
and other results of my observations. Ten interviewees had previous experience in being
interviewed; for the other four participants, it was the first experience in being
interviewed. Despite these minor experience differences, all participants were open and
friendly throughout our interview time. All participants operated on facts and examples
from their lived experiences. However, parliamentarians often referred to theoretical and
conceptual things, citing examples from various areas of public administration. Auditors
mainly referred to applied things, but similar to the parliamentarians they operated on
examples from various areas of public administration. Managers generally referred to
legislative aspects and particularities in narrow (specific) areas of public administration.
Private areas for face-to-face interviews and times of phone interviews, which were out of
the business day, contributed to completing all 14 interviews effectively.
Evidence of Trustworthiness
By applying approaches described in Chapter 3, I ensured the trustworthiness of
my research guided by credibility, dependability, transferability, and confirmability
criteria.
Credibility
Data of 14 semistructured phenomenological interviews were the central sources
of information used for my study. Therefore, and to properly manage subjectivism issues
that might lead to risks of gathering misinformation or inaccurate data, I used group
characteristics and single sufficient case sampling methods to find and recruit the most
suitable study participants. I checked the compliance of selected participants with both
general and specific inclusion criteria when I prepared the list of potential participants
and I verified these inclusion criteria during our first communication. I received
exhaustive responses to all my questions which are, in turn, logically aligned with the
research question.
I carefully listened to all interviewees, made notes in my journal, and applied the
member checking techniques during interviews by asking clarification and probing
questions. As a result, 12 transcripts and two interview notes were compiled with rich
descriptions of my participants’ experience in performance auditing. As such, I created a
reliable contextual database with which to interpret meanings from their lived experiences
closely aligned as to how they were shared. I also used prolonged engagement tactics to
ensure the credibility of research findings. I communicated with each participant at least
three times to recruit them (providing them with information related to my study),
interview them (providing detailed explanations related to informed consent), and
member check the accuracy of their stories.
Transferability
Performance auditing practices of internal auditors, private audit companies,
local-level public audit organizations in other regions of Kazakhstan as well as
performance auditing practices in other countries were out of the scope of my study.
Nevertheless, using obtained thick descriptions and audit trails allowed me to illustrate
for future scholars and practitioners how to use the research results assessing their
applicability within other situations, other populations, and other timeframes. Instead of
studying the lived experiences of a specific group of participants, I focused on and
studied the lived experiences of three groups of participants who each play different roles
in performance auditing and public administration (i.e., public auditors, parliamentarians,
and senior-level managers of central ministries and local departments). Thus, the findings
from my research may be replicable for exploring unstudied phenomena related
specifically to performance auditing and to the wider public administration discipline.
Finally, I was able to improve transferability by following critical guidance and constructs
from the phenomenological approach and interpretive analyses.
Dependability
Like credibility and transferability, dependability was supported by using thick
descriptions and audit trials. I created detailed documentation to make my research work
transparent. Using the alignment between research stages based on my constructed
literature review and maintained through the constructs of my theoretical framework, I
was able to collect, organize, and manage data systematically. For instance, I created
separate working files for my primary and secondary data and the interview transcripts
were saved separately as Microsoft Word and Microsoft Excel working files. I stored all
the documents as password-protected files on my computer.
To analyze and interpret data analysis results, I used techniques on the
visualization of a large amount of complex and multi-sources information, e.g.,
comparative tables, dynamic charts, and cyclic diagrams. Finally, I used both manual and
computer-based data processing and analyzing methods.
Confirmability
To derive the results of my research from the lived experiences of my participants,
I used bracketing approach to collect and preliminarily analyze collected data and, then,
‘unbracketing’ approach. I applied the unbracketing approach to evaluate data collected
from multiple sources in a neutral and objective approach. Even being committed to
performance auditing ideology, I recognize that the performance auditing is not a
panacea. As a researcher, my objective was to collect data from multiple sources without
bias (see Moustakas, 1994; Patton, 2015, for more). Therefore, I collected data from all
key participants and users of performance audits including auditors, parliamentarians, and
managers of auditees.
I deployed multiple methods to reduce bias during both collecting and analyzing
data. During my communications with my participants, I remained neutral and did not
exert pressure on them. Awareness of my participants related to their rights on voluntary
participation, the possibility to withdraw from the study at any stage, for any reason and
without explanations, and exclusion of sanctions for withdrawal significantly contributed
to building a trusting relationship. Using the study participants’ quotes contributed to the
confirmability as well (see Lincoln & Guba, 1985, for more). Each participant, along with
their responses to my questions, was allowed to speak freely regardless of the questions
asked.
To ensure the reliability of used resources, truthfulness, and applicability of my
research results, I used triangulation to investigate related themes and potential
differences (see Patton, 2015, for more). Together with the interview data, including
transcripts and my journal entries for observing interviewees, I used peer-reviewed and
official documents of government organizations that are related to the topic of my
research. For systematic data collection, I used my created tools to include the Interview
Guide, Interview Protocol, and Interview Matrix. By monitoring and analyzing the
similarities and differences between the participants’ data, between the participants’ data
and secondary data, as well as marking the repeated information and the lack of new
information, I was able to draw conclusions related to data saturation.
Data Analysis
Applied Strategies and Techniques
I analyzed the collected data by using strategies defined at the initiation stage of
my study. Firstly, I analyzed participants’ data and secondary data separately; secondly, I
intergraded the results of these analyses to create a holistic and reliable view on the
phenomena under study.
I adopted Moustakas’s (1994) method on analyzing the phenomenological data.
Moustakas (1994) suggested using a modification of Van Kaam’s or Stevick-
ColaizziKeen’s method for data analyses and interpretation (pp. 120-124). These methods
differ by application of a researcher’s experience with phenomenological data
interpretation. I combined these two modified methods in the following way:
1) I used bracketing to keep a distance from my experience during data
collection and preliminarily data analysis, i.e., before the interpretation of
final results;
2) I incorporated the thematic analysis method to generate and validate themes
associated with the studied phenomena;
3) I used the ‘unbracketing’ approach to critically interpret specific results of my
study, including discrepancies between data from different sources;
4) I kept to the following algorithm:
a. To be familiar with the raw data (epoch);
b. To generate themes based on significant statements
(phenomenological reduction);
c. To create textual and structural descriptions related to the phenomena
under study (imaginative variation);
d. To consolidate the research findings by the composition of textual and
structural descriptions. (Moustakas, 1994; Van Manen, 2014) For data
analysis, I used an inductive approach that implies making analytical
interpretations based on the collected data and by referencing my
personal experience (see Moustakas, 1994, for more). While the
modification of the Van Kaam’s method implies distancing from a
researcher’s and, vice versa, the modification of StevickColaizzi-
Keen’s method implies using a researcher’s experience; I did not use
my knowledge and experience as a source of information to analyze. It
would lead to cognitive dissonance between the two methods. So, I
modified my data analysis strategy by referring to my experience to
interpret discrepancies between the data collected from different
sources.
I analyzed data manually and by utilizing NVivo 12 Plus for qualitative data
analyses. I wrote and printed 12 transcripts and two interviews notes in Microsoft Word.
Before transcribing, I listened each audio records at least two times. Utilizing NVivo 12
Plus, I translated all significant statements and results of my manual analysis from the
Kazakh and Russian languages into the English language. Like the primary data, I
analyzed the secondary data in the original language of the document, i.e., in the Kazakh
or Russian languages; I then translated all significant statements and results of manual
coding into the English language. The translation of all data was inadvisable since they
were voluminous and contained insignificant statements. I reviewed translations multiple
times to ensure and minimize the risks of loss or alteration of original data due to
translation errors or contextual misinterpretations between the spoken interview scripts
and their eventual reduction to English text.
All codes were assigned based on my collected data thus avoiding a process of
creating pre-codes for thematic matching. Table 8 illustrates how I assigned both
descriptive and conceptual codes based on my in vivo coding approach, i.e., codes
derived from my participants’ lived experiences as well as a value coding approach.
Table 8
Examples of Assigned Descriptive and Concept Codes
Participants’
Codes
Raw data (extracts from
interview transcripts)
Concept codes
Descriptive codes
AC01
In other words, we decide on the
conduction of performance
audits when we develop the list
of auditees for a planning year.
performance
audit list of
auditees
planning
making decision on
performance audits
annual planning of
performance audits
PC06
Sometimes we request public
audit organizations to provide as
with the information about the
results of their audits. But we
have right to send our request to
any other organizations.
public audit
organizations
results of
audits
mandate other
organizations
legislative body’s
procedure on requesting
audit results legislative
body’s procedure on
requesting other
information; [results of
audits are in demand:
parliamentarians’
perspective]
AL03
In 2018 we conducted
performance audits in response
to citizens’ complaints…
increasing their complaints. We
know about their complaints
from i-komek system.
performance
audit citizens’
complaints i-
komek
system
making decision on
performance audits;
[performance audits are in
demand: citizens’
perspective] using open
government
systems to plan
performance audits and
respond to citizens
ML/C14
Any auditors’ visits are stressful
for me. The main problem is that
we have many tasks that should
be done in any case... I mean it
doesn’t depend on auditors’
visits… But auditors add tasks.
auditor
problem
workload tasks
interactions between
managers of auditees and
auditors stress response
to audit impact on
working order
I used a value coding approach as it helped to evaluate and provide context to
perceived demand for and effects of performance audits and my participant’s attitude
toward these phenomena. When assigning codes and generating themes, I considered
each participants’ statement to be of equal importance, and I then conducted a deeper
review to identify those statements illustrating more significance to the collective
participant group than others. I used alignment with the research question as a landmark
for confirming the significance of statements and reliability of themes.
I used Microsoft Word in both cycles of coding, identification of significant
statements, selection of the study participants’ quotes in order to use them further as my
evidence base. Color coding, tables, and diagrams were used to assign codes, formulate
categories, and generate themes. Next, I used Microsoft Excel to evaluate codes,
categories, and themes in terms of their validity and alignment with the research question.
I used spreadsheets to identify similarities and differences among codes and categories
and to create the descriptive statistics of the research results. For instance, I quickly
calculated the number of codes and identified the dominance of specific themes. At the
final stage, I utilized NVivo 12 Plus to check the results of manual analytical work and
provide a visualization of the results analyses.
Consolidated Statistics of Data Analysis Results
I assigned 7,494 codes as the result of the 1st cycle of coding and 1,085 concept
and descriptive codes as the results of the 2nd cycle of coding. I then created 97
categories focusing on the research question. Based on textual and structural descriptions,
I generated 11 themes related to the perceived factors of demand for performance audits
in Kazakhstan and seven themes and two sub-themes related to the perceived impact of
performance audits on Kazakhstani system of public administration. Eighteen themes and
two sub-themes were divided into six patterns associated with the studied phenomena in
general. As demonstrated in Figure 11, I derived six patterns associated with the studied
phenomena from 7,494 codes, applying the technique of phenomenological reduction.
Figure 11. The aggregated results of phenomenological reduction
I created all categories based on significant statements. Whereas participants
belonged to three different groups, a specific theme might be nonessential for one group
and essential for the other two groups. Therefore, I generated themes under the following
conditions:
(a) the frequency of categories associated with a potential theme is one or more,
in one group of participants; and
(b) the frequency of categories associated with a potential theme is nine or more
in three groups of participants.
I identified that participants of performance audits and other key participants of
public administration create the demand for performance audits (the 1st pattern). The
features of the system of public auditing (the 2nd pattern) and problems in public
administration (the 3rd pattern) emerged as influential factors helping to explain the
demand for performance audits in Kazakhstan. According to the lived experience of
participants, performance audits impact performance auditors, managers of auditees, and
parliamentarians (the 4th pattern), audit organizations and auditees, i.e., audited
organizations (the 5th pattern), and these audits have macrolevel impact (the 6th pattern),
as it is illustrated Figure 12, which has been created by using the initial outline described
in Chapter 3, and in Table 9.
Figure 12. Visualization of patterns, themes, and sub-themes
Table 9
Aggregated Results of Data Analysis
The research question:
What are the lived experiences and perceptions of performance auditors, managers of auditees,
and parliamentarians, the key participants and users of performance audits, regarding the
demand for and impacts of these audits on Kazakhstan’s public administration?
The perceived factors of demand for
performance audits in Kazakhstan
The perceived impact of performance audits on
Kazakhstani system of public administration
code name
code
Name
Who creates demand for
Pattern 1 audits
Pattern 1
Impact on participants of
audits
Auditors prefer performance
Theme 111 audits
Theme 211
Impact on auditors
Parliamentarians need in
Theme 112 performance audits
Theme 212
Impact on managers of
auditees
President prioritize
Theme 113 performance audits
Theme 213
Impact on
parliamentarians
Managers of auditees use Impact on audit
Theme 114 Pattern 2
performance audits’ results organizations and auditees
Theme 115
Citizens and media shape the
demand for performance
audits
Theme 221
Changes in audit
organizations
Pattern 2
Features of the system of
public auditing
Theme 222
Changes in auditees
Theme 121
Legitimization of performance
audits
Pattern 3
Macrolevel impact
Theme 122
Adaptation of international
experience
Theme 231
Improved budget process
Pattern 3
Problems in public
administration
Theme 232
Changes in laws and
regulations
State planning
Theme 131
Sub-theme 233 Self-contribution to 3E
Systems’ problems
Theme 132
Risks of new areas
Theme 133 Requested changes in
Sub-theme 234 performance auditors’
Needs in evaluating the work
Theme 134
effectiveness
Results
Perceived Central Concepts
Auditors, when sharing their experiences, most often used the words audit (audits)
and performance. I found these words were used most in their sense of performance
audit, and it is explained mainly by auditors’ central role in a performance auditing
process – starting from initiation and planning audits till submission of audit reports to
the end users. Next by frequency were the words compliance, auditees, and budget. The
word compliance was frequently used in the sense of compliance audit. In most cases,
auditors used this concept to express their preferences between compliance and
performance audits. The words budget and auditees were often used by auditors in the
sense object of and subject of performance audits, respectively. Figure 13 illustrates how I
distilled central concepts related to my studied phenomena from auditors’ perspectives
based on their [concepts] thematic size (weighted percentage of the words audits (audit)
was 4.23%, performance – 2.37%, compliance – 0.70%, auditees – 0.65%, and budget –
0.58%), applying NVivo Plus analysis techniques.
Figure 13. Perceived central concepts of auditors
Like auditors, parliamentarians often used the words audit (audits) and
performance in the sense of performance audit. However, within the parliamentarian
group these descriptive words were mainly explained to be related to the implementation
of parliamentary oversight functions. Next by frequency was the word budget.
Parliamentarians often used that concept to share their experiences in using auditors’
work to approve budgets of the central and local governments and audit organizations’
annual reports submitted to legislative bodies as a monetary measure of performance
success. Parliamentarians also often used the word result to share their perceptions related
to the expected effectiveness of government programs and projects. Finally, as
demonstrated in Figure 14, parliamentarians frequently used the words auditors,
parliamentarians, and government with the same frequency.
Figure 14. Perceived central concepts of parliamentarians
As illustrated in Figure 15, the third and final participant group, managers of
auditees, often used the word auditors. This descriptive word was mainly related to
managers’ perceptions associated with their interactions with auditors. Next by frequency
were the words audit (audits) and performance, and managers used these words in the
sense of performance audit less often than auditors and parliamentarians. The concept of
performance audit was used by managers of auditees to share their views about the
absence of significant differences between performance and compliance audits. Often,
managers used the concept of performance to tell stories about their organizations’
achievements. Managers often used the word budget to share their experiences on
participation in budget processes, such as planning and implementing the budget
programs.
Figure 15. Perceived central concepts of managers of auditees
The concept of performance audit was central for many of the participants, as it is
demonstrated in Figure 16; that concept was introduced in the course of implementing
NPM (see Jacobs, 1998; Funnell, 1997; Morin, 2001, for more). Participants’ perceptions
associated with the budget concepts are also related to NPM; e.g., auditors shared their
experiences in evaluating the effectiveness of budget programs, parliamentarians – on
oversight over the budget process, and managers – on adapting result-oriented budgets.
Finally, unlike completing the processes, achieving the results is one of the central
performance indicators introduced by NPM (see Hood, 1991, for more).
In sharing their lived experiences, 100% of interviewees identified specific
aspects of auditors, auditees, and parliamentarians’ interactions. Auditors and managers
mainly focused on their interpersonal interactions, whereas parliamentarians more
specifically discussed the interaction of government’s role and responsibilities. In all
cases, participants’ experiences were linked with PAM, i.e., the model of interactions
between agents and principals (see Waterman & Meier, 1998, for more).
Figure 16. Perceived central concepts of all participants
Perceived Factors of Demand for Performance Audits
According to experiences of 12 participants (85.7%), the scope of performance
audits conducted in Kazakhstan in 2016–2019 across the entire public sector was
increased. Secondary data analysis illustrated this same trend. In 2016–2019, Kazakhstani
public audit organizations undertook more than 400 performance audits covering more
than 1,600 auditees per year. As demonstrated in Table 10, the audited areas of public
The perceived central
concepts
Weighted
percentage (%)
performance audit
)
audits
(
7.57
budget
1.11
results
0.79
auditors
1.71
auditees
0.63
parliamentarians
053
government
0.53
administration included state planning (i.e., performance audits of state programs,
programs of development of regions, strategic plans of central ministries, and
development plans of state-owned companies), budget planning, tax administration,
infrastructure and investment projects in core sectors of the economy (i.e., subsoil use,
agriculture, construction, transport and communications, healthcare, and education), and
programs specific to social care for citizens.
Table 10
Performance Auditing Statistics in Kazakhstan
2016 2017
2018
2019
Accounts Committee
Completed performance
audits, weighted (%)
65
65
55
60
Areas of public
administration covered
by performance audits
state planning,
budget
planning,
infrastructure
projects
state planning,
tax
administration,
education
programs
tax
administration,
healthcare
programs,
investment
projects
infrastructure
and investment
projects, social
care for
citizens,
quasipublic
companies
Revision Commissions
Completed performance
audits, weighted (%)
40 45
45
60
Areas of public
administration covered
by performance audits
healthcare
and healthcare education
programs and programs, and
providing providing social care
for social care for citizens
citizens
state planning,
infrastructure
projects,
education
programs, land
use
state planning,
healthcare
programs, and
providing
social care for
citizens
Note: ACCERB (n.d.); RCNC (n.d.); RCAR (n.d.); RCAR (n.d.); RCAR (n.d.); RCAR (n.d.); RCWKR (n.d.); RCZR
(n.d.); RCKR (n.d.); RCKR (n.d.); RCKR (n.d.); RCMR (n.d.); RCPR (n.d.); RCNKR (n.d.); RCEKR (n.d.); RCAC
(n.d.).
Pattern 1: Who creates demand for audits. Themes 111 to 115 illustrated
evidence that auditors, managers of auditees, parliamentarians, and the key participants of
public administration, such as the President of the Republic of Kazakhstan, citizens, and
media create demand for performance audits, as it is demonstrated in Figure 17.
Figure 17. Visualization of themes on Pattern 1 by the frequency of categories
21
18
15
9
14
7
32
5
5
3
7
7
10
4
2
Theme 111 - Auditors
prefer performance
audits
Theme 112 -
Parliamentarians need in
performance audits
Theme 113 - President
prioritize performance
audits
Theme 114 - Managers
of auditees use
performance audits'
results
Theme 115 - Citizens
and media shape the
demand for performance
audits
Managers' perceptions
Parliamentarians' perceptions
Auditors' perceptions
The need of parliamentarians for performance audits is the most influential factor
of demand for these audits; all five themes emerged in three groups of participants with
different levels of frequencies, as it is demonstrated in Table 11.
Table 11
Pattern 1: Matrix of Themes and Frequency of Associated Categories
Theme Frequency of themes by participants Weighted
Group
1:
Auditors
Group
2:
Parliamentarians
Group
3:
Managers
Total
percentage:
Frequency
of
categories
Theme 112:
Parliamentarians need in
performance audits
5 of 5
(100%)
4 of 4
(100%)
2 of 5
(40%)
11 of 14
(78.6%)
35.8%
Theme 111: Auditors 5 of 5 4 of 4 4 of 5 13 of 14
22.0%
prefer performance audits (100%) (100%) (80%) (92.9%)
Theme 113: President
prioritize performance
audits
5 of 5
(100%)
2 of 4
(50%)
4 of 5
(80%)
11 of 14
(78.6%)
18.9%
Theme 114: Managers of
auditees use performance
audits’ results
5 of 5
(100%)
2 of 4
(50%)
2 of 5
(40%)
9 of 14
(64.3%)
11.9%
Theme 115: Citizens and
media shape the demand
for performance audits
5 of 5
(100%)
2 of 4
(50%)
3 of 5
(60%)
10 of 14
(71.4%)
11.3%
Parliamentarians need in performance audits. All auditors stated that
parliamentarians use performance audits. Two auditors more specifically described that
parliamentarians, who are members of the central legislative body, do not directly request
conduction of performance audits; however, they often request information about
performance audits. In turn, three auditors noted that members of Maslikhat request the
local audit organization to conduct performance audits and actively participate in the
meetings of this organization on considering performance audits’ results. AL04 argued
that parliamentarians are more interested in performance audits rather than in compliance
audits. “Performance audit provokes intensive discussions, and that’s why it is interesting
to deputies. We audit areas with unresolved issues, problematic issues that are topical.
Then, these problems become high on the agenda of parliamentary hearings” (AL04,
personal communication, January 10, 2020). All auditors stated that parliamentarians are
interested in the results of performance audits included in annual reports of audit
organizations submitted to the Parliament and Maslikhats. According to three of the five
auditors, parliamentarians are awaiting reports from their organizations longingly.
Auditors’ perceptions were generally in line with parliamentarians’ views. Three
of the four parliamentarians send requests to audit organizations in order to provide them
[parliamentarians] with information about performance audits, and all four
parliamentarians used the results of performance audits in their policy making activities.
PC07 shared experiences on using the results of performance audits to lodge deputy
inquiries and send related requests to the Prime Minister, central ministries, and agencies.
PL09 shared experiences on using the performance audits’ results in making legislative
proposals. Three parliamentarians opted for performance audits comparing the
compliance and financial audits with performance audits. “It is important to conduct more
and more performance audits… Compliance and financial audits is an area of interests for
ministers and heads of departments. They all are about routine things. I am not very
interested in such things” (PL08, personal communication, January 20, 2020).
Three parliamentarians reasoned that performance audits will be in demand in the
future.
So far, we have not practiced sending the requests on the conduction of
performance audits to the Accounts Committee… [thinking]… but in the future,
probably, it will be realized, because we are preparing the package of bills to
strengthen parliamentary oversight. In my point of view, the strengthening of
parliamentary oversight should also include the practice of requesting audits.
(PC06, personal communication, January 16, 2020)
Unlike auditors and parliamentarians, managers of auditees were less active on
Theme 112. Two of the five managers shared their opinion about using the performance
audits by parliamentarians. They gained related experiences when they receive deputy
inquiries and attend the meetings of audit organizations on considering the performance
audits’ results.
Auditors prefer performance audits. All auditors are experienced in initiating the
performance audits, and they emphasized that suggestions on the conduction of
performance audits, as a rule, are supported by auditors and senior-level managers of their
organizations. “Since we are a collegial body, all decisions made collectively... almost
always, proposals on performance audits are accepted unanimously” (AL04, personal
communication, January 10, 2020). Telling about the conduction of compliance and
performance audits, auditors noted the advantages of compliance audits in six cases and
the advantages of performance audits in 14 cases. When I asked them to select their
preference of compliance or performance audits, all five auditors opted for performance
audits.
There were no situations when my colleagues did not support ideas on
performance audits… I suggested a few topics for the future, for example,
performance audits on the Sustainable Development Goals. We cannot manage
this kind of audits today because of a lack of resources, but we will do them in the
future. (AC/L05, personal communication, January 15, 2020)
AC01 and AL04 also noted that a few decisions on postponing the proposed
performance audits for the midterm were made due to a lack of auditors. Therefore, like
parliamentarians (Theme 112), auditors might continue to create the demand for
performance audits (Theme 111).
The frequency of categories on Theme 111 in the groups of parliamentarians and
managers was 7 in each group (20.0% out of the frequency of categories associated with
Theme 111 in all three groups). Unlike auditors, they were less active in discussing the
auditors’ interests in performance audits. As PL08 noted, auditors conduct more
performance audits because of the “political motivations of leaders of audit
organizations” (PL08, personal communication, January 20, 2020). “A performance audit
is the most popular trend among auditors; performance audits are mainstream activities of
all audit organizations in our country” (ML12, personal communication, February 15,
2020).
President prioritize performance audits. Eleven interviewees (78.6%) were
experienced in performance audits conducted by audit organizations in response to the
instructions of the President.
AC01 stated that including the performance audits in audit organizations’ plans is
a priority since “…per the recent message from the Head of State, we need to get away
from conduction of current controls and we have to conduct more and more and more
performance audits” (AC01, personal communication, January 8, 2020). Similar
references to prioritizing the performance audits by the President made the other three
auditors. AL04 emphasized that “if we receive the instruction of the President
Administration to undertake a performance audit, then we include that audit in our plan
without fail” (AL04, personal communication, January 10, 2020). All five auditors noted
that they participated in performance audits requested by the President Administration;
three auditors shared their experiences on performance audits conducted in response to
the President’s instructions during the days of our interviews. “Now, I’m participating in
the audit of quasi-public companies requested by the President. I know that other audit
organizations in parallel with us work on the same requests” (AC02, personal
communications, January 8, 2020).
Influencing the demand for performance audits by the President’s instructions as
the theme on Pattern 1 emerged in the experiences of two parliamentarians. PC07 stated,
“multiple times I and my colleagues from deputy corps questioned the effectiveness of
our national companies, but only now the auditors are stepping up and this, of course, is
because they received an order from President” (PC07, personal communication, January
17, 2020).
Four of the five managers associated conducting the performance audits with the
President’s instructions. “Audit organizations, no difference, Accounts Committee or
Revisions Commissions, they all in the case of President’s criticism of the work of
ministries or state companies or instructions from the President Administration respond
by their audits” (МL/С14, personal communication, February 28, 2020). Two managers
shared their experience of working together with auditors on the instructions from the
President Administration. In one case, health professionals were included in the team of
auditors, and internal auditors of the Ministry of Healthcare transferred their reports to the
Accounts Committee. In another case, several central ministries create the working group
on the evaluation of state-owned companies.
Citizens and media shape the demand for performance audits. Based on the
lived experiences of nine participants (64.3%), I concluded that citizens and media, to a
certain degree, shape the demand for performance audits in Kazakhstan.
All five auditors stated that some performance audits were conducted in their
organizations due to citizens’ constraints and needs. “There are a lot of complaints year
after year. In 2018 we conducted performance audits in response to citizens’ complaints
on the provision of urban amenities; they were unhappy with the quality of landscaping
and so forth” (AL03, personal communication, January 9, 2020).
We initiated the performance audit on safety-net programs for people with
disabilities. All supported that suggestion without any objections because it is
important, and we did it. And, recently, measures on social support for large
families, pensioners, and children have been very much discussed or even
criticized in our city. That’s why we conduct our audits. (AL04, personal
communication, January 10, 2020)
Three auditors mentioned that they use media publications in making decisions on
performance audits.
We study journalists’ articles and monitor media publications. It is required by our
rules. It helps us to make our decisions on our audits. Today, working with the
government’s reports or auditees’ accounting documents is not enough to propose
a good performance audit. I mean a useful performance audit. (AC/L05, personal
communication, January 15, 2020)
Three auditors shared their experiences on interactions with media; according to
their perceptions, the media actively use their performance audits’ results. As AC02
noted, journalists accept invitations to the meetings of audit organizations and actively
use audits’ results in their investigations.
Two parliamentarians referred to the needs of citizens, including users of public
services, when they shared their views about the importance of performance audits.
Citizens’ needs are important. By auditing the effectiveness of the efforts of our
governments in addressing citizens’ problems, auditors provide us with good materials.
Auditors should continue to focus their efforts on social problems, problems of residents
of our city. (PL09, personal communication, January 23,
2020)
Two managers shared their opinions on using the performance audits by media; in
both cases, their experiences were related to publishing resonant articles based on auditors’
work.
Managers of auditees use performance audits’ results. Unlike the previous four
themes on Pattern 1, Theme 114 is less highlighted; but, 10 participants (71.4%) shared
views on using the results of performance audits by managers of auditees.
Four auditors (80%) stated that managers of auditees did not request performance
audits formally, and two auditors shared their expediencies when managers of central
ministries requested audits informally. “I know that they ask for audits, but they did not do
it by official requests, by sending letters. They ask for audits informally by phone calls…
[smiling]…” (AC02, personal communication, January 8, 2020). Two auditors mentioned
that newly appointed ministries request audits to be aware of operations completed before
their appointment.
They must minimize the risks of financial violations, but they are interested in all
things that could be criticized by performance auditors as well. Here, I want to
add that only new ministries request audits, and they do it informally. (AC/L05,
personal communication, January 15, 2020)
All five auditors argued that managers of auditees used the results of their works
because most of their recommendations were implemented by auditees. AC02 shared the
experience of using the performance audits’ results by managers of auditees. I see that
now they perceive our work as a means to achieve their objectives or address their
problems. It was, let’s say, managers’ poor dealing with their old problems, and they
were unable to solve these problems themselves, without us.
(AC02, personal communication, January 8, 2020)
Two parliamentarians stated that using the results of audits depends on both
managers and auditors. “Ministries or chairs of central agencies are different. Few
ministries use the auditors’ work in the right way, I see it because of their reactions to our
questions, but the others are not so active” (PC06, personal communication, January 16,
2020). “Some chairs are pro-active, while the others, unfortunately, they represent the
majority, respond to performance audit results sluggishly. Honestly, I have to say that in
some cases auditors’ work caused that kind of reaction” (PL08, personal communication,
January 20, 2020).
“About using the audits, it is the right direction. As one of the key players in
budget processes, our ministry is interested in performance audits ... criticism, of course,
is unpleasant. But we are interested in an external assessment” (MC10, personal
communication, February 11, 2020). Other managers’ perceptions, however, mainly relate
to the requirements of legislation on implementing auditors’ instructions. “For us, it is
like a functional task because we must review that audit report, we must implement
instructions. We try to implement auditors’ instructions, but we have more freedom with
recommendations” (ML12, personal communication, February 15, 2020).
Pattern 2: Features of the system of public auditing. According to the lived
experience of participants, features of the system of public auditing in Kazakhstan is
another factor that explains the demand for performance audits, including the
legitimization of performance audits (Theme 121) and adapting an international practice
(Theme 122). As demonstrated in Table 12, the legitimization of performance audit is
more of an influential factor than demanding the performance audits to be done.
Legitimization of performance audits. According to the experience of 12 (85.7%)
participants, demanding performance audits in Kazakhstan is related to the legitimization
of these audits, i.e., audit organizations conduct more performance audits because they
were empowered to do so by the legislation.
Table 12
Pattern 2: Matrix of Themes and Frequency of Associated Categories
Theme Frequency of themes by participants Weighted
Group
1:
Auditors
Group
2:
Parliamentarians
Group
3:
Managers
Total
percentage:
Frequency
of
categories
Theme 121:
Legitimization of
performance audits
5 of 5
(100%)
4 of 4
(100%)
3 of 5
(60%)
12 of 14
(85.7%)
73.2%
Theme 122: Adaptation of
international experience
3 of 5
(60%)
3 of 4
(75%)
2 of 5
(40%)
8 of 14
(57.1%)
26.8%
When I asked auditors about making decisions on performance audits, all five
auditors started sharing their experiences from the planning audits, making multiple
references on the Public Auditing Law or on the Rules on Conduction of Public Audits by
External Public Audit Organizations. As AC/L05 noted, “We have the mandate … to
express an opinion on the Government’s report over execution of the republican budget
and, so, we have to annually conduct performance audits on budget revenues…”
(AC/L05, personal communication, January 15, 2020). Auditors, when answering other
questions, also referred to mandates of their organizations to conduct performance audits.
Three auditors claimed that their organizations plan and conduct more performance audits
because the introduction of these types of audits is a priority state policy spelled out in
strategies and concepts. Finally, AL04 stated that “to comply with recommendations of
the Coordinating Council of Public Audit Bodies, established under the auspices of the
Accounts Committee, audit organizations should strive to ensure that more than 70% of
their audit activities related to performance audits” (AL04, personal communication,
January 10, 2020).
Like auditors, all parliamentarians connected the increased number of
performance audits with empowering of audit organizations. In most cases,
parliamentarians referred to audit organizations’ conclusions on governments’ reports
detailing execution of the republican and local budgets. All four parliamentarians stated
that audit organizations’ conclusions are made based on performance audits results; they
support it. Two parliamentarians reasoned that audit organizations have a wide range of
mandates, and it allowed those audit organizations to enhance influence in
parliamentarian decision making.
In describing the experience of interactions with auditors, three of the five
managers referred to auditors’ mandates to undertake performance audits.
They visit our ministry every year. Every year. Before we saw them less often. Of
course, I understand their objectives. They visit us to gather facts on budget
execution and to report to Parliament. Auditors also need to report on how they
cope with their mandates… [smiling]. (MC10, personal communication, February
11, 2020)
Adaptation of international experience. The adaptation of audit practices of
foreign countries, as revealed in the experience of 8 participants (57.1%), also can be
interpreted as one of the influential factors of demanding the performance audit in
Kazakhstan. In particular, auditors and parliamentarians shared their perceptions about
the practices of United Kingdom (three times), the United States and Latvia (two times),
and Australia and Canada (one time).
According to the experiences of three auditors, introducing performance audit
developments in their organizations are due to adapting international practices. I
participated in international seminars, and I liked how auditors from various countries, for
example, Canada and Latvia, presented their achievements in studying the environmental
issues within their performance audits. We did not conduct such audits. So, I suggested to
conduct environmental performance audits, and all supported my idea. (AC/L05, personal
communication, January 15,
2020)
However, negative perceptions were shared by auditors in terms of adapting these
international experiences. As AL04 mentioned, “… we use international practice to
conduct our performance audits, study different methods, and we try to apply them, but
we do not succeed in some cases…” (AL04, personal communication, January 10, 2020).
Three parliamentarians were well aware of the adaptation of international
practices by audit organizations. As PL08 noted, “I know that auditors travel to foreign
countries, study international experience, they do it to develop their own rules and
internal work to improve their performance audits” (PL08, personal communication,
January 20, 2020). Unlike auditors, all three parliamentarians spoke positively about the
practice of adapting world experience, and two parliamentarians argued the need to
continue implementing the best international practice. “In Great Britain, they also have a
unitary state. Nevertheless, their Parliament reviews the materials of each performance
audit. The same practice in Latvia. I would also like to have such a practice in our
country” (PC06, personal communication, January 16, 2020). Two managers also shared
their experiences associated with international practice.
We worked on the draft concept on the introduction of public auditing. The
implementation of this concept implies using international practice, including the
benchmarks in performance auditing. But our auditors are still very far from the
best experiences. It is my, may be very subjective, but my view. (MC 10, personal
communication, February 11, 2020)
Secondary data analysis on Theme 121 resulted in the following conclusion. The
national standards on performance auditing are generally in line with the Performance
Audit Principles, also known as ISSAI 300. Review of themes illustrated few differences
between the compared standards. These differences are interpreted to be not significant;
they are not-matching aspects of ISSAI 300 as regulated by the National standards on
public auditing or the Rules of conduction of external public audit, as it is illustrated in
Table 13.
I also compared the provisions of performance audits reports of the Accounts
Committee and the Revision Commission on Nur-Sultan city (by two reports from each
organization) published in January and February 2020 with the provisions of performance
audits reports of the SAIs of Australia, the United States, and United Kingdom (by two
reports from each country) published in the same period, and two reports of the SAI of
Canada published in December 2019. Table 14 illustrates the main results of comparing
these SAI’s performance audit reports.
Table 13
Results of Comparing Performance Auditing Standards
National standards International standards
100. Procedural standard – extracts
and descriptions
ISSAI 300 - Performance Audit
Principles – extracts
Conclusions
Audit objective
… expression an independent, competent, and …constructively to promote no objective
opinion on economy, effectiveness, economical, effective and significant and
efficiency in an audited area... efficient governance... differences
Audit scope
tax and custom administration, budget
planning and execution, assets
management, implementation
strategic documents, pricing and public
procurement, public debt
management, using the grants,
investments, and loans, environmental
protection, information technologies,
auditees’ activities
… need not be limited to specific
programmes, entities or funds but can
include activities… or existing
situations… Examples might be service
delivery by the responsible
parties or the effects of government
policy and regulations on
administration, stakeholders,
businesses, citizens and society.
no
significant
differences
Audit approach
Audit team selects result-, problem- or
system-oriented approaches or a
combination these approaches.
… a result-, problem- or
systemoriented approach, or a
combination thereof…
no
differences
Audit criteria
Audit team use the basic criteria,
including the criteria related to the
principles of economy, efficiency and
effectiveness, and specific criteria.
… suitable criteria which correspond to
the audit questions and are related to
the principles of economy, efficiency
and effectiveness.
no
differences
Strategic planning
Audit organizations develop Auditors should select audit topics
perspectives plan for performance through the SAI’s strategic planning
auditing through identifying potential process by analysing potential topics
performance audit topics based on the and conducting research to identify study
of public management system. risks and problems.
no
significant
differences
Recommendations and follow-up
recommendations and
follow-up related are
included in the list of
objectives of performance
audit
…provide constructive recommendations that are
likely to contribute significantly to addressing the
weaknesses or problems identified by the audit
Auditors should follow up previous audit findings
and recommendations wherever appropriate.
no
differences
Audit report
*Provided references to the Rules Auditors should seek to make their reports
of conduction of external public widely accessible, in accordance with the
audit. mandate of the SAI.
no
significant
differences
Note: INTOSAI, 2016a; ISLARK, 2016b.
Table 14
Results of Comparing Performance Audit Reports
Australia Canada Kazakhstan The United United Conclusions
States Kingdom
Audit objective is related to evaluation of economical, effective and efficient governance
Yes Yes Yes Yes Yes no significant differences
Auditee scope is not limited specific programmes, entities or funds
Yes Yes Yes Yes Yes no significant differences
Audit criteria is related to evaluation of economical, effective and efficient governance
Yes Yes Yes Yes Yes no significant differences
Recommendations is related to addressing the weaknesses or problems identified by the audit
Yes Yes Yes Yes Yes no significant differences
Follow-up related info incorporated into audit reports
Yes Yes Yes Yes Yes no significant differences
Note: Australia: Audit report 1: Aboriginal and Torres Strait Islander Participation Targets in Major Procurements,
National Indigenous Australians Agency (ANAO, 2020a); Audit report 2: Management of Spectrum Reallocation to
Support the Deployment of 5G Services, Department of Communications and the Arts Australian Communications and
Media Authority (ANAO, 2020b). Canada: Audit report 1: Call Centres, Department for Employment and Social
Development Canada (OAG, 2019a); Audit report 2: Departmental Progress in Implementing Sustainable Development
Strategies, Across Departments (OAG, 2019b). Kazakhstan: Audit report 1: Using the Republic Budget’s Funds and
Assets Allocated for Exploration and Subsoil Use Sectors, The Ministry of Energy of the Republic of Kazakhstan
(ACCERB, 2020a); Audit report 2: Using Budget Funds Allocated for the Development of Kostanay Region
(ACCERB, 2020b); Audit report 3: Implementing the Action Plan for State Program on Development of Education and
Science in the Republic of Kazakhstan for 2016-2019 and Material and Technical Equipment of Educational
Organizations in Nur-Sultan city (RCNC, 2020a); Audit report 4: Communal Property and Assets Management of the
JSC “Social-Entrepreneurial Corporation “Astana” (RCNC, 2020b). The USA: Audit report 1: Enhanced Federal
Information Sharing on Coordination Could Improve Rural Transit Services, The Federal Transit Administration
(USGAO, 2020a); Audit report 2: Opportunities Exist to Address Water Quality Problems, International Boundary and
Water Commission (USGAO, 2020b). UK: Audit report 1: Business support schemes, Department for Business, Energy
& Industrial Strategy (UKNAO, 2020a); Audit report 2: Local authority investment in commercial property, Ministry of
Housing, Communities & Local Government (UKNAO, 2020b).
While performance auditing practices and approaches to write performance audit
reports vary depending on the features of public administration, powers, and traditions of
public audit organizations, I made fragmentary analysis using a few core aspects of
performance auditing outlined in ISSAI 300 (see Barton et al., 2019; Jantz et al., 2015;
Rosa et al., 2014a; Tillema & Bogt, 2016; Torres et al., 2019, for more). So, as
demonstrated in Table 14, this content evaluation illuminated some similarities in
approaches to performance audits of the compared countries.
Pattern 3: Problems in public administration. Problems in public
administration (Pattern 3) is the most influential factor on demanding performance audits
in Kazakhstan compared to the previous two patterns. According to the lived experience
of participants, the demand for performance audits is explained by problems in state
planning (Theme 131), systems problems in various sectors of the economy (Theme 132),
risks of new areas (Theme 133), and the need in the evaluation of the effectiveness of
government programs and activities (Theme 134). As demonstrated in Figure 18 and
Table 15, the most highlighted theme is Theme 132, at the same time, all four themes on
Pattern 3 emerged in all participant groups with different levels of frequency.
Figure 18. Visualization of themes on Pattern 3 by the frequency of categories
Table 15
Pattern 3: Matrix of Themes and Frequency of Associated Categories
Theme Frequency of themes by participants Weighted
Theme 131: 3 of 4 3 of 5 10 of 14
25.8%
State planning (80%) (75%) (60%) (71.4%)
Theme 134:
Needs in evaluating the
effectiveness
4 of 5
(80%)
3 of 4
(75%)
1 of 5
(20%)
8 of 14
(57.1%)
16.3%
Group
1:
Auditors
Group
2:
Parliamentarians
Group
3:
Managers
Total
percentage:
Frequency
of
categories
Theme 132:
Systems’ problems
5 of 5
4 of 4
(100%)
4 of 5
(80%)
13 of 14
(92.8%)
45.5%
14
36
20
11
22
44
4
22
18
15
2
1
Theme 131 - State planning
Theme 132 - Systems'
ploblems
Theme 133 - Risks of new
areas
Theme 134 - Needs in
evaluating the effectiveness
Auditors' perceptions
Parliamentarians' perceptions
Managers' perceptions
Theme 133:
Risks of new areas
5 of 5
(100%)
2 of 4
(50%)
1 of 5
(20%)
8 of 14
(57.1%)
12.4%
Systems’ problems. When asked about the justification of performance audits and
their advantages, 13 participants (92.8%) specifically shared their views about problems
in public administration, such as issues in public financial management, including budget
deficits and ineffective budget spending, regional problems, and sectoral development
lags.
During the interviews, all auditors mentioned that they conducted performance
audits in response to a specific problem. Often, auditors pointed to ineffective budget
spending and poor performance of government entities.
I agree that we have to do it today, have to, because in the case of dealing with
budget deficits, we often ‘close’ the republican budget with a deficit, we must
conduct performance audits to know about how efficiently very limited budget
money is used… (AC01, personal communication, January 8, 2020)
Three auditors shared their opinions about the importance of conduction of
performance audits in quasi-public companies since their activities are not effective, and
they do not achieve the expected results. AL03 shared in detail experience on a selection
of auditees to conduct performance audits of their activities.
To select organizations that should be audited, we assess their risks. I mean, for
example, if we see that departments did not spend, did not use transfers, received
from the republican budget, then, we have to visit them with our performance
audits. These departments as administrators of local budget programs are
ineffective, and they labeled as auditees with high risks. (AL03, personal
communication, January 9, 2020)
Four auditors focused on gaps in regions, including the weak control over the
budget process and lack of qualified specialists, that increase the risks of ineffective
budget spending.
Theme 132 was the most highlighted in parliamentarians’ experiences. All four
parliamentarians talked about systems problems, sharing their views about the importance
of performance audits.
It is… due to fluctuations of demand for oil… tense moments arise on budget
execution, … money invested to development of non-oil sectors do not lead to
dynamic and sufficient increases of budget revenues, that’s why we are very
concerned about how budget funds, that were allocated to government, are being
used. (PL08, personal communication, January 20, 2020)
Well, the main thing is that performance audit … [thinking] … it provides us with
a wide vision of whether our government ready to use budget funds… Often, they
throw money away. In that direction, on that problem performance audits give us
causes for discussions and making decisions within the Parliament. (PC06,
personal communication, January 16, 2020)
Like auditors, parliamentarians shared their views on the importance of
responding to regional issues. PL09 claimed that auditors took into account social
problems such as lack of housing, lack of hospitals and schools, and other problems
caused by the rapid development of the capital.
Unlike auditors and parliamentarians, managers of auditees shared their
experiences associated with problems in public administration more carefully. Most of the
managers confirmed the perceptions of auditors and parliamentarians about the gaps in
regional development.
We could probably request performance audits on some problems because we
cannot manage them ourselves. We are the local executive body, and our activities
are strictly regulated, from one hand, and we face the lack of qualified specialists,
from another hand. We are in such a position that we give explanations and
explanations… (ML13, personal communication, February 15,
2020)
Two managers briefly shared their experiences on being deal with problems of
quasi-public companies.
State planning. The state planning system in Kazakhstan implies the application
of a strategic approach to select long-term development goals connected to funding
budget programs of central ministries, agencies, and local departments that play the role
of administrators of central or local budget programs (ISLARK, 2017a, 2017b). Ten
participants (71.4%) shared their perceptions about problems in the state planning, such
as gaps in policy papers, ineffective implementation of state programs, programs on
development of regions, development plans of national companies, and budget programs.
According to auditors’ perceptions, the gaps in budget programs, ineffective
implementation of state programs, and programs on the development of regions are
significant reasons for performance auditing. All five auditors, when answering on my
question about justification of performance audits, shared examples of poor budget
planning. For instance, AC01 shared two examples of ministries’ approaches to planning
the budget programs that did not lead to the expected results and caused forgone benefits.
If, for example, we see that they planned budgets incorrectly, I mean, without proper
justification, then, we study the related budget programs. It might be the budget programs
aimed at implementing the state programs or using the transfers from central ministries.
Then, we conduct related performance audits because we saw problems in budget
planning. (AL03, personal communication, January 9,
2020)
One more example of a performance audit in healthcare… We focused on
programs on healthy lifestyle programs… Annually, we spend a significant
amount of budget funds to implement the state program, there is a specific budget
program on promoting a healthy lifestyle, but, unfortunately, the morbidity
indicators do not tend to be improved. (AC/L05, personal communication,
January 15, 2020)
Three parliamentarians emphasized problems in the state planning that request
performance audits.
More audits should be conducted in terms of planning. Let’s take the Strategic
plan on the development of our republic till 2025. According to that strategic
document, 5.4% of economic growth is planned per year, but when our
government drafts the republican budget, they plan the economic growth at 3.1%
per year. I don’t see any sense in it, except setting goals that are easy to achieve.
(PC07, personal communication, January 17, 2020)
Two parliamentarians reiterated their opinions about the ineffective
implementation of state programs. In particular, they criticized the quality of construction
of clinics and schools within the implementation of state programs on healthcare and
education.
If we build a hospital with departments divided by street or if we build a new
school next to the functioning schools, instead of building that school on the other
side of the river where there are no schools, … [thinking]… it means that we have
pressing problems with our state planning. (PL08, personal communication,
January 20, 2020)
Two parliamentarians shared their views about problems with the approval of the
development plans of national companies that lead to the inefficient implementation of
their investment programs.
Managers were less critical about problems in state planning. Two of the five
managers were challenged with problems in developing their budget programs; one
manager was dealing with problems in preparing budget reports; and three managers
faced problems in implementing the state programs, programs of development of regions,
and budget programs. All these problem areas were shared by managers of auditees as
examples of performance auditors’ findings. “We had problems with performance
indicators established by the state program and the program of development of our city.
Auditors showed us these problems, and, of course, we tried to improve our program”
(ML13, personal communication, February 15, 2020).
Needs in evaluating the effectiveness. In describing the experience of planning
performance audits, AL04 noted, “… mostly, we are transferring to the conduction of
more performance audits to evaluate the effectiveness of our auditees’ performance and
provide them with recommendations on how to increase their effectiveness” (AL04,
personal communication, January 10, 2020). Seven participants (50%) expressed similar
perceptions. According to their experiences, it is existed needs in evaluating the
effectiveness of government entities, quasi-public companies, and policies in some
specific areas of public administration.
Four of the five auditors specifically noted that decisions on conducting of
performance audit were made due to contradictory information on the performance of
government entities or government, in general.
Or, reports of government. Usually, reports consist of data on positive things, their
great achievements… however, if we listen to the users of public services, they
criticize the government and they have reasons for criticism… No access to water,
no access to centralized gas supply… So, while we have such conflicting data, we
need to make our evaluations. (AC/L05, personal communication,
January 15, 2020)
In discussing the importance of performance auditors’ work, three of the four
parliamentarians shared their concerns related to the lack of objective evaluation of the
effectiveness. “Everyone wants to know how efficiently budget funds are spent” (PL08,
personal communication, January 20, 2020). PC07 more specifically admitted the needs
to evaluate the effectiveness of all state programs, government’s measures on support for
entrepreneurship, investment projects of free economic zones, and budget planning.
We should be provided with a reliable, objective, and competent evaluation of the
Government’s funding needs… What about the effect of state programs? We spent
more than 6 trillion tenges to implement these programs… What about desired
investments that should be attracted by our free zones? (PC07, personal
communication, January 17, 2020)
“Well, why are we dealing with ineffective budget spending? Did we make a
wrong decision on the allocation of budget funds? Maybe, other ministers would do
better? And, here, performance auditors have to say their words” (PC06, personal
communication, January 16, 2020).
Risks of new areas. Like the previous theme, Theme 133 was less highlighted in
the participants’ experience compared to the other two themes on Pattern 3. According to
eight participants (57.1%), the demand for performance audit is explained by the risks of
new areas, including new projects and unaudited areas of public administration. The
following new areas were identified as the scope of performance auditors’ work by
participants: (a) financial funds and institutions, (b) subsoil use, (c) environmental issues,
(d) public debt, (e) social support for vulnerable groups, (f) public-private partnership
projects, and (g) large national companies’ activities.
AC01 shared experiences on the conduction of performance audits of the National
Fund of the Republic of Kazakhstan and the Unified Accumulative Pension Fund. We
conducted performance audits of these two funds for the first time, that is to
say, it was unaudited areas... I have to say that we started other performance
audits in areas or in organizations that were unaudited before as well. (AC01,
personal communication, January 8, 2020)
According to AC/L05, new areas in public financial administrations should be
audited because annual performance audits on tax revenues already contributed to solving
old problems, and auditors have to find new gaps.
Relating to revenues of the republican budget… It is better here and now to
evaluate the effectiveness of tax administration… So, the idea came up to assess
the tax gap to evaluate and understand the situation with a real performance of tax
bodies, and this work was the first of its kind. (AC/L05, personal communication,
January 15, 2020)
Two parliamentarians briefly shared their views on the conduction of performance
audits in new areas, focusing mainly on investment and innovation projects, and national
companies. One manager added the experience of being deal with risks in new areas
explained by a lack of qualified specialists.
Perceived Impact of Performance Audits
Based on the participants’ verbalized experiences, I concluded that performance
audits in Kazakhstan impact on participants of a performance auditing process (Pattern
1), audit organizations and auditees (Pattern 2). Additionally, performance audits have
specific impacts at the macrolevel (Pattern 3).
(100%)
(100%)
(100%)
5
of
5
(100%)
of
5
5
(100%)
13
of
14
(92.3%)
2
5
of
2
5
of
8
14
of
Pattern 1: Impact on participants of performance audits. I generated three
themes on Pattern 1. Figure 19 illustrates how I identified that performance audits impact
on managers of auditees to a greater extent compared with auditors and parliamentarians.
Figure 19. Visualization of themes on Pattern 1 by the frequency of categories
As demonstrated in Table 16, three themes on Pattern 1 emerged with different
frequencies in three groups of participants.
Table 16
Pattern 1: Matrix of Themes and Frequency of Associated Categories
Theme Frequency of themes by participants Weighted Group
Group Group Total percentage:
1: 2: 3: Frequency
Auditors Parliamentarians Managers of
categories
Theme 212: Impact on 5 of 5 4 of 4 5 of 5 14 of 14
43.4% managers of auditees (100%)
Theme 211: 3 of 4
41.2%
Impact on auditors (75%)
Theme 213: Impact on 4 of 4
11.3%
parliamentarians (40%) (100%) (40%) (57.1%)
89
51
6
4
28
44
56
78
6
Theme:211 - Impact on auditors
Theme:212 - Impact on managers of
auditees
Theme:213 - Impact on
parliamentarians
Auditors' perceptions
Parliamentarians' perceptions
Managers' perceptions
Impact on managers of auditees. According to auditors’ opinions, performance
audits have both negative and positive impact on managers of auditees. I grouped the
associated categories in the following way:
1) perceived positive impact of performance audits on managers of auditees
associated with better understanding the performance audits objectives,
receiving new and useful information, learning, and positive reaction on
auditors’ work;
2) perceived negative impact of performance audits on managers of auditees
associated with misunderstanding the performance audits objectives, a
negative reaction on performance audits, and objections to performance audit
results.
I identified that the perceived impact of performance audits on managers was
negative in 28 cases, i.e., the frequency of associated categories, and positive in 23 cases.
Figure 20 illustrates the distribution of the perceived impacts that performance audits
have on managers of auditees by related categories.
According to four of the five auditors, managers in most cases do not understand
the objectives of performance audits and, as a result, they negatively respond to audits
and resist performance audits’ results.
I think that they do not see differences, they don’t care that we conduct
performance or compliance audits. I mean, they perceive us as before, no changes.
Now, an audit has come, that is, the inspection. End of the story! Now, they will
find violations. (AC02, personal communication, January 8, 2020)
Figure 20. Auditors’ perception on impact of performance audits on managers
“They give requested documents with a delay. Honestly, they delayed deliberately.
But the active resistance begins at the stage of discussion of audit reports. In
10 out of 10 cases, we receive their objections to our reports” (AC/L05, personal
communication, January 15, 2020). At the same time, all five auditors admitted that
performance audits have a positive impact on managers of auditees, allowing them to
learn and receive new information. For instance, in describing the experience of
participating in meetings of audit organizations on considering performance audits’
results, AC01 argued that managers became aware of shortcomings in their organizations
because of auditors’ work.
In principle, the vice-minister positively responded to our audit’s results because
we allowed them to hear about shortcomings in their local organizations. There
was even such a phrase, “New. I learned a lot of new things at today’s meeting”.
It was the vice-minister’s words. (AC01, personal communication, January 8,
2020)
Four auditors noted that when they interacted with managers of organizations that
were audited before, they observed desired changes in managers’ reactions.
As in the case with auditors, I grouped the perceived impact of performance audits
on managers (according to their views) by the following associated categories:
1) perceived positive impact of performance audits on managers of auditees
associated with better understanding the performance audits objectives,
receiving new and useful information, learning, and promotion;
2) perceived negative impact of performance audits on managers of auditees
associated with complicating their work, stress, increased distrust of audits,
the uselessness of audits results (caused by the absence of new information in
performance audit reports).
Unlike auditors, managers of auditee shared positive perceptions in 18 cases and
negative perceptions in 60 cases. Figure 21 illustrates the distribution of these perceptions
by related categories.
Figure 21. Managers’ perception on impact of performance audits on them
One of the five managers noted that the previous experience of being audited
contributed to a better understanding of objectives of audits. Four managers discussed the
positive impact of performance audits, such as learning and receiving the new
information used in their practice. As ML13 stated, “We receive some comments from
auditors with pleasure, because in 2016, due to the change of the head of our department,
the continuity in transferring of functions was disappeared. Auditors indicated that fact
very clearly” (ML13, personal communication, February 15, 2020). One manager
expressed the view about promotion perspectives. “In case of the absence of violations or
serious deficiencies in the list of auditors’ findings, you might be promoted. Sure. It
demonstrates that you are good at your job” (ML/C14, personal communication, February
28, 2020). In all cases of sharing the positive perceptions, managers repeated that it was
rare cases.
Three managers noted that, in most cases, performance auditors do not provide
them with new or useful information. In describing the experience of interactions with
auditors, four of the five managers stated that, because of audits, they are dealing with
work complications, distractions from their main duties to prepare documents for
auditors. Most of the managers expressed distrust in performance audits and auditors’
contributions.
In addition to violations, now, auditors identify, as they call it, systematic issues.
So, now, we are dealing with more problems with the discussion of auditors’
opinions. Fewer discussions are needed to object auditors’ opinions related to
violations, because we follow specific laws, rules, or sectorial regulations. But in
case of… [laying stress] … systematic issues, it is impossible to make competent
judgments about our effectiveness without special knowledge and a deeper
understanding of our business processes. (MC10, personal communication,
February 11, 2020)
I can give you a real example. They said that our public sector is too big, there are
unprofitable organizations in the quasi-public sector, and they concluded that we
have to liquidate some quasi-public companies. They are wrong. There are other
benchmarks for state-owned companies. (MC11, personal communication,
February 14, 2020)
Two managers noted that they stressed because of audits, and all five managers
noted that their subordinates stressed as well.
I honestly tell you, when auditors come, the first association is about the stress,
stress for me, stress for all members of our department. Stress… As in Gogol’s
play ‘The Inspector-General’, we are scared when we know that auditors will visit
us… [laughing]. I don’t know, maybe, it will change someday, but now
everywhere and everyone has the same reaction. (ML13, personal
communication, February 15, 2020)
In 14 cases parliamentarians shared perceptions on positive impact of performance
audits on managers, and in 14 cases their perceptions were negative. Three
parliamentarians, detailing their experiences on interactions with managers of auditees
within discussions of annual reports of audit organizations, noted that, as a rule, managers
do not understand the objectives of performance auditors’ work; two parliamentarians
linked that facts with managers’ incompetence.
That’s why from the side of both government entities and quasi-public companies,
often, we face giving the runaround (unsubscribe) or resentment of auditors’
work. They have such rhetoric: The auditors came to us and wrote in their report
about things, that is not true, the auditors are wrong. (PL08, personal
communication, January 20, 2020)
Two parliamentarians shared their observations associated with changes in
managers’ reaction. “I see that ministers react actively to auditors’ critiques if the
critiques are based on conceptual and important things” (PC06, personal communication,
January 16, 2020).
Impact on auditors. According to auditors, performance audits have both negative
and positive impact on them. I grouped the associated categories in the following way:
1) perceived positive impact of performance audits on auditors associated with
mastering new skills, professional development, promotion, and being deal
with more interesting works;
2) perceived negative impact of performance audits on auditors associated with
the increased number of requirements, workload, demotivation, and staff
turnover.
I identified that the perceived impact of performance audits on auditors was
positive in 40 cases and negative in 49 cases. Figure 22 illustrates the distribution of the
perceived impacts of performance audits by related family categories.
Figure 22. Auditors’ perception on impact of performance audits on them
The positive perceptions of auditors are associated with mastering new skills (five
of the five auditors) and the fact that their work becomes more interesting (four of the
five auditors). Three auditors, explaining the advantages of performance audits, admitted
that auditors have opportunities for professional development since they conduct
performance audits in different areas. “You better understand how everything works in
healthcare, education, construction…” (AL03, personal communication, January 9,
2020). Two auditors shared their experiences on being promoted due to the participation
in performance audits. Promotion in their cases was associated with a demonstration of
the abilities of performance auditors to cope with complicated tasks.
We can develop our analytical skills. We look deeply and identify the causes of
violations or shortcomings. Next, we consider and evaluate achieving the desired
results, and how these results impact the auditee’s performance, specific sector,
and our economy as a whole. (AC02, personal communication, January 8, 2020)
What I like now is that our work becomes more interesting. The results of our old
audits… [smiling]… were of the same type. They violated here; they didn’t fulfill
their plans here; they didn’t pay here; and so forth. And repeat in a circle. Now,
our audit reports, not all, some audit reports are very interesting to read. (AC/L05,
personal communication, January 15, 2020)
In describing the changes that occurred in audit organizations, all five auditors
noted that the workload and number of requirements to auditors’ work increased
significantly. The workload increase was explained as the consequence of lacking
qualified auditors and the scale of performance audits to be conducted.
Lack of auditors still exists, but today we feel it stronger. The front of our work is
constantly increasing. Performance audits require more resources. There are some
mechanisms for attracting experts, but we cannot use them often, because of a
lack of resources. (AC/L05, personal communication, January 15, 2020)
Three of the five auditors more specifically shared their views on increasing the
number of non-audit functions and tasks and, so, they linked the workload with these
changes as well.
For four of the five auditors, the increased number of requirements to auditors
included keeping financial control mechanisms in place, or more precisely, focusing on
finding the violations and sanctioning. According to AC02 and AC/L05, requirements on
using financial controls coupled with the unfair evaluation of performance auditors’ work
leads to demotivation.
Today, to be honest, we are not removed from the requirement of identifying
violations…, and you spent your time on performance auditing, but you didn’t
find violations, violations expressed in monetary terms, then you will get a low
rating. And, vice versa, if others spent their time on finding the violations, then
they will get a high rating. (AC02, personal communication, January 8, 2020)
Two auditors shared examples of staff turnover in their organizations. In both
cases, they linked staff turnover with increased workloads and, to a greater extent, with
the unfair evaluation of performance auditors’ work. “Sometimes we did too many things
to come to the endpoint, while we can drive faster instead of walking through a
windbreak. Walking through a windbreak, we lose qualified auditors” (AC01, personal
communication, January 8, 2020).
I revealed that the perceived impact of performance audits on auditors was
positive in 12 cases and negative in 44 cases; Figure 23 illustrates categories on the
related perceptions of managers of auditees.
One of the five managers stated that auditors changed their methods of auditing
and became more objective in their judgments.
Figure 23. Managers’ perception on impact of performance audits on auditors
Auditors changed. I think that some auditors do their work without punitive
purposes, they want to help… My first reaction was scary. I was afraid that they
find violations. But, day after day, I observed..., and they were fair. They
identified some deficiencies in our work, made recommendations, discussed all
things, and they were very open. (ML13, personal communication, February 15,
2020)
Two managers shared similar experiences; however, they noted multiple times that
these changes in auditing approaches were rare cases. “I was surprised during the last
audit because auditors ask such questions as: Can you fulfill our recommendations? How
long will it take? The pleasant novelty in the work of auditors…” (ML12, personal
communication, February 15, 2020).
Four managers (80%) stated that, in most cases, there were no positive changes in
auditors’ work, and, as before the introduction of performance auditing, they are dealing
with incompetent auditors. According to managers’ experience, auditors do not listen to
auditees and make subjective conclusions about auditees’ performance.
There are so many specific things in our work, and I know that an auditor cannot
understand these things without professional training, or if before he worked only
with accounting papers. So, …. we face auditors, who made incompetent
conclusions; they do not take into account or don’t understand the numerous
explanations on our part. (MC 10, personal communication, February 11, 2020)
“Auditors vigorously criticized us for the state program; it was the State program
of development of healthcare ‘Healthy Kazakhstan’. But for that program, we were
highly praised by World Bank experts, experts in the field of health” (MC 11, personal
communication, February 14, 2020). ML12 argued that there are significant differences
between declared objectives of performance audits and real work of auditors.
Even auditors tell you, that they want to find deficiencies or problems, we know
that, when they find violations or facts for transferring to law enforcement bodies,
it makes them happy. They believe that they did a good job. (ML12, personal
communication, February 15, 2020)
Three of the four parliamentarians shared their perceptions of the positive impact
of performance audits on auditors. The positive impact included mastering new
(analytical) skills by auditors and conduction of interesting performance audits.
Generally, Theme 211 was less highlighted in the group of parliamentarians; the
frequency of associated categories is 7 (2.6%).
Impact on parliamentarians. Unlike the previous two themes, Theme 213 is less
highlighted; the frequency of associated categories is equal to 15.6% of the frequency of
categories within all three themes on Pattern 1. This theme emerged in the lived
experiences of eight participants (57.1%). Two auditors and two managers of auditees
shared experiences when performance audits allowed the parliamentarian to implement
their functions, or more precisely, facilitate their task on parliamentary oversight. The
other two auditors and managers noted that parliamentarians use performance audits in
their political interests. “I think that deputies use our reports to scold government. No, not
even government as a whole, but some ministers. Lets’ say, sometimes deputies worked
on cameras. Probably, they also have PR goals” (AC02, personal communication, January
8, 2020).
Four of the four parliamentarians shared perceptions about the impact of
performance audits on their perspectives; the frequency of associated categories is equal
to 78.6% out of the frequency of all categories on Theme 213. Three parliamentarians
positively perceived the impact of performance audits on their perspectives, while one
parliamentarian noted that the impact of performance audits is limited. According to the
lived experience of parliamentarians, performance audits facilitate implementing their
duties on parliamentary oversight (four of the four interviewees), on ensuring the
accountability of the government, central and local ministries (four of the four
interviewees), approval of the budget (three of the four interviewees), and making
legislative proposal (three of the four interviewees).
We, using our deputy powers and based on audit reports, ask our questions to the
heads of local departments. Why did they bring the situation to such a state? Why
was the inefficient use of budget funds allowed? Sure, it helps us. (PC09, personal
communication, January 23, 2020)
“We, of course, use the results of performance audits. We need performance
audits. It helps us to consider the draft budget, as I said. It helps us to make decisions, to
accomplish our parliamentary functions” (PC06, personal communication, January 16,
2020).
I need in more performance audits since, by using the results of these audits, I will
be aware of macroeconomic issues; I will study regional issues and relations
between central and local governments. All these conceptual things are about our
duties and responsibilities. (PL08, personal communication, January 20, 2020)
Pattern 2: Impact on audit organizations and auditees. According to the lived
experience of most of the participants, performance audits impact on audit organizations
and auditees, as demonstrated in Figure 24 and Table 17; two themes on Pattern 2
emerged with different frequencies in all groups of participants.
(100%)
(100%)
(100%)
of
5
5
of
5
5
12
14
of
Figure 24. Visualization of themes on Pattern 2 by the frequency of categories
Table 17
Pattern 2: Matrix of Themes and Frequency of Associated Categories
Theme
Frequency of themes by participants
Group Group Group Total
1: 2: 3:
Auditors Parliamentarians Managers
Weighted
percentage:
Frequency
of
categories
Theme 221: Changes in 5 of 5 4 of 4 5 of 5 14 of 14
58.7% audit organizations (100%)
Theme 222: 2 of 4
41.3%
Changes in auditees (100%) (50%) (100%) (85.7%)
Changes in audit organizations. Unlike personal impacts (Themes 211, 212, and
213), it is impossible to clearly distinguish positive and negative perceptions on
organizational impacts, since participants shared mixed perceptions related to a specific
change (e.g., changing the methodology), or participants shared their experiences without
43
19
25
3
40
54
Theme 221 - Changes in audit organizations
Theme 222 - Changes in auditees
Auditors' perceptions
Parliamentarians' perceptions
Managers' perceptions
expression of their attitude to a specific change (e.g., changes in organizational
structures).
Thus, as illustrated in Figure 25, I divided the participants’ perceptions in the
following way: (a) organizational changes include changes in organizational structure and
methodology; (b) perceived negative impact includes keeping financial control and
deterioration of the reputation of audit organizations; (c) perceived positive impact
includes excepting financial control and improved reputation of audit organizations.
Figure 25. Auditors’ perception on impacting audit organizations
Five auditors (100%) shared their experiences on how introducing the
performance audits impacted their organizations. According to their perceptions,
changing the methodology is the most influential impact of performance audits on audit
organizations.
In our organization… [thinking]… I would not say that I observed obvious
progress. The number of methodological recommendations, internal rules, and
regulations increased. Often, the internal rules are changed. Of course, the
requirements on the application of methodological papers are changed as well.
(AC02, personal communication, January 8, 2020)
The other four auditors shared the same view; they more specifically noted that
the number of methodological papers increased due to performance audits. For instance,
AL04 argued that many different methodological papers were developed because of
performance audit introductions and “it is the first time when auditors have
methodological papers for all occasions” (AL04, personal communication, January 10,
2020). According to AC/L05, despite the development and approval of methodological
guidelines on performance auditing, not all auditors in both central and local audit
organizations apply these guidelines. All five auditors shared their views on
complications the methods of their organizations on the conduction of performance
audits.
In most cases, auditors referred to an increased number of job functions including
non-audit functions, such as preparing the information for external users, quality control,
and legal expertise.
Unfortunately, our possibilities on changing our organizational structure are
extremely limited, our structure has not been changed from 2011, while we are
transferring to performance audits. We were required to introduce many different
functions, we have quality control functions, we have functions on legal expertise,
and so on. (AL04, personal communication, January 10, 2020)
Two auditors directly linked these changes with the increased number of
performance audits, while others stated that there are also other reasons for these changes.
In describing the changes that occurred in their organizations’ structure, four of the five
auditors admitted that there are no desired changes associated with performance audits
since their organizations did not create separate performance audit departments.
Four auditors (80%) negatively perceived keeping financial control functions with
its strengthened requirements for finding financial violations during performance audits.
Three auditors more specifically noted that tasks on finding violations lead to
complications of interactions with auditees and, therefore, audit organizations are
perceived as “controllers from the Soviet Union who are pursuing the goal of punishing”
(AC02, personal communication, January 8, 2020). Two of the five auditors shared their
views on the negative impact of performance audits on their organizations’ reputations. In
both cases, auditors noted that negative reactions on their performance audits took place
when they received substantiated objections to audit result findings.
According to all five auditors, there are also some positive changes in audit
organizations associated with performance audits. In all cases, auditors emphasized that
these changes are changes from recent days. AC01 noted that both the central and local
audit organizations’ approach to performance auditing is changed “little by little… and
people now understand what does performance auditing mean” (AC01, personal
communication, January 8, 2020). Two of the five auditors noted that performance audits
lead to the improved reputation of their organizations. “I think… our commission became
recognizable, and it is thanks to our performance audits” (AL03, personal
communication, January 9, 2020).
In describing the experience of interaction with auditors, five managers of
auditees listed changes in auditors’ methods.
There are some other changes. Before auditors interact with our accountants and
financiers, they didn’t go to our department or other departments. Now they
interact with all departments. Like our accountants and financiers, we closely
interact with auditors. I see that their tasks are changed and increased. (ML/C14,
personal communication, February 28, 2020)
“Now, we may pay tribute to the auditors, they started to write about our good
practices in their audit reports. I support it… [smiling]” (ML12, personal communication,
February 15, 2020).
Four of the five managers noted that performance audits positively impact on
audit organizations’ reputation. In three cases, managers commented upon the positive
impacts with audit organizations’ practice on attracting qualified experts. But there
were a couple of good points. I do believe that they do a good job because it was a
strong team of auditors with external experts. Then, and this, perhaps, was a one-time
case, the audit report was different, it was the better
report compared with all previous reports. (MC10, personal communication,
February 11, 2020)
Like auditors, managers of auditees shared negative perceptions associated with
keeping financial control mechanisms. “For us, there are no significant differences
between old and performance audits. Why? Because the results of all audits are the same.
All is about violations, and, anyway, violations are important performance indicators for
audit organizations” (ML/C14, personal communications, February 28, 2020). Three
managers more specifically noted that, like compliance audit reports, performance audit
reports consist of facts of financial violations, transferring auditors’ findings to law
enforcement bodies, and administrative violation protocols.
In the group of parliamentarians, changes in audit organizations are associated
with using new methods of audits (two of the four parliamentarians) and keeping
financial controls (three of the four parliamentarians). As PL09 stated, “I don’t perceive
our local audit commission as a body of financial control because their reports focused on
recommendations. Yes, of course, they include identified violations in their reports, but
they put more effort into recommendations” (PL09, personal communication, January 23,
2020). Two parliamentarians shared their positive perceptions about audit organizations
achievements. “I know that today local revision commissions actively conduct
performance audits. They have good results” (PC06, personal communication, January
16, 2020).
Two parliamentarians noted that there are no significant changes in audit
organizations since, as before, they primary focus is on violations.
Each time they report on violations, report on billions of tenges of violations. But
if you look carefully, you see that these billions of tenges are accounting errors.
Then, they start their explanations that they classified violations by financial
violations and procedural violations… Anyway, these kinds of results are results
of financial or compliance audits. (PC07, personal communication, January 17,
2020)
Changes in auditees. The changes in auditees effected by introduction of
performance audits are more highlighted in the experience of managers and auditors, and
less highlighted in the experience of parliamentarians. Five auditors (100%) positively
perceived changes in auditees, whereas managers of auditees shared both positive and
negative perceptions. In describing their experience in performance audits conducted in
2016–2019, auditors noted that implementation of their recommendations by auditees led
to improved performance, better discipline, enhancing the control over the execution of
budget programs, and changes in organizational structures. In two cases, auditors stated
that their work contributed to the improved reputation of auditees as well. Figure 26
illustrates the distribution of the perceived impacts of performance audits by related
family categories.
As AC03 shared, performance indicators of local departments that are responsible
for managing the landscaping projects improved as a result of the implementation of
auditors’ recommendations.
Figure 26. Auditors’ perception on impacting auditees
The chief accountant and public service inspector reported on streamlining
internal business processes in their organization after our audit. The chief
accountant said, “Now, we can establish more real deadlines and justified
requirements for our contractors, and they don’t stamp their feet anymore.” It was
a good feedback. (AL03, personal communication, January 9, 2020)
AL04, more specifically, shared the experience of changing the organizational
structures of auditees. All five auditors stated that auditees’ discipline including financial
discipline, i.e., compliance with the regulations on the execution of budget programs, is
significantly improved. Four auditors shared their experience of enhancing controls over
budget program execution by audited departments. “Auditees develop. I see that audit by
audit, they become better. Today, I have to say… violate to reap the fruits of their
mediocrity or illiteracy… [thinking]… there is no such thing today” (AC01, personal
communication, January 8, 2020).
Managers shared their positive perceptions associated with the impact of
performance audits on their organizations in 33 cases and negative perceptions in 21
cases. Figure 27 illustrates the distribution of the perceived impacts of performance audits
by related categories.
Figure 27. Managers’ perception on impacting auditees
As demonstrated in Figure 27, for most managers, changes in auditees included
improved performance, better discipline, and enhanced control. Additionally, five
managers of auditees (100%) noted that performance audits contributed to the readiness of
their organizations to new audits.
I believe that year to year we complicate the task of auditors because we are
improving… If we will see violations, they will be eliminated. It will be
monitored. And if somebody commits the same violation repeatedly, then more
severe measures will be applied to him. (ML12, personal communication,
February 15, 2020)
“If auditors’ comments were addressed to other administrators of budget
programs, even to other administrators, we ‘have our finger on the pulse’ to receive less
or no comments from auditors in the future” (MC10, personal communication, February
11, 2020). “I’m sure that we will have audits again and again… [smiling]… for example,
they may decide to audit our obligatory health insurance system. Well, we will be ready”
(MC11, personal communication, February 14, 2020).
Answering my question about changes that occurred in auditees as a result of
performance audits, managers shared negative perceptions associated with workloads,
demotivation, staff turnover, and financial consequences, such as budget cuts and
penalties. Managers of auditees admitted that workloads are a direct result of performance
audits since, unlike compliance auditors, performance auditors request more information
and have more questions. For instance, MC10 stated that during the performance audits
“all our organization work for auditors preparing the requested documents, a huge amount
of information, a huge amount of analytical data, and we provide them with multiple
explanations” (MC10, personal communication, February 11,
2020). Similar views were shared by other managers as well.
In turn, managers connected staff demotivation and turnover, and financial
consequences with financial control measures undertaken within performance audits.
ML12 shared in detail experiences of auditors applying financial control mechanisms. “It
is critical and sensitive things for us, for any others who under audits because if our staff
received the administrative violation protocol, auditors’ decisions on penalties, it leads to
negative consequences. It demotivates our staff” (ML12, personal communication,
February 15, 2020). Two managers shared stories about the dismissals of qualified
specialists following the results of audits. “It was a case when that qualified specialists
did not agree with the measures taken to them, and there are other organizations in
regions which are in similar ‘deplorable’ situations” (ML/C14, personal communication,
January 28, 2020).
In sharing the views related to changes in auditees, parliamentarians referred to
improved financial discipline. However, three of the four parliamentarians mentioned that
there are no significant changes in audit organizations that could be defined as “a net
result of audit organizations” (PL08, personal communication, January 20, 2020).
Pattern 3: Macrolevel impact. For most of the participants, the macrolevel
impact of performance audits included improved budget process (Theme 231) and
changes in laws and regulations (Theme 232). After completion of data analyses, I also
identified that improved budget processes and changes in laws and regulations, viewed by
the majority of participants, are not attributed solely to the impact of performance audits
(Sub-theme 233). Fourteen participants (100%) shared their views relating to changes in
auditors’ work needing to facilitate a more positive impact of performance audits on
public administration in Kazakhstan (Sub-theme 234), as it is demonstrated in Figure 28.
Figure 28. Visualization of themes on Pattern 3 by the frequency of categories
20
14
6
24
20
9
22
17
12
8
34
7
Theme 231 - Improved
budget process
Theme 232 - Changes in
laws and regulations
Sub-Theme 233 - Self-
contribution to 3E
Sub-Theme 234 - Requested
changes in auditors' work
Managers' perceptions
Parliamentarians' perceptions
Auditors' perceptions
5
5
of
(100%)
4
5
of
Table 18 illustrates the themes and sub-themes on Pattern 3 emerging with
different levels of frequency in three groups of participants.
Table 18
Pattern 3: Matrix of Themes and Frequency of Associated Categories
Theme/ Sub-theme
Frequency of themes by participants
Group Group Group Total
1: 2: 3:
Auditors Parliamentarians Managers
Weighted
percentage:
Frequency
of
categories
Theme 231: 5 of 5 4 of 4 4 of 5 13 of 14
62.7%
Improved budget process (100%) (100%) (80%) (92.9%)
Theme 232: Changes in 4 of 4 4 of 5 13 of 14
37.3%
laws and regulations (100%) (80%) (92.9%)
Sub-theme 233: 4 of 4 5 of 5 13 of 14
56.4%
Self-contribution to 3E (80%) (100%) (100%) (92.9%)
Sub-theme 234: Requested
changes in performance
auditors’ work
5 of 5
(100%)
3 of 4
(75%)
3 of 5
(60%)
11 of 14
(78.3%)
43.6%
Improved budget process. Thirteen participants (92.9%) shared their experiences
of improved budget processes as a result of performance auditing. That perceived effect
included changes for better planning and execution of budgets, preparing reports by
administrators of budget programs, and enhancing controls over execution of the central
and local governments’ budgets.
Four parliamentarians (100%) emphasized that using the performance audits by
legislative bodies significantly contributed to enhancing controls over government
entities, more precisely over the spending of budget funds by the central ministries and
local executive departments. Three parliamentarians argued that auditors’ work facilitated
the Parliament and Maslikhats in ensuring accountability within the public and
quasipublic sectors.
All deputies, all ministers and other top-managers from central ministries, heads
of akimats, and even chairs of big national companies, all take part these
meetings, all become more aware of the performance audits results, on the one
hand, and about the effectiveness in using the budget funds, on the other hand.
(PC06, personal communication, January 16, 2020)
“It contributes to discipline not only in audited ministries or agencies, in all other
organizations as well” (PL09, personal communication, January 23, 2020).
Three of the four parliamentarians noted that performance audits contributed to
changes in budget planning by facilitating correct redistribution of funds between
administrators of budget programs. PC06 and PL09 shared their experiences on making
decisions on reallocations of budget funds between ministries and departments that used
budget funds ineffectively or were unprepared for implementation of investment projects.
There were such situations when we used the facts of violations and ineffective budget
spending as a rationale to make decisions on cutting the allocated budget funds of related
administrators. We redistribute these funds between other ministries, we allocate them to
other priority sectors of the economy, to the social needs of the state. (PC06, personal
communication, January 16, 2020)
We use the performance audit as a tool to understand and to find the right ways on
how we can prevent ineffective spending. By using the results of audits, we can
prevent ineffective spending; we can do it when we consider and approve budget
programs, proposals on investment projects, and so on. I think, that there were
many examples when …[thinking]…, we saved budget funds by making our
decisions on budget approval. (PL08, personal communication, January 20, 2020)
Like parliamentarians, auditors positively perceived the impact of their
performance audits on budgeting processes. However, they mainly referred to improved
quality within budget program execution. Five auditors (100%) noted that their efforts
contributed to prevention or minimizing the risks of negative consequences of ineffective
spending.
For example, after our audits the budgets are replenished with unpaid taxes,
service providers carry out restoration work. They did something poorly, we
pointed them out, they fixed it. Of course, let’s say so, we are not the only ones to
do it. Anyway, our audit is very beneficial for our state. (AC02, personal
communication, January 8, 2020)
Four of the five auditors mentioned the improved quality of budget programs as a
result of their performance audits.
There are many, too many, situations when ministries plan to spend huge budget
funds that could lead to ineffective spending or even to negative results. We
indicated this problem, they corrected their documents, and the others learn. Then,
all can prepare good budget programs and budget applications. (AC01, personal
communication, January 8, 2020)
AL03 shared, in detail, examples of saving budget funds as a result of
performance audits and using central budget transfers by local departments. In describing
their experiences on monitoring implementation of auditors’ recommendations, all five
auditors argued that these results contributed to better budget planning when
recommendations were implemented.
Two auditors shared their experiences on contribution to addressing systems’
issues, such as lack of financing or inefficient financial management. In two cases,
auditors emphasized that their work is beneficial as governments and parliaments use
these audit results to promote reforms in areas of public financial administration. “Thanks
to our audit reports, they have new opportunities to address big problems or find new
ways – increase funding, reduce funding, or apply a public-private partnership. I have to
say that they do it” (AL04, personal communication, January 10, 2020).
On my question about the changes that occurred in areas of public administration
supervised by their organizations (i.e., public finances and healthcare) after the
implementation of performance audits, managers of auditees often described these
measures as positively impacting budget reimbursement of unused or misappropriated
funds or elimination of work defects by service providers. Three managers noted, after
auditors’ work, their organizations strengthened control over all administrators of local
budget programs. Two managers shared their experience of changing performance and
budget reports. “We prepare and submit our performance and budget reports to the
Ministry of Healthcare. After audits, we improved our reports” (ML13, personal
communication, February 15, 2020). ML/C14 shared an example of improved
interdepartmental cooperation in using state loan funds at the local level.
Changes in laws and regulations. In relation to perceived changes in laws and
regulations, I interpreted participants’ perceptions on providing auditees and authorized
bodies with recommendations on changing the laws and regulations as (a) agreed by
auditees and authorized bodies or (b) implemented by auditees and authorized bodies, as
well as (c) perceptions of participants on using performance audit results by
parliamentarians or governments in making their legislative proposals.
According to five auditors (100%), performance audits are useful since they
contributed to improved laws and regulations. Three auditors shared their experience of
making recommendations related to changes in the Budget Code, the Tax Code, and the
Environmental Code. “We recommended changes in rules, regulations, laws, and codes…
Our recommendations were related to the budget process, related to public procurement...
Some recommendations were supported and agreed by the Ministry of Justice and the
Prime Minister’s Office” (AC01, personal communication, January 8, 2020). AC/L05
more specifically shared examples of making the amendments to the Budget Code, the
strategic plan of the Ministry of Finance, and regulations of the Ministry of National
Economy.
Our recommendations on using the tax gap as the performance indicator by tax
body and related changes to the Budget code were agreed by the auditee. Of
course, at the first stage of our discussions, they objected, but then they
understand and agreed with us. Moreover, our recommendations were supported
by parliamentarians. (AC/L05, personal communication, January 15, 2020)
Two auditors shared their experiences in making recommendations in the
following areas: (a) state programs on healthcare, (b) programs concerning regional
development, and (c) programs for waste management.
In sharing their experiences in using auditors’ work, four parliamentarians (100%)
referred to changes in laws and regulations. In particular, they listed the Budget Code,
laws on public procurements and republican budgets, and local legislative body’s
decisions on approval of local budgets. PC06 shared experiences of making amendments
to the Law of the Republic of Kazakhstan about the Republican Budget; parliamentarians
in detail shared their views on changes within budget parameters and costs of ministries
and agencies. PL08 referred to similar experience and noted that “performance audit
results should continue to motivate us to change laws or write new laws, we need to do it
to eliminate all risks of violations, deficiencies, and mainly – ineffective spending of
taxpayers’ money” (PL08, personal communication, January 20, 2020).
Three parliamentarians described how they make legislative proposals based on
auditors’ work; two parliamentarians, however, mentioned that it was a rare case. “As part
of the promotion of legislative initiatives, I can also rely on their materials. But to be
honest, I don’t do it often. I work with the government directly. Why? Because we receive
audit materials after the fact” (PC07, personal communication, January 17, 2020).
Four managers (80%) shared their experiences on implementation activities of
auditors’ recommendations. Their perceptions are mainly associated with changing of
rules and regulations. “There were internal local documents of our ministry, which were
aligned and improved… Yes, we work on the bugs” (MC10, personal communication,
February 11, 2020). MC11 mentioned that some regulations, including rules for
purchasing of medical equipment, were changed because of auditors’ recommendations.
Generally, themes associated with changes in laws and regulations were less
highlighted in parliamentarians and managers experiences, which was opposite to
auditors’ experiences. I concluded these differences to be associated with
parliamentarians and managers often referring to their contributions to improvements
whereas auditors were focused on direct observations and compliance experiences.
Therefore, two trends emerged as subthemes on Pattern 3, including sub-theme 233
“Self-contribution to 3E” and sub-theme 234 “Requested changes in performance
auditors’ work”.
Self-contribution to 3E. I entitled sub-theme 233 as self-contribution to 3E since
perceptions of all parliamentarians and managers about their contributions might be
interpreted as contributions to the implementation of principles of 3E (i.e., economy,
effectiveness, and efficiency). Four auditors (80%) referred to contributions of the
parliament and government.
All parliamentarians explicitly described their activities on changing the laws and
regulations without references to auditors’ work. “In some cases, we, the deputy corps,
reveal shortcomings in our legislation... We make recommendations on improving the
legislation” (PC06, personal communication, January 16, 2020). PL09 shared examples
of helping veteran policymakers to put forth legislative proposals. The following
statement encapsulated the parliamentarians’ perceptions: “If the parliamentary institute is
not oriented on real improvements focusing on the electorate, then discussions of the
audit results become a small-scale game. To avoid playing to the gallery, we have not to
rely only on performance auditors” (PL08, personal communication, January 20, 2020).
Like parliamentarians, all managers shared their views on their organization’s
achievements. “We introduced the result-oriented budget, strategic planning, and we
improved the budget reports, the transparency of our performance results is significantly
increased” (ML12, personal communication, February 15, 2020). “We were the first who
introduced many innovations… We open departments for premature babies… We
transferred our new practices to other regions” (ML13, personal communications,
February 15, 2020). Four managers shared examples of changing regulations by their
organizations. Managers of auditees did not connect all these cases with the performance
audits.
Requested changes in performance auditors’ work. Eleven participants (78.6%),
based on their experiences in performance auditing, reasoned that changes are needed in
performance auditors’ work. Five auditors (100%) admitted that changing their approach
to performance auditing will contribute to the usefulness of their audits. Four auditors
emphasized the need for changing audit methodology, making it more flexible. Most of
the auditors suggested changes in using financial control mechanisms since “it is
impossible to undertake an ideal performance audit with keeping the financial control
measures inherited from the Soviet Union” (AC02, personal communication, January 8,
2020). All auditors highlighted that the amount of identified violations should not be used
as auditors’ performance indicators.
Three parliamentarians (75%) also called for changes. Like auditors,
parliamentarians discussed needed changes in financial control mechanisms.
“Performance audit should not be carried out to identify violations of 20 billion tenges…
But it often happens to us” (PL08, personal communication, January 20, 2020).
Parliamentarians suggested changes in submitting the performance audit reports to
legislative bodies.
I would like to bring the process of reviewing audit results by the deputies’ corps
closer to the completion of audits. Today, we need to wait for annual reports, we
consider the results of all audits completed a year or several months ago. We have
to change it. (PC06, personal communication, January 16, 2020)
Three managers of auditees (60%) discussed needed changes in audit methods and
reports, and they called for competent recommendations and objective audit opinions.
“We face auditors who evaluate us solely based on identified violations, they don’t pay
attention to our achievements. Making the audit conclusions based on negative points is
unfair” (MC11, personal communication, February 14, 2020).
Summary
Chapter 4 consists of detailed descriptions of implementing the field stage of my
study, beginning from participant recruiting through synthesizing research findings to
address my posed research problem and research question. I generated 18 themes
associated with my research question through data analyses of 14 phenomenological
interviews and examination of secondary data. As highlighted in the lived experiences of
participants, the demand for performance audits in Kazakhstan is explained through
preferences, interests, and needs of performance audit users, as well as through
legitimization of performance audits and adoption of international audit best practices.
The most influential factors on demanding performance audits were found to be directly
associated with problems in public administration in Kazakhstan.
According to participants’ perceptions, performance audits have personal,
organizational, and macrolevel impacts. In describing changes resulting from
performance audits, participants shared both positive and negative perceptions.
Nonetheless, there were no significant discrepancies found between analyzed data.
Overall, mixed perceptions are explained by features found through transitioning from
old financial control systems to new public auditing systems in Kazakhstan. Generally,
auditors and parliamentarians shared positive perceptions associated with performance
audits’ influence, while managers of auditees often noted negative aspects associated with
performance auditors’ work. More specifically, all parliamentarians and managers shared
their views about their organizations’ contribution to changes without references to
performance auditors’ work (the first sub-theme). The majority of participants were
unanimous in requesting changes on the application of financial control mechanisms by
performance auditors (the second sub-theme). Further interpretation of these research
findings, considerations about positive social change impact, future research, suggested
public policy changes, and other political implications will be described in Chapter 5.
Chapter 5: Discussion, Conclusions, and Recommendations
Introduction
To explore factors that might explain the demand for performance auditing in
Kazakhstan and the perceived effects of performance audits on Kazakhstani public
administration, I studied the lived experiences of 14 participants of a performance
auditing process. Additionally, I reviewed and analyzed more than 200 official documents
related to the studied phenomena, including audit reports issued by the Accounts
Committee and local public audit organizations in 2016–2020, as well as audit reports
issued by SAIs of Australia, Canada, the United States, and United Kingdom within the
same timeframe. Guided by a combination of NPM and PAM theoretical frameworks, I
addressed a gap in understanding the reasons for demanded performance audits and
perceived impacts that these audits have on public administration in Kazakhstan.
I determined problems in public administration, needs and interests of
parliamentarians, public auditors, the President, citizens, and media as the most
influential factors of highly requested performance audits in Kazakhstan. To a lesser
extent, increasing the number of performance audits is explained by interests of managers
of auditees and features of Kazakhstani public auditing system, such as legitimization of
performance auditing practice and adaptation of international experience.
Despite their novelty, performance audits already had instrumental, conceptual,
tactical, and strategic impacts; these impacts manifested at micro-, meso-, and
macrolevels. Conclusively, I found that performance audits led to changes in public
administration, impacting positively and negatively (a) those involved in these audits, (b)
audit organizations and auditees, and (c) contributing to improved budget process and
changes in laws and regulations.
Comparing my research findings with the reviewed literature results, I concluded
that they [findings] are consistent, partially consistent, or inconsistent with previous
studies’ findings and conclusions. There are a few relatively new findings that actualize
further theoretical and practical considerations. Chapter 5 consists of a detailed
interpretation of my research findings, descriptions of limitations of the study,
recommendations, and study implications.
Interpretation of the Findings
Empirical Context and Evidence
Factors of highly demanded performance audits. Like many developed and
developing countries, Kazakhstan is within a megatrend on expanding their performance
audits (see Alwardat & Basheikh, 2017; Bunn, Pilcher, & Gilchrist, 2018; Flesher &
Zarzeski, 2002; Funnell, 2004a; Glynn, 1985; Levy, 1996; Loke et al., 2016; Pollitt,
2018; Reichborn-Kjennerud & Johnsen, 2018; Torres et al., 2019; Yamamoto &
Waranabe, 1989, for more). According to the perceptions of the 14 (100%) participants,
more performance audits are being conducted in various areas of public administration by
both the central and local public sector audit organizations. In the last 18 years, audit
organizations increased their share of performance audits starting from 0 to more than
60% of their audit portfolios, as it is illustrated in see Table 10, and, thus, performance
auditing practice flourished in Kazakhstan as well (see Pollitt, 2018, p. 167, for more).
Answering questions regarding the reasons behind performance audit requests, I
identified that these audits are in high demand in Kazakhstan due to the needs and
interests of parliamentarians, auditors, the President, citizens, and media. Previously,
participants or users of performance audits within public administrations were not defined
as influential factors of demand for performance audits and at such a ranking. However,
these findings are consistent or partially consistent with previous studies’ results.
Parliamentarians need and use performance audits to implement their duties, such
as ensuring the accountability of governments, making budgetary decisions, and
promoting legislative proposals. This finding is consistent with results of studies on using
performance audit results (Arthur, Rydland, & Amundsen, 2012; Funnell, 2004a, 2011,
2015; Funnell & Wade, 2012; Funnell et al., 2016; Guthrie & Parker, 1999; Morin, 2001,
2008, 2016; Pollitt & Summa, 1997; Reichborn-Kjennerud, 2013a, 2013b, 2014a, 2014b;
Reichborn-Kjennerud & Johnsen, 2011; Reichborn-Kjennerud & Vabo, 2017). For
instance, one parliamentarian stated that reports of audit organizations are the only
sources about the quality of implementing the budget process and ministries’
effectiveness that allow a deputy corps of parliamentarians to make informed decisions
about budgets and laws (PL06, personal communication, January 16, 2020). This
perception is in line with considerations of Reichborn-Kjennerud (2014a, 2014b) related
to providing elected officials with information about using public money through auditing
the performance of executives. Arthur et al. (2012) similarly opinioned that performance
audit results are necessary to making parliamentary decisions. Funnell (2015) argued that
parliamentarians use performance audits to legitimize their decisions, and that argument
was confirmed by the experience of study participants, who shared views on pursuing
political interests by members of the Parliament of Kazakhstan via performance audit
results (PL08, personal communication, January 20, 2020; AC02, personal
communication, January 8, 2020; AC/L05, personal communication, January 15,
2020).
In Australia, increasing the number of performance audits was motivated by
representatives of parliamentary committees for public accounts who “complement and
enhance the role of performance auditors primarily by providing the public forum for
further investigations into government practice and performance” (Hoque and Sharma as
cited by Parker, Jacobs, & Schmitz, 2019, p. 284). In Kazakhstan, parliamentarians praise
the work of auditors as well. However, they use different tactics to shape the demand for
performance audits depending on levels of public administration. Members of the central
legislative body request information about performance audits results, while local
parliamentarians, like legislators in western public administrations, directly request
conduction of performance audits and actively participate in meetings considering
performance audits results. Despite these differences, my conclusion on the demand for
performance audits due to parliamentarians’ needs is consistent with definitions of
parliamentarians as motivators of performance audits in Australia (Parker et al., 2019)
and as external initiators of performance audit ideas in New Zealand (Nath, Radiah, &
Laswad, 2019).
Further, both the central and local public audit organizations, in most cases (i.e.,
except requesting audits by parliamentarians and the President), select themes, areas, and
auditees for conducted audits at their sole discretion. Of those interviewed, all auditors
opted for performance audits, shared their experience in proposing performance audits,
and predominantly supported proposals to conduction more performance audits by their
colleagues. These findings were consistent with recent considerations of Nath et al.
(2019), who defined audit organizations and audit teams as the main internal initiators of
performance audit ideas. Then, auditors’ preferences as the influential factor of demand
for performance audits are due to their active positions (see Norton & Smith, 2008, for
more). Performance auditors are ready to initiate important audits and issue audit reports
even when these audits are not requested by governmental or any other organizations
(Wheat as cited by Norton & Smith, 2008).
In turn, an active position of Kazakhstani performance auditors might be defined
as a manifestation of a responsive approach to performance auditing (see Arthur et al.,
2012; Knaap, 2011; Rosa et al., 2014a; Rosa, Morote, & Prowle, 2014, for more).
Applying that approach means that performance audits are initiated and conducted in the
interests of users, e.g., crime victims, employers, and disabled people (Arthur et al.,
2012). In the lived experience of auditors, the needs of citizens were highlighted as
reasons for increasing performance audits, opposite to the lived experience of
parliamentarians or managers of auditees. Particularly, auditors claimed that performance
audits are needed because of citizen complaints and the existence of social problems,
including the unaddressed needs of people who are most in need of governments’ support,
such as pensioners, large families, and disabled people (AC01, personal communication,
January 8, 2020; AL03, personal communication, January 9, 2020;
AL04, personal communication, January 10, 2020; ACL05, personal communication,
January 15, 2020). According to the lived experience of 7 (50%) participants, the media,
to a certain degree, shape the demand for performance audits as well.
The generated theme “Citizens and media shape the demand for performance
audits” is consistent with defining the public (the public complaints) and media as
external initiators or motivators of performance audits (see Nath et al., 2019; Parker et al.,
2019, for more). At the same time, cases from Australia and New Zealand studied by
Parker et al. (2019) and Nath et al. (2019), respectively, differ from the case of
Kazakhstan. For instance, auditors in Kazakhstan consider citizens’ complaints posted in
i-komek, i.e., the information system of local authorities in planning performance audits
(AL03, personal communication, January 9, 2020), while auditors in New Zealand
respond to citizens’ complaints submitted directly to their organizations by conduction of
performance audits (Nath et al., 2019). Regarding the media, auditors in Australia do not
use the media to disseminate performance audit reports, while audit organizations in
Kazakhstan invite the media to share the results of performance audit results (AC02,
personal communication, January 8, 2020). In the end, the media shape the demand for
performance audits in Kazakhstan, like in other countries (see Justesen & Skaerbaek,
2010; Morin, 2008; Nath et al., 2019; Parker et al., 2019; Tillema & Bogt, 2010, for
more).
Nath et al. (2019) analyzed situations when themes of performance audits in
healthcare were defined by SAI of New Zealand, taking into account views of
representatives of the Ministry of Health and the District Health Boards. The authors
concluded that it was a useful experience. My study participants, i.e., managers of
auditees and auditors, shared opposite positions regarding the agreements or joint
initiations of performance audits in Kazakhstan. All managers and all auditors stated that
they did not experience any performance audit initiations or took part in planning
performance audits by auditees’ managers. It was explained by (a) frequent visits of
auditors caused by increasing number of audits or annually repeated performance audits
(MC10, personal communication, February 11, 2020; MC11, personal communication,
February 14, 2020; ML12, personal communication, February 15, 2020; ML13, personal
communication, February 15, 2020; ML/C14, personal communication, February 28,
2020); (b) requesting audits by newly appointed ministries to check financial violations,
in rare cases (AC02, personal communication, January 8, 2020; ACL05, personal
communication, January 15, 2020). Therefore, Kazakhstani practice on performance
auditing differs and are a far contrast from the experiences of western countries where
performance audits are requested not only by parliamentarians, but also by auditees or
governments (see Funnell 2015; Knaap, 2011; Nath et al., 2019; Parker et al., 2019, for
more). A few managers shared their experience on interactions with external public
auditors when they execute the President Administration’s order (MC10, personal
communication, February 11, 2020; ML12, personal communication, February 15, 2020).
Participants did not perceive that experience as a voluntary action, but, in their opinion, it
was a positive experience. Thus, the statements about conduction of useful performance
audits through agreed actions between governments, auditees, and auditors made by Nath
et al. (2019) and almost 20 years ago by Morin (2001) are to some extent, relevant to the
Kazakhstani case.
Finally, I defined the President’s policy on prioritizing performance audits as one
of the more influential demand factors for these audits in Kazakhstan. This finding is
relatively new since as my literature review offered no studies that explored the influence
of presidents (head of the states) on performance auditing, and presidents were not
defined as initiators or motivators of performance audits before. This strong presidential
power, which implies using mandates of both legislative and executive bodies
(Nazarbayev, 2017; OECD, 2017a), explains why, in the lived experience of 11 (78.6%)
participants, the President’s order emerged as high priority demand factor for
performance audits in Kazakhstan. Additionally, based on my literature and documentary
review results, I concluded that heads of states request performance audits in other post-
Soviet republics including Russia (The Accounts Chamber of the Russian Federation
(ACRF), 2020), but their influences on demand for performance audits were not studied.
There are other influential factors that explain the phenomenon of highly
requested performance audits in Kazakhstan. As described in Chapter 2, based on
linkages between the emergence and developments of new public management and
performance auditing, I defined NPM as a classical trigger for performance audits. Due to
changes in principles of the public sector functioning (Hood, 1991; Hood, 1995), public
auditors started to focus on evaluations and results in order to assess achievements
(Guthrie & Parker, 1999; Levy, 1996; Pollitt & Summa, 1997), and by pursuing
objectives related to promotion of improvements and enhanced accountability in audited
areas (Alwardat et al., 2015; Alwardat & Basheikh, 2017; Barrett, 2012; Flesher &
Zarzeski, 2002; Funnell et al., 2016; Funnell & Wade, 2012; Funnell, 2015; Morin &
Hazgui, 2016; Morin, 2016; Reichborn-Kjennerud & Johnsen, 2018;
ReichbornKjennerud & Vabo, 2017). Similar to other countries, highly demanded
performance audits in Kazakhstan are due to NPM (see Linn, 2014; Nazarbayev, 2017;
Vakulchuk,
2016, for more).
Firstly, in response to reforms in public management, audit organizations extended
their mandates and added performance audits into their audit portfolios
(English, 2003; Free, Radcliffe, & White, 2013; Funnell, 2015; Glynn, 1985; Hossain,
2010; Morin, 2003; Morin & Hazgui, 2016; Torres et al., 2019; Yamamoto & Waranabe,
1989). Different countries empowered their audit organizations with mandates to
undertake performance audits differently. In Canada, performance audits evolved steadily
(Free et al., 2013), whereas, in Australia, mandates on performance auditing of SAI were
expanded, reduced, and expanded again (Hossain, 2010). Governments resisted
expanding public auditors’ mandates (Aucoin as cited by Morin & Hazgui, 2016) and,
therefore, public auditors undertook performance audits de facto, i.e., they exceeded their
de jure mandates (Hossain, 2010; Parker et al., 2019). Despite these differences,
increasing the number of performance audits, as a rule, was preceded by empowering
audit organizations with mandates to conduct performance audits. According to the lived
experiences and perceptions of all parliamentarians, all auditors, and three managers in
my participant group, the number of performance audits increased in Kazakhstan due to
empowering audit organizations with a mandate to conduct that type of audit.
For all countries, except for pioneers in performance auditing, like Canada, the
United States, and United Kingdom, performance auditing is an adapted practice
(Alwardat & Basheikh, 2017; Flesher et al., 2003; Glynn, 1985; Grönlund et al., 2011;
Levy, 1996; Loke et al., 2016; Reichborn-Kjennerud, 2013a; Weihrich, 2018). Many
countries increased the number of performance audits as a result of using the practice of
pioneers or leaders in performance auditing. Three auditors in the participant group
shared examples of using an international practice in their planning and conducting of
performance audits; three parliamentarians also referred to international practice,
including two parliamentarians who argued that adapting an international practice on
performance audits should be continued (PC06, personal communication, January 16,
2020; PC07, personal communication, January 17, 2020).
Trends on adapting international practice are still topical. As described in Chapter
2, INTOSAI contributes to sharing experiences among country-members. Pierre and
Licht (2019) argued that proving the professionalism of SAI’s staff and quality of
auditors’ work “urged SAIs to conform to the INTOSAI’s norms and rulings” (p. 228),
that includes recommended standards and guidelines on performance auditing (INTOSAI,
2019). Thus, my research findings related to Pattern 2 “Features of public auditing
system” includes themes on the legitimization of performance audits and adapting an
international practice being consistent with previous research findings.
Secondly, problems in public administration also shape the demand for
performance audits. Moreover, unlike the needs and interests of participants and users of
performance audits (themes on Pattern 1) and legitimization of performance audits and
adapting the international practice (themes on Pattern 2), problems in public
administration are the most influential factors of highly demanded performance audits in
Kazakhstan (themes on Pattern 3). Systems problems, shortcomings in state planning,
needs in evaluating the effectiveness, and risks of new areas are the main explanations of
demand for performance audits in Kazakhstan. Comparing these findings with the
reviewed literature, I concluded that they are related to both classical and specific
triggers, described in Chapter 2, including global financial crises, governments’ initiatives
on addressing lack of financial resources, structural problems, and environmental issues.
The factors of demanded performance audits associated with a classical trigger
include systems problems and shortcomings in state planning. Conceptually, these
findings are consistent with Jacobs’ (1998) argument that performance auditing emerged
and developed to solve problems. When justifying the conduction of more performance
audits, auditors and parliamentarians indicated deficits of the republican budget and tax
revenue reduction caused by global financial crises as reasons that support their positions
(AC01, personal communication, January 8, 2020; PL08, personal communication,
January 20, 2020). Worsening financial conditions and fiscal stress caused by global
financial crises were identified as reasons to conduct performance audits earlier (Athmay,
2008; Free et al., 2013; Guthrie & Parker, 1999; Jacobs, 1998; Rosa et al., 2014b;
Yamamoto & Waranabe, 1989). Particularly, analyzing correlations between sizes of
public sectors, tax revenues, and expenses, Yamamoto and Waranabe (1989) concluded
that performance auditing widely spread when public administrators were challenged by
lack of financial resources.
In my research, I found that performance auditing in Kazakhstan supported the
relevance of Yamamoto and Waranabe’s previous findings despite their research being
conducted over 30 years ago. For Kazakhstan, triggering performance audits due to
financial stresses is consistent with the most recent considerations made by Torres et al.
(2019) and Parker et al. (2019). As Parker et al. (2019) stated, “the growth of PA
[performance auditing] has been the product of a ritualistic response to a broader social
demand for financial assurance driven by cycles of financial crisis and associated
governmental budgetary constraints” (p. 281).
For my study participants, systems problems also included ineffective budget
spending, poor performance of government organizations, social problems, and lags in
regional development. The same explanations for emerging and faster performance audit
developments were provided by scholars in the 1990s and in more recent studies. For
instance, Jacobs (1998) argued that, in the public administration of New Zealand,
performance auditing emerged in response to problems in government financial
management. Free et al. (2013), based on cases from Canada, shared the same view. The
inferior performance of public sector organizations of both developed and developing
countries was also defined as reasons for performance audits (Common and Minogue, as
cited by Athmay, 2008). In continuation of these views, Alwardat et al. (2015), based on
the case of United Kingdom, concluded that the development of performance auditing
was stimulated by intensions to maximize outcomes in terms of reducing governments’
spending.
Shortcomings in state planning, identified as an influential factor of demand for
performance audits, are also related to NPM, or, more precisely, to post-NPM, since
Kazakhstani state planning system was established within NPM’s initiatives and it
implies using the strategic planning approaches (ISLARK, 2017a, 2017b). These findings
are consistent with Knaap’s (2011) view about undertaking performance audits due to
disappointing agency or program performance. As shared by my study participants,
documents of state planning (i.e., strategies, state programs, programs on regional
development, development plans of national state-owned companies, and budget
programs) and the quality of their implementation are reasons to conduct more
performance audits. In turn, responding to problems on state programs implementation by
performance audits are in line with the well-established practice of SAI of the United
States on performance auditing of state programs (USGAO, 2019).
Nine (64.3%) participants shared their views and perceptions about needs in
evaluating the effectiveness of government entities, quasi-public companies, and
programs’ implementation. All five auditors justified their organizations’ practice on
planning and conducting more performance audits by the necessity to provide users with
reliable evaluations, especially in terms of existing controversial information (ACL/05,
personal communication, January 15, 2020). In most cases, the perceptions of my
participants were related to ensuring the government’ accountability for budget planning
and program implementation, and auditees for using budget funds as well. For instance,
one parliamentarian listed a few programs of the central government, including programs
on providing benefits to small and medium-sized businesses, that should be objectively
and independently evaluated by performance auditors (PC07, personal communication,
January 17, 2020). Thus, the generated theme “Needs in evaluating the effectiveness” is
directly related to realizing the accountability principle and, therefore, consistent with
some associated studies (Barrett, 2012; Funnell, 2004a, 2004b, 2015; Kells, 2011; Morin
2014, 2016; Reichborn-Kjennerud, 2011b, 2014b).
Theme “Risks of new areas” was generated based on two family categories. The
first family categories included unaudited organizations (e.g., the National Fund and the
Pension Fund) or unaudited areas of public administration (e.g., regional waste
management programs) in Kazakhstan. This finding is in line with Lapsley and Pong’s
study (2000) that resulted in a conclusion about expanding performance auditing
coverage within the public sector. Auditors’ views and experiences on the conduction of
performance audits in organizations that do not receive budget funds (AC01, personal
communication, January 8, 2020) are matched with Bini’s (2019) considerations related
to managing the scope of performance audits to avoid limitations, such as controlling
budget money. The second family categories included new areas in public administration,
defined in Chapter 2 as specific triggers. Particularly, public-private partnership projects,
national companies, and financial institutions were listed as subjects for performance
audits by my participants. According to auditors and parliamentarians’ perceptions, these
areas should be audited due to risks of negative consequences or even losses in budget
funds. These findings are consistent with Reichborn-Kjennerud and Johnsen’s (2011)
conclusions about selecting performance auditing themes by auditors depending on the
risks for public funds. In sum, regarding the identified factors that explain the highly
demanded performance audits in Kazakhstan, I concluded that some research findings are
consistent or partially consistent with views and conclusions of authors of the reviewed
studies, while others are relatively new, as it is demonstrated in Table 19.
Impacts of performance audits. Despite the novelty, performance audits already
had made impacts on public administration in Kazakhstan in line with Lonsdale’s et al.
(2011) classifications, as described in Chapter 2. Performance audits had instrumental
impact; e.g., all auditors discussed their tasks within performance audits, such as
evaluation of achieving the expected outcomes in implementing budget programs, and, in
turn, managers shared their experiences on changing reports on budget program
implementation in their organizations as effects of auditors’ work (see Lonsdale et al.,
2011, for more). Tactical (e.g., managers shared their experience on implementation of
auditors’ instructions on restoration works) and strategic (e.g., both auditors and
managers shared examples of changing budget programs and programs on regional
development as results of an implementation of auditor’s recommendations) impacts also
emerged in the lived experience of study participants (see Desmedt et al., 2017; Lonsdale
et al., 2011, for more). Most of the participants shared their views about auditors’
methods, demonstrating an understanding of performance audit objectives and changes in
requirements to auditees’ performance, i.e., examples of conceptual impact (see Lonsdale
et al., 2011, for more).
Table 19
Aggregated Data on Empirical Context and Evidence: Demand for Performance Audits
Patterns and themes Authors of previous studies
Conclusions related
to consistency the
research findings
Who creates demand for audits
Auditors prefer
performance audits
Nath et al., 2019; Norton & Smith, 2008
consistent
Parliamentarians
need in performance
audits
Arthur et al., 2012; Funnell, 2004a, 2011, 2015;
Funnell & Wade, 2012; Funnell et al., 2016;
Guthrie & Parker, 1999; Morin, 2001, 2008,
2016; Pollitt & Summa, 1997;
ReichbornKjennerud, 2011, 2013a, 2013b; 2014a,
2014b;
Reichborn-Kjennerud & Vabo, 2017
Parker et al., 2019; Nath et al., 2019
consistent
partially consistent
President prioritize There are no studies related to presidents’ influence on demand for
performance audits performance audits.
Managers of auditees
use performance
audits’ results
Funnell 2015; Knaap, 2011; Morin (2001); Nath et
al., 2019; Parker et al., 2019
partially consistent
Citizens and media
shape the demand for
performance audits
Arthur et al., 2012; Knaap, 2011; Nath et al.,
2019; Parker et al., 2019; Rosa et al., 2014a;
Rosa et al., 2014b
Justesen & Skaerbaek, 2010; Morin, 2008;
Tillema & Bogt, 2010
partially consistent
consistent
Features of the system of public auditing
Legitimization of
performance audits
English, 2003; Free et al., 2013; Funnell, 2015;
Glynn, 1985; Hossain, 2010; Morin, 2003;
Morin & Hazgui, 2016; Torres et al., 2019;
Yamamoto & Waranabe, 1989
consistent
Adaptation of
international
experience
Alwardat & Basheikh, 2017; Flesher et al., 2003;
Glynn, 1985; Grönlund et al., 2011; Levy, 1996;
Loke et al., 2016; Pierre & Licht, 2019;
Reichborn-Kjennerud, 2013a; Weihrich, 2018
consistent
Problems of the system of public administration
State planning
Knaap, 2011; USGAO, 2019
consistent
Systems’ problems
Alwardat et al., 2015; Free et al., 2013; Guthrie
& Parker, 1999; Jacobs, 1998; Parker et al.,
2019; Rosa et al., 2014b; Athmay, 2008; Torres
et al., 2019; Yamamoto & Waranabe, 1989
consistent
Risks of new areas
Lapsley & Pong, 2000; Bini, 2019;
ReichbornKjennerud, 2011
consistent
Needs in evaluating the
effectiveness
Barrett, 2012; Funnell, 2004a, 2004b, 2015;
Kells, 2011; Morin 2014, 2016;
ReichbornKjennerud, 2011b, 2014b
consistent
Political-legitimizing impact was also manifested in the lived experience of
participants; they did not refer to exact examples of using the performance audits to
legitimize adopted policies or previous political decisions (see Funnell, 2015; Lonsdale et
al., 2011, for more). However, some managers shared examples including points about
good practices of their organizations in performance audit reports, supporting that
approach; parliamentarians shared their experience of using the performance audits for
making legislative proposals.
Performance audits impacted at micro-, meso-, and macrolevels (see Lonsdale et
al., 2011, for more). I defined generated Themes 211, 212, and 213 as examples of impact
at microlevel, Themes 221 and 222 as examples of impact at mesolevel, and Themes 231
and 232 as examples of impact at macrolevel (see Desmedt et al., 2017; Lonsdale et al.,
2011, for more).
According to participants’ perceptions, performance audits impacted auditors
(100% participants), managers of auditees (92.9%), and parliamentarians (57.1%), and
the impact on managers of auditees was more highlighted compared to impacts on
auditors and parliamentarians. Based on evidence-based findings from my literature
review, I concluded that this specific research finding is relatively new given that
previous studies did not show evidence related to identifying personal impacts of
performance audits based on simultaneously exploring the perceptions of auditors,
managers of auditees, and parliamentarians. Particularly, influences of performance audits
on managers were explored based on perceptions of managers (Desmedt et al.,
2017; Morin, 2008, 2014), and partially by Alwardat and Basheikh (2017) and Justesen
and Skaerbaek (2010). Some aspects of personal impact on managers were fragmentarily
analyzed by scholars who studied reactions of auditees on performance audits (Funnell &
Wade, 2012; Reichborn-Kjennerud, 2013b, 2014a) and perceptions of auditors and
managers about usefulness of performance audits at meso- and macrolevels (Alwardat et
al., 2015; Funnell et al., 2016; Morin 2016; Reichborn-Kjennerud, 2013a, 2014b,
Reichborn-Kjennerud & Johnsen, 2018; Reichborn-Kjennerud & Vabo, 2017).
One of my key findings related to microlevel impacts is that participant’s
perceptions about the impact of performance audits on managers of auditees vary. All
three groups of participants shared examples of both negative and positive impacts.
Firstly, according to the lived experience and perceptions of managers of auditees,
in most cases, performance audits negatively impact on their perspectives. Wherein,
negative perceptions were related to work complication and negative emotions, such as
stress, increased distrust of auditors’ work, and perceived performance audit results as
uselessness. Other scholars (see Pollitt as cited by Morin, 2008, for more) discussed work
complications as a result of auditor visits. Justesen and Skaerbaek (2010) identified stress
and discomfort as negative emotional consequences of performance audits for both
managers and employees of audited organizations. Distrust to auditors’ work due to their
technical competencies (Alwardat et al., 2015; Funnell et al., 2016; Morin 2001;
Pendlebury and Shreim as cited by Loke et al., 2016), usefulness and problematic nature
of performance audits (Bawole & Ibrahim, 2016; Kells, 2011; Lapsley & Pong, 2000) are
inherent aspects of interactions between managers and auditors and serve as examples of
negative impacts. For instance, Alwardat et al. (2015) stated, “auditors’ competence,
skills, experience and knowledge of the public bodies’ activities… had negative effects on
the ability of the external VFM [value-for-money] auditors to influence the audited
bodies” (p. 213). All five managers’ stories about situations when they may experience
fear, non-acceptance, recognition, and gratitude are consistent with observations made by
Funnell and Wade (2012), who concluded that fear and animosity are emotional
perceptions of performance audits by auditees, while some auditees may experience
understanding and appreciation as well.
Positive perceptions of managers regarding the impact of performance audits were
related to learning, receiving new information, and, in fewer cases, promotion and
understanding performance audits’ objectives. These findings are also consistent with
previous studies. For instance, Reichborn-Kjennerud and Johnsen (2018), by testing their
hypotheses related to positive impacts of performance audits in Norway, concluded, “An
audit might reveal new facts that trigger debates and lead to a shift in paradigms. It might
also provide new information that will lead auditees to see solutions in a new light” (p.
1430). Alwardat and Basheikh (2017), Desmedt et al. (2017), and Morin (2008, 2014)
also identified some positive personal impacts on managers and employees of audited
organizations, including career development opportunities. However, analyzing
components of positive personal impacts in detail, I concluded that my findings are only
partially consistent with these author’s results. Particularly, unlike the Belgian case
studied by Desmedt et al. (2017) and the Canadian case studied by Morin (2008, 2014),
negative perceptions of managers prevailed in the Kazakhstani.
Like managers of auditees, auditors perceived the impact of performance audits on
managers as mainly negative; wherein, negative perceptions were related to objections
concerning performance audits results, a negative reaction on conducting a performance
audit itself, and misunderstanding performance audits’ objectives. Auditors’ perceptions
about personal impacts that performance audits have on managers of auditees were not
studied previously. However, there are few consistencies and inconsistences between my
findings and results from Funnell and Wade (2012) and Morin and Hazgui (2016). For
example, Funnell and Wade (2012) discussed auditors’ perceptions about auditees
reactions, which may include personal attacks, to their performance audits. In contrast to
Funnell and Wade (2012), I concluded that auditors faced less aggressive reactions of
managers, especially if the latter [managers] gained more experience in performance
audits. For instance, some auditors shared their experience when managers of auditees
positively reacted to their audits and demonstrated an understanding of performance
audits objectives (AC01, personal communication, January 8, 2020; AL03, personal
communication, January 9, 2020; AL04, personal communication, January 10, 2020).
Morin and Hazgui (2016) explained confrontations between auditors and auditees through
an expectation gap, i.e., auditors and auditees do not gain desired results from each other
in the course of their interactions (e.g., managers doubt auditors’ contribution, while
auditors want to be perceived as assistants). These considerations are relevant to the
Kazakhstani.
Secondly, in auditors’ opinions concerning performance audits’ impact on their
perspectives, negative perceptions outweigh positive perceptions. The negative impacts
include increased requirements, workload, and, in fewer cases, demotivation, and staff
turnover. Auditors directly related increased requirements and workload with the
introduction of performance audits in their organizations, while demotivation and staff
turnover were mainly associated with requirements on financial control and applied
(unfair) methods of evaluation of performance auditors’ work. The positive impacts
include mastering new skills, finding interesting work, and, to a lesser degree,
professional development, and promotion. This finding is relatively new. Alwardat et al.
(2015), Funnell and Wade (2012), and Loke et al. (2016) explored auditors’ perceptions
about usefulness of their work, Morin (2001, 2003) and Morin and Hazgui (2016)
analyzed roles that they [performance auditors] play in public administration. These
studies resulted in predominantly positive rhetoric about performance auditors’
contributions.
Managers of auditees, in most cases, expressed negative perceptions about how
performance audits impact on auditors. Particularly, managers claimed that auditors did
not change their methods of auditing, preferred to find financial violations; they
[auditors] are still perceived as controllers or auditors who, as before, make unfair and
subjective conclusions. These findings are consistent with related discussions and
explanations provided in previous studies (Funnell & Wade, 2012; Funnell et al., 2016;
Morin 2003, 2008, 2014, 2016; Reichborn-Kjennerud, 2013b, 2014a). Stereotypical
perceptions are manifested in relation to performance auditors who operate in western
public administrations (Funnell & Wade, 2012; Pollitt, 2003) and in Kazakhstan as well.
But, in the first case, performance auditors were perceived as “characterless men in grey
suits” (Pollitt, 2003, p. 168), while, in the second case, they perceived as inspectors who
return from the Soviet Union (ML/C14, personal communication, February 28, 2020).
Thirdly, unlike auditors and managers of auditees, parliamentarians were less
active in sharing their experience and perceptions about the personal impacts of
performance audits. In their lived experience, negative and positive influence on
managers highlighted equally, whereas impacts on auditors were perceived as positive
mainly. These findings are generally consistent with some previous studies. For instance,
Morin (2016), and most recently Parker et al. (2019), argued that legislatives praise the
work of auditors. Both positive and negative perceptions of parliamentarians associated
with personal impacts on managers are partially consistent with discussions shared by
Funnell (2015), Morin (2016), and Reichborn-Kjennerud (2013a, 2014b). Impacts of
performance audits on perspectives of parliamentarians from their viewpoints were not
studied before.
Next, according to the lived experiences of study participants, performance audits
impacted audit organizations (100% of participants) and auditees (85.7% of participants).
Wherein, audit organizations were impacted more significantly compared with auditees.
In turn, organizational impacts on legislative bodies did not emerged as major themes in
the lived experience of my participants, actualizing the need for further studies in order to
explore mesolevel impacts of performance audits.
In auditors’ opinions, the most influential impacts of performance audits on their
organizations are associated with changes in audit methodology and organizational
consequences. While changes in audit methodology were solely related to the
introduction of performance audits, organizational consequences were due to other
factors. Some managers of auditee and parliamentarians also referred to changes in
methods of auditors’ work, e.g., auditors’ focus expanded from accounting departments’
performance to other core departments in ministries (ML13, personal communication,
February 15, 2020; ML/C14, personal communication, February 28, 2020; PL09,
personal communication, January 23, 2020). Changes in methods and organizational
structures of SAIs and local audit organizations as results of performance audits were
discussed by Barrett (2010, 2011a), English (2003), Grönlund et al., (2011), Hossain
(2010), Jacobs (1998), Keen (1999), Kells and Hodge (2011), Lonsdale (2000), Pollitt
and Summa (1997), Pollitt (2003), Rosa et al. (2014a), and Tillema and Bogt (2010).
However, these authors’ conclusions are results of reviewing practices of audit
organizations, i.e., they [conclusions] were not derived through studying perceptions of
participants of performance audits.
Further, auditors shared (a) positive perceptions associated with the introduction
of performance audits, such as the improved reputation of their organizations and changes
in approach to auditing in general, and (b) negative perceptions, including the keeping of
financial control that leads to deterioration of their organizations’ reputation. The negative
attitude toward using mechanisms of financial control in performance audits was shared
by managers of auditees and parliamentarians; it is almost the only case of consensus in
study participants’ opinions, i.e., when everyone expressed a unanimous opinion.
On one hand, my research findings are partially in line with Lonsdale’s (2000,
2008), who argued that performance audits influence audit organizations’ reputation.
However, the negative impact of performance audit on audit organizations were related to
using financial control in Kazakhstan, while in United Kingdom it was due to political
risks associated with close links between audit organizations, governments, and
parliaments (Lonsdale, 2000, 2008).
On the other hand, my research findings are also partially consistent with
considerations of dual roles of audit organizations, i.e., watchdogs or controllers and
modernizers or catalysts for change and improvement (Morin 2003, 2008, 2010, 2016;
Morin & Hazgui, 2016). Like Lonsdale (2000, 2008), these authors also did not consider
audit organizations’ roles in the context of financial control, i.e., sanctioning for
violations and deficiencies; they analyzed the roles of auditors as controllers, i.e.,
compliance auditors, and change agents, i.e., advisers who help by providing auditees
with recommendations (Morin 2003, 2008, 2010, 2016; Morin & Hazgui, 2016). Despite
these different aspects in exploring performance audits’ impacts, my research findings
about auditors’ desire to be welcomed and perceived as advisors (AC02, personal
communication, January 8, 2020; AL03, personal communication, January 9, 2020) and
managers’ distrust to auditors’ new roles and perceptions of interactions with compliance
auditors as easier opposite to interactions with performance auditors (MC10, personal
communication, February 11, 2020; ML12, personal communication, February 15, 2020)
are consistent with research findings of Morin (2003) and Morin and Hazgui (2016). Both
managers and parliamentarians wait for audit organizations’ recommendations before
addressing problems, rather than acting on facts of identified violations (PC07, personal
communication, January 17, 2020; PL08, personal communication, January 20, 2020;
ML13, personal communication, February 15, 2020; ML/C14, personal communication,
February 28, 2020).
All auditors claimed that performance audits positively impacted auditees, and
positive impacts included improvements in performance and financial discipline,
enhanced internal control, and, in fewer cases, improved reputation and organizational
changes. Auditors emphasized that virtually every audit event led to improved prospects
for audited organizations. These positive auditors’ perceptions are consistent with the
results of studies by Alwardat et al (2015), Funnell and Wade (2012), Loke et al. (2016),
Morin and Hazgui (2016), and most recent publications of Jeppesen et al. (2017) and
Johnsen et al. (2019).
Unlike auditors, managers of auditees shared both positive (readiness to new
audits, better discipline, improved performance, and enhanced control) and negative
(workload, demotivation, staff turnover, and financial consequences) perceptions
associated with impacts that performance audits had on their organizations. Managers’
perceptions related to impacting financial discipline within their organizations are in line
with Pollitt and Summa’s (1997) conclusions and are related to changes in internal
business processes in concert with Lapsley and Pong’s (2000) arguments. Workload and
demotivation are also earlier identified as consequences of performance audits; “the
presence of auditors added considerably to auditees usual workload” and lead to “a drop
in motivation” (Pollitt et al. as cited by Morin, 2008, p. 703).
Morin (2004, 2008, 2014) argued that performance audits contribute to changes in
audited public administrations. To a greater extent, performance auditors impacted
administrations through making attention to problems, preparing useful audit reports,
providing relevant recommendations, and, to a lesser extent, through preventive effects,
changing in management practices, taking actions, contributing to better relations with
stakeholders, organizational and personal consequences (Morin, 2014). In the lived
experience of managers of auditees, ‘controversial’ pictures emerged aligned with
Morin’s (2014) categories. For instance, in managers’ opinions, the positive impact of
performance audits, in most cases, manifested as preventive effects (i.e., the improved
financial discipline and readiness to future audits) and, in fewer cases, by useful reports
or relevant recommendations being offered as a useful audit outcome. However, some
managers shared examples of only rarely receiving quality reports with applicable
recommendations (ML13, personal communication, February 15, 2020; ML13, personal
communication, February 15, 2020).
Further, Morin (2008, 2014) and Desmedt et al. (2017) argued that performance
audits do not lead to deteriorations in public administrations. In contrast to these
conclusions, managers of auditees frequently listed negative consequences of
performance audits, such as workload, demotivation, staff turnover, and financial
consequences. That consideration is based on comparing research results achieved by
applying different research designs, i.e., qualitative and quantitative, and, therefore,
drawing any conclusion should be cautiously done. Further research is needed. Morin’s
(2008) found that “the visits of auditors are not guaranteed to produce the desired
changes in the management of government organizations” but generally performance
audits “seem to have done more good than harm to audited organizations” (p. 712),
similar to Desmedt et al. (2017), and are relevant to Kazakhstani.
Based on the Saudi Arabia SAI’s case and though application of Morin’s (2014)
research instrument, Alwardat and Basheikh (2017) concluded that public administrators
make more efforts to comply with laws, enhance control, establish acceptable indicators,
improve the quality of performance reports, change strategic and operational plans, and
maintain effective relations with their superiors and subordinates, as a result of
performance audits. Similar examples of the positive impact of performance audits on
audited organizations were shared by managers of auditees and auditors; however, unlike
public administrators in Saudi Arabia, managers emphasized that these positive examples
were rare cases in Kazakhstan.
Reichborn-Kjennerud and Vabo (2017) stated that, in Norway, the positive
impacts of performance audits, in managers’ opinions, include increased documentation
and reporting, changes in approaches to internal control and risk management, improved
strategies, and planning. These findings are partially relevant to Kazakhstan, with
similarities including changes in reporting, internal control, strategies, and budget
planning.
My findings related to impacting auditees are also partially consistent with
considerations of scholars who argued that the positive (or negative) impact of
performance audits on auditees is achievable subject to specific conditions. In particular,
all managers emphasized that if they agree with the recommendations of the auditors and
find them useful, they are interested in their implementation, i.e., it is consistent with the
results of testing related hypotheses of Reichborn-Kjennerud (2013а) and
ReichbornKjennerud and Johnsen (2018).
Finally, at macrolevel, performance audits impacted budget processes at both
central and local levels of public administration in Kazakhstan and contributed to changes
in laws and regulations. Unlike impacts that performance audits had at micro- or
mesolevels, macro-impacts were not associated with negative perceptions of my study
participants. Auditors and parliamentarians mainly shared examples on positive impacts,
while managers of auditees shared their opinions on impacting performance audits on
public administration neutrally. Most of the participants emphasized that judging about
performance audits’ impact is difficult, and that point was discussed by Funnell and Wade
(2012), Morin (2008), and Pollitt (2003). At the same time, nine (64.3%) participants,
when asked my question related to impacts of performance audits on public
administration in general, stated that these audits are useful.
Specifically, according to the perceptions and opinions of most participants,
performance audits impacted budget planning, administration of budget programs, and
budget reports. Focusing participants’ perceptions on budget affairs is explained by
features of the accountability system and practices on using performance audits. Audit
organizations use their performance audits to prepare annual reports, i.e., conclusions
about the execution of central and local budgets by governments; their reports are
considered and approved by parliamentarians. Wide use of performance audits begins
from discussions of audit organizations’ annual reports in parliaments with both
governments and media participation. That approach is in line with western systems on
accountability when “the presentation of reports is merely one stage in a cycle of
accountability that begins with the budget process and ends with the presentation of
reports to parliament” (English, 2003, p. 56). My research findings are also consistent
with views about performance audits’ impacts through stimulating public hearings (Morin
2001, Reichborn-Kjennerud, 2013a). As Reichborn-Kjennerud (2013a) argued,
performance audits might effectively impact public administration, enhancing the
accountability subject to the involvement of parliament, media, and other users of audit
reports.
As my literature review offered, the identified impact of performance audits on
budget processes was not explored or explained before as a self-sufficient impact.
However, impacts that performance audits have on public financial management and
specific chains of a budget process, e.g., making budgetary decisions, were discussed by
Johnsen et al. (2001), Lonsdale (2000), and most recently by Olaoye and Adedeji (2019).
In contrast to Olaoye and Adedeji (2019), who noted that performance audits lead to
budget efficiency, I concluded that impacts of performance audits on budget processes are
more procedural rather than result-oriented. In turn, Lonsdale’s (2000) considerations
about the adaptation of budget bills as a result of using performance auditors’ work are in
line with the experiences of the Kazakhstani parliamentarians (PC06, personal
communication, January 16, 2020; PL09, personal communication, January 23, 2020).
Alwardat and Basheikh (2017), Thompson and John (2019), and Torres et al. (2019)
explored impact of performance audits on public sector organizations’ financial
behaviors, which inherently linked with a budget process. Their results are also consistent
with my research findings.
All participants discussed changes in laws and regulations as positive
consequences of performance audits. This research finding is consentient with the
research of Morin (2008, 2014), Alwardat and Basheikh (2017), and Desmedt et al.
(2017). These authors interpreted changes in laws and regulations as mesolevel impacts,
i.e., as a variable of concrete actions undertaken by auditees after performance audits.
However, their findings are not conflicting with my findings since the impacts of
performance audits at different levels are interrelated, e.g., impacts at mesolevel lead to or
imply changes at macrolevel (see Desmedt et al., 2017, for more). My research-based
findings related to impacting improvements of laws and regulations as a result of
auditors’ interventions are also consistent with well-tested multiyear practices of some
SAIs. For instance, GAO uses data on changes in laws and regulations made based on
their recommendations as non-financial benefits, i.e., as a performance indicator of its
audits (Bawole & Ibrahim, 2016; USGAO, 2019).
Reviewing the literature on impacts of performance audits in different countries
and at different stages of development of performance auditing practice, I concluded that
most recent studies are focused on or revealed impacts on accountability, political and
democratic processes, whereas impacts on policies and programs were explored within
more pioneering studies (Glynn, 1995; Johnsen et al., 2019; Levy, 1996; Morin, 2016;
Pollitt & Summa, 1997; Torres et al., 2019). A few ideas associated with the
accountability and political developments were shared by my study participants, but the
frequency of associated categories was not enough to generate relevant themes. These
differences in comparable research findings are due to the novelty of performance
auditing in Kazakhstan, and identified impacts on budget processes, laws, and regulations
are predominantly related to changes in policies and programs. Thus, impacting political
and democratic processes and accountability by performance auditors’ interventions
might be proposed as perceptive areas for future studies.
Further, the novelty of performance audits or, more precisely, beginning
performance auditing practice is the main explanation of why a sub-theme on needs in
changes in performance auditors’ work emerged in the lived experience of the 14 (100%)
participants. One example is that parliamentarians expressed their opinions on expanding
the scope of performance audits and focusing on making recommendations, supporting
Barrett’s (2011) statement that, “A major issue for performance auditing is the
maintenance of parliamentary and public confidence in the coverage, timeliness and
outcomes of such audits, as well as effective action to ensure proper implementation of
any recommendations and conclusions” (p. 130). Another example is that all auditors
called for changes in audit methodology, pointing to overregulated methods on doing
performance audits today. Auditors’ concerns are consistent with a generally accepted
views that, “Streamlined procedures, methods and standards may in fact hamper the
functioning and the progress of performance audit... standards – as well as quality
assurance systems – that are too detailed should be avoided. Progress and practices must
be built on learning from experience” (INTOSAI as cited by Knaap, 2011, p. 355). In
turn, views about changes in the use of financial control within a performance audit
shared by all participants required responses from policymakers.
Finally, the generated sub-theme regarding contributions to public sector
performance by parliamentarians and managers of auditees without using the
performance audits, i.e., their self-contributions to the economy, effectiveness, and
efficiency, are consistent with previous studies’ results about using performance auditing
as one of the NPM’s tools on improving the public sector performance (Alwardat et al.,
2015; Azuma 2003, 2005; English, 2003; Flesher & Zarzeski, 2002; Funnell, 2015;
Funnell et al., 2016; Funnell & Wade, 2012; Lapsley, 1999; Lonsdale et al., 2011; Morin,
2016; Morin & Hazgui, 2016; Power, 1996; Reichborn-Kjennerud & Johnsen, 2018).
Thus, as demonstrated in Table 20, findings on the impact of performance audits
on Kazakhstani public administration are consistent or partially consistent with
conclusions of authors of some previous studies, while others are relatively new.
Theoretical Context and Alignment
As outlined in Chapter 1 and particularly described in Chapter 2, I used a
combination of Hood’s NPM and Waterman and Meier’s PAM as theoretical frameworks
to explore the phenomena of demanded performance audits in Kazakhstan and their
impact on national public administration system (see Hood, 1991; Waterman & Meier,
1998, for more).
Table 20
Aggregated Data on Empirical Context and Evidence: Impact of Performance Audits
Patterns and
themes
Authors of previous studies
Conclusions on
consistency
Impact on participants of audits
Impact on auditors
There are no studies related personal impacts of performanc
performance auditors.
e audits on
Impact on
managers of
auditees
Alwardat & Basheikh, 2017; Alwardat et al., 2015;
Desmedt et al., 2017; Funnell et al., 2016; Funnell &
Wade, 2012; Justesen & Skaerbaek, 2010; Loke et al.,
2016; Morin, 2001, 2008, 2014; Morin & Hazgui,
2016; Reichborn-Kjennerud & Johnsen, 2018
partially
consistent
Impact on There are no studies related personal impacts of performance audits on
parliamentarians parliamentarians.
Impact on audit organizations and auditees
Changes in audit
organizations
English, 2003; Grönlund et al., 2011; Hossain, 2010;
Jacobs, 1998; Keen, 1999; Lonsdale, 2000, 2008;
Morin 2003, 2008, 2010, 2016; Morin & Hazgui, 2016;
Pollitt & Summa, 1997; Pollitt, 2003; Rosa et al.,
2014a
partially
consistent
Changes in
auditees
Alwardat et al., 2015; Alwardat & Basheikh, 2017;
Bawole & Ibrahim, 2016; Desmedt et al., 2017; Funnell
& Wade, 2012; Jeppesen et al., 2017;
Johnsen et al., 2019; Kells, 2011; Lapsley & Pong,
2000; Loke et al., 2016; Morin, 2004, 2008, 2014;
Morin & Hazgui, 2016; Pollitt & Summa,1997;
Reichborn-Kjennerud, 2013а; Reichborn-Kjennerud &
Vabo, 2017; Reichborn-Kjennerud & Johnsen, 2018
consistent
and
partially
consistent
Macrolevel impact
Improved budget
process
Alwardat & Basheikh, 2017; Johnsen et al., 2011;
Lonsdale, 2000; Olaoye & Adedeji, 2019; Thompson et
al., 2019; Torres et al., 2019
partially
consistent
Changes in laws
and regulations
Alwardat & Basheikh, 2017; Bawole & Ibrahim, 2016;
Desmedt et al., 2017; Morin, 2008, 2014; USGAO, 2019
consistent
Self-contribution
to 3E
Alwardat et al., 2015; Azuma 2003, 2005; English, 2003;
Flesher & Zarzeski, 2002; Funnell, 2015; Funnell et al.,
2016; Funnell & Wade, 2012; Lapsley, 1999; Lonsdale et
al., 2011; Morin, 2016; Funnell, 2015;
Morin & Hazgui, 2016; Morin, 2016; Power, 1996;
Reichborn-Kjennerud & Johnsen, 2018
consistent
Requested
changes in
performance
auditors’ work
Barrett, 2011; Knaap, 2011
consistent
To address problems on ineffective public management, public administrators
apply approaches and means encapsulated by Hood in his seven doctrines, and it explains
why NPM becomes a widely using public management reform: “From Denmark to New
Zealand, from education to health care, from central to local government and quangos,
from rich North to poor South…” (Hood, 1991, p. 8). In turn, performance auditing is one
of the tools used to address problems in ineffective public management (Barrett, 2012;
English, 2003; Funnell, 2015).Thus, identified factors of highly demanded performance
audit in Kazakhstan, including problems in public administration, confirm the
reasonableness of Hood’s idea on labeling NPM as ‘public management for all seasons’
since local and other variations do not change the essence of NPM and NPM is an
‘apolitical’ framework (Hood, 1991, p. 8).
Auditors and parliamentarians, when justifying increased number of performance
audits, referred to shortcomings in state planning, systems’ problems, and risks in new
areas of public administration at both central and local governments levels in Kazakhstan
(Themes 131, 132, and 133). In particular, the study participants explained the needs in
performance audits by ineffective budget spending and weak control over the execution
of budget programs. These views are in line with Hood’s doctrines including explicit
standards and measures of performance and greater emphasis on output controls (Hood,
1991, 1995). Needs in evaluating the effectiveness (Theme 134) are important in making
decisions related to effective and efficient allocation and using the budget funds, or, in
“cutting costs and doing more for less as a result of better-quality management” (Hood,
1991, p. 15).
The arguments of study participants associated with budget deficits (AC01,
personal communication, January 8, 2020), reduction in tax revenues caused by fall in oil
prices (PL08, personal communication, January 20, 2020), and complaining of a drop in
investment (PC07, personal communication, January 17, 2020) are in line with Hood’s
consideration about the development of NPM and its tools in response to fiscal stress and
poor macroeconomic performance (Hood, 1995). According to most of the participants,
enhanced financial discipline and better implementation of budget programs are examples
of positive impact of performance audits. These views are consistent with the seventh
doctrine of Hood – “stress on greater discipline and parsimony in resource use” (Hood,
1991, p. 5). Like Hood’s (1991, 1995) study of cases of OECD countries in the 1990s,
and more recently studied cases of OECD countries, including countries of Eastern
Europe experienced in building national systems of public administration following the
fall of the Soviet Union, China, and Vietnam (Bao et al., 2012; Esposito et al., 2018;
Hammerschmid et al., 2018; Reiter & Klenk, 2019; Verbeeten & Spekle, 2015), using
performance audits in Kazakhstan offered legitimization (Theme 121) and importing
international experiences in the course of NPM adaptation (Theme 122; see Barrett, 2012;
English, 2003; Funnell, 2015, for more).
The needs and interests of auditor, parliamentarians, and managers of auditees
(Themes 111, 112, and 114), who play the roles of principals and agents within a
performance auditing process, also explain the demand for performance audits (see
Waterman & Meier, 1998, for more). When sharing the experience of interactions with
managers, auditors demonstrated specific features of roles of principals to managers, e.g.,
setting accountability requirements, such as preparing and providing information about
performance results, or requirements on implementing auditors’ instructions, such as
carrying out restoration works. Then, unlike parliamentarians or managers of auditees,
auditors play both principal and agent’s roles, and it explains their perceptions about
using the performance audits results and alignment their explanations with Waterman and
Meier’s (1998) considerations.
As Waterman and Meier (1998) argued, in some cases, legislators as principals
through oversight processes may have more access to information opposite to agents. Or,
managers as agents may have more access to information opposite to principals, and,
therefore, “they will almost always have a better understanding of how their agencies
operate than will outside principals” (Waterman & Meier, 1998, p. 184). Thus, PAM was
a good framework for understanding parliamentarians’ references to using their mandates
on request of information from different sources, and their positions on being not limited
to performance audits result in implementing their duties as elected officials.
Analogically, auditors shared their experience of using auditees’ data as well as media
publications, citizens’ complaint tracking information systems, and other sources to make
decisions related to performance audits. In turn, all managers argued that they are better
aware of the functioning of their ministries or departments, criticizing auditors for their
biased or unfair conclusions or explaining parliamentarians’ critics as a way of using
auditors’ work in their political objectives.
The lived experiences of 13 (92.9%) participants, including four parliamentarians,
five auditors, and four managers, demonstrated how information asymmetry works within
conflicting interests of principals and agents (see Waterman & Meier, 1998, for more).
For instance, different level of access to information and conflicting goals explains
variations in the reaction of principals and agents toward performance audits (see
Maggetti & Papadopoulos, 2018; Waterman & Meier, 1998, for more). Four managers of
auditees perceived interactions with auditors as a functional, but not voluntary, duty,
while auditors shared experiences on managers’ resistance to performance audits and
their experiences were confirmed by both managers and parliamentarians.
Further, like participants of performance audits, President (Theme 113), citizens,
and media (Theme 115) influence the demand for performance audits in Kazakhstan. Or,
in the language of PAM, there are multiple principals and agents in a chain of interactions
within the performance auditing process (see Waterman & Meier, 1998, for more).
Like other interaction processes (e.g., interactions between elected politicians and
public officials with the involvement of other parties such as state legislatures, governors,
regulated interest groups or regulated independent agencies, consumers, and citizens),
President, citizens, and media are influential participants in PAM, as demonstrated in
Figure 29 (see Gerber & Teske, 2000; Maggetti & Papadopoulos, 2018, for more). Thus,
according to my research findings, performance auditing in particular and public auditing
system, in general, are suitable political domains for the implication of dynamic and
multilevel PAM (see Gerber & Teske, 2000; Maggetti & Papadopoulos, 2018; Waterman
& Meier, 1998, for more).
Note: Developed based on Barzelay (2001), Reichborn-Kjennerud (2013a), and Waterman and Meier (1998).
Coupled with information asymmetry, conflicting goals, and involvement of
multiple principals and agents, dynamics (i.e., changes in access to information) and
accountability are other aspects of PAM (Gerber & Teske, 2000; Maggetti &
Papadopoulos, 2018; Waterman & Meier, 1998). It explains the experience of managers
on their constructive cooperation with public auditors in implementing the President’s
order (MC10, personal communication, February 11, 2020; ML12, personal
communication, February 15, 2020) because the same level of access to information leads
to constructive cooperation even within conflicting goals and interests. Accountability
Figure 29.
Updates on PAM based on the research findings
PRINCIPALS
AGENTS
AGENTS
President
Citizens
Media
PRINCIPALS
parliamentarians
managers of auditees
performance auditors
principles incorporated into PAM emerged in stories of all participants, and most of the
cases were associated with the experience of parliamentarians who argued that
governments should be accountable for results in implementing programs and auditors
should be accountable for results of their performance audits.
A central point of interactions between principals and agents in the context of
NPM is the agreed objective on improving public sector performance that is both
achievable and provable and subject to accountability (Barzelay, 2001; Hood, 1991, 1995;
Kapucu, 2006). My research findings are also in line with these considerations. The lived
experiences of all participants demonstrated that there are no significant differences in
perceptions related to achieving better (desired) results in terms of implementing budget
processes and policy programs through performance audits. Therefore, the combination
of NPM and PAM provided deep insights on perceptions of participants related to
performance audits’ impacts depending on their [participants] roles, mandates, and
responsibilities of their organizations (Themes 211, 212, 213, 221, and 222). Last but not
least, features of interactions between participants of the performance auditing process
and other users of performance audits, e.g., variations in levels of their influence, explain
macrolevel impacts of performance audits in Kazakhstan (Themes 231 and 232).
Limitations of the Study
My research findings on the causes of highly demanded performance audits and
perceived impacts of these audits on public administration in Kazakhstan are results of
analyzing the data of 14 phenomenological interviews and documentary reviews. Thence,
there are some limitations on using the results of my study to other target groups (e.g.,
internal auditors of government entities or managers of quasi-public companies) or
general populations (e.g., users of public services or leaders of non-governmental
organizations) due to sample size.
Exploring the perceptions of managers was limited to two areas of public
administration, i.e., public finance and healthcare, whereas perceptions of managers from
other areas (e.g., agriculture, education, transport, and information technology) may differ
from perceptions of managers who work in ministries and departments for finance and
healthcare. I interviewed auditors, parliamentarians, and managers of auditees, as the key
participants of performance audits conducted at both central and local government levels.
However, the lived experience and perceptions of other those involved in auditing from
other government and public sectors were not studied. Despite recruiting participants who
are highly experienced in performance auditing and who openly and impartially
expressed their position, it would be interesting and useful to study the lived experience
of other participants of performance auditing who do not represent rich-cases. As it was
revealed, besides the key participants of performance audits, the President’s policy, needs
of citizens, and interests of media create the demand for performance audits in
Kazakhstan. Since the roles of the President, citizens, and media significantly differ from
the roles of auditors, managers, and parliamentarians, exploring their lived experience
and perceptions may add insights to the studied phenomena.
Localization is a limitation of the study as well. The local level perspectives were
studied based on the case of Nur-Sultan city. Despite functioning within similar political
domains, there are some regional features and traditions in public administration and
performance auditing, in particular, that may influence the perceptions and lived
experiences of auditors, parliamentarians, and managers who live and work in the other
16 regions of Kazakhstan.
As described in Chapters 1 and 2, in my research, I focused on performance audits
conducted in 2016–2019; for documentary review purposes, I extended that timeframe.
So, like sample size, scope, and localization, the timeframe presents one more limitation
of the study, since (a) despite to changes in audit approaches, performance auditing
practice started in Kazakhstan in 2002, and (b) performance auditing practice continued
during and after interviews affected by the normal course of business and unforeseen
circumstances.
I used group characteristic and single sufficient case sampling methods and other
strategies, as it was assumed at the initial stage of my study and described in Chapter 3, to
minimize risks associated with the listed limitations, i.e., gathering incorrect and biased
information. I managed all listed limitations and risks of both researcher and participants’
biases, and they did not impact the research findings’ reliability. Limitations associated
with small sample size, scope, location, and timeframe are manageable through
recommendations provided in the next section of this chapter.
Recommendations and Implications
Theoretical Implications
My study is the first research attempt on exploring factors of demanding
performance audits in Kazakhstan and the impacts of these audits on public
administration. Further research is still needed to obtain a more comprehensive
understanding of phenomena under study addressing the described study limitations in
hopes to further expand upon the findings of my research. More specifically, the research
findings related to involvement of the President, citizens, and media into a performance
auditing process, who were identified as principals in PAM, actualize exploring their
lived experiences and perceptions about performance auditing because they shape
demand for performance audits in Kazakhstan. It is also advisable to expand the target
population by adding other principals and agents. Studying the lived experience of
auditors, managers of auditees, and parliamentarians from other regions in Kazakhstan
may add a deeper understanding of factors of demand for and impact of performance
audits, taking into account regional specifics. Studying the lived experience and
perceptions of managers from other areas of public administration, i.e., other than public
finance and healthcare that were covered by my research, may obtain additional
information, taking into account sectoral specifics.
As described in Chapter 2, in addition to external auditors, internal auditors are
also empowered to undertake performance audits in the public and quasi-public sectors in
Kazakhstan (ISLARK, 2015). Including internal auditors as participants of future studies
may shed additional light on my research findings from a relatively passive position of
managers of auditees in terms of requesting or initiating performance audits. In particular,
explaining their opinions about no real need for a request for external performance audits,
managers of auditees referred to mandates of internal auditors of ministries and local
executives on conduction of performance audits.
In the lived experience of parliamentarians, i.e., elected public officials, and
managers of auditees, i.e., implementers of public officials’ policy, the needs and interests
of citizens emerged as factors of demanded performance audits in fewer cases opposite to
the lived experience of auditors. To a certain degree, it is a contradictory (resonant) fact,
given that the NPM approach on achieving results associated with citizens’ satisfaction on
the quality of public services, is actively declared and implemented in Kazakhstan.
Therefore, exploring the perceptions of both parliamentarians and managers of auditees
about a responsive approach to performance audits (i.e., initiating and undertaking
performance audits in the interests of citizens) will provide additional information in
relation to research and practice gaps between perceptions of parliamentarians and
managers of auditees and perceptions of auditors.
My research findings also may be used to identify to what extent demand for
performance audits and their impacts on public administration are interrelated. The
generated themes and family categories related to factors of demand for and impacts of
performance audits may be used as variables to test hypotheses within future quantitative
studies, e.g., to what extent increased demand for performance audits contribute to
positive (or negative) impacts on public administration. Or, vice versa, to what extent
positive (or negative) impacts of performance audits on public administration lead to
changes in demand for performance audits.
Further, based on my literature review results, I concluded that personal impacts
of performance audits are understudied, especially impacts on auditors and
parliamentarians. My study resulted in conclusions with both negative and positive
impacts that performance audits have on auditors’ perspectives, according to perceptions
of auditors, as well as managers of auditees and parliamentarians. It is unknown how
performance audits in both developed and other developing countries impact auditors and
parliamentarians from the perceptions of all participants of performance audits. Thus,
new studies on exploring perceptions of users of performance audits about personal
influences of performance audits will provide additional information and learning
opportunities related to microlevel impacts of these audits.
As described in the previous section, I also identified perspective areas for
scholarly works on exploring meso- and macro-impacts of performance audits. While
negative perceptions of my study participants about performance audits’ impact on
auditees were related to using financial controls and sanctions by audit organizations in
the course of their performance audits, it will be useful to explore to what extent these
controls and measures contribute to changes in audited organizations and whether these
changes are positive in terms of achieving better organizational performance.
Additionally, it will be beneficial to explore impacts that performance audits have on
legislative bodies and their organizational performance. At macrolevel, it will be useful to
explore whether performance audits contribute to the political and democratic processes
in Kazakhstan, using my research instrumentations and findings (e.g., exploring the
perceptions of influential users of performance audits).
Finally, both quantitative and mixed methods may be used to measure the impact
of performance audits on public administration in Kazakhstan. Specifically, it is possible
to quantitatively measure the impact of performance audits that they have at micro-,
meso-, and macrolevels in Kazakhstan, applying research instrumentation of Morin
(2008, 2014) tested in the cases of developed (Desmedt et al., 2017) and developing
(Alwardat & Basheikh, 2017) countries, or by a modification of that instrumentation
though incorporating my research findings. Mixed method research approaches provide
added insights through data triangulation as a foundation for increasing study validity.
Public Policy Implication
Performance auditing is one of the prioritized directions of public policy in
Kazakhstan (ISLARK, 2013); it was confirmed by studying the lived experience and
perceptions of my study participants. However, further considerations and responses from
the side of policymakers are needed specifically in identified areas for further
improvements.
Firstly, the current practice of considering and using the results of performance
audits by parliamentarians as integrated parts of annual reports of audit organizations
(i.e., once a year) reduces opportunities for receiving more benefits from performance
audits. Views of three parliamentarians (75% of interviewed representatives of a deputy
corps) on considering the performance audits results right after audits completion are
consistent with previous empirical evidence on parliamentarians’ contribution to making
performance audits more useful (Funnell, 2004a; Morin 2001, 2003, 2004, 2008, 2016;
Morin & Hazgui, 2016, Reichborn-Kjennerud & Johnsen, 2018; Reichborn-Kjennerud &
Vado, 2017).
Secondly, I revealed situational discrepancies on responding to performance audits
by auditees through exploring perceptions of auditors, managers, and parliamentarians.
All auditors and most of the parliamentarians believed that managers of auditees resist
performance auditors’ work mainly due to misunderstanding of performance audits
objectives. Whereas, all interviewed managers of auditees demonstrated right
understanding of performance audits objectives; their reactions, such as resistance or
objections to auditors’ work, were due to the facts that auditors did not change their
approaches regardless of introduction of performance audits. A key observation brought
to light in my research is a legacy stereotypical attitude, predominantly kept, and more
precisely translated to performance audits (i.e., from revisions of controllers from the
Soviet Union to newly adapted practice of performance auditors) due to auditors
defaulting their auditing practices to financial control measures. This is evidence of
ineffective or unconstructive communications in reality because the 14 (100%)
participants unanimously expressed doubts regarding the reasonableness of applying
financial control mechanisms in performance audits. Therefore, I recommend to the key
policymakers in public auditing system to make agreed political decisions on changes in
approaches to performance auditing in terms of applying sanctions as measures on
addressing deficiencies in audited areas and organizations.
Thirdly, this Kazakhstani case is at the beginning stage of the transformation of
auditors’ classical role that implies focusing on problems into an advocate role that
contributes to changes in political reforms and developments (see Morin & Hazgui, 2016;
Pierre & Licht, 2019, for more). There are research-based pieces of evidence sources that
agreed on actions of auditors (audit organizations), managers of auditees (governments
and their executive entities) in terms of identifying priorities in performance audits and
evaluation criteria ultimately lead to useful performance audits (Morin, 2003; Nath et al.,
2019). In this regard, and based on my research findings, I recommend to concerned
public administrators and, in particular, to leaderships of audit organizations to start these
tested actions, maintaining adherence to the principles of independence and excluding
any possible corruption-related risks. I believe that it will allow moving to a new level
performance audit impact – from solving local or specific problems to the desired
political transformations and further democracy developments in Kazakhstan.
Fourthly, predominantly negative views of my study participants related to
measuring the audit organizations performance by facts of identified violations or
measures on sanctioning may be used as further justifications to change performance
indicators of audit organizations. Audit organizations’ performance should be measured
by their real contributions to positive changes in public administration; generated themes
“Improved budget process” and “Changes in laws and regulations” indicated the
manifestation of such positive effects of performance audits in Kazakhstan.
In sum, all mentioned public policy implications are doable through making
amendments or updating the concept on the introduction of public auditing in Kazakhstan
adopted in 2013 (ISLARK, 2013) and not changed regardless of the results of its
implementation.
Methodological and Practical Recommendations
According to my study participants’ perceptions, introduction of performance
audits in their organizations was associated with methodological changes and increased
regulations (requirements) on audit methods. From a theoretical perspective, it is a not
new or unique finding (see Knaap, 2011; Lonsdale, 2000, for more); however, it requires
both methodological and practical changes. I recommend to the key decision-makers in
the public auditing system in Kazakhstan, including methodologists, to change
approaches to methodological support of performance audits, based on my study findings.
Methodological problems emerged in the lived experience of all auditors and the
frequency of associated categories was 14. Whereas performance audits are perceived as
an opportunity for professional development by three auditors and the frequency of
associated categories was only five. So, I propose to be guided by the following
consideration – opposite to strictly regulated methods (in all interviewed auditors’
opinion, it takes place), professional development of performance auditors will facilitate
the initiation and conduction of better and more useful performance audits. Performance
audits should not be perceived as a daunting methodological task. It should be perceived
as an opportunity to develop professionally to provide users with competent and reliable
conclusions regarding the economy, efficiency, and effectiveness of an auditee or audited
area of public administration.
Next, perceptions on negative impacts of performance audits, such as
demotivation and staff-turnover caused by unfair evaluation of performance auditors’
works, are also required changes in methods and practices. Evaluation of auditors’
performance should be incorporated into practical objectives of performance audits. A
main objective should be that performance evaluations motivate auditors to achieve
results that are valuable in the context of an ideology of performance audits, for example,
facilitating public hearings on important social issues instead of writing more
administrative violation protocols.
Despite to empirical evidence about auditors’ capacities to conduct useful
performance audits (Loke et al., 2016; Morin & Hazgui, 2016), performance auditing
implies attraction of highly qualified experts (Loke et al., 2016; Mathur, 2018), for
example, public debt management experts or health experts. This objective need is due to
the fact that it is highly improbable to have auditors with sufficient level of qualifications
in dissimilar areas of public administration that are subject to performance audits.
However, only one auditor referred to attracting experts as a necessary addition to
auditing processes, but emphasized that frequent involvement of experts is not possible
due to resource constraints. At the same time, positive perceptions of managers of
auditees of performance audits were explained by cases of conduction of performance
audits by audit teams consisted of experts.
Positive Social Implications
As evidenced by my research, the introduction and expanding performance audits
in the public sector of Kazakhstan is mainly due to intentions on achieving positive
changes. More specifically, it was the intention to solve problems, minimize risks of
ineffective actions and budget spending, as well as ensuring accountability in terms of
using public funds, and implementing government programs that drove performance audit
adoption. Performance audits are highly demanded not only due to auditors’ preferences,
needs of parliamentarians, and auditees’ behaviors, but also due to the needs of public
administration with and through the interests of the President, citizens, and media. These
key findings of my research work are perceptive from positive social change
implications.
The positive social change implications include raising awareness about who and
what demand performance audits, and how. Public politicians may use my research-based
data to assess reasoning for highly requested performance audits in Kazakhstan in making
decisions in the area of public auditing (e.g., legitimize responsive approach to
performance audits), as well as in general public administration (e.g., enhance
parliamentary oversight powers through integrated performance audits).
Shared deep insights about the key motivators and initiators of performance audits
in Kazakhstan and identified differences by compared countries (see Nath et al., 2019;
Parker et al., 2019, for more) may help in planning needed performance audits; identified
problems that triggered performance audits may help in planning and undertaking useful
performance audits. Newly added knowledge about performance audits’ macro-impacts
will contribute to the development of more effective strategies and tactics on public
budgeting, modernizing laws and regulations. I also hope that the results of my study will
inform public policy leaders about existing reserves for expanding a spectrum for
performance audits’ positive influences at the macrolevel through using these audits as a
tool for improvements in the interests of citizens and improving their well-being.
My research findings also might be applied by public administrations to ensure
and enhance the positive impacts of performance audits at organizational and personal
levels. The leadership of audit organizations and auditees, as well as executives and
legislatures, may use the study findings related to impacting performance audits on public
sector organizations’ performance to identify future effective and efficient strategies on
organizational developments. In particular, through learning the perceptions of auditors
and managers, I identified that performance audits lead to challenges in human resource
management in both audit organizations and auditees (i.e., staff-turnover and
demotivation). Therefore, the results of my study might be used to improve policies on
human resource management through better understanding the personal impacts of
performance audits and maintaining a proper level of employee motivation.
A contribution to the development of a new profession in Kazakhstan, i.e., the
profession of performance auditor, is one more positive social implication of my study.
Identified aspects of performance auditing, including interactions between all participants
in the course of audits, may help to identify future training tactics of both audit and
educational organizations. In turn, parliamentarians and managers of auditees may use my
research findings to effectively manage their perspectives associated with their future
experiences in performance auditing.
Conclusions
The central findings of my research work are explanations for highly demanded
performance audits in Kazakhstan and the perceived impacts that these audits have on
auditors, managers of auditees, and parliamentarians, on audit and audited organizations,
and on budget process, laws, and regulations. The significance of my study was related to
the investigation of unknown reasons for demanded performance audits in Kazakhstan
and the impacts that these audits have on national public administration. That information
gap increased risks of missed opportunities for better public sector governance through
using performance audits and understanding their impacts.
The study results illustrated that highly requested performance audits in
Kazakhstan were not due to isomorphism phenomena, however, an adaptation of popular
and widely used practice, to a certain degree, explained why performance audits are in
demand in Kazakhstan. The main explanation and reason for increasing the number and
expanding the scope of performance audits are problems in public administration. It was
also revealed that performance audits had both positive and negative impacts at micro-
and mesolevel, and positive impact at macrolevel in Kazakhstan. These findings are
consistent, partially consistent, or inconsistent with previous studies’ findings associated
with the case of developed and developing countries. The inconsistencies or, more
precisely, differences in reasons for and impacts of performance audits explained by
public administration styles, i.e., western and emerged after the breakup of the Soviet
Union, and maturity of performance auditing practice, i.e., active use during more than 50
years in western countries and less than 10 years in Kazakhstan.
Triggers, initiators, and motivators of performance audits were found to be very
similar regardless of public administration styles, but ways and levels of their influence
differed. There are also significant differences in the perceived usefulness of performance
audits depending on the roles that participants play in performance auditing. Overall my
research illustrated theoretical interpretations, useful public policy change opportunities,
and positive social change implications for improved government.
