Business Ethics Journal Article
sandeep1234
ContentServer.asp.pdf
JANUARY-APRIL 2020 DOI NUMBER: 10.19255/JMPM02212
JMPM Issue #22 | Vol.07 Num.04
Measuring PMO Performance – Application of the
Balanced Scorecard in a Collaborative Research
Context
Simon P Philbin
London South Bank University
Rajneet Kaur
Imperial College London, United Kingdom
Abstract: This article provides a discussion and analysis of the successful deployment by a
university-based project management office (PMO) of the balanced scorecard as a performance
measurement tool. The research study builds on a supporting literature review on the balanced
scorecard along with background material on collaborative research projects. This is followed
by a case study investigation of the design and implementation of the scorecard for a university
PMO over a 4-year period. Various managerial insights have been generated that have value to
project management professionals engaged in the roll-out of a performance measurement
system to support the management of research projects. There is a need to carefully adapt
scorecard metrics to the academic requirements regarding the management of a portfolio of
research projects. Furthermore, although data collection is necessary for the sustained use of
the scorecard to support team operations, it is also essential to consider the people or social
dimensions when utilizing the scorecard approach. The article also includes specific details on
how scorecard’s key performance indicators have been derived through distilling strategic
objectives into operational requirements.
Keywords: Project Management Office, PMO, Balanced Scorecard, Performance
Measurement, University
JAN-APR 2020 JOURNALMODERNPM.COM
2 Measuring PMO Performance …
Introduction
The PMO (project management office) is an organizational unit that is established in order to
standardize how projects are managed and to secure efficiencies through generating best
practices from the delivery of a portfolio of projects (Desouza and Evaristo, 2006). The PMO
approach to project management has been deployed in different sectors, such as construction
information systems (Ward and Daniel, 2013), the public sector (Esquierro et al., 2013), and
research administration (Wedekind and Philbin, 2018). This is because many organizations
increasingly deliver activities according to projects; indeed, projects as temporary
organizations can be viewed as a production capability, as a unit for allocating resources to
change initiatives, and as a unit for managing uncertainty (Turner and Müller, 2003). We can
also consider that projects may be delivered for clients as part of funded contracts as well as
there being internally commissioned projects, such as those associated with business and digital
transformation. While projects and project management are highly prevalent in the modern
organization, some projects continue to encounter difficulties, for instance, in the case of IT
(information technology) projects (Nelson, 2007).
The project management office provides a mechanism to allow organizations to maximize the
likelihood of project success. Furthermore, the PMO is able to develop project management
best practices through the successive delivery of multiple projects. This knowledge can be
augmented by various tools, techniques, and standard operating procedures that a PMO
develops to support the project management community within the organization (do Valle et
al., 2008). In regard to the positioning of a PMO, the Project Management Body of Knowledge
(PMBOK®) has identified three different types of PMOs, which are supportive, controlling,
and directive (PMI, 2013). Each of these types has a different level or extent of involvement in
the delivery of project management within the organization, where supportive PMOs are more
advisory in nature, controlling PMOs provide governance for projects, and directing PMOs
provide both of these functions in addition to the provision of resourcing for the project
management function. Moreover, Dai and Wells (2004) identified a set of standard features for
a PMO, which are as follows: Developing and maintaining project management standards and
methods; developing and maintaining project historical archives; providing project
administrative support, providing human resource/staffing assistance, providing project
management consulting and mentoring, and providing or arranging project management
training.
Projects are traditionally delivered according to the so-called ‘iron triangle’, i.e. delivery of the
project to meet the schedule, budget and specification criteria while achieving a required
quality level (Atkinson, 1999), but the successful functioning of a PMO will rest on many
factors. There is the ability of the PMO to secure new projects along with the various financial
metrics associated with the development and subsequent delivery of a portfolio of projects;
there are factors associated with the PMO resourcing, such as resource deployment and
training; there are process and systems related factors, and there are yet further factors
associated with the expectations of both internal and external stakeholders of the PMO. The
adoption of an integrated performance measurement system should, therefore, be able to
JAN-APR 2020 JOURNALMODERNPM.COM
3 Measuring PMO Performance …
accommodate a range of different factors, and in this context, the balanced scorecard is well
suited to this application.
Collaborative research projects offer a number of benefits, such as enabling an exchange of
knowledge between collaborators, increasing the scope and scale of data generated by research
studies, and allowing access to complementary research infrastructure (such as experimental
equipment and numerical models). However, there are also certain challenges associated with
research collaborations (Cummings and Kiesler, 2005), and these include coordination costs,
development of multidisciplinary teams, ensuring alignment of the collaborative partners, and
generating the required level of impact from the project. Moreover, if collaborative research
and technology projects are to generate potential solutions to address societal needs, across
areas such as improved healthcare solutions, mitigating the effects of climate change and new
forms of transportation, it is important that such projects can be managed to a high degree of
success. This level of performance can be viewed in terms of achievement of the project
milestones in the required timeframe and according to the quality, budget and schedule
requirements, but the performance is also predicated on the research including the necessary
creativity and scientific freedom to facilitate developments in the particular scientific or
engineering discipline. In this context, it is important to have the necessary systems and
processes to enable the effective performance measurement of research projects and especially
in the case where a portfolio of projects is managed by a PMO.
This article includes details of an investigation into the use of the balanced scorecard to support
performance measurement for a PMO focused on the delivery of a portfolio of collaborative
research projects. The structure of the article is as follows. After the introduction, there is a
literature review on the balanced scorecard, and this is followed by background material on
collaborative research projects. This is followed by the method section and the results from a
case study investigation of a university-based PMO, where the balanced scorecard has been
successfully implemented and used to support operations over a four-year period. The case
study includes a discussion of how the scorecard was designed from a strategic perspective and
how it was deployed from an operational perspective. This is accompanied by lessons learned
from the case, conclusions, and future work.
Literature review on the balanced scorecard
The balanced scorecard was originally developed to support the strategy development process
and help organizations to measure business unit performance (Kaplan and Norton, 1996). The
scorecard has four perspectives (namely financial, customer, internal business processes, and
learning and growth), which provide a balanced view of the operating performance of a
business unit along with the corresponding drivers contributing to future performance. A
scorecard is a tool that offers a number of organizational benefits, and these can be summarised
as follows: It provides a top-down perspective on an organization’s mission and overall
strategy; it captures current as well as future success and so is inherently forward-looking; it is
broad-based through integrating both internal and external perspectives; it allows organizations
JAN-APR 2020 JOURNALMODERNPM.COM
4 Measuring PMO Performance …
to focus only on the performance metrics that are the most important and have the highest
priority (Kaplan and Norton, 1998).
A core feature of the scorecard is clearly the balanced approach – it seeks to move beyond the
historical viewpoint of managing organizations through solely relying on tracking financial
performance (Kaplan and Norton, 2001). Finances are, of course, necessary to all organizations
– industrial companies need to make a profit from operations, and even charities need to
generate revenues to cover operational costs and ideally make a surplus for reinvestment into
the organization. The scorecard, therefore, links operational performance with both customer
and financial performance, while capturing metrics on innovation, employee capabilities,
technology, organizational learning, and customer success (Kaplan, 2009).
The balanced scorecard has been investigated and applied in many different organizational
situations and applications including, for instance, the healthcare sector (Zelman et al., 2003),
textile industry (Cebeci, 2009), the retail sector (Biggart et al., 2010), manufacturing
(Fernandes, 2006), banking sector (Wu, 2012) as well as government and non-profit agencies
(Niven, 2011). Despite such widespread investigation and usage of the tool to support a range
of different types of organizations, there have also been issues raised about the scorecard.
While Mooraj et al. (1999) have reported that the scorecard adds value through the provision
of both relevant and balanced information in a concise manner for management, they also
highlight that use of the scorecard is reliant on both formal and informal processes and they
suggest that further research is required to understand the cost-benefit outcome for adoption of
the scorecard.
Others argue that in the public healthcare sector while the scorecard can be adapted for strategic
performance management purposes, it does not give sufficient weight to the political aspects
and wider context for public sector organizations (Chang, 2007). It has also been pointed out
that there have been numerous iterations of the scorecard over the last 20 years and that there
is a need for an agreed taxonomy for such iterations, which would provide a comparison of the
research findings (Perkins et al., 2014). Nevertheless, the balanced scorecard continues to be
deployed in organizations to support performance measurement and advocated as a potential
enabler for strategy implementation (Atkinson, 2006).
In regard to the higher education sector, there are a number of research studies that have
examined the scope and application of the balanced scorecard (Lawrence and Sharma, 2002;
Karathanos and Karathanos, 2005). The scorecard has been implemented as a tool for
measuring the impact of industry-university collaborations (Al-Ashaab, 2011), where there is
a need to ensure adequate measurement of the outputs of research and development (R&D)
arising from collaborative research projects. The scorecard has been implemented to support
the operational management of a university institute that also included a need for management
of the university-industry interface in order to ensure the requirements of the institute funders
were accommodated (Philbin, 2011).
Pertaining to the strategic context for higher education institutions, there is a need for such
organizations to meet the stated missions by offering educational value to the students. This
often has to be achieved while management processes are improved along with adequately
JAN-APR 2020 JOURNALMODERNPM.COM
5 Measuring PMO Performance …
containing costs – moreover, universities need to effectively and efficiently use scarce
resources, including intellectual capital, funding from government and other sources as well as
people and the available time. The balanced scorecard has been described as a useful approach
to help higher education institutions to operate in such an arena (Beard, 2009).
The utility of the scorecard has been explored as a strategic management tool to support
performance evaluation to help the Taiwanese higher education sector to address the significant
challenge in regard to supply and demand (Chen et al. 2006). In India, the scorecard has been
found to provide higher education institutions with an opportunity to assemble a cascade of
measures to translate the institution’s mission of knowledge creation, sharing and utilization
into a single framework that is comprehensive, coherent, communicable and mobilizing for
institutional stakeholders (Umashankar and Dutta, 2007).
In terms of scorecard derivatives, it has been argued that the scorecard, which is a performance-
oriented approach can be combined with a process-oriented approach (such as ‘Hoshin Kanri’)
in order to create synergy and this has been examined in the context of an engineering
management graduate programme at a higher education institution (Serdar Asan and Tanyaş,
2007). More broadly a strategy formulation framework for vocational education has been
investigating through integrating the balanced scorecard with SWOT (strengths, weaknesses,
opportunities, and threats) analysis, QFD (quality function deployment) methodology and
MBNQA (Malcolm Baldrige National Quality Award) education criteria (Lee et al., 2000).
These studies point to the extended utility of the scorecard and the potential for it to be
integrated with other management support tools and processes.
In regard to the research reported herein, we have sought to recognize the merit of applying the
scorecard as part of the management infrastructure to support the strategic orientation and
operational delivery for a university-based PMO. Indeed, Kutsch et al. (2015) have used the
scorecard as an analytical lens to focus on the contribution of the project management office
through leading to an increase in project efficiency, reduction in project costs as well as
improved success rates for project delivery.
Background on collaborative research projects
In terms of the broader context for collaborative research projects, there is an increasing trend
towards research being collaborative in nature. Indeed, bibliometric research by Wuchty et al.
(2007), which was based on an analysis of 19.9 million papers and 2.1 million patents over 5
decades, demonstrated that teams increasingly dominate solo authors in the production of
knowledge (see Table 1).
JAN-APR 2020 JOURNALMODERNPM.COM
6 Measuring PMO Performance …
Table 1: Knowledge production patterns by academic field (source of data: Wuchty et al.,
2007).
Fields Nfields Increasing
team size
RTI > 1
(with self-citations)
RTI > 1
(no self-citations)
Nfields % Nfields % Nfields %
Science &
engineering
171 170 99.4 167 97.7 159 92.4
Social
sciences
54 54 100.0 54 100.0 51 94.4
Arts &
humanities
27 24 88.9 23 85.2 18 66.7
Patents 36 36 100.0 32 88.9 – –
This large-scale study included ISI (Institute for Scientific Information) Web of Science data
covering research publications from science and engineering since 1955, social sciences since
1956, arts and humanities since 1975, and the US-registered patents since 1975. In this regard,
Table 2 highlights the number (N) and percentage (%) of the subfields that exhibit larger team
sizes in the last 5 years compared to the first five years. The data was based on the relative
team impact (RTI) for a given time period and field, where RTI is the mean number of citations
obtained by team-authored work divided by the mean number of citations obtained by solo-
authored work. The data highlights that in the case of no self-citations and for science and
engineering, 99.4% of the 171 subfields have experienced increased levels of collaborative
activity, with the corresponding figures for social sciences, arts & humanities, and patents
being 100.0%, 88.9%, and 100.0%. Therefore, this data indicates that there is an increasing
trend of scientific research studies being delivered by collaborative teams.
In regard to the trend towards international collaboration, we can consider data from the United
Nations Educational, Scientific, and Cultural Organization (UNESCO, 2015). Figure 1
provides data on the percentage (%) of publications with international co-authors from 2008-
2014. The data is based on total publications across all major fields of science and is provided
for the top 12 countries based on the total number of international publications. This highlights,
for instance, that 55.9% of scientific publications from the United Kingdom (UK) involve
international co-authors, while the figures for the USA and Japan are 34.8% and 27.1%,
respectively. Although there are variations between countries, it is evident that international
research collaboration is a major feature associated with modern science and technology.
JAN-APR 2020 JOURNALMODERNPM.COM
7 Measuring PMO Performance …
Figure 1: Level of international co-authored publications per country (source of data:
UNESCO, 2015).
Focusing on the case for collaborative research projects, a number of benefits for the
researchers involved have been identified, and this includes researchers based at higher
education institutions as well as those at other organizations, such as government labs, research
institutes, and industrial companies. These benefits have been assembled based on the authors’
experience in managing collaborative research projects over the past 20 years and are
summarized in Table 2.
Table 2: Benefits for collaborative research (source: author’s experience).
Benefits for conducting collaborative research projects
• Enable researchers to work together in order to complement their skills and knowledge.
• Allow the exchange of information and knowledge between collaborators.
• Enable multidisciplinary research studies to address major technical challenges, e.g. societal, industrial, or academic challenges.
• Provide access to specialized equipment, numerical models or other research infrastructure.
• Support the expansion of the field of data available for research studies.
• Provide scope to explore joint publishing opportunities.
• Support researcher mobility between institutions.
• Help secure research funding designated for supporting collaborative projects.
While there are compelling reasons to undertake collaborative research projects, there are also
certain challenges associated with managing such projects. These challenges can be articulated
in terms of the technical, commercial, and people-based (or social) aspects of the projects.
Table 3 provides a summary of the challenges for managing collaborative research projects,
and these insights are based on the authors’ experience in this area. In the context of both the
benefits as well as challenges for managing collaborative research projects, it can be observed
that the PMO is an ideal supporting mechanism to enable the delivery of a portfolio of research
projects. Moreover, it is important to implement an effective performance measurement system
for the PMO, and the balanced scorecard is well suited to this application.
24.4
27.1
34.8
46.0
47.8
50.4
51.6
52.6
54.3
55.9
58.3
68.9
0.0 20.0 40.0 60.0 80.0
China
Japan
USA
Italy
Spain
Canada
Australia
Germany
France
United Kingdom
Netherlands
Switzerland
Percentage (%) of Publications with International Co-authors (2008-14)
Percentage (%) of publications
JAN-APR 2020 JOURNALMODERNPM.COM
8 Measuring PMO Performance …
Table 3: Challenges for collaborative research projects (source: author’s experience).
Technical Commercial People (social)
• Availability of research resources (staff, facilities,
materials).
• Maintaining research quality while meeting
project requirements
(schedule, cost, quality).
• Generating sufficient ‘impact’ over the required
timeframe.
• Inadequate planning of technical aspects of the
project.
• Ensuring alignment with industry needs (in regard to
products and services).
• Availability of funding (industry, government, or
other).
• Ensuring financial costing captures all the project
costs.
• Lack of a robust business case to support the project.
• Flexibility in commercial arrangements, e.g. for IPR
(intellectual property
rights).
• Other legal matters, especially for international
contracts.
• Managing contributions from multiple partners.
• Managing delivery across international borders,
different languages, and
cultures.
• Insufficient leadership of the project, either academic
or commercial.
• Establishing effective multidisciplinary teams.
• Maintaining regular and open communications with
partners.
Method
The method employed in the research study is based on a case study investigation (see Figure
2). The method involves three main stages, namely literature review and background, case
study investigation, and deductive reasoning. A process of reflective analysis (Schön, 1987)
was employed to ascertain how the scorecard was designed and implemented at a university-
based PMO in the United Kingdom. The method also involves a process of deductive reasoning
(Johnson-Laird et al., 2017) in order to derive the findings from the case study that are
structured according to three main areas, namely strategic perspective – design of the scorecard
(1), operational perspective – deployment of the scorecard (2), and lessons learned from the
case (3).
The authors were responsible for the direction and operational management of the PMO team,
and the scorecard was utilized over a 4-year period (2014 to 2018), and this involved production
of a scorecard report on a monthly basis and consequently 48 successive monthly reports were
generated. The scorecard reports were used to implement the strategic objectives for the PMO
and support the operational delivery of a substantial portfolio of collaborative research projects.
JAN-APR 2020 JOURNALMODERNPM.COM
9 Measuring PMO Performance …
Figure 2: Research methodology employed in the study.
Case study investigation
The PMO featured in the case study was established with a mission to provide high-quality
project management services to academic teams at the university engaged in the delivery of
collaborative projects. The projects were funded from a range of different sources, including
the European Commission’s Horizon 2020 programme (European Commission, 2019), as well
as other international organizations engaged in collaborative funding projects. The PMO team
provided a range of integrated services to support the academic units at the university, and this
included consortium management, project management, and commercial services as the
primary capabilities alongside a distinct focus on process and systems management. Figure 3
provides a schematic view of the core capabilities of the PMO.
Figure 3: Schematic view of the core capabilities of the PMO.
(1). Literature review and background
• Supporting data and information on PMO and the need for performance measurement
• Literature review carried out on the balanced scorecard
• Background material on collaborative research projects
(2). Case study investigation
• Reflective inquiry of PMO case study at a university
• Managerial insights captured on PMO structure and management processes for measuring the performance of a portfolio of research projects
(3). Deductive reasoning
• Findings from the case study organised in three main areas
• Strategic perspective – Design of the scorecard (1)
• Operational perspective – Deployment of the scorecard (2)
• Lessons learnt from the case (3)
C o
n so
rt iu
m
m a
n a
g e
m e
n t
P ro
je ct
m
a n
a g
e m
e n
t
C o
m m
e rc
ia l
se rv
ic e
s
Process and systems management
Academic value generated
PMO level coordination
JAN-APR 2020 JOURNALMODERNPM.COM
10 Measuring PMO Performance …
The PMO provided both coordination and oversight of research projects that were ultimately
driven by the need to deliver academic value across the university. Furthermore, the PMO
adopted a series of management processes that were geared towards the needs of academic
teams engaged in the delivery of collaborative research projects. The structures and processes
of the PMO team were designed according to recognized best practices for project management
to ensure that benefits were realized and project risks were properly managed (Office of
Government Commerce, 2002).
The PMO supported academic teams across the university to deliver two main types of
technical projects; these were European Commission-funded consortium projects and
academic-driven commercial projects. Both of these types of projects directly underpinned the
achievement of the university’s organizational strategy. Moreover, it was important that the
PMO’s scope of activities and projects are strongly aligned with the strategic direction of the
university. This alignment and positioning of the PMO, as well as the two main types of
projects supported, is conceptualized in the strategy diagram in Figure 4.
Figure 4: Strategic alignment of the PMO and projects supported.
The resources available to the PMO include staff and non-staff areas as part of a clearly defined
organizational structure. The PMO team included a team leader (PMO director), operations
manager, project managers, the back-office administrative team as well as other specialists,
such as contracts manager. The team worked together to enable support to be provided to the
two main types of projects and in accordance with the PMO level strategy that is closely aligned
with the university’s organizational level strategy. The team included the required levels of
experience and knowledge to ensure high-quality project management is available, and this
includes the necessary project management certification (namely the European PRINCE2
standard) held by team members. As mentioned previously, the PMO provided project
management for two types of projects, and it is useful to provide further details on these
projects (see Table 4).
University Strategy • Strategy includes research, education and knowledge exchange activities geared to meet the strategic objectives of the univer sity • PMO ensures close alignment of projects delivered to meet academic needs and according to overall strategy of university • This alignment is maintained through an integrated set of governance and reporting arrangements
U n
iv e
rs it
y
st ra
te g
y
European Commission Funded Consortium Research Projects • Leading research in strategically important areas, such as healthcare, security, sustainable energy
and integrated transport systems • Ability to attract high quality PhD students as well as supporting researcher mobility
Academic-Driven Commercial Projects • Research and technical services delivered for a range of partners • Projects based on delivery of knowledge -based services across different sectors (healthcare,
engineering, and other areas) for international partners
PMO • Supporting European Commission funded consortium research projects and academic-driven commercial projects • Professional services with significant experience of international projects as well as commercial services • Process driven approach, standard operating procedures, and supporting systems to enable project delivery
A ca
d e
m ic
p ro
je ct
s d
e li
v e
re d
b y
u n
iv e
rs it
y P
M O
S
u p
p o
rt
JAN-APR 2020 JOURNALMODERNPM.COM
11 Measuring PMO Performance …
Table 4: Features and details of project types supported by the PMO.
Project type Project features Sector/areas Project outputs PMO services
Consortium
research projects
Projects involve a
number of
consortium
partners
(universities,
research
institutes, and
companies)
working together,
typically over 4-5
years to achieve a
series of research
goals.
Projects across
various aspects of
healthcare,
including
pediatrics and
other areas (low
TRL). Also,
engineering
projects
involving the
development of
new technologies
(higher TRL).
Research outputs,
including PhDs,
graduated,
publications,
papers, etc.
The impact
generated across
societal aspects
(improved
healthcare),
industrial
development and
economic
outputs.
Provision of
project
management for
the overall
consortium,
managing project
delivery and
periodic
reporting,
communication
and knowledge
dissemination
activities.
Academic-driven
commercial
projects
Projects involve
the delivery of
technical services
for partners based
on knowledge-
driven activities,
such as analysis,
advisory services,
and testing.
Projects are
across healthcare
and engineering
sectors, typically
at a high TRL
level in most
cases.
Knowledge-
exchange with
commercial
partners, capacity
building with
international
partners,
improved
technical capacity
and knowledge
provision.
Provision of
project
management,
including
tracking
deliverables and
milestones. Also,
commercial and
financial
administration
activities.
As can be observed, the two main types of projects have different features and characteristics.
The consortium projects were research-based, with much of the project management being
focused on supporting the delivery of consortium activities across the various partners. The
projects included fundamental research at a lower TRL (technology readiness level), such as
TRL1-2, as well as technology development projects with a mid-level TRL of 3-4. Conversely,
the academic-driven commercial projects were typically at the higher TRL level of 5-6. For
further information on the use of technology readiness levels to characterize research at the
different stages of development, see the work of Mankins (2009). Also, the commercial
projects involved the delivery of various knowledge-based services, such as technical analysis,
advice, and testing. Both types of projects were supported by project management delivered by
the PMO, although the emphasis of the management support varies and was tailored according
to the needs of the project and the academic team. In the case of the consortium projects, the
project management was focused on supporting the project’s PI (principal investigator) to
deliver the research objectives across the consortium of partners. Conversely, in the case of the
commercial projects, the project management included traditional management of deliverables
and milestones combined with commercial services, including contractual and financial
administration.
Delivery of the project portfolio was based on the provision of project management across the
full project lifecycle, including supporting pre-award (proposal development) and post-award
(project delivery) activities. The delivery of high-quality support to academic teams requires
JAN-APR 2020 JOURNALMODERNPM.COM
12 Measuring PMO Performance …
close working with other professional services teams at the university as well as providing
close support to the principal investigators. This work was supported by internal
communications (including in-reach events, workshops, and other activities) as well as external
communications (including the use of social media and websites). Adoption of an integrated
communication strategy, therefore, supports the development of new projects, maintaining the
project portfolio, and helping to underpin the financial sustainability of the PMO team.
Moreover, the project portfolio was delivered through a series of tailored management
processes designed to ensure projects are properly controlled, and progress is monitored
through project and team level reports (including use of key performance indicators and the
balanced scorecard). The provision of high-quality project management (as evidenced by
academic and customer level feedback as well as achievement of project objectives) was also
dependent on access to the necessary management tools, standard operating procedures (SOPs),
and systems including appropriate ICT (information and communications technology)
infrastructure.
Discussion of the case
Strategic perspective – Design of the scorecard
The scorecard used by the PMO team was designed to enable the PMO to deliver its strategic
objective of providing high-quality proposal and project management support to academic
teams involved in the proposal writing and delivery of collaborative research projects across
the university. The PMO was responsible for ensuring that projects were delivered according
to the budget, schedule, and scope as well as quality requirements. This involved a series of
project management tasks focused on the planning, organizing, controlling and monitoring of
activities for collaborative research projects. Consequently, commercial project management
practices were required to support this process but the PMO was operating in a university (i.e.
non-profit) environment, and therefore care was needed in how commercial approaches were
adopted as a university should be focused on knowledge creation and sharing as a primary
driver and not the generation of commercial benefits.
In order to provide clarity on the strategy formulation process for the PMO and how this relates
to the four perspectives of the balanced scorecard, Figure 5 illustrates the cause-and-effect
links for the PMO team, adapted from Mooraj et al. (1999, p. 483). This diagram translates the
strategy and objectives of the PMO team into broad areas of requirements for the scorecard
according to the four specified perspectives. This approach also highlights the interconnectivity
between these broad areas and how the different perspectives are aligned to support the
achievement of strategy through the eventual achievement of financial performance by the
PMO. In order to give clarity on the scorecard metrics for the university-based PMO, Table 5
provides more detailed requirements along with definitions, according to Kaplan and Norton
(1996, p. 54).
JAN-APR 2020 JOURNALMODERNPM.COM
13 Measuring PMO Performance …
Figure 5: Cause-and-effect links for the PMO team according to the scorecard perspectives,
adapted from Mooraj et al. (1999, p. 483).
Perspective Definition Requirements for scorecard metrics
Financial “To succeed financially,
how should we appear
to our shareholders?”
• There is a need to monitor the performance of
the PMO team regarding the financial value of
proposals submitted and awarded.
• Other financial areas to be considered include
outstanding debt levels on projects as well as
the value of charges against PMO managed
projects.
Customer “To achieve our vision,
how should we appear
to our customers?”
• There is a need to monitor the level of
engagement of the PMO team with the ‘internal
customer base,’ which is the academic
community.
• Other areas include the number of new projects
that have been initiated with external customers
as well as communications and social media
activities with stakeholders.
Le a
rn in
g a
n d
g
ro w
th C
u st
o m
e r
F in
a n
ci a
l In
te rn
a l b
u si
n e
ss
p ro
ce ss
e s
Financial performance
of projects
Skills development
Project management capabilities
New projects awarded
Team sustainability
Standard operating
procedures
Social media activities
ICT systems
Project opportunities identified
Academic engagement
Communications and dissemination
events
JAN-APR 2020 JOURNALMODERNPM.COM
14 Measuring PMO Performance …
Internal
business
processes
“To satisfy our
shareholders and
customers, what
business processes must
we excel at?”
• There is a need to monitor the performance of
the PMO team regarding the development of
new processes and procedures as part of
capturing best practices for project management
and other functional areas.
• Other areas include the use of new ICT systems
by the PMO team to support efficiency
improvements.
Learning and
growth
“To achieve our vision,
how will we sustain our
ability to change and
improve?”
• There is a need to monitor the performance of
the PMO team regarding the level of training
undertaken across the whole team.
• Other areas include the need to measure the
number of external presentations given by team
members as part of their professional
development.
Table 5: Requirements for scorecard metrics, with definitions according to Kaplan and
Norton (1996, p. 54).
These requirements can be further refined to provide a specific set of key performance
indicators that were part of the scorecard. Consequently, Table 6 provides a summary of the
balanced scorecard developed to support the university PMO and the 15 KPIs across the four
perspectives. This set of KPIs was carefully assembled to reflect the relative areas of
‘performance concentration’ according to the strategy and objectives of the PMO team. It can
be observed that the scorecard includes an uneven number of KPIs across the four perspectives,
which reflects there is a greater concentration of performance in the financial perspective (6
KPIs) and customer perspective (5 KPIs) when compared to the internal business processes
perspective (2 KPIs), and learning and growth perspective (2 KPIs). However, this does not
mean that monitoring the performance of the team in these latter two perspectives is a lower
priority than the former two perspectives, but rather it is a reflection of the concentration and
number of areas that need to be measured to properly monitor and control the performance of
the team. Conversely, implementation of the scorecard for another team in a different
organizational scenario may result in a different set of KPIs with a corresponding different
level of performance concentration across the four perspectives.
Perspective KPI Description Comments
Financial 01 Value of proposals submitted to
funding bodies.
• This is the financial value (£) of
proposals submitted to funding bodies
during the month.
• The financial value (£) is tracked
against an annual target.
02 Value (consortium) of new
projects awarded.
• This is the financial value (£) of new
projects awarded (based on the
university and collaborative partner
allocations) during the month.
JAN-APR 2020 JOURNALMODERNPM.COM
15 Measuring PMO Performance …
• The financial value (£) is tracked
against an annual target.
03 Value (institutional) of new
projects awarded.
• This is the financial value (£) of new
projects awarded (based on the
allocation for the university only)
during the month.
• The financial value (£) is tracked
against an annual target.
04 Value (team) of new projects
awarded.
• This is the financial value (£) of new
projects awarded during the month.
• The financial value (£) is tracked
against an annual target.
05 Value of staff charges to PMO
managed projects.
• This is the financial value (£) of staff
charges (or bookings) made to PMO
managed projects during the month.
• The financial value (£) is tracked
against an annual target.
06 Value of past due to receivables
on PMO managed projects.
• This is the financial value (£) of
outstanding debt from customers on
PMO managed projects during the
month.
• The financial value (£) is tracked
against an annual target.
Customer 07 The number of university
academics supported by the
PMO team on proposals (pre-
award).
• This is the number of academic
members of staff (principal
investigators and co-investigators)
that are engaged in the development
of proposals and negotiation of
contracts during the month.
• The number of academics is tracked
against an annual target.
08 The number of university
academics supported by the
PMO team on projects (post-
award).
• This is the number of academic
members of staff (principal
investigators and co-investigators)
that are engaged in the delivery of
projects during the month.
• The number of academics is tracked
against an annual target.
09 The number of new projects
initiated by the PMO team.
• This is the number of new projects
that have been initiated during the
month.
• The number of projects is tracked
against an annual target.
10 The number of events and
communications activities
delivered by the PMO team to
• This is the number of events, such as
workshops and other significant
communications activities that were
JAN-APR 2020 JOURNALMODERNPM.COM
16 Measuring PMO Performance …
grow project management
engagement.
delivered by the PMO team during the
month.
• The number of events and
communications activities is tracked
against an annual target.
11 Level of social media activity to
raise the profile of PMO both
internally and externally.
• This is the cumulative number of
followers on the PMO team's social
media sites.
• No annual target is set for this KPI.
Internal
business
processes
12 The number of new PMO
processes captured,
documented, and disseminated.
• This is the number of new standard
operating procedures (SOPs) issued
by the PMO team during the month.
• The number of SOPs is tracked
against an annual target.
13 Level of ICT system used to
improve efficiency and
effectiveness across PMO
projects.
• This is a qualitative assessment in
regard to the adoption and
deployment of new ICT systems to
support PMO operations during the
month.
• No annual target is set for this KPI.
Learning
and growth
14 Level of training carried out by
team members to grow the
skills-base and intellect of the
PMO team.
• This is the number of days of training
undertaken by PMO team members
during the month.
• The number of training days is
tracked against an annual target.
15 The number of external
presentations delivered by PMO
team members at project
meetings, conferences, and
external events.
• This is the number of external
presentations delivered by PMO team
members at project meetings,
conferences, and external events
during the month.
• The number of external presentations
is tracked against an annual target.
Table 6: Summary of balanced scorecard and KPIs developed to support the university PMO.
Operational perspective – Deployment of the scorecard
In terms of the operational use of the scorecard, the data for the KPIs were gathered on a
monthly basis to summarise the performance of the team for the previous month. Data was
assembled every month through coordinating the contributions from project managers in the
PMO team and also through interrogation of a number of ICT (information and
communications technology) systems, including the university’s ERP system (enterprise
resource planning), CRM (customer relationship management) system, project management
system as well as local records held in spreadsheets. Consequently, there was a certain
administrative requirement associated with operating the scorecard. Initially, when the
JAN-APR 2020 JOURNALMODERNPM.COM
17 Measuring PMO Performance …
scorecard was adopted, an investigation was carried out over whether the scorecard KPIs could
be integrated into one of the existing corporate ICT systems with an automatic generation of
the required KPI data every month. While this would have understandably reduced the
administrative effort required, it was deemed to be excessively costly to undertake the required
systems integration work. Therefore, it was decided to operate the scorecard through a
spreadsheet template that was populated with data every month. This also had the benefit of
being readily accessible to all of the team members and still allowed all the KPIs and monthly
updates to be available on a single screenshot view.
Each month and once the scorecard had been assembled, it was forwarded to the divisional
director in order to ensure effective oversight and governance of the PMO team. The scorecard
and the performance of the team according to the specific KPIs were also displayed and
discussed at the monthly PMO team meeting. At the end of each academic year, the
performance of the team against the annual KPI targets was also identified, and this information
was shared across the team and with senior management. Additionally, the scorecard KPIs
were reviewed on an annual basis to ensure they were still aligned to the team strategy and
objectives; where required, adjustments were made both to the KPIs themselves and also to the
annual numerical targets.
Lessons learned from the case
The following lessons have been learned through considering the findings from the case study
investigation into how the balanced scorecard was deployed at a university-based PMO.
• The balanced nature of the scorecard provides an ideal mechanism to adopt a holistic
view of the work and functioning of a PMO and the projects that are delivered, and this
includes the project, process, and people considerations.
• The essential need for reporting financial metrics is highly consistent with the needs of
a PMO to gather and report on the performance of projects, but this is augmented
through also capturing KPIs from the other perspectives, i.e., customer, internal
business processes, and learning and growth.
• The utility and benefits gained from using the scorecard to guide operations and track
team performance need to be weighed against the administrative requirements for
servicing the scorecard and gathering the required data. In the case study, monthly
updates of the scorecard provided the optimal balance, but in other instances, the
frequency could be different, such as weekly or quarterly (i.e., every three months).
• In the ideal situation, it is suggested that the scorecard should integrate with the main
corporate IT systems that hold the data for the KPIs, such as the ERP, CRM, and project
management systems. This integration was not possible in the case, and hence the data
capture exercise did carry an administrative requirement. Conversely, where a ‘low
tech’ option, such as gathering scorecard KPI data in a document table or spreadsheet,
is required for a straightforward application, such an option can be readily adopted
without the need for extensive systems integration activities. In such a scenario, more
complex roll-outs of the scorecard can still be implemented in a phased approach as
JAN-APR 2020 JOURNALMODERNPM.COM
18 Measuring PMO Performance …
and when the organization has the resources available for a more complex system, or
when the circumstances of the team or organization have sufficiently changed.
• The team leader (or business unit leader) needs to be accountable for the performance
of the scorecard KPIs, although responsibility for the performance of specific KPIs or
groups of KPIs can be allocated (or delegated) to specific members of the team.
• Regular review of the scorecard and the performance of the KPIs with team members
helps to focus the attention of the team and socialize any issues associated with
underperformance in a particular area. This is important as it encourages discussion on
corrective action within the team, and any actions required to address such
underperformance can be shared across all the relevant team members. Team leaders
need to guide this process, but team members also need to properly engage with the
process and any follow-on activities. A further reflection is that the scorecard served as
a motivation tool and helped ignite an element of healthy competition between the team
members.
• Capturing data on learning and growth, such as training undertaken as well as
presentations given by team members, helps, in particular, to further motivate team
members. This is because an individual can clearly observe how their achievements and
efforts are being recognized; conversely, it can also spur other team members on to
undertake training or volunteer for an initiative or presentation that extends beyond
their current role.
• Adopting the balanced scorecard in a sustainable manner requires the commitment of
the team, and implicitly the team leader will need to drive forward both the development
and delivery of the scorecard – a continuing focus is then required if the scorecard is to
remain a part of the team’s core management processes. Where possible, the use of the
scorecard should be integrated with wider governance processes, such as those at the
divisional and corporate levels.
Conclusions
This article has reported on the findings from a case study investigation into how the balanced
scorecard was designed and deployed at a university PMO over a 4-year period. The application
of the scorecard was to enable performance measurement for a substantial portfolio of
collaborative research projects managed by a PMO team and working in conjunction with
academic teams across the university. This research study provides a supporting literature
review on the balanced scorecard along with accompanying background material on
collaborative research projects, which illustrates the context for the study and helps underpin
the need for the PMO in this project management application.
The balanced scorecard has been adopted widely over the last 20-25 years, and this has
extended across different types of organizations and applications. The scorecard continues to
be a powerful tool to help management track the performance of a team or business unit through
reporting on the progress of the team to senior management and controlling the allocation of
resources to ensure that progress is maintained. While there have been some authors that have
JAN-APR 2020 JOURNALMODERNPM.COM
19 Measuring PMO Performance …
raised certain issues to be addressed in regard to the scorecard, the continued use of the
scorecard as strategic development and operational management tool remains obvious. It is
important though to properly adapt the scorecard to the specific organizational circumstances
– this may result, for instance, is there is a higher concentration of KPIs in certain perspectives
of the scorecard, but this is an essential feature of aligning the scorecard perspectives to the
strategy for the business unit.
The design and operational use of the scorecard are very much associated with the collection
of data and information, and in the case of a PMO, there is a need for a significant amount of
data associated with project management. But in regard to the successful adoption of the
scorecard, while such data collection is, of course, necessary, it is the people or social
dimension that has a pivotal role to play in regard to the sustainable use of the scorecard over
the medium-to-long term. As a further parallel, the use of the tool to monitor the performance
of the team and subsequently control the allocation of resources across the team is a pre-
requisite use for the scorecard, but again from a ‘soft viewpoint,’ the scorecard can be equally
valuable in regard to helping to motivate team members to maintain and even increase
performance levels. The scorecard, therefore, provides benefits that extend beyond the initial
performance measurement aspects into more fundamentally important areas relevant to the
sustainability of the team or business unit.
In regard to future work, it is suggested that further longitudinal studies are undertaken on how
the balanced scorecard has been adopted in different organizational settings over extended
periods of time. This will provide greater insights into how the scorecard can be adapted for
different applications and inform future areas of research.
References
Al-Ashaab, A., Flores, M., Doultsinou, A., & Magyar, A. (2011). A balanced scorecard for measuring
the impact of industry–university collaboration. Production Planning and Control, 22(5-6), 554-570.
Atkinson, H. (2006). Strategy implementation: a role for the balanced scorecard?. Management
Decision, 44(10), 1441-1460.
Atkinson, R. (1999). Project management: cost, time and quality, two best guesses and a phenomenon,
its time to accept other success criteria. International Journal of Project Management, 17(6), 337-342.
Beard, D. F. (2009). Successful applications of the balanced scorecard in higher education. Journal of
Education for Business, 84(5), 275-282.
Biggart, T. B., Burney, L. L., Flanagan, R., & Harden, J. W. (2010). Is a balanced scorecard useful
in a competitive retail environment?. Management Accounting Quarterly, 12(1), 1.
Cebeci, U. (2009). Fuzzy AHP-based decision support system for selecting ERP systems in textile
industry by using balanced scorecard. Expert Systems with Applications, 36(5), 8900-8909.
Chang, L. C. (2007). The NHS performance assessment framework as a balanced scorecard
approach. International Journal of Public Sector Management, 20(2), 101-117.
JAN-APR 2020 JOURNALMODERNPM.COM
20 Measuring PMO Performance …
Chen, S. H., Yang, C. C., & Shiau, J. Y. (2006). The application of balanced scorecard in the
performance evaluation of higher education. The TQM Magazine, 18(2), 190-205.
Cummings, J. N., & Kiesler, S. (2005). Collaborative research across disciplinary and organizational
boundaries. Social Studies of Science, 35(5), 703-722.
Dai, C. X., & Wells, W. G. (2004). An exploration of project management office features and their
relationship to project performance. International Journal of Project Management, 22(7), 523-532.
Desouza, K. C., & Evaristo, J. R. (2006). Project management offices: A case of knowledge-based
archetypes. International Journal of Information Management, 26(5), 414-423.
do Valle, J. A. S., e Silvia, W. D. S., & Soares, C. A. P. (2008). Project management office (PMO)-
Principles in practice. AACE International Transactions, PM71.
Esquierro, J. C., Valle, A. B., Soares, C. A. P., & Vivas, D. C. (2014). Implementation of a project
management office in a public sector organization: A case study involving a sanitation
institution. International Review of Management and marketing, 4(1), 1-12.
European Commission (2019), Presentation on Horizon 2020 – The New EU Framework Programme
Research and Innovation, 2014 to 2020, website accessed on 5th January 2020:
https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/281113_Horizon%202020%20s
tandard%20presentation.pdf.
Fernandes, K. J., Raja, V., & Whalley, A. (2006). Lessons from implementing the balanced scorecard
in a small and medium size manufacturing organization. Technovation, 26(5-6), 623-634.
Johnson-Laird, P. N., Goodwin, G. P., & Khemlani, S. S. (2017). Mental models and reasoning.
In International handbook of thinking and reasoning (pp. 346-365). Routledge.
Kaplan, R. S., & Norton, D. P. (1996). Linking the balanced scorecard to strategy. California
Management Review, 39(1), 53-79.
Kaplan, R. S., & Norton, D. P. (1998). Putting the balanced scorecard to work. The Economic Impact
of Knowledge, 27(4), 315-324.
Kaplan, R. S., & Norton, D. P. (2001). Transforming the balanced scorecard from performance
measurement to strategic management: Part I. Accounting horizons, 15(1), 87-104.
Kaplan, R. S. (2009). Conceptual foundations of the balanced scorecard. Handbooks of management
accounting research, 3, 1253-1269.
Karathanos, D., & Karathanos, P. (2005). Applying the balanced scorecard to education. Journal of
Education for Business, 80(4), 222-230.
Kutsch, E., Ward, J., Hall, M., & Algar, J. (2015). The contribution of the project management office:
A balanced scorecard perspective. Information Systems Management, 32(2), 105-118.
Lawrence, S., & Sharma, U. (2002). Commodification of education and academic labour—using the
balanced scorecard in a university setting. Critical Perspectives on Accounting, 13(5-6), 661-677.
Lee, S. F., Lo, K. K., Leung, R. F., & Ko, A. S. O. (2000). Strategy formulation framework for
vocational education: integrating SWOT analysis, balanced scorecard, QFD methodology and MBNQA
education criteria. Managerial Auditing Journal, 15(8), 407-423.
JAN-APR 2020 JOURNALMODERNPM.COM
21 Measuring PMO Performance …
Mankins, J. C. (2009). Technology readiness assessments: A retrospective. Acta Astronautica, 65(9),
1216-1223.
Mooraj, S., Oyon, D., & Hostettler, D. (1999). The balanced scorecard: a necessary good or an
unnecessary evil?. European Management Journal, 17(5), 481-491.
Nelson, R. R. (2007). IT project management: Infamous failures, classic mistakes, and best
practices. MIS Quarterly Executive, 6(2), 67-78.
Niven, P. R. (2011). Balanced scorecard: Step-by-step for government and nonprofit agencies, John
Wiley & Sons.
Office of Government Commerce (2002). Managing Successful Projects with PRINCE2, The United
Kingdom Stationery Office.
Perkins, M., Grey, A., & Remmers, H. (2014). What do we really mean by “Balanced
Scorecard”?. International Journal of Productivity and Performance Management, 63(2), 148-169.
Philbin, S. P. (2011). Design and implementation of the Balanced Scorecard at a university
institute. Measuring Business Excellence, 15(3), 34-45.
Project Management Institute (2013). A Guide to the Project Management Body of Knowledge
(PMBOK®), Fifth Edition, PMI.
Schön, D.A. (1987). Educating the reflective practitioner, San Francisco: Jossey-Bass.
Serdar Asan, Ş., & Tanyaş, M. (2007). Integrating Hoshin Kanri and the balanced scorecard for
strategic management: The case of higher education. Total Quality Management, 18(9), 999-1014.
Turner, J. R., & Müller, R. (2003). On the nature of the project as a temporary
organization. International Journal of Project Management, 21(1), 1-8.
Umashankar, V., & Dutta, K. (2007). Balanced scorecards in managing higher education institutions:
an Indian perspective. International Journal of Educational Management, 21(1), 54-67.
UNESCO (2015). UNESCO Science Report: Towards 2030 by United Nations UNESCO (United
Nations Educational, Scientific and Cultural Organization). Paris: UNESCO Publishing, 820 pp.
Ward, J., & Daniel, E. M. (2013). The role of project management offices (PMOs) in IS project
success and management satisfaction. Journal of Enterprise Information Management, 26(3), 316-336.
Wedekind, G. K., & Philbin, S. P. (2018). Research and Grant Management: The Role of the Project
Management Office (PMO) in a European Research Consortium Context. Journal of Research
Administration, 49(1), 43-62.
Wu, H. Y. (2012). Constructing a strategy map for banking institutions with key performance indicators
of the balanced scorecard. Evaluation and Program Planning, 35(3), 303-320.
Wuchty, S., Jones, B. F., & Uzzi, B. (2007). The increasing dominance of teams in production of
knowledge. Science, 316(5827), 1036-1039.
Zelman, W. N., Pink, G. H., & Matthias, C. B. (2003). Use of the balanced scorecard in health
care. Journal of Health Care Finance, 29(4), 1-16.
JAN-APR 2020 JOURNALMODERNPM.COM
22 Measuring PMO Performance …
About Authors
Simon P Philbin is Professor and Director of the Nathu Puri Institute for Engineering and Enterprise at London South Bank University (LSBU) in the United Kingdom. He joined LSBU in 2018 and previously worked at Imperial College London. Prior to joining Imperial in 2003, he was at the UK Ministry of Defence. He is published across several areas, including project management, research & technology management, and chemistry. Previous academic roles include Visiting Fellow at Imperial College Business School and Visiting Fellow at Birkbeck, University of London. He holds a BSc and Ph.D. in chemistry as well as an MBA.
Rajneet Kaur is an Associate Director at Imperial College London in the United Kingdom, looking after the delivery of academic-driven commercial projects. Before joining Imperial College London in 2014, Rajneet worked at one of the Imperial's subsidiary companies, Imperial Consultants, in the capacity of HR and Office Manager from 2010. Prior to joining Imperial Consultants, Rajneet has worked in other private and public sector organizations in various HR roles. Rajneet holds a B Tech in Electronics and Communication Engineering from India, an MSc in Engineering Management from Brunel University, and an MBA from Imperial College London.
Copyright of Journal of Modern Project Management is the property of Editora Mundo and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
