Please note: The pdf of this article is available on the CPMA Member Portal for members. The illustrations and figures are all available in the pdf.
by Stephen Jenner, Director FCMA, MBA --- Director, Criminal Justice IT (CJIT)
Context
The CJS IT approach to Portfolio Management won the 2007 UK Civil Service Financial Management award, is the subject of a case study by Gartner, and has been commended by Booz Allen Hamilton and is cited as best practice in reports to the OECD and European Commission. This article outlines the approach.
Introduction
Organisations invest in IT projects and programmes to deliver, or more usually to enable, benefits in terms of regulatory compliance, cost savings, increased revenue and/or performance improvements. This applies as much in the public as it does in the private sector - for example, the Office for Government Commerce[1] state, "The fundamental reason for beginning a programme is to realise the benefits through change" and the Cabinet Office[2] agree, "It is only possible to be sure that change has worked if we can measure the delivery of the benefits it is supposed to bring."
Yet the track record in government is not good - the Public Accounts Committee for example, concluded in 1999 that[3]:
"...for more than two decades, implementing IT systems successfully has proved difficult...has resulted in delay, confusion and inconvenience to the citizen and, in many cases, poor value for money to the taxpayer."
High profile failures in recent years indicate that the situation has not improved significantly and the issue is not restricted to the public sector or to the UK - KPMG for example, report in their 2005 Global IT Project Management Survey that, "project success appears to equate to achieving an acceptable level of failure or minimizing lost benefits." The issue was summed up most notably by the Nobel prize winning economist, Robert Solow[4], who commented, "You can see computers everywhere but in the productivity statistics."
This is however hardly surprising when as Gartner[5] report, "Business cases are... generally viewed only as documents for gaining funding. Once approved they are put away....few track the business benefits the projects actually achieve."
By 2004, the Criminal Justice System IT programme (a £2 billion investment between 2003 and 2008) faced a familiar situation - costs were escalating, delivery was delayed and benefits were falling significantly. Criminal Justice IT's response[6] was to develop and implement an approach that integrates Portfolio Management techniques derived from modern finance theory, with more effective approaches to benefits realisation that embed responsibility within the business rather than with the project. This approach, which combines robust scrutiny of investments throughout the project life cycle, with one that captures all forms of value created, has borrowed extensively from best practice both here in the UK and abroad - see Figure 1.
Figure 1 An integrated, active, repeatable process
This article outlines the key principles, tools and processes that are, it is argued, applicable to major investments beyond the criminal justice and the IT contexts.
Investment Appraisal
In allocating funding and assessing performance, it is crucial that projects are appraised on a level playing field against agreed criteria. A first step is therefore to agree a set of ‘Investment Principles' against which all existing and potential projects will be appraised. The nine CJS IT Investment Principles agreed at Ministerial level are as follows:
1. All programmes and projects that are funded by the CJS IT ring-fence budget are governed by the Spending Review Settlement Conditions and the CJS IT governance arrangements.
2. We should not fund new projects that do not contribute to the CJS Vision or add value to multiple Criminal Justice Organisations (CJOs).
3. What has been started should be finished i.e. funding should continue for those projects that are on time and to budget, otherwise previous investment would be wasted.
4. New projects should not be funded unless they can demonstrate a positive return on investment and the benefits are agreed, in principle, by all parties.
5. Where possible, options selected should represent: the shortest implementation timeframe; the lowest cost and; deliver a reasonable amount of benefit to the CJS.
6. Overspends should be borne by the relevant organisation rather than the CJS IT ring-fence.
7. Continued funding will be contingent upon project performance, in terms of meeting delivery milestones and realising benefits, and will be reviewed periodically.
8. Projects will not be funded if critical issues are unresolved.
9. Projects should not be started if the assessments of their achievability and attractiveness do not meet minimum standards.
The principles of ‘Attractiveness' and Achievability' underpin the approach and reflect an application of the concepts of risk and return derived from Portfolio Theory, to the Project and Programme Management environment where:
•Ø ‘Attractiveness' is appraised in terms of the contribution the project or programme will make to strategic priorities and financial/economic return measured by it's net present value (NPV); and
•Ø ‘Achievability' is assessed as the: degree of innovation and complexity; whether there is sufficient capability and capacity to deliver the project; the adequacy of risk assessment and contingency; and stakeholder commitment.
Investment appraisals are however dependent on reliable data - as Cooper and Edgett[7] say, "The best project selection system in the world is worthless unless the data is sound." Unfortunately, experience indicates that, as one practitioner says[8], "Business cases contain untested assumptions masquerading as facts". This view is supported by eempirical research which indicates that optimism bias is a reality i.e. "There is a demonstrated, systemic, tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors...appraisers tend to overstate benefits, and underestimate timings and costs"[9].
It is therefore essential that such claims are subject to robust scrutiny. An independent Portfolio Unit has therefore been established within CJIT to review business cases and evaluate them against the agreed Investment Principles. This unit triangulates and validates cost and benefits data by:
1. Firstly, appraising the cost/benefit case against: the HM Treasury ‘Green Book' rules for economic appraisal (ensuring that costs and benefits are valued wherever feasible and are adjusted for Optimism Bias); the hurdle rates of return specified by the current Spending Review Settlement conditions; and the CJS IT Benefits Eligibility Framework (see below);
2. Secondly, ensuring that benefits are agreed, at least in principle, by the recipient agencies, strategic and efficiency planners - for example, the project's impact on each strategic target is rated as: Mission Critical, Highly Desirable, Desirable or Minimal - and these ratings are agreed with the relevant Strategic Planners; and
3. Thirdly, using the Proving ModelTM, the only OGC/APMG Gold accredited investment appraisal tool. The model is used to assess projects' attractiveness and achievability using a series of questions derived from a two year research study undertaken at Cranfield University into the causes of programme failure. This research concluded that in the overwhelming majority of cases, these factors were manifest prior to business case authorisation. In other words, failing projects tend not to have brilliant business cases and strong stakeholder commitment - and early identification of these factors has a massive payback in terms of detecting the causes of failure prior to major investment. The appraisal includes a detailed review of project documentation and stakeholder interviews. The output is a summary report - See Figure 2.
Figure 2: The Proving Model assessment
The Proving Model scores are then combined with the financial/economic analysis and benefits reviews in an Investment Appraisal report (see Figure 3). This ensures that rather than having to wade through 100 page business cases, governance bodies receive a summary focusing on the salient facts - how much is the project going to cost, what benefits are claimed, what is the contribution to strategic priorities and what degree of confidence do we have in these costs and benefits?
Figure 3: The Investment Appraisal Report
Portfolio Management and Prioritisation
Portfolio Management is defined by the OGC as[10], "a corporate, strategic level process for co-ordinating successful delivery across an organisation's entire set of programmes and projects." The objectives of the process include:
1. To obtain the highest return (in financial and/or performance terms) from available resources given an acceptable level of risk
2. To ensure balance - in terms of investment types and organisational strategies
3. To ensure funding allocations reflect business priorities
4. To reallocate funds when performance deteriorates and/or priorities change
5. To manage dependencies, constraints and minimise double counting of benefits
6. To manage Portfolio-level risk and uncertainty
7. To provide transparent reporting on performance
A key element in the process is the management of constraints - and a key factor in this regard is the third ‘A' of Affordability. Portfolio Prioritisation exercises therefore need to be undertaken taking account of not only projects' NPVs and ‘Attractiveness' and ‘Achievability' ratings, but also ‘Affordability' i.e. projects' relative rankings need to be considered in the context of available funding.
The assessments of new and existing projects are plotted on an Attractiveness & Achievability matrix to demonstrate their relative rankings (see Figure 4).
Figure 4: Portfolio Analysis - Attractiveness and Achievability
Consideration however also needs to be given to additional factors such as:
1. The existence of any mandated or regulatory requirements. Such investments are by definition non-discretionary and therefore need to be funded prior to allocations to discretionary projects.
2. Balance across the Portfolio in terms of investment in: the various parts of the Criminal Justice System; supporting the various strategic targets; and in the types of investment - infrastructure and applications.
3. The existence of interdependencies between projects. Infrastructure projects in particular, tend to score poorly in terms of financial return on investment but can enable applications that have strong strategic alignment and positive NPVs (see Figure 5).
Figure 5: Portfolio Dependency Analysis
The output of this exercise is an allocation of funds to existing and proposed investments according to the following categorisation: Invest/Commit; Hold (i.e. invest when funding becomes available); or Reject/Cull. The conclusions of the prioritisation exercise are then reported in a Delivery Plan which is subject to approval by Ministers.
This is however not just a once and for all activity - formal Portfolio Prioritisation exercises are undertaken at the start of each new Spending Review period (usually every two years) but the Delivery Plan and funding allocations are reviewed at least every six months. In this way, the CJS IT Portfolio operates a series of "Gates with Teeth" - review stages that operate at two levels:
Ø At the individual project level (where formal reviews are undertaken at Outline and Full Business Case stage and then throughout the project life cycle) so that funding allocations are: incremental; only guaranteed only to the next review; increased as confidence in outcome grows; and linked to performance.
Ø At the Portfolio level - prioritisation is re-visited on a periodic basis to:
§ Assess performance in terms of delivery against agreed milestones, costs and whether the anticipated benefits are being realised;
§ Identify and resolve any emerging issues that might prevent the Portfolio from meeting it's objectives; and
§ Ensure the Portfolio remains aligned to strategic priorities.
Active Benefits Management
The OGC estimate that 30-40% of systems to support business change deliver no benefits whatsoever[11]. Experience indicates that the reality may be worse: benefits are often poorly defined in business cases; success in many projects is defined in terms of deployment rather than usage; planning for benefits realisation is rudimentary; and little effective action is taken when benefits are not delivered. Too often benefits management is seen as an unnecessary overhead that gets in the way of delivery. Yet this attitude is itself at the heart of why so many projects fail to deliver the promised benefits.
The CJIT Portfolio Unit has addressed these issues by establishing a Value Management Office (VMO) with an approach that combines robust scrutiny and validation of claimed benefits with research and analysis to ensure that all relevant benefits are captured. The approach:
Ø applies a consistent benefits framework across the programme;
Ø focuses on benefits throughout the project lifecycle, from outline business case through to post-deployment;
Ø adopts a joined up approach to the identification and valuation of cross system benefits;
Ø embeds responsibility for benefits realisation, not only with the relevant projects, but also those in the business able to influence their realisation; and
Ø includes regular review points to ensure that, if benefits can no longer be achieved in a cost-effective manner, then appropriate action is taken and, if necessary, resources are re-directed to more deserving projects.
Independent Scrutiny and Validation - ensuring Benefits are robust and realisable
Costings are generally subject to detailed rules about what can and can't be counted but the same is not true of benefits. CJIT has consequently developed a ‘Benefits Management Framework' that encompasses:
1. A ‘Benefits Classification Framework';
2. The ‘Benefits Eligibility Framework';
3. A quarterly Benefits Integrity Check; and
4. A framework with clear accountability and responsibility for the realisation of benefits.
Each of the above will be addressed in turn.
A ‘Benefits Classification Framework' has been developed that values benefits on two dimensions: firstly, by type - benefits are classified into efficiency (cashable and non-cashable) and effectiveness (the economic value of improved performance) categories; and secondly, by recipient i.e. the organisation, group or sector that actually receives the benefit. In this way benefits are represented in a ‘Benefits Grid' where the horizontal figures are required to reconcile to the vertical totals - see Figure 6.
Figure 6: The Benefits Grid
The ‘Benefits Eligibility Framework' represents the set of detailed rules about what benefits can, and can't, be claimed and how they should be valued. As such it ensures a methodologically sound approach to measuring and valuing benefits; minimises double counting; allows a consistent approach across the Portfolio - a level playing field; and so enables comparisons over time and between projects.
Benefits claims and reports are subject to a regular ‘Benefits Integrity Check'. On a quarterly basis, updated benefits reports are prepared by each contributor project - ‘contributor' in the sense that projects contribute benefits to stakeholders (organisations, groups or individuals). One page formats have been developed to focus on the key benefit information - see Figure 7.
Figure 7: The Quarterly Contributor Project 1 Page Benefits Report
These reports are checked against the ‘Benefits Eligibility Framework' and are reconciled with benefits reports from each recipient organisation - see Figure 8.
Figure 8: The Quarterly Recipient Organisation 1 Page Benefits Report
It can be seen that whilst the focus of the Project report is on the key benefits forecast over the full life cycle, the recipient organisation report is focused on what benefits have been and will be realised in the current Spending Review (3 year) period. Where there is a difference in the reported benefits, responsibility for approving them ultimately lies with the recipient agency rather than the ‘contributor' project. This is a fundamental principle that ensures benefits claims are robust and realisable.
Efficiency benefits are also tracked through and agreed with the departmental efficiency planners.
Number 1 in the National Audit Office/Office of Government Commerce's list of Common Causes of Project Failure is: "Lack of a clear link between the project and the organisation's key strategic priorities" Performance/effectiveness benefits recorded on the Benefits reports are therefore re-cut to show the contribution the Portfolio is making to the achievement of each strategic target - and these claims are validated with the relevant Strategic Planners via Strategic Target Benefits Realisation Plans - see Figure 9
Figure 9: The Strategic Target Benefits Realisation Plan
The output of this quarterly Benefits Integrity Check is a detailed report consisting of the project, organisation and strategic target reports referred to above. The position is then summarised in a Portfolio level Benefits Scorecard to provide a one page overview of the current position on benefits forecast and realisation - see Figure 10.
Figure 10: The Benefits Scorecard
This process is supported by a governance and accountability framework within which the OCJR Operational Board (CEO level) have accepted accountability for the realisation of benefits in their organisations from the investment in CJS IT. They are supported by nominated Benefits Realisation Leads (BRLs) who are responsible for validating/signing off benefits claims and ensuring that the benefits they have agreed are actually realised - benefits not just from projects sponsored by their organisation, but from the wider CJS IT portfolio. These BRLs meet on a monthly basis as the Benefits Working Group to review progress and address emerging issues.
Ensuring all benefits are captured
International research indicates that e-Government initiatives often struggle to demonstrate a positive financial or economic return on investment. It is not that these investments do not have a return, rather that traditional financial appraisal techniques do not capture all forms of value added. This puts at risk continued investment and the realization of benefits from the investment made to date. It is therefore crucial that this missing value is recognised - whether it relates to cross system performance improvements; social and political value; or what is termed foundation value.
Cross-system Performance/Effectiveness Benefits
Realisation of benefits in a cross-organisational portfolio is problematic in that benefits to one organisation may well be dependent on business change elsewhere in the system. The Cabinet Office e-Government Unit have commented that[12], "Business cases were found to be particularly strong on the assessment of costs and benefits related to the lead department itself...However, the identification and quantification of external benefits, for example to users or other departments, were less well handled resulting in business cases that often understated the benefits, and provided an incomplete base for tracking future 3rd party benefits through to realisation."
This issue has been addressed by the development of a ‘Root Cause Model' (RCM) as a basis for agreeing, quantifying and valuing cross-system benefits. The model, which is developed from a series of practitioner workshops and is agreed by the Benefits Working Group, identifies the forecast impact of the Portfolio as a whole on the root causes of key problems in the Criminal Justice System and the contribution to Public Service Agreement (PSA) targets. As such it represents a detailed cause and effect chain from IT project through to PSA target. See extract from the RCM at Figure 11.
Figure 11: Extract from the Root Cause Model
The model is underpinned by cost and measurement data that enables the performance impacts forecast by the model to be quantified in economic terms - what are referred to as the Combined Effectiveness Impact (CEI) benefits. These benefits are recorded on the Recipient Organisation and Strategic Target Benefits Realisation reports referred to above - in this way, these benefits are subject to the same validation and tracking mechanisms as the benefits delivered from ‘silo' projects.
The model has also been found to have a number of other advantages:
• It helps identify double counting in project business cases;
• The process gains stakeholder buy in and fosters cross-system understanding; and
• It promotes discussion of how projects can impact on performance and the business changes necessary to realise those benefits
Article continued. Click here to be taken to the conclusion.
DIGG
NEWSVINE
DEL.ICIO.US
REDDIT
TECHNORATI
Print this