4.3 Findings
4.3.1 Planning for the project
Three business cases
Over time, the Ultranet project has been the subject
of three business cases, with one rejected and two endorsed by government.
None of these business cases provided a robust needs
analysis and also did not provide a sound basis for approval of the project.
They did not answer the ‘Why invest?’ question for the Ultranet.
The first business case was developed in early 2007
and was rejected by government. The second business case, submitted in July
2007, was a revised version of the first. This formed the basis of the (failed)
tender process initiated in 2007.
The third business case was presented to government in
May 2009 after the second procurement process had been initiated and almost
immediately before the Ultranet contract was awarded to the service provider in
June 2009.
This was a serious departure from the usual process of
developing a business case prior to embarking on a procurement process. The
Department of Treasury and Finance’s (DTF) Lifecycle Investment Framework
states that the development of a business case must occur before the
procurement phase.
Needs analysis
The 2007 and 2009 business cases stated a number of
business drivers for the development of the Ultranet described in Figure 4A.
Figure 4A
Identified needs for the Ultranet
2007 business case
|
2009 business case
|
Critical need for a customised learning environment, i.e.
teachers are unable to centrally source and share learning resources.
|
Same
|
Poor quality of information available to education
stakeholders, i.e. teachers and school leaders have difficulties in tracking
student progress from year to year and between schools.
|
Same
|
Failure to maximise return on previous information
and communications technologies (ICT) investments, i.e. existing
disparate school systems inhibit interoperability with other systems.
|
Same
|
Massive inefficiencies in
administration for schools, i.e. too much time is spent on time‑consuming
tasks that take the focus away from learning and teaching.
|
Not included – administration was dropped as an
Ultranet functionality in the 2009 business case.
|
Source:
Victorian Auditor-General’s Office.
The purpose of a needs analysis in a business case is
to identify the fundamental need for an investment including the problems,
issues or circumstances of the current situation that needs to be addressed.
Both business cases failed to provide sufficient
material to explain the context and extent of the identified issues. For
example, with regard to ‘massive inefficiencies in administration’, the
business case did not explain how much teachers’ time would be released through
the automation of these administrative tasks, nor did it estimate the extent to
which this could improve teachers’ productivity.
Options analysis
Three options were considered in both the 2007 and
2009 business cases.
Figure 4B
Options analysis for the Ultranet
2007 business case
|
2009 business case
|
Do nothing—allow schools to determine and purchase
their own learning software solution ($67 million estimated cost)
|
Do nothing—allow schools to determine and purchase
their own learning software solution ($67 million estimated cost)
|
Further develop and deploy early version of Ultranet
trialled in several schools in 2006 ($72 million estimated cost)
|
Negotiate
contract with preferred tenderer with DEECD/contractor as integration
partners ($60.4 million estimated cost)(a)
|
Develop
and deploy an Ultranet solution ($83 million estimated cost)(a)
|
Negotiate contract/s with preferred tenderer and a third
party as systems integrator ($68.6 million estimated cost)
|
(a)
Preferred option.
Source:
Victorian Auditor-General’s Office analysis of Department of
Education and Early Childhood Development documents.
In the 2009 business case, there was little
relationship between the options presented and the needs identified for the
proposed project. Instead of presenting options on the appropriate type of
investment to address the identified needs, the options were types of
contractual arrangements that DEECD could have with the preferred tenderer.
The options presented for 2009 should have been similar
to those of the 2007 business case, albeit in a de-scoped and reconfigured
version of the concept, as per the decision in May 2008 to endorse the
reconfiguration of the Ultranet to fit the original $60.5 million announced
budget.
The various Ultranet business cases did not answer the
‘Why invest?’ question nor did they provide a sound basis for the project’s
approval.
Estimated costs do not reflect full
project costs
The 2009 business case estimated the cost of the
preferred option at $60.4 million. This is not an accurate estimate of costs
for this option.
By the time the May 2009 business case was prepared,
DEECD knew that the Ultranet contract would cost $64.6 million. This means that
in line with the business case’s figures, a more accurate estimate for the
preferred option would have been $69.6 million (i.e. given that $5 million
from DEECD’s project development costs had been incurred up to June 2009).
Further, this $69.6 million estimate does not include
the costs to be incurred by DEECD as systems integrator. Based on DEECD’s
estimates, this would be about $5 million.
This means that the estimated cost for the preferred
option in the 2009 business case should have been about $74.6 million and not
$60.4 million.
The disconnection between the options presented and
the identified needs for the Ultranet, and the failure to accurately estimate
the costs for the preferred option, show that the options analysis conducted
for the Ultranet was very poorly considered.
This, together with the weakness of the needs analysis
to provide sufficient material to explain the context and extent of identified
issues, means that the Ultranet business cases failed to provide a sound and
reliable basis for government investment in the Ultranet project.
4.3.2 Project delivery
Procurement process
Consistent with public sector practice, the Department
of Premier and Cabinet (DPC) and DTF provided advice to government at key
decision points over the life of the Ultranet project.
The project continued despite advice from central agencies
that it should cease or be delayed. There is no trail of documentary evidence
to explain whether or how DEECD addressed the many critical issues raised by
DPC and DTF.
It is difficult to understand why the Ultranet
procurement was able to proceed to contract execution, given the significant
concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD
had received from various Gateway reviews since the project first commenced.
Further, this audit detected a number of serious
process and probity issues in relation to the tendering and procurement for
Ultranet. DEECD has advised that it has commenced a number of actions and
further detailed investigations in response to these matters. In May
2008—after the failure of the first Ultranet request for tender to
receive a suitable bid from the market—DPC recommended that the Ultranet
project be stopped. DTF also expressed concerns about the future of the
Ultranet.
Subsequently, the government decided to de-scope and
redefine the Ultranet project in alignment with the originally announced $60.5
million budget. After this endorsement, the originally required number of
Ultranet functions (e.g. automatic text messages to parents when their children
are absent or late) was reduced from 1 260 to 131.
In May 2009—just prior to the final approval of
contract execution—government received further advice from both DPC and
DTF counselling against proceeding with the Ultranet project.
For example, DTF queried whether DEECD was breaching
due process and major project protocols by seeking approval to enter into a
contract without any documentation of final positions reached on threshold
issues.
DTF also advised that external scrutiny and
reporting was warranted so that DEECD could:
-
clarify how it intended to deliver a project with a fixed budget
of $60.5 million when the preferred tenderer’s offer was $77.3 million
-
explain how, when it had concerns about the preferred tenderer’s
ability to perform the role of system integrator, it had decided to be
‘integration partners’, considering that this arrangement would increase
DEECD’s risks and responsibilities in managing and implementing the Ultranet
-
explain why it had not provided the required implementation plan
and risk management frameworks
-
make government fully aware of all Ultranet functionalities that
will not be delivered as previously publicly committed, noting the
significantly reduced project scope.
It is the responsibility of the sponsoring department
to verify the data and assumptions contained in business cases, as well as
follow up any questions from central agencies. Further, there is no trail of
documentary evidence to explain whether or how DEECD addressed the many
critical issues raised by DPC and DTF.
This external scrutiny requested by DTF did not take
place. No further advice was provided by DEECD on the issues raised before the
execution of the contract with the service provider one month later.
It is difficult to understand why the Ultranet
procurement was able to proceed to contract execution, despite these
significant concerns raised by DPC and DTF, as well as the many adverse ratings
that DEECD had received from various Gateway reviews since the project first
commenced.
This situation demonstrates a clear lack of
accountability and integrity in the checks and balances in place to scrutinise
project approvals and expenditure.
Gateway reviews
In addition to objections raised by central agencies,
the project received a number of adverse Gateway ratings. Figure 4C shows that
the Ultranet received a ‘red’ rating for four of the five Gateway reviews
conducted during the planning phase.
Figure 4C
Gateway reviews during the planning phase
Date
|
Phase
|
Rating
|
Outcome
|
May 2007
|
Gateway 2 – business case review
|
RED
(critical and urgent, action on recommendations should be
taken immediately)
|
In all instances, the project proceeded to the next phase
with little evidence that issues identified were rectified and recommended
actions were implemented.
|
May 2007
|
Gateway 3 – procurement strategy review
|
RED
|
Apr 2008
|
Gateway 4 – contract decision review
|
RED
|
Oct 2008
|
Gateway 3 – procurement strategy review (second)
|
AMBER
(critical and not urgent, actions on recommendations
should be carried out before further key decisions are taken)
|
Feb 2009
|
Gateway 4 – contract decision review (second)
|
RED
|
Source:
Victorian Auditor-General’s Office analysis of Department of
Education and Early Childhood Development documents.
The DTF-commissioned Gateway Review Team expressed
concern that there was a high risk that the Ultranet was not going to be
achieved within budget because the $60.5 million project budget had not been
informed by a full scoping exercise.
In several instances, the review team also noted that
it was unable to perform a comprehensive assessment because the necessary
documentation was not complete and had not been provided prior to the review.
DEECD did not adequately address more than 60 per
cent of the recommendations in the Gateway reviews for the planning phase. For
example, the following recommendations were not actioned when developing the
revised 2007 business case:
-
clarify the value of expected efficiencies and/or cost reduction
benefits
-
clearly define the educational outcomes which the project aims to
deliver, and its effect in schools
-
update the budget, review the training and change management
required, and more effectively capture the costs of the change program.
Appropriate action from DEECD on these recommendations
may have led to a better planned and implemented Ultranet project.
Process and probity issues in the
evaluation of tender bids
DEECD’s evaluation of tender bids for the second
request for tender in 2009 raises some serious questions about DEECD’s decision
to award the delivery of the Ultranet to the preferred tenderer.
In February 2009, the Ultranet Project Board suspended
the contracted service provider as a bidder (due to what DEECD described as
untruthfulness in some tender responses) until just before the final stage of
the evaluation process.
DEECD could not explain how its tender evaluation
process could have selected a sole short-listed tenderer that had been
previously suspended from the process.
Tender evaluation documents repeatedly note DEECD’s
doubts about the ability of the service provider to deliver the Ultranet. Serious
concerns were also raised about its lack of industry experience as a systems
integrator and capability to deliver a project of the size and scale of the
Ultranet. The proposal also did not follow the preferred prime contractor
arrangement requested in the request for tender.
Coaches and support for Ultranet
implementation
While DEECD has provided a significant level of
support to teaching staff to adopt and integrate learning technologies in
teaching practice, DEECD has not provided that support for long enough.
The Ultranet coaching program was poorly timed in that
it commenced too early, i.e. two years before the Ultranet started being
deployed in schools. Significant technical issues and a delay in delivering the
Ultranet until Term 3 of 2010 meant that teachers were unable to make maximum
use of the help provided by the coaches when it was most required. While
teachers found the training useful, its timing was ineffective because they
were unable to apply the learning within a live system.
DEECD did not understand that teachers would require
regular and sustained coaching and training over a period to properly adopt and
integrate learning technologies. This could have been achieved in a number of
different ways, such as by providing sufficient resources to support the lead
teacher program on an ongoing basis.
DEECD implemented a lead teacher program so that at
least two trained teachers were available at each government school to provide
on-site support. This program had mixed results, and depended on the school
leadership team’s commitment to using the Ultranet. It also depended on
teachers’ willingness to use the inherent capabilities of the VicSmart network
and the broader internet and Edumail email systems to provide other learning
tools.
On 9 August 2010, DEECD organised a pupil-free day, or
‘Big Day Out’, for teachers to trial the Ultranet. Teachers across the state
were asked to sit at their desks to participate in this live trial.
As already discussed in the Background chapter of this
report, the Ultranet system did not operate as planned, and teachers were
unable to trial the system as expected. However, teachers and principals at
some schools implemented a ‘Plan B’ activity with Ultranet coaches to enable
some learning to take place.
Teachers from schools visited as part of this audit,
as well as reviews commissioned by DEECD, have confirmed that the Big Day Out
experience significantly tarnished the reputation of the Ultranet among
teachers. Since then, many teachers have refused to try using the system again.
A DEECD-commissioned study found that even with
support and training, teachers reported being overwhelmed by the idea of using
the Ultranet in the classroom. They also felt unable to put the training
provided by Ultranet coaches into practice.
A common theme from school visits was that teachers
needed time to assimilate all the information they were receiving and decide
how to make best use of the Ultranet in the classroom environment. Teachers
also said that limited available class time for teaching lessons did not allow
enough time for system crashes, multiple logins and generally slow Ultranet
performance.
Technical support has not been sufficient. There has
been no clear process to raise technical problems and there have been significant
delays in receiving responses to issues raised.
DEECD did not consider that the change management
process from introducing the Ultranet would entail greater initial workloads
for teachers as they learned to use and integrate the Ultranet with their
teaching practice.
Even in the longer term, as teachers begin to
incorporate more aspects of the Ultranet into their teaching, they will need
more time to access relevant training programs to continually update their ICT
skills. Currently there is no provision to support teachers to do this
professional development work.
DEECD expects schools to appropriately enable parents
to access the Ultranet, including providing login assistance and training to
use the Ultranet. This responsibility also adds to teachers’ workload.
4.3.3 Benefits realisation
Transition from project to business
as usual
The Ultranet transitioned from a project to business
as usual (BAU) in April 2012.
Transition to BAU means that responsibility for the
Ultranet technical solution now rests with DEECD’s IT Division, while the
business owner and business team is the Student Learning Outcomes Division. BAU
also means that the contractor should now be solely responsible for hosting and
maintenance activities.
However, not all of the contracted functionalities for
the Ultranet were delivered prior to transition to BAU. For example, records
management, timetabling and student reporting are yet to be delivered to
schools. DEECD continues to work with the contractor to deliver these
functionalities.
The Ultranet contract ends on 30 June 2013. Unless the
contract is extended, DEECD will need to conduct an open and competitive tender
for the continued hosting and maintenance of the Ultranet.
Costs of Ultranet implementation
Based on DEECD-provided data, capital and operating
expenditure for the Ultranet had reached $97 million by June 2012. By June
2013, it is estimated that it will have cost nearly $109 million, or some $50
million (83 per cent) more than the $60.5 million budget first announced in
November 2006.
However, based on a further review of DEECD’s
financial data, we estimate that actual capital and operating expenditure was
approximately $162 million by June 2012, and by June 2013 it is likely to have
cost approximately $180 million.
Figure 4D
Variance in budgeted versus estimated costs for Ultranet
Note:
The project budget up to 2011–12 includes $60.5 million in
total estimated investment, including $21.8 million for Ultranet coaches, and
$9.7 million for ongoing funding for software and licensing fees in
2011–12. In addition, the project budget up to 2012–13 includes
$9.7 million for ongoing funding for software and licensing fees in
2012–13.
Source:
Victorian Auditor-General’s Office.
The variance between the DEECD and VAGO estimates is
explained by DEECD’s failure to record and track all costs relating to the
Ultranet. For example, DEECD has not classified all relevant Ultranet costs
(such as project team staff costs) to the Ultranet cost centre code.
Publicly available information on the Ultranet, such
as data provided by DEECD on its website, continues to refer to the Ultranet as
a $60.5 million government project. This is misleading and does not accurately
represent the full cost of the Ultranet.
Despite this significant overall expenditure, no
cost-benefit analysis has been conducted by DEECD to determine whether the
Ultranet provides value for money, or whether the same functionality could have
been delivered more cost effectively.
Functionality: promised versus
actually delivered
Although the Ultranet has been transitioned from a
project to BAU, much of the functionality promised in November 2006 has either
not been delivered or has been de-scoped from the original specifications.
Figure 4E sets out commitments from ministerial
statements and business cases versus the actual Ultranet capability that has
been deployed.
Figure 4E
Assessment of VicSmart’s expected benefits
2007 government commitment
|
Achieved
|
Victorian parents will get virtual access to their child’s
classroom 24 hours a day, seven days a week under Labor’s $60.5 million plan
to revolutionise learning.
|
No
|
The Ultranet will allow parents to log in to their child’s
classroom, check lesson plans, homework and results, attendance, and even
communicate directly with teachers via email.
|
No
|
Every classroom in a school and every school within the
state will be linked, starting in 2007, with the rollout completed within
three years.
|
Yes/No(a)
|
Students would have their own virtual work spaces with
homepages and personally tailored lesson plans accessible, via the internet,
from anywhere.
|
Yes(b)
|
The Ultranet will consolidate school administrative
functions into the one system and lighten the burden on teachers
|
No
|
Parents will be able to monitor school attendance, which
will be recorded twice a day, with automatic notification via text, email or
phone call of non‑attendance.
|
No
|
(a)
Although all schools are now linked to the Ultranet, this started happening in
2010, not 2007.
(b) While this
functionality is now available in the Ultranet, very few students have in fact
set up their personal work spaces.
Source: Victorian
Auditor-General’s Office.
Ultranet use
At present, use of the Ultranet is well below
expectations. DEECD documentation shows that, on average, there were fewer than
11 000 teacher and 52 000 student unique monthly logins for the period
from February 2011 to September 2012.
This represents only 10 per cent of students and 27 per
cent of teachers, or 4.2 per cent of the 1.5 million total expected
Ultranet users projected in both the 2007 and 2009 business cases. There is no
equivalent data on parent usage available from DEECD.
Departmental documentation suggests that real teacher
logins may actually be much lower. This is because some schools have
established the practice of having staff regularly log in, so that their
presence is centrally recorded, even if they do not undertake any activity in
the Ultranet. This means that, at best, DEECD’s data is on the high side.
DEECD data also shows that unique monthly student
logins are below 5 per cent for at least 50 per cent of the 1 587 schools
across the Victorian public school system for the period November 2010 to
August 2012. The percentage of schools whose students did not access the
Ultranet at all has also increased from 15 per cent in 2010 to
21 per cent in 2012.
While significant online materials and training
support has been provided to teachers, these have been neither sufficient nor
sustained. DEECD has not fully considered the demand the Ultranet would have on
teachers’ time and workload, particularly in terms of teachers’ roles in
supporting and guiding parents on how to use the Ultranet.
System problems have also discouraged uptake across
the government school system. These include the Ultranet’s technical
unreliability and poor performance, the lack of appropriate support to develop
principals’ and teachers’ ICT competency and confidence, and ageing ICT
infrastructure and devices (which can affect online access to the Ultranet).
Use of third-party non-Ultranet
software in schools
Prior to the development of the Ultranet concept, many
government schools had purchased and were using third-party ICT software,
applications and platforms, such as Moodle and Compass, for classroom learning.
This practice has continued after the rollout of the
Ultranet, with DEECD data confirming that the number of schools purchasing and
using these third-party applications has remained at about the same level as
before the introduction of the Ultranet.
While both the 2007 and 2009 business cases were
underpinned by the assumption that all schools would use the Ultranet, DEECD
has advised that government schools are not mandated to use the Ultranet. DEECD
asserts that this position reflects the government’s policy of school autonomy.
DEECD has also decided to allow schools to have
greater control over their Ultranet deployment schedules in response to strong
feedback from schools.
In addition, the teachers’ union informed DEECD that
it had resolved in March 2010 that schools should be allowed to determine when
and to what extent they should use the Ultranet, because the significant issues
encountered by teachers when using the Ultranet. The union has also recommended
to its members various work bans in relation to the Ultranet as part of its
current pay and conditions campaign.
Stakeholder consultations and analysis of
documentation confirm that the Ultranet was intended to be used by all
government schools. The change in DEECD’s position, allowing schools to opt in
or opt out as they please, means that it is highly unlikely that the Ultranet’s
objective of educational collaboration and providing transferring students a
seamless transition from school to school will be achieved.
The achievement of these objectives relies on all
schools across the government system using the Ultranet, either by mandating
its use or putting significant effort into assuring that the system’s
functionality, implementation and issues management are given appropriate
consideration to maximise its use.
The November 2011 speech by the Minister for Education
Victoria as a Learning Community
refers to the need to complete the functionality and assuring the quality of
the Ultranet in order to encourage its widespread use.
DEECD needs to give greater consideration and
attention to the current ambiguity in regards to the Ultranet opt-out issue.
Without near-universal take-up by teachers and students—as forecast in
all the business cases—most of the benefits will not be achievable, and
the significant investment of taxpayers’ funds to date will therefore be
wasted.
Recommendations
The Department of Education and Early Childhood
Development should:
- urgently review its investment in the Ultranet, with a particular
focus on:
-
assessing whether the contractor has delivered
all functionality as required by the contract and what action, if any, needs to
be taken to enforce the state’s rights
-
rigorously assessing its financial management
practices and identifying the real, current cost of the Ultranet to determine
the extent to which further investment is warranted
-
identifying and addressing the underlying causes
of low take-up rates across the school system by teachers, students and parents
-
providing advice to government on the
cost-benefit of decommissioning the system now against continuing to fund and
rectify the system so that it can be implemented as originally expected
-
conduct an agency-wide review of its internal tendering, probity
and financial management practices in light of the serious issues identified by
this audit
-
expedite the provision of guidance to schools on the current
status of the Ultranet as the department’s key learning technology investment,
and clarify the policy context of schools’ autonomy in purchasing non-Ultranet
learning technologies.
