Census staff have spoken out about how budget cuts and indecision paralysed preparations for the 2016 Census and contributed to the system’s embarrassing failure on Census night.

Australian Bureau of Statistics (ABS) staff have lifted the lid on events leading up to Census night on August 9 when thousands of Australians could not complete their Census forms online after denial of service attacks crashed the website.

The comments came from public servants and contractors, who were also members of the Community and Public Sector Union (CPSU), and were contained in the union’s submission to the Senate Standing Committee on Economics, which is conducting an inquiry into what went wrong during this year’s Census.

What emerges is a picture of the titanic statistics agency careening towards disaster, something its staff said was painful but predictable after work was delayed because of speculation around the Census.

Then Prime Minister Tony Abbott was exploring possibilities that the Census could be carried out every decade, instead of every five years, or even pulled completely to save money. The cash was needed to fund an urgent upgrade of the Bureau’s ICT system that was on its knees and vulnerable to attack or error.

A public backlash to the proposed changes meant the Census was retained but the CPSU argued that delays had by then irretrievably hobbled the Census by interrupting ICT work.

The Union also said that its members reported that only half the money needed to drive the ICT transformation program found its way to the ABS.

One staffer said: “The decision to try to save money by trying to cancel the Census in 2016 stopped planning for six months at a critical juncture. It was then restarted too late to ensure systems would be ready.”

Another said: “The decision to proceed was too late, resulting in extreme work pressures, programme cuts and patchwork outputs.”

The disjointed workflow meant that an under-tested product was unleashed on Census night.

“The end result was that systems testing were still more than six months behind and decisions to prioritise key issues had to be made. A lot of the systems being used are a long way short of ideal, simply because we did not have enough time to build them properly,” said one.

Another agreed the testing process had been rushed and incomplete, “We only ran tests on a couple of targeted systems and processes.”

CPSU Deputy National Secretary Melissa Donnelly said in the union’s submission:

“The preparation, administration and management of the 2016 Census was significantly affected by uncertainty in the ABS about the funding, scope and future of the Census.”

She said there were 700 fewer staff at the ABS now than when the Census was last conducted in 2011. This had meant the loss of experienced staff in key positions, increased used of casual staff and falling morale of those left behind shouldering the extra work.

One worker said they had worked at least half the weekend for months, banking more than 150 hours of flexitime.

Donnelly said: “Members spoke about the continual loss of expertise and knowledge. The constant shedding of staff has meant that corporate knowledge is not maintained, which makes it increasingly difficult to address issues when they arise.”

The Union made it clear that it did not blame ABS workers for the problems and said they displayed “a high level of professionalism under difficult circumstances.”

“Had the ABS been funded in a correct and appropriate manner, many of the problems that arose may have been avoided,” Donnelly said.

But is seems likely the Bureau will suffer further budget cuts because it is undergoing a review in November as part of the Commonwealth Contestability Programme.

Failure inevitable: Staff

The union said that a number of its members pointed to the inevitability the system would fail and highlighted the lack of a Plan B if it did.

There have also been criticisms levelled at IBM, who won the $10 million contract, and the Bureau over the lack of back-up systems or an upstream provider to keep the site accessible to the public in the event of an attack.

“I was alarmed by how ill prepared ABS was for the major incident,” said one employee. “It seemed like the approach to risk managing was to risk assess the hell out of it – “yep, it could go pear-shaped” but there did not seem to be any preparations or contingencies ready to go.”

They said there was little planning should a critical incident take place, “we had nothing and had not been required to prepare anything or ‘practice’ any kind of responsiveness ahead of the main event. We spent a few days scrambling to pull something together.”

Another person said: “It was clear from the start of my contract that the Census would probably fail. Very few aspects of the project were free of mistakes and delays. Obstacles that were obvious to me were ignored.”

IT nightmares

The union’s submission makes it clear that “an overwhelming number” of its members are deeply concerned about the under resourcing of ICT at the Bureau with staff cuts meaning that anything except urgent work cannot get done.

One person said the Bureau’s IT problems were ‘crippling.’

“We experience constant and ongoing technical problems with all our systems. This severely affects our ability to do our work in a timely way and with high quality.”

Another union member said they could not get a response from their IT section regarding core services they should provide.

“I think they’re just run off their feet. I think many sections across the ABS have been overstretched, understaffed and under delivering for several years.”

Doing it better next time

Union members were asked what would make the process smoother next time. They named funding, better planning and reviewing the questions contained on the Census form.

Skilling up Bureau staff on ICT and obtaining better external specialist support was also mentioned with one member saying: “The ABS is drastically under-skilled in the areas of modern web-based ICT and public relations. Given this, they sought outside advice, which turned out to be insufficient but still seems like the right call. Increasing skills across these areas should be a priority.”

It will be interesting to discover how much the Census cost to run this time round.

The 2011 Census cost about $440 million but this was projected to fall by $100 million in the 2016 Census with the move to more people completing it online. The ABS target was to double the e-census return rate to 65 per cent of Australians, or 16 million.

However, it is likely that more paper forms may have been returned than originally predicted given problems accessing the system and privacy concerns about linking datasets and personal information being kept for up to four years.

Submissions to the senate inquiry on the Census closed this week and the final report is due on November 24.