A picture is beginning to emerge of what went wrong during Australia’s Census when denial of service (DDOS) attacks on the evening of August 9 paralysed the system and meant thousands of Australians could not complete their forms online for almost two days.

The Senate Economics Reference Committee hearing today (Tuesday) gave Australian Bureau of Statistics (ABS) top bods a thorough grilling about the events leading up to and on Census night, including how the system was tracked, remedial action taken, communication with the public and reports to Minister Michael McCormack.

The hearing, which is part of the Committee’s inquiry into the 2016 Census, also provided some insight into the likely content of the report written by the Prime Minister’s Special Adviser on Cyber Security, Alistair McGibbon.

Mr McGibbon handed his report, which aims to ferret out the reasons for Censusfail and where blame should fall, to Prime Minister Malcolm Turnbull on October 14 but it has not yet been made public.

The government’s cyber security mandarin said the eCensus collapse on Census night was down to the coalescing of three factors:

• DDOS attacks
• A router which was not properly configured and did not fire back up after being switched off
• The misinterpretation of the traffic and the load monitoring system

“Any one of those wasn’t really sufficient to lead us to this Commission today. It was the three combined that led us here,” Mr McGibbon said, although he added that the third factor was the most significant.

Questions were also raised about the procurement process ABS followed before it engaged IBM for the eCensus contract, which was worth around $10 million. The total bill for this year’s Census was $470 million, most of this spent on wages for field staff.

The global tech giant already had a long relationship with the Bureau and the contract did not go to open tender, something that Senator Nick Xenophon said appeared to flout Commonwealth procurement rules.

But the Bureau’s Deputy Statistician Trevor Sutton denied that this was the case and said he had followed guidance from the Department of Finance, although he agreed that time pressures influenced the decision to go with IBM – a trusted partner – where the risk was seen as lower.

“If IBM had not provided value for money or met our requirements we would have gone to open tender,” Mr Sutton argued.

He said that the ABS had already gone to market in an RFEOI in February 2012 and later tested the Blaze system it already used to find out if it was scalable for the Census 2016 but neither approaches had yielded results.

Mr McGibbon said, “I believe that there was an element of vendor lock-in. There could have been other paths that the ABS had taken but chose not to. I’ve come to some conclusions and recommendations around that.”

The content of the contract that the Bureau had with IBM also came under examination.

Mr McGibbon said that the ABS had included a clause about what should be done to prevent a DDOS in the contract with IBM but that the Bureau could done more to find out what the company had actually put in place, including what action would be taken if the attacks occurred.

“They [the ABS} could’ve gone and had more third party testing done. They may have asked more questions of IBM to proof of what they were delivering, the services they were contracted to do, absolutely. The ABS could have done more. Clearly mistakes were made.”

He is likely to recommend a third party audit of the architecture of systems, which may have found that DDOS provisions were not sufficient, be carried out for the next Census and that more testing should have been done before the system went live.

However, he backed the ABS’ decision to wait for around 40 hours before getting the system up and running again.

“The only one thing worse after four DDOS was to get the site back up and have it knocked down again. Quite rightly there was extreme concern about making sure when the site went back up it was robust enough to cope with whatever the internet would throw at it.”

ABS Chief Statistician David Kalisch admitted the Bureau had made some poor judgements and vowed to correct its mistakes before the next Census in 2021, including identifying and mitigating some of the risks and improving communications with the public, particularly around changes to the Census and also on Census night so people understood they had weeks to fill their forms in.

“We had the capability and we had the capacity for people to complete the Census on the night and the DDOS event just shouldn’t have occurred,” Mr Kalisch said.

He said federal government funding of $257 million would help fund the changes, including examining the Bureau’s internal capacity to carry out the next eCensus.

“We expect to have five clear years to plan and implement a very successful 2021 Census.”

Field staff payments have been paid, says ABS

Mr Kalisch addressed media reports alleging that Census field staff had not yet been paid saying that elaborations and assumptions had been made and people should not “believe everything that you read in the papers.”

“There is a much smaller number and it’s against the backdrop of most staff having been paid,” Mr Kalisch said.

The outstanding sums were mostly to cover additional hours and mileage stemming from problems with the eCensus, he said.

The Bureau said that that 4700 extra claims had been made and nearly 3000 of these had been paid up to November 3. The remainder would be paid within four to six weeks.

 Xenophon quizzes ABS over Census fines

Senator Nick Xenophon asked the ABS how many infringement notices it had issued to those people who had refused to complete the Census. He also wanted to know whether people who had completed their Census form but omitted their name and would be fined. Mr Xenophon famously promised he would not put his name on his Census form, as a protest against the Bureau keeping data for longer and linking it to other data sets.

The ABS said it had encountered 10,531 refusals and issued only 239 formal letters (notice of directions) and none of these were sent to people who had not filled in their name. This was lower than in 2011 when 13,000 people refused to complete their form. The Director of Public Prosecutions will decide whether the issue goes any further.