SLED getting $5M tech overhaul

“We discovered a lot of things the hard way,” Keel told The Associated Press. From the initial outage, “everything that could go wrong after that went wrong. Everything we tried to do — it was like playing chess — every move we made, it seemed to backfire on us.”

To prevent a similar mishap, the agency is spending more than $250,000 to update and double up on its backup power system — about $100,000 has been spent so far — as it also revamps its entire computer infrastructure using money the Legislature designated last year. About $5 million worth of purchase orders went out last week.

The debacle began with a July 5 system crash, caused by an explosion in the battery backup system that’s supposed to keep computers running until the generator kicks in. Keel said he’s still not sure why the equipment failed, but it wasn’t lightning, as the agency initially told the public. Unfortunately, he said, somebody’s offhanded guess circulated around so much, it became the official explanation.

Among the lessons learned, he said, is “making sure when we tell somebody something we’re telling them the right thing.”

With the uninterruptible power supply not working, officials worried that storms could cause another outage, so the agency shut all computers down July 11 to switch to generator power. But then the generator failed — twice. Another controlled shutdown put the system back on commercial power.

While other servers were brought back up each time, the database of DUI breath tests and videos — usually accessible online to lawyers — remained unavailable until Aug. 7. The process of recovering more than 20 terabytes of DUI data was underway when the first controlled shutdown abruptly ended it.

Emails obtained by the AP through a public records request show that SLED officials were warned that would be a problem.

“Do you feel this is really acceptable? We have had NO notice. The outage WOULD, and WILL be detrimental!” wrote Tom Guzik, CEO of IRSA Video, which built and maintains the DUI evidence system, to SLED officials less than two hours before the shutdown. The underlined sentence was part of a series of urgent emails.

Keel said it was a matter of trying to protect the entire system, which ranges from lab results to fingerprint identification to the database officers across the state access during traffic stops — what Keel called his top priority.

“That system is more critical to me to have up than anything else. If that person’s wanted for murder or just robbed a bank and we can’t get that message to that trooper, that may put him in a life or death situation,” Keel said. “Obviously, DUI videos are very important to the administration of justice. But to me, when I look globally, the most important thing to me is officer safety, and that comes with keeping our switch up.”

IRSA’s system, which includes 157 breath test machines in 116 locations, represents just one of SLED’s 185 servers, he said.

With the power so unstable, IRSA’s final restoration attempt didn’t start until Aug. 4.

While the DUI system was down, the emails show, IRSA and some SLED officials worried that breath test machines across the state would reach capacity and shut down, since they couldn’t transfer data to SLED, and evidence would be lost. But Keel said those fears were unfounded. According to an IRSA audit of the system, all archived data was recovered through tape backups, and local machines never exceeded their on-site storage.

The emails also pointed to problems with the agency’s overall computer system — issues Keel says will be solved with that $5 million overhaul. The agency also is requesting about $2.5 million in next year’s budget for additional equipment, storage and software, and to replace a fourth of the agency’s computers, putting SLED’s technology on a replacement cycle.

“I didn’t try to paint anybody a pretty picture about where we were IT-wise,” Keel said about seeking the money in the 2012-13 budget. “Our systems are antiquated. We’ve got some technology over there over 30 years old. We had servers so old, and no maintenance contracts on them, when one would go down, we’d have to rob parts off of others to try to get things back up. But we were without dollars to do anything about it.”

Even after the Legislature provided the money, he said, he couldn’t do anything with it right away, largely because he lacked the staff — including no director for information technology. When IT director Greg Meetze did come aboard last November, Keel said, his top priority was securing the system from a cyberattack. SLED recently contracted out for an infrastructure consultant to help Meetze manage the project. He started last week. The overhaul should be in full swing after the holidays, Meetze said.