Why Are Data Glitches Continually Skewing Hawaii’s Covid-19 Numbers?
The state’s tracking system is like a complex information highway, moving data from many labs to a central reporting point. But some say there are too many traffic jams.
On Aug. 26, Francis Chan, the chief executive of the Hawaii Health Information Exchange, received troubling news.
The on the data highway that moves Covid-19 information from testing labs to the Hawaii Department of Health, and eventually to the public.
Batches of Covid-19 test data, Chan learned, had gotten held up on the info path from one of the testing labs. As a result, the numbers that had been reported to the public over more than a week were wrong.
And when the health department did report the backlogged cases on Sunday, the numbers appeared to show an alarming spike in cases — some 1,678 total.
The case count was alarming enough that it prompted Hawaii鈥檚 Covid-19 liaison, Lt. Gov. Josh Green, to spring into action to figure out what was going on. He was soon explaining to a concerned public and the media that some 600 to 700 cases were part of the backlog and that the situation was not nearly as bad as it seemed.
Although the biggest data glitch, it wasn鈥檛 the first or last: in fact, another just happened on Wednesday and Thursday. The Department of Health reported 455 cases on Wednesday but said that didn鈥檛 include everything. And Thursday, the count indeed jumped more than 100% to 1,068.
Such fluctuations, Green says, pose problems for government leaders and others trying to make real time decisions during a crisis.
鈥淭here are some days when we feel we鈥檙e flying blind on a rocket cycle,鈥 he said.
Are Data Glitches Unavoidable?
Such glitches underscore a sobering reality: the state鈥檚 system for collecting data on positive cases is being stressed from the volume of information staff is handling, and epidemiologists and information technology specialists are having to adapt on the fly.
It鈥檚 not just the large quantity of test data posing a challenge, it鈥檚 also that the data is coming from a staggering number of sources, not all of which are reporting the info in exactly the same way.
Before Covid-19, the health department received test results mainly from a handful of big labs, including Diagnostic Laboratory Services, Clinical Labs of Hawaii and some large medical centers, said Jonathan R. Johnson, Electronic Laboratory Capacity IT Specialist with the Hawaii Department of Health. And most of those flowed through the health information exchange.
Now there are upward of 100 labs reporting results, Johnson said.聽These include small labs and test sites in Hawaii as well as national players, like out-of-state labs and pharmacies that are part of Hawaii鈥檚 Safe Travels program.
The result is that every one of these labs has to be set up to send information to the health department, and if there鈥檚 a problem it can take significant time and effort to fix it 鈥 even if that particular lab reports a relatively small percentage of the state鈥檚 cases.
鈥淚t鈥檚 not that it鈥檚 less work,鈥 Johnson said. 鈥淚t鈥檚 the same amount of work鈥 fixing a small lab鈥檚 problem as it is fixing one arising from a bigger lab.
In theory, sending Covid test data here and there should be easy and seamless. That鈥檚 because it鈥檚 all supposed to be sent using a standard computer language known as , or HL7. But one challenge is not every system works immediately with HL7.
鈥淵ou don鈥檛 just turn it on and it works,鈥 said Dan Hall, chief information officer for Clinical Labs. Instead, he said, IT professionals need to integrate it into existing systems for the various labs.
What鈥檚 more, Chan said, HL7 has different rules for different data elements, and even a small data entry error can cause a glitch.
鈥淓ven the formatting of, say, a date — once in a while for whatever reason, the date becomes invalid, either because of a data entry error or whatever,鈥 he said.
The result, he said, can be the information highway equivalent of a traffic accident during rush hour.
鈥淚t could hold up everything,鈥 he said.
Chan wouldn鈥檛 say what caused the system backup leading to the Aug. 29 glitch. Neither would health department officials or Hall.
But Chan did provide a schematic diagram to show where in the information pipeline the Aug. 29 glitch happened.
The diagram indicates information from a lab got stuck on its way to another organization, a company called Datahouse, which the schematic calls an aggregator. Datahouse executives did not return a call for comment.
A schematic diagram created by the Hawaii Health Information Exchange shows how data flows from testing labs to the Department of Health. The item marked “B” shows where in the information pipeline a problem occurred that caused a major reporting problem on Aug. 29.
Even with automated systems, Chan said, mistakes can happen.
鈥淪everal hundred things can potentially go wrong with every message,鈥 he said.
And as much information as the health exchange has to manage, Chan said, it is relatively easy because it still comes from a relatively small pool of labs.
鈥淲hat we face is less stressful than what the Department of Health has to deal with,鈥 he said.
Epidemiologists: Use Seven-Day Averages To Make Decisions
When the public goes to the , it sees an array of stats showing latest case counts and seven-day averages, among other things. What the public doesn鈥檛 see is what goes into checking the integrity of the data before the agency posts it.
David Johnston is a veteran epidemiologist in charge of the health department’s data dashboard. As Johnston explains it, checking test data before reporting it is far more complex than it might seem.
For example, people who test positive often take multiple tests, and the department needs to make sure it isn鈥檛 counting the same person twice 鈥 something that can be especially difficult if, say, one test document misspells the person鈥檚 name.
鈥淭here鈥檚 still a lot of manually touching of the data that has to go on,鈥 he said.
All of this might help explain the risks and challenges associated with what might seem like a simple task of reporting positive Covid-19 tests in close to real time. But it doesn鈥檛 explain what government officials and the public should do when looking at the numbers.
Joshua Quint, a health department epidemiologist, offered two pieces of advice. One, he said, is for lay people to avoid the temptation of using public health data to try to project or predict where the virus is going 鈥 a task that鈥檚 vexing even for people with advanced public health degrees.
And, he says, look at seven-day averages instead of data from just a day or two. The averages will smooth out the variations that at times swing wildly.
Green said he appreciates that perspective, but he urged the professionals collecting and reporting data to do better. Seemingly flawed or incomplete data from public health institutions, he said, can fuel conspiracy theorists trying to undermine the credibility of the institutions, he said.
In addition, elected officials often must quickly respond to the concerns of constituents, and it can be hard to get the public to ignore alarming daily numbers and focus on seven-day trends.
鈥淚 appreciate what they鈥檙e saying, but human nature doesn鈥檛 work that way,鈥 he said.
