ÌìÃÀÊÓƵ

Against all odds — including ever-deeper education budget cuts and the infamous Furlough Fridays — student scores on the Hawaii State Assessment rose for the fourth year in a row this year.

But one man is not celebrating.

Dave Rolf, a business owner and father of two public school graduates, voiced concerns to members of the Hawaii State Board of Education‘s committee on curriculum this month that the current state test, implemented in 2007, was manipulated during its development to make it easier to pass.

The charges that Rolf levels at the board and the Hawaii Department of Education are serious ones. Members of the public rely heavily on the state test scores to tell them what is happening in our schools, and the federal mandate uses them to determine whether schools are meeting annual progress benchmarks. Every public school’s standing under federal requirements is determined by the percentage of students who meet proficiency levels on the state’s assessment. Schools that don’t measure up face severe consequences, up to and including restructuring.

The education board on April 19, 2007 approved the new assessment, Rolf said, based on the knowledge that under its standards, a much higher percentage of students would meet proficiency levels.

“Those are some very serious allegations, and it’s irresponsible to make them without even understanding the process for developing the tests and setting the cut scores,” said Cara Tanimura, director of the Hawaii Department of Education‘s Systems Accountability Office, in an interview with Civil Beat. “The short answer is no, we do not manipulate the scores.”

The state assessment is actually more accurate today than it was five years ago, administrators say, and to support their claim that it hasn’t inflated scores they point out that student scores on the national assessment are rising in tandem with the upward trend on the state test.

(A related story explores how Hawaii standards compare nationally and internationally.)

Rolf is now the president of his own advertising and public relations firm. But he has served on one national and two state education task forces, including a 1999 task force on education reform for then-Gov. Ben Cayetano, and has long supported the implementation of a statewide curriculum in Hawaii’s schools.

The current test was developed by to align more closely with the , which guide educators on what to teach at each grade level. Federal regulations dictated the shift to a more sophisticated test, Tanimura said.

The previous test was aligned with an earlier set of the standards and assessed students’ mastery of all the standards within multiple grade levels. For example, identical tests were administered to all the students in a span of three grades, but students in each grade were expected to answer a different percentage of the same questions correctly. The new grade-specific assessment has a unique test for each grade level.

Rising Student Scores

After the new HSA was implemented, Hawaii saw in the percentage of students who met the state’s new proficiency benchmarks.

Hawaii State Assessment — Percent Proficient

	Reading	Math
2003	41%	20%
2004	45%	23%
2005	49%	25%
2006	47%	27%
2007	60%	38%
2008	62%	42%
2009	65%	44%
2010	67%	49%

The 13-percent jump in reading proficiency and 11 percent jump in math proficiency between 2006 (the old test) and 2007 (the new one) are what raised Rolf’s concerns. He attributes the jump to score inflation caused by lower proficiency standards on the new tests — or, as he calls it, manipulation. He says the board voted to approve lowering , the minimum scores required to pass the test.

Rolf’s theory would work if the students’ on the test had remained the same or dropped, but they didn’t. Scale scores are the conversion of students’ raw scores — or the number of questions they got correct — placed on a common scale that allows for numerical comparisons from year to year, according to the . Hawaii students’ scale scores continued their gradual climb, according to data from AIR.

Reading scores:

Math scores:

Tanimura and student assessment administrator Kent Hinton say the grade-specific assessments more accurately reflect how students are doing with the material they are expected to know — and not how well sixth-graders are doing with eighth-grade material. They point out that the gains show not only on the state test, but on national measures as well.

“What you don’t want to see is the kids doing so well on the state assessment and then scoring really low on the ,” Tanimura said.

The NAEP is the only nationwide assessment that can help states gauge their student performance against national averages. It is also commonly used to estimate the relative difficulty, or rigor, of state tests, because each state develops its own tests and sets its own standards. But the national assessment tests only a sample of students in three grades — 4, 8 and 12 — every two years,

“The gap between the NAEP and our state test is actually smaller since we developed the new test in 2006,” Tanimura said. “NAEP is very rigorous, which is why we feel very confident that our state test and cut score is right where it should be.”

It’s hard to show the national and state tests’ scores side by side, because the NAEP is only administered every two years to grades 4, 8 and 12, while the the HSA only goes back to 2002 and reported results for only grades 3, 5, 8 and 10 before 2006. Although Hawaii’s national assessment coordinator successfully in a report to the board of education in 2007, the report only included score data through 2005.

Despite the difficulty of comparing pre-2006 state assessment scores with national assessment scores, it’s easy to show that Hawaii student performance on the national test shows a long-term rise. And at a faster rate than the national average. In the year 2000, the average score among Hawaii fourth-graders on the national math test was eight points lower than the national average. In 2009, the difference was only three points. In 2002, Hawaii eighth-graders were scoring 11 points lower on the reading test than the national average. By 2009, they had narrowed the gap to seven points. Similar gains are evident in fourth-grade reading and eighth-grade math results.

Hawaii Reading 4:

Hawaii Reading 8:

Hawaii Math 4:

Hawaii Math 8:

Hawaii gained the most ground on the national test in 2007 and 2009 — right after implementing its new state assessment. Administrators say that’s because the new test caused teachers to instruct to higher standards.

“Hawaii is not the highest-achieving state in the nation, and it faces a lot of challenges,” said Jon Cohen, senior vice president and director of assessment at the American Institutes for Research, which helped develop both the Hawaii and Oregon state assessments. “But the state has stuck with some very high proficiency standards.”

How Standards Are Set

Standards and cut scores for the HSA haven’t changed since the education board and department , Tanimura said, adding that there are each state has to follow when developing its assessments.

The Hawaii education department forms two committees for each of the three subject areas (math, reading and science) — one for elementary grades and the other for secondary grades. Each committee examines the data and test items prepared by the AIR for a specific content area. The committees have 24 members representing all geographic areas and demographics of the community; and educators with expertise in the content area they are evaluating. These committees establish the assessment’s standards and cut scores. The higher the cut scores, the more difficult it is for students to achieve “proficiency.” The department and AIR look at the recommendations, Tanimura said, but they honor the committees’ decisions.

“It’s a very transparent process,” she said. And it’s one used all over the country, Cohen added. (A similar one is used even to set standards for the national assessment.)

But it is an imperfect one, as Tanimura acknowledged. A couple of states have been caught making their state tests easier in recent years. is one of the most recent and notable examples. After the education department confessed, it brought in third-party analysts to examine and reset its cut scores.

The education board gives the final nod before the standards and cut scores are used for the state assessment. Rolf maintains that the board — like New York’s education officials — approved lowered cut scores in that 2007 meeting.

Cohen said the new cut scores are more accurate, since they reflect what a student is expected to know at each grade level, rather than lumping several grade levels’ worth of standards into one test with different cut scores.

“States are asked to set their own proficiency levels, and we don’t set them low,” Tanimura said. “When I open the standards-setting meeting, I tell them we have to look at this as in terms of what is best for the students — not just what’s going to make ourselves look good. You can’t say the kids are really smart if you’ve set the standards really low, so (former) Superintendent (Patricia) Hamamoto said to make them rigorous, make them difficult and make them challenging.”

The entire process of developing and vetting the test items and establishing cut scores must be documented for the U.S. Department of Education. The Hawaii education department is held accountable by a six-member technical advisory committee consisting of national testing experts.

That committee stated in 2007 that Hawaii’s procedures for developing the HSA were “appropriate and technically rigorous,” according to an by AIR to the Hawaii State Board of Education.

“We meet with them twice a year and they go over the test from beginning to end,” Tanimura said. “We have to submit documentation for everything we do: who sat on the committee, where they came from, etc. And if they saw anything like us playing games, we would get slapped on the wrist.”