A team of education officers in Samoa has completed the first phase of a three-month training programme in the development of test instruments facilitated by the Pacific Community (SPC).
The officers, who are part of the newly formed National Tool Development Unit at Samoa’s Ministry of Education, Sports and Culture (MESC), are mostly new to the specialised area of assessment work. The unit, a part of the ministry’s Assessment and Examinations Division, focuses on the development of assessment items and test blueprints. A test blueprint is a framework or outline of learning outcomes to be assessed in the test to be designed.
MESC Chief Executive Officer Dr Karoline Afamasaga-Fuatai, during the Pacific Board for Educational Quality’s annual meeting last month, thanked the SPC’s Educational Quality and Assessment Programme (EQAP) for its quick response to their request for training support for the new team.
She said the restructure meant that there was now a team dedicated to the creation of assessment items. Previously, these officers were responsible for curriculum design but undertook the extra work of designing assessments. After thorough consultation earlier this year, a training schedule organised in blocks of two-hours per day was confirmed. The training is spread over three months.
Toloai Tipi, the most experienced officer in the unit, said the new unit recognised the additional assessment work. The training, he said, was thus a refresher course for him, although certain aspects were new.
“Everything in the training is valuable to the work,” he said. “As a tool developer, the ability to read and understand data from these variable (item) maps helps a lot. These variable maps provide reliable results which reflect the types of question designed and the information tells me the story of either side of the graph: the level of difficulties of a question versus the level of ability each individual candidate has.”
The item maps reflect the spread of students’ abilities at the different cognitive levels of the assessment. Tool developers analyse these maps to ensure the assessment items are targeted well.
The other tool developers undergoing the training are Aloiafi Ma’alo and Leutogi Faleva’ai. Faleva’ai, who taught visual arts and English for nine years before working in the ministry as a curriculum officer for the past 10 years, said the creation of a codebook and the data mining aspects of the training intrigued her the most.
“I was previously trained on developing and coding items, but not a codebook,” she said. “Everything (in the training) was exciting, but the codebook and reading data are what stands out the most to me.” Faleva’ai added that her colleagues from their data unit sometimes joined the trainings whenever they could.
The team’s training in the analysis and development of item maps was conducted by EQAP’s assessment officers Adrian Alamu and Tuila Seniloli. Alamu said the first phase of training focused on the fundamentals of developing targeted test instruments, which includes the analysis of item maps. He said the training was progressing well and the Samoan team was finding the training rewarding.
EQAP’s curriculum and assessment team leader Mere Vadei said the two areas of work (curriculum and assessment item development) are closely interrelated.
“The two areas of work require different skillsets, but good knowledge of the curriculum is critical to good item development,” said Vadei. “It is not uncommon to find people who are skilled in both and who do both. However, the workload tends to be too heavy.”
Vadei added that while having separate teams meant quality time could be dedicated to each area of work, the two groups would need to meet regularly to ensure their work was in sync. This is to ensure that the framing of assessment items is informed by the analysis of corresponding item maps.
Samoa has also transitioned to fully adopt a multiple-choice item format for its low-stake examinations, except for its story writing or composition items. For the multiple-choice scripts, students’ answers are captured by a special scanner and digitally processed for scoring.
As for the story writing items, these are still scored manually. “I still train scorers to score story writing separately and (these are) scanned later,” said Tipi, who is the primary tool developer for languages and social studies. Tipi was part of previous trainings in coding and item writing conducted by EQAP. Faleva’ai and Ma’alo are the primary tool developers for TVET, and for Mathematics and Science, respectively.
The next phase of training will focus on item development.