Showing Race Fairness in Hogan’s Australian and New Zealand Assessments

Showing Race Fairness in Hogan’s Australian and New Zealand Assessments

Organisations that use assessments to make decisions trust those assessments will provide helpful insights (who to hire, where to focus during coaching) without unfairly discriminating against demographic groups. Hogan shares these organisations’ concerns and is committed to ongoing reviews of our assessments’ fairness across age groups, genders, and ancestries/ethnicities. Because of the lasting effects of colonialism on the people of Australia and New Zealand, it is important assessment scores that contribute to employment decisions don’t cause additional harm to marginalised groups.

Many clients ask if Hogan sees differences in personality and values scores for various groups. Our research indicates that, although small differences exist, we find no meaningful differences between ancestries/ethnicities. In this blog, we first explain how we determine if our assessments unfairly impact underrepresented groups. Then, we summarise results for Australian and New Zealand ancestries/ethnicities.

How to verify Hogan scales and items’ fairness

To evaluate our assessments’ fairness among ancestries and ethnicities in Australia and New Zealand, we compared the scores of each group to those of European descent. We examined mean differences between groups, and we conducted differential item functioning (DIF) and differential test functioning (DTF) analyses. DIF and DTF analyses help determine if items or scales, respectively, may favour or discriminate against certain groups, even if those individuals have the same level of ability or trait being measured by an assessment. For example, a math test question may be unfairly easier for men compared to women, despite both genders having the same mathematical ability. That question would be said to have DIF. We conducted DIF and DTF analyses to evaluate if the HPI, HDS, and MVPI items or scales are biased in a way that unfairly favour certain ancestries/ethnicities.

To conduct those analyses, a minimum number of test-taker responses is required. Research recommends a minimum sample size of 250 people (Lai et al., 2005) to conduct these analyses, and Hogan generally recommends a sample of at least 500 people for greater representativeness. Hogan does not have enough data to conduct DIF and DTF analyses for the Sub-Saharan African (Australia) and Pacific Peoples (New Zealand) groups. We will wait to analyse these groups until additional data are collected. We also note the Aboriginal and Torres Strait Islander and North African or Middle Eastern groups are also below Hogan’s recommended sample size, therefore, we regard their results as tentative until we collect more data.

Australia: HPI, HDS, and MVPI fairness across ancestries

We first examined mean score differences on the HPI, HDS, and MVPI scales between European and Asian, Aboriginal and Torres Strait Islander, North African or Middle Eastern, and People of the Americas. We found no meaningful differences between ancestries in Australia.

We did find one moderate mean score difference between the Aboriginal and Torres Strait Islander and European groups on the HDS scale Dutiful. Therefore, we examined if the Dutiful scale functioned differently (using DTF analyses) for the Aboriginal and Torres Strait Islander group. We did not find any difference compared to the European group.

These results show little difference between the groups we examined and provide strong evidence for the fairness of the HPI, HDS, and MVPI. Although we do see some differences between Aboriginal and Torres Strait Islander and European test-takers on the Dutiful scale, the lack of differential functioning at the scale level leads us to believe these differences result from higher levels of some personality traits in the Aboriginal and Torres Strait Islander population.

Second, we looked at item fairness. Most items (99%) function similarly across ancestries and appear to be fair. We conducted over 2000 item comparisons to determine if we should replace or modify items on the HPI, HDS, or MVPI. Of those comparisons, 1% were flagged for further investigation. Most of the flagged items belonged to the Aboriginal or Torres Strait Islander and North African or Middle Eastern groups, which contained less people than our recommended threshold of 500 people per group. Because of this fact, and considering the results at the scale level, we don't currently need to replace any items, though will continue to monitor these specific items as we collect more data.

Mean Differences Across Ancestries (Australia)

Read more about Australian Race/Ethnicity Equivlence of the HPI, HDS, and MVPI

New Zealand: HPI, HDS, and MVPI fairness across ethnicities

We conducted the same analyses for New Zealand ethnicities. We first examined mean score differences on the HPI, HDS, and MVPI scales between European and Asian and Māori. We found no meaningful differences between ethnicities in New Zealand.

We found some moderate score differences between the between the Asian and European groups (Inquisitive, Bold, Diligent, Recognition, Power, Altruism, Security, and Commerce. When we examined if any of the scales functioned differently (using DTF analyses) for the Asian group, we did not find any difference in functioning compared to the European group.

These results show little difference between the groups we examined and provide strong evidence for the fairness of the HPI, HDS, and MVPI in New Zealand. Although there are some differences between Asian and European test-takers’ scores, the lack of differential functioning at the scale level leads us to believe these differences results from higher levels of some personality traits or values in New Zealand’s Asian population.

Second, we looked at item fairness. Most items (98%) function similarly across ethnic groups and appear to be fair. We conducted over 1000 item comparisons to determine if we should replace or modify items on the HPI, HDS, or MVPI. Considering the results at the scale level, we don't currently need to replace any items, though will continue to monitor these specific items (2%) as we collect more data. Hogan will also verify our assessments’ fairness for the Pacific Peoples when we have enough test-taker responses to conduct DIF and DTF analyses.

Mean Differences Across Races/Ethnicities in New Zealand

Read more about New Zealand/Ethnicity Equivalence of the HPI, HDS, and MVPI

In conclusion, our research indicates that, although there may be minor differences, there are no meaningful disparities between ancestries and ethnicities in personality and values scores in Australia and in New Zealand. This blog has provided an explanation of our efforts to verify and mitigate any potential impact on underrepresented groups, as well as a summary of our analyses specific to Australia and New Zealand ancestries/ethnicities. By prioritising fairness and inclusivity, we aim to support organisations in making informed and unbiased decisions.

References

Lai, J.-S., Teresi, J., & Gershon, R. (2005). Procedures for the Analysis of Differential Item Functioning (DIF) for Small Sample Sizes. Evaluation & the Health Professions, 28(3), 283–294. https://doi.org/10.1177/0163278705278276

This article was authored by Hogan Assessments’ Data Science Team.