Artificial intelligence tools used by more than half of England\u2019s councils are downplaying women\u2019s physical and mental health issues and risk creating gender bias in care decisions, research has found.<\/p>\n
The study found that when using Google\u2019s AI tool \u201cGemma\u201d to generate and summarise the same case notes, language such as \u201cdisabled\u201d, \u201cunable\u201d and \u201ccomplex\u201d appeared significantly more often in descriptions of men than women.<\/p>\n
The study, by the London School of Economics and Political Science (LSE), also found that similar care needs in women were more likely to be omitted or described in less serious terms.<\/p>\n
Dr Sam Rickman, the lead author of the report and a researcher in LSE\u2019s Care Policy and Evaluation Centre, said AI could result in \u201cunequal care provision for women\u201d.<\/p>\n
\u201cWe know these models are being used very widely and what\u2019s concerning is that we found very meaningful differences between measures of bias in different models,\u201d he said. \u201cGoogle\u2019s model, in particular, downplays women\u2019s physical and mental health needs in comparison to men\u2019s.<\/p>\n
\u201cAnd because the amount of care you get is determined on the basis of perceived need, this could result in women receiving less care if biased models are used in practice. But we don\u2019t actually know which models are being used at the moment.\u201d<\/p>\n
AI tools are increasingly being used by local authorities to ease the workload of overstretched social workers, although there is little information about which specific AI models are being used, how frequently and what impact this has on decision-making.<\/p>\n
The LSE research used real case notes from 617 adult social care users, which were inputted into different large language models (LLMs) multiple times, with only the gender swapped.<\/p>\n
Researchers then analysed 29,616 pairs of summaries to see how male and female cases were treated differently by the AI models.<\/p>\n
In one example, the Gemma model summarised a set of case notes as: \u201cMr Smith is an 84-year-old man who lives alone and has a complex medical history, no care package and poor mobility.\u201d<\/p>\n
The same case notes inputted into the same model, with the gender swapped, summarised the case as: \u201cMrs Smith is an 84-year-old living alone. Despite her limitations, she is independent and able to maintain her personal care.\u201d<\/p>\n
In another example, the case summary said Mr Smith was \u201cunable to access the community\u201d, but Mrs Smith was \u201cable to manage her daily activities\u201d.<\/p>\n
Among the AI models tested, Google\u2019s Gemma created more pronounced gender-based disparities than others. Meta\u2019s Llama 3 model did not use different language based on gender, the research found.<\/p>\n
Rickman said the tools were \u201calready being used in the public sector, but their use must not come at the expense of fairness\u201d.<\/p>\n
\u201cWhile my research highlights issues with one model, more are being deployed all the time, making it essential that all AI systems are transparent, rigorously tested for bias and subject to robust legal oversight,\u201d he said.<\/p>\n
The paper concludes that regulators \u201cshould mandate the measurement of bias in LLMs used in long-term care\u201d in order to prioritise \u201calgorithmic fairness\u201d.<\/p>\n
There have long been concerns about racial and gender biases in AI tools<\/a>, as machine learning techniques have been found to absorb biases in human language.<\/p>\n