This story is part of Undertones, Global Voices’ Civic Media Observatory‘s newsletter. Subscribe to Undertones here. Welcome to Undertones, the newsletter where we decode narratives. In this edition, we cover how people speak about the (mis)use of AI in the Netherlands, thanks to researcher Leon Ingelse. Data scientist Ingelse analyzed the discussions about algorithms in the Netherlands. After all, it is a big deal in the Low Lands: in 2021, an entire government fell after years of mishandling a case of AI fraud detection among childcare beneficiaries. Thousands of families, primarily of migrant backgrounds, had been wrongly accused of cheating the system. The Netherlands, a country that takes pride in efficiency and the use of technology in government services, is a case study of how the misuse of AI can hurt the lives of individuals and undermine trust in authorities. Nevertheless, algorithms continue to be used in municipalities throughout the Netherlands to determine who is more or less likely to commit welfare fraud — an endless hot topic in Europe. However, nearly all of this AI tech is opaque to journalists and researchers as authorities claim privacy and intellectual property concerns. Only Rotterdam, the second-largest Dutch city, recently agreed to share the skeleton of their algorithm with journalists from investigative media Lighthouse Reports and Follow The Money. They accidentally also shared the training data sets, which provided invaluable insight into how the algorithm works from the inside out. The journalists have found that the algorithm is a scoring card fed with personal criteria such as age, sex, and marital status, as well as elements of behavior and language skills. Rotterdam initially developed its algorithm with the help of consulting firm Accenture. “The data fed into the algorithm ranges from invasive (the length of someone’s last romantic relationship) and subjective (someone’s ability to convince and influence others) to banal (how many times someone has emailed the city) and seemingly irrelevant (whether someone plays sports),” Lighthouse writes. In other words, if you are a) a woman, b) plurilingual, c) twenty or thirty-something, and d) a parent, the algorithm would file you as someone at risk of committing fraud, independently of any actual fraudulent behavior in the past. If you appear shy to the social worker, you will be even worse off. You will not know whether you’re on the blacklist, or why. Still, uncomfortable investigations by local authorities would ensue, as well as potentially losing welfare benefits. Screenshot of the risk scores calculated by Rotterdam's welfare fraud algorithm, ask reconstructed by Lighthouse Reports. “The Dutch tend to believe they don't have structural racism, whereas this algorithm clearly shows otherwise, as people with limited Dutch proficiency are discriminated against,” Ingelse says. Other discriminatory algorithm uses in the Netherlands have also come to light, with the profiling of nationality and ethnicity in visa applications. Journalists have pointed out the intrinsic biases in algorithmic risk scores elsewhere. As early as 2016, an investigation by ProPublica revealed that within the U.S. criminal justice system, Black individuals were twice as likely to be erroneously identified as “high risk” for reoffending; in contrast, white individuals were twice as likely to be wrongly categorized as “low risk.” In the Rotterdam case, experts argue that AI performance is only marginally better than randomly selecting welfare beneficiaries for investigation, according to Lighthouse. AI algorithms exhibit bias due to training on skewed or insufficient real-world data and the influence of human developers’ inherent prejudice. Biases enter the system in other ways too. In the Dutch AI scandals, the public debate does not tend to happen via original tweets or posts, but rather only on comment sections, which, surprisingly, show an interesting array of opinions. Most of those launching the debate are journalists. “It could be that the Dutch are tired after the Toeslagenaffaire [childcare benefits scandal], and are more focused on ChatGPT,” Ingelse says. “Also, public figures that have been part of the scandals tactically avoid answering questions about their involvement.” He adds that the only people burdened with these specific algorithms are minorities, who are often invisibilized. “Notice how these fraud algorithms are never used in profit taxes, for example, which would target the rich,” Ingelse says. We have reached out to local Rotterdam authorities as well as the newly created group of families impacted by the algorithm to understand their perspectives, but have not received answers.