Dr Rumman Chowdhury wants us to imagine a world where no digital information, not even the video call we’re using for our interview, can be trusted. How do we prepare for such a world? What safe ecosystem must we build to withstand the “massive” impact of AI-powered manipulation?
Chowdhury knows it sounds dark. Dystopian even. But she’s a tech optimist who believes in disaster planning. Imagining the worst-case scenario, she explains, allows us to work backwards. To look beyond a particular defence technology and understand what solutions, institutions, investment and public engagement we need.
Chowdhury, the former US Science Envoy on AI in the Biden administration, spoke to Athena this week. One of the world’s leading thinkers on responsible AI, she’s led AI ethics teams at Twitter (now X) and Accenture, and is the co-founder of Humane Intelligence, a nonprofit dedicated to making AI systems that serve society.
Power and responsibility
Chowdhury says the threat of AI-powered disinformation is now “incredibly high”, as it moves beyond fake content and into the realm of fake provenance (complete with invented personas and websites that appear to corroborate claims). AI is also pulling us further away from original sources of information, she said, leaving private companies, with opaque processes, as the “arbiters of what truth is”.
Under pressure from the public, frontier AI labs are showing more responsibility, Chowdhury said, but are facing the same difficult problems social media companies tackled, and often failed to solve. She says AI labs tend to throw money at the problems, believing they can solve them all in-house.
“What I don’t see enough of is engagement with civil society,” said Chowdhury. “Consensus building is how democracy happens. It will be messy and it will be difficult, but that is the nature of what it is to do these things well.”
Chowdhury’s research for UNESCO showed how easily AI tools could be used to create material for gender-based harassment campaigns. She says AI agents make it easier still for armies of bots to act like people and shape perspectives or beliefs.
A hidden secret
The flood of AI-generated content creates another problem for frontier labs, she believes. Chowdhury wonders how they’ll find enough high-quality information to train their models and avoid “model collapse”. She suspects this may be their “hidden secret”.
“We’re not living in a world in which, suddenly, models showed up that were hyper capable and we were unprepared,” Chowdhury said. “We are living in a world in which we heavily invested in model capabilities, but did not invest in the parallel safety and security measures.”
Hours after we spoke to Chowdhury, news broke about the size of Anthropic’s blockbuster IPO, valued at $1.4 trillion. Interestingly, Chowdhury sees a path to accountability in private companies going public, saying shareholders can demand more transparency – the kind that points a bright beam at “hidden secrets”.
“Companies are not addressing the questions that people are asking… The average person is asking how their data is being used, they’re asking to not be manipulated, they’re asking to not be sold things, they’re asking for their children not to be engaged in parasocial relationships because of harmful AI. But they [the AI companies] don’t talk about those problems.”
Breaking a depressing cycle
Chowdhury believes middle power countries such as the UK can be a “force for good governance”. They may not be at the forefront of building LLMs, but the UK, she says, has a massive talent pool and was an early leader in AI governance. What’s missing? Cash. Lots of it.
Chowdhury would put that money into AI assurance, the work of testing whether AI systems are safe and reliable before people depend on them. For Chowdhury, testing critical systems - an NHS chatbot for example - before they go live is part of defending the information people rely on. She also sees room for investment in tools that spot false or fraudulent information, and in cybersecurity.
Is social media dying?
According to Chowdhury, governments have a role in protecting people and tackling false advertising or false election information (see next story). She’d keep that role tightly defined though, to avoid any administration wielding too much power over what people can say online. She wants independent watchdogs to have a bigger role in holding AI companies to account.
She also sees a use for AI tools in the hands of defenders, drawing a parallel with cybersecurity, where new technology tends to flow from offence to defence. Chowdhury’s other observation is that “social media is kind of dying”. She believes younger people are starting to spend less time on social media and are thinking more carefully about where they get their information.
An analogue defence
In the US, she’s encouraged by what she sees happening offline, with people getting involved in local politics and encountering different opinions. For her, the answer can be as ordinary as joining a book club, getting involved in a city council, or running for a seat on a school board or homeowners’ association. Spend time with people whose lives differ from yours, she says, and you get some practice sitting with disagreement.
“Human-to-human contact does more to stem mis- and disinformation than any sort of sophisticated technical mechanism we could possibly come up with,” she said.
Can we stem the tide of disinformation? Can we stop a “depressing cycle” of online disinformation, where one AI system churns out misleading posts while another takes them down. Chowdhury isn’t sure. But if the day comes when even a video call can’t be taken at face value, a “non-zero likelihood” scenario in her opinion, she wants us to have built the tools and institutions we’ll need to work out what we can trust.
Key takeaways:
Bring the public in: give global communities a say in how AI is tested and governed.
Meet IRL: spend time with people you might never encounter in your feeds.
Sovereignty matters: middle powers like the UK can lead the way in good governance because information is critical infrastructure.
Invest, invest, invest: for instance in AI testing and cybersecurity and detection.
Listen to Athena’s full interview with Rumman Chowdhury:
South Africa tests new election defence
Ahead of last week’s local government elections, the Electoral Commission of South Africa (IEC) introduced a new, legally binding disinformation code.
The guidance forced political parties and candidates to report suspected online disinformation within 36 hours of discovering it. It was in place from early September to the results being declared.
The code, it seems, was justified. Africa Check exposed a fake AI-generated image of ANC president, Cyril Ramaphosa, handing out “cash for votes”. A real video of ballot collections was misrepresented to raise outrage about cheating. And the IEC warned that “misleading social media posts are threatening the integrity of the local government elections”. To facilitate reporting, the IEC designated a platform, Real411.
South Africa’s landmark 1994 election, which paved the way for Nelson Mandela to become president, hold a special place in Jessica Cecil’s heart. A month after launching Athena, Jessica shared her reflections on what it takes to fight the kind of disinformation that worried the IEC enough to introduce a new code.
You can read Jessica’s Daily Maverick op-ed here.
For an excellent take on what the election outcome means, read this analysis from Daily Maverick columnist Stephen Grootes.
News: What changed this week
AI impersonation is at the centre of an alleged $129 million fraud. US prosecutors say three men used AI audio fakes to impersonate chief executives and lawyers, tricking employees at companies around the world into transferring money for what they believed were confidential deals. The suspects were arrested in Britain and Spain and face charges in the United States.
Read The New York Times report · Read the US prosecutors’ account
Fake news reports blamed migrants for unrest in France. An internal EU assessment seen by Reuters found that a Russian network impersonated Western news outlets to spread false claims about last month’s riots. The videos tried to make the French government look as though it had lost control. Athena previously examined how Russian networks exploited homegrown anger in France.
Read Reuters’ report · Read Athena’s French anger, Russian fingerprints
Concerns grow over the independence of US government AI testing. The Associated Press reports that White House oversight of staff at the US AI safety centre has raised concerns about whether political priorities could influence which risks the centre investigates and what it makes public.
Read the Associated Press report
Solutions:
Humane Intelligence, the nonprofit co-founded by Rumman Chowdhury, brings the public into the work of testing AI. It runs exercises known as red teaming, where participants deliberately probe models for failures, from biased answers to misleading claims.
The Ada Lovelace Institute is an independent UK research institute that involves the people affected by AI in decisions about how it’s used and governed. Chowdhury recommends its work because it connects policy with practice and keeps the focus on what AI means for people in their everyday lives.
Real411 gives people in South Africa a place to report suspected disinformation, hate speech and other online harms. Complaints are reviewed, and readers can look up cases to see how they were resolved.
Singapore’s AI Verify Foundation develops open-source tools that help organisations check whether their AI systems are safe and reliable. That’s AI assurance in practice. The nonprofit also brings developers and independent testers together to assess how AI applications perform in real-world settings.