Use AI? Sure. Trust AI? Never.

Dr. Timnit Gebru, PhD, has three words of advice for dealing with artificial intelligence systems: Don’t trust them.

“Even if these systems work as intended, they can still be harmful,” Gebru, an expert in the area of ethical AI, said last week in a keynote talk at Alberta Innovates’ Inventures. The annual event, held May 31 to June 2 in Calgary, attracted over 3,000 people from more than 60 countries. Attendees included tech industry leaders, entrepreneurs, venture capitalists and others in the innovation ecosystem.

Gebru, who described herself as having a “much more expansive view” of the harms AI can cause, told the audience that multiple harms can occur at any place in the process of creating AI systems. This includes asking whether a particular system should be built in the first place, how systems are built and tested, how they are deployed, and who they are deployed on.

Gebru is the founder and executive director of the California-based non-profit Distributed Artificial Intelligence Research Institute (DAIR) for independent and community-rooted AI research. Prior to founding DAIR, she was fired by Google, where she was co-lead of the Ethical AI research team, in December 2020 for raising issues of AI-enabled discrimination in the workplace.

AI systems rely on huge datasets used to train the systems in how to recognize images and – as with OpenAI’s ChatGPT chatbot – generate text, Gebru said. “It’s not magic. There’s no sentient being over there. It’s a whole bunch of data that is labelled by a whole bunch of people.”

The AI industry employs hundreds of thousands people to sit and label millions of images — paying them a couple of cents per image, she added. The job includes screening out hateful, abusive, and toxic data.

“These workers, because they see so much of this content, they get traumatized, they have PTSD (post-traumatic stress disorder),” Gebru said. “This is harm that we’re inflicting on people during the development process.”

Harm also can occur in the large amount of data used to train AI systems. The AI industry “scrapes” or gathers data supplied by people around the world and often from everything available on the internet, including biased and toxic information.

 “The internet represents hegemonic views,” she noted, referring to the dominant social, cultural, ideological and economic perspectives of a nation or a society. .

Despite the enormous amount of data used to develop some AI systems, “size doesn’t guarantee diversity,” she said. “Many people in the world don’t have the internet. So their views are not being represented [in the data].”

For example, she suggested, information on Wikipedia, one of the main sources of data for AI systems, reflects the dominant geographical locations and languages of the world, while at the same time  under-representing women.

“So when you’re training [AI systems], whether it is anything related to images or anything related to text, using data from the internet you’re going to be replicating hegemonic views, even though you think because you have so much data it’s going to be diverse.”

Gebru’s research at Google included studying AI-enabled face-recognition systems developed by IBM, Microsoft and other companies. She found such systems achieved essentially perfect accuracy for lighter-skinned men, but were very inaccurate for darker-skinned women.

The reason, she said, is that many of the AI face-recognition systems were trained on images scraped from the internet that were predominantly of lighter-skinned men.

The lesson for AI developers is they should curate their datasets when they’re training AI systems, and encode the desired ethical values they want into their datasets, Gebru said.

AI systems can further perpetuate inequities

Harm can also occur during testing of AI systems. Companies didn’t realize the problems with their AI face-recognition systems because their evaluation datasets showed very high accuracy in recognizing faces, based on the large dataset of lighter-skinned men.

“You can get 97-per-cent accuracy on this evaluation dataset and still have zero per cent accuracy on certain sub-groups,” Gebru said.

She recommended that all AI and machine-learning systems should be tested using an “intersectionality” process, by testing and comparing an AI system with as many diverse groups as possible, including sub-groups under-represented in data on the internet.

Another harm is that deploying AI systems can further perpetuate inequities in society, such as face-recognition systems that result in increased surveillance of people or groups already subject to surveillance, such as Black neighbourhoods in Detroit, she said.

“The first question we need to ask, before we build any systems, is: ‘Should this system even exist?’”

Gebru noted that IBM got sued for and had to remove an AI-enabled gender-recognition system after scraping data without first getting people’s consent, including photos from Flickr and photos of transgender people from YouTube. Microsoft had to abandon its AI gender-recognition system after complaints it scraped data of darker-skinned people.

“What many of these organizations didn’t ask, before going to all this trouble, is: ‘Why should we even have this gender-recognition system? Why is it even built?’”

The lesson for AI developers is to involve a diverse group of people, including those most likely to be harmed, in deciding what systems to build or not build, Gebru said.

Despite AI’s potential to cause harms, “there is an AI gold rush right now” for venture capitalists and investors and companies that think they need AI to get investor funding, she said. “I think it’s good to cool off [with saying] that everybody has to have AI.”

“Automation bias” can be extremely dangerous

A potentially extremely dangerous harm of AI involves “automation bias,” where humans tend to over-trust automated systems such as face-recognition systems or even Google maps, Gebru said.

In a series of experiments led by robotics engineer Dr. Ayanna Howard, PhD, at Georgia Tech, researchers simulated a building fire and used a robot to guide building occupants to safety. Even after the machine proved itself unreliable —sometimes guiding people toward rather than away from the fire — and even after some participants were told the robot had broken down, test subjects continued to follow instructions from the robot.

“The people just would follow the robot, regardless of where it took them,” Gebru said.

Similarly, police using AI face-recognition systems have wrongfully arrested people, including in one case arresting a person who was not even in the same city when and where the crime was committed. Because they over-trusted the AI system, she said, “the police did not even check his height or weight to make sure it matched the suspect.”

Generative AI systems such as ChatGBT “are literally created to make stuff up," Gebru argued. "They’re ingesting lots of stuff from the internet and put out the most likely sequences of words."

In one recent case, a lawyer used ChatGBT to write his legal brief, but it ended up containing fake legal cases and bogus citations.

In another case that Gebru documented in her research, a Palestinian man wrote “Good morning” in Arabic, but Facebook translated the writing as “Attack them.” The man got arrested and police didn’t bother to check what was originally written.

“This over-trusting of [AI] systems can be extremely dangerous,” she said.

“Don’t trust these systems,” whether you’re a consumer or a business or an organization utilizing AI, Gebru advised. “Put some guardrails in place so that people don’t over-trust these systems.”

Last week, international AI researchers and leaders in the industry, including the CEOs of OpenAI and Google Deepmind, warned in an online statement that AI is an existential threat that poses the risk of extinction of the human species, like other societal-scale risks such as pandemics and nuclear war.

Researchers in Quebec studying AI have told Research Money that the federal government needs to regulate AI quickly and also support “socially beneficial” AI. However, only 24 per cent of the Canadian population trusts Ottawa to implement effective policies to regulate AI, according to Matt Malone, an assistant professor of law at Thompson Rivers University in Kamloops.

Others AI researchers and AI chatbot users have told Research Money they’re more sanguine about AI systems.

R$