By the end of this year already, the market for artificial intelligence (AI) in South Africa is projected to reach a size of $2.4 billion, showing an annual growth rate of 21% between now and 2030. Locally, the technology has the potential to mitigate security risks, enhance decision-making, address legacy challenges, and have a significantly positive societal impact. Despite the impressive applications and implications, Anna Collard, SVP Content Strategy & Evangelist at KnowBe4 AFRICA, warns of the associated risks that need to be considered.

“Generative AI models are trained on data from various sources,” she explains, highlighting that these sources are not all verified, lack sufficient context, and are not regulated. “AI is incredibly helpful in handling the mundane administrative tasks associated with spreadsheets and statistics. However, it becomes concerning when we rely on it to make decisions that have the potential to influence people’s lives.”

AI is an algorithmic construct built on the bones of human creative endeavours and data that is often flawed and biased. “As Kate Crawford, a professor at the University of Southern California and Microsoft researcher, pointed out, AI is not truly artificial or intelligent. This poses risks that can have long-term consequences if users are unaware of them,” explains Collard.

Here are six of the most concerning risks:

01: AI hallucinations: Earlier this year, a New York attorney used a conversational chatbot for legal research. The AI deceitfully incorporated six fabricated precedents into his filing, falsely attributing them to prominent legal databases. This is a perfect example of an AI hallucination, where the output is either fake or nonsense. These incidents happen when prompts are outside of the AI’s training data and so the model hallucinates or contradicts itself in order to respond.

02: Deepfakes: The implications of fake images extend to various areas. With the rise of fake identities, revenge porn, and fabricated employees, the range of potential misuse for AI-generated photographs is expanding. One particular technology called Generative Adversarial Network (GAN) is a type of deep neural network capable of producing new data and generating highly realistic images by using random input. This technology opens up the realm of deepfakes, where sophisticated generative techniques manipulate facial features and can be applied to images, audio, and video. This form of digital puppetry carries significant consequences in political persuasion, misinformation or polarization campaigns.

03: Automated and more effective attacks: This taps directly into the potential of GAN mentioned before, as cybercriminals make use of deepfakes in more sophisticated attacks. They use it in impersonation attacks, where fake voice or even video versions of someone can be used to manipulate victims into paying or following other fraudulent instructions. Cybercriminals also benefit from jailbroken generative AI models to help them automate or simplify their attach methods, such as for example automating the creation of phishing emails.

04: Media equation theory: This refers to the fact that human beings have a tendency to attribute human characteristics to machines and develop feelings of empathy towards them. This tendency becomes even stronger when the interactions with machines seem intelligent. Although this can positively impact user engagement and support in the service sector, it also carries a risk. People become more vulnerable to manipulation, persuasion, and social engineering because of this over-trust effect. They tend to believe and follow machines more than they should. Research has shown that people are likely to alter their responses to queries in order to comply with suggestions made by robots.

05: The manipulation problem: AI, through the use of natural language processing, machine learning, and algorithmic analyses, can both respond to and simulate emotions. By gathering information from various sources, agenda driven AI chatbots for example can promptly react to sensory input in real time and utilise it to accomplish specific objectives, such as persuasion or manipulation. These capabilities create opportunities for the dissemination of predatory content, misinformation, disinformation, and scams.

06: Ethical issues: The presence of bias in the data and the current absence of regulations regarding AI development, data usage, and AI application all raise ethical concerns. Global efforts are underway to tackle the challenge of ethics in AI and reduce the risks of AI poisoning, which entails manipulating data to introduce vulnerabilities or biases. “However, South Africa currently lacks momentum in addressing these issues. This must change, as managing and detecting the risk of polluted AI data before it causes long-term harm is essential.” Says Collard.

“It is important to be mindful of the information we share with AI chatbots and virtual personal assistants. We should always question how our data is being used and by whom,” concludes Collard. “There is a risk of sharing sensitive personal and business information with data training models.

“While AI is a valuable tool, it is crucial to use it with critical thinking and mindfulness, and only rely on it in situations where it provides the most value and has been fact checked.”