Artificial intelligence (AI) encompasses various technological systems with differing levels of complexity. On a basic level, we understand what the algorithm is doing. For instance, some AI uses unsupervised learning, where the AI learns by encountering large quantities of data and identifies patterns. After analyzing many images of a camera, the system can associate features, such as a lens and a shutter, with the concept of a camera.

We also know that machine learning models can process vast amounts of data to drive decision-making, functioning in descriptive (explaining data), predictive (anticipating outcomes) or prescriptive (recommending actions) ways. Like the human brain, AI is not merely retrieving stored data but is generating new output based on its training. Deep learning relies on complex neural networks that mimic the human brain, but these networks are so vast and intricate that tracking the process is challenging. Even the top engineers cannot fully explain how AI moves from input to output, raising questions such as what steps occur once a prompt is entered. This problem is often referred to as the “black box” problem.

The entire process is not clouded, however, as AI can often identify the sources it relies upon and explain aspects of its reasoning, although not the complete path to its conclusions. In some cases, this may be sufficient for the average user, although this should be considered on a case-by-case basis, such as biases in hiring algorithms.

The lack of transparency in how AI systems operate, exemplified by the black box problem, creates challenges for trust and reliability and raises accountability concerns in the practical and legal context for the companies creating these systems. Transparency is critical when AI malfunctions, whether it is an automatic weapon misfiring, a self-driving car failing to stop, a face being incorrectly recognized, or a medicine being incorrectly prescribed. In high-stakes scenarios, understanding the decision-making process is essential for accountability and problem-solving.

Moreover, the black box problem means that AI companies may not be aware when their dataset has been poisoned, i.e. victimized by a type of cyberattack where bad actors manipulate or corrupt the training data used to develop artificial intelligence (AI) and machine learning (ML) models. Despite these concerns, a level of secrecy can be important. For instance, if AI had full transparency, users would be able to access the materials on which AI was trained as well as the developers’ algorithms, thereby exposing valuable intellectual property. AI’s full transparency would also raise national security, privacy and safety concerns.

As of this writing, more than half of Americans use AI multiple times a week, and that number is growing. Due to hallucination, malfunction, accountability and security concerns, mitigating risk with due diligence when it comes to high-risk situations is critical. For further reading from our website on the topics discussed here, see the following insights and IP Bits & Pieces®: AI Can’t Hold Copyrights and our Copyright and AI FAQs.