Source: https://victorzhou.com/series/neural-networks-from-scratch/
With neural networks and deep learning architectures becoming an essential component of many operational systems, adversarial manipulation of such systems has given rise to the field of adversarial machine learning. Vulnerabilities in neural networks are fundamentally different from common software bugs such as memory corruption or syntax errors. Adversarial examples exploit mathematical properties of pattern recognition in high-dimensional space to subvert the intended operation of a neural network both during training and the inference phase. Prompt injection, data poisoning, and model inversion attacks are three particularly concerning phenomena that impact modern artificial intelligence systems in security-sensitive ways. Prompt injection attacks are a unique threat to generative artificial intelligence and large language models that cause a system to accept potentially unsafe instructions from users. Prompt injection represents a model's inability to distinguish between internal and external information, and can take two primary forms: direct and indirect. Direct prompt injection occurs when a user deliberately provides override instructions to a language model, whereas indirect prompt injection embeds additional directives in a web page, document, or tool response that the model assumes originated from the user. Due to how prompting-based instruction tuning works, hidden prompt injections could exfiltrate session data or perform unintended application programming interface calls. Defenses against prompt injection should not rely on simple regular expression matching but instead utilize restricted tool use, apply the least-privilege principle, use per-request nonces, and require human confirmation of risky operations. Unlike prompt injections, which impact the inference stage, data poisoning attacks target the training data of a machine learning system. Attackers could modify examples in poisoned training data or insert backdoors that cause the network to trigger specific hidden behaviors when presented with particular inputs during inference. Data poisoning poses a serious threat to the accuracy of computer vision systems and medical diagnosis tools, as poisoned images could drastically reduce the accuracy of a vision-based model. Defenses against data poisoning typically involve auditing training data for potential manipulation, sanitizing input pipelines, and utilizing robust aggregation techniques. Federated learning systems are also adopting new methods to identify and discard updates from rogue clients that would undermine the integrity of the training data. Model inversion and extraction attacks attempt to exfiltrate sensitive data from a machine learning system performing inference against the targeted model. The problem of model inversion is particularly acute for large language models, which are often trained on private datasets: an attacker could query the application programming interface of a language model to extract sensitive training data that should not be publicly available. Model inversion exploits the fact that neural networks do not forget information that is used to update their weights during the training period. Such a breach of confidentiality could result in serious regulatory and financial consequences for the affected organization. Differential privacy during the training phase presents a potential defense by adding carefully tuned encrypted noise to model parameters or updates, thus preventing individual training records from being reconstructed.
Sources
https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf https://www.sysdig.com/learn-cloud-native/prompt-injection