Securing the Generative Frontier: A Systematic Analysis of Training Data Poisoning and Prompt Engineering Vulnerabilities in Large Language Models
DOI:
https://doi.org/10.32628/CSEIT251112159Keywords:
Generative Artificial Intelligence, Data Poisoning, Prompt Engineering Security, Training Data Vulnerabilities, Cybersecurity Risk MitigationAbstract
The rapid advancement and widespread adoption of generative AI technologies have introduced unprecedented cybersecurity challenges in data management and system security. This article presents a comprehensive analysis of security vulnerabilities in generative AI systems, with particular emphasis on training data poisoning attacks and prompt engineering exploits. Through systematic investigation of current security frameworks and emerging threat vectors, this article identifies critical vulnerabilities in training data management processes. It examines potential information leakage mechanisms in large language models. The article employs a multi-layered analytical approach to evaluate both intentional and unintentional security risks, ranging from malicious data manipulation to inadvertent exposure of sensitive information through model responses. Additionally, the article proposes a novel framework for threat detection and mitigation, incorporating adaptive security protocols and robust validation mechanisms for prompt engineering. The findings contribute to the growing cybersecurity literature by providing actionable insights for organizations implementing generative AI systems while establishing a foundation for future research in AI security standardization and self-healing machine learning models.
Downloads
References
M. Gupta et al., "From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy," IEEE Access, vol. 12, pp. 12345-12356, 2023. [Online]. Available: https://ieeexplore.ieee.org/document/10198233/keywords#keywords
S. Oh and T. Shon, "Cybersecurity Issues in Generative AI," IEEE Xplore, vol. 11, no. 4, pp. 6789-6792, 2023. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10255179/authors#authors
IEEE Standards Association, "Standard for Security and Trustworthiness Requirements in Generative Pretrained Artificial Intelligence (AI) Models," 2023. [Online]. Available: https://standards.ieee.org/ieee/7018/11306/
Y. Liu et al., "Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study," arXiv preprint arXiv:2305.13860, 2023. [Online]. Available: https://arxiv.org/abs/2305.13860
Y. Roh, G. Heo, and S. E. Whang, "A Survey on Data Collection for Machine Learning: a Big Data - AI Integration Perspective," IEEE Xplore, 2018. [Online]. Available: https://arxiv.org/abs/1811.03402
V. Oleshchuk, "Trust-enhanced data integrity model," IEEE 1st International Symposium on Wireless Systems (IDAACS-SWS), 2012. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/6377645
B. Biggio, G. Fumera, and F. Roli, "Security evaluation of pattern classifiers under attack," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 4, pp. 984-996, 2014. [Online]. Available: https://ieeexplore.ieee.org/document/6494573
X. Huang, A. D. Joseph, B. Nelson, B. I. P. Rubinstein, and J. D. Tygar, "Adversarial machine learning," Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, 2011. [Online]. Available: https://dl.acm.org/doi/10.1145/2046684.2046692
N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, "Practical black-box attacks against machine learning," Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security. [Online]. Available: https://dl.acm.org/doi/10.1145/3052973.3053009
D. Rosch-Grace and J. Straub, "Considering the Implications of Artificial Intelligence, Quantum Computing, and Cybersecurity," IEEE Conference Publication, 2022. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10216430
D. A. Alber et al., "Medical large language models are vulnerable to data-poisoning attacks," Nature Medicine, 2024. [Online]. Available: https://www.nature.com/articles/s41591-024-03445-1
Downloads
Published
Issue
Section
License
Copyright (c) 2025 International Journal of Scientific Research in Computer Science, Engineering and Information Technology

This work is licensed under a Creative Commons Attribution 4.0 International License.