Securing the Generative Frontier: A Systematic Analysis of Training Data Poisoning and Prompt Engineering Vulnerabilities in Large Language Models

Authors

  • Manojava Bharadwaj Bhagavathula University of Pittsburgh, USA Author

DOI:

https://doi.org/10.32628/CSEIT251112159

Keywords:

Generative Artificial Intelligence, Data Poisoning, Prompt Engineering Security, Training Data Vulnerabilities, Cybersecurity Risk Mitigation

Abstract

The rapid advancement and widespread adoption of generative AI technologies have introduced unprecedented cybersecurity challenges in data management and system security. This article presents a comprehensive analysis of security vulnerabilities in generative AI systems, with particular emphasis on training data poisoning attacks and prompt engineering exploits. Through systematic investigation of current security frameworks and emerging threat vectors, this article identifies critical vulnerabilities in training data management processes. It examines potential information leakage mechanisms in large language models. The article employs a multi-layered analytical approach to evaluate both intentional and unintentional security risks, ranging from malicious data manipulation to inadvertent exposure of sensitive information through model responses. Additionally, the article proposes a novel framework for threat detection and mitigation, incorporating adaptive security protocols and robust validation mechanisms for prompt engineering. The findings contribute to the growing cybersecurity literature by providing actionable insights for organizations implementing generative AI systems while establishing a foundation for future research in AI security standardization and self-healing machine learning models.

Downloads

Download data is not yet available.

References

M. Gupta et al., "From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy," IEEE Access, vol. 12, pp. 12345-12356, 2023. [Online]. Available: https://ieeexplore.ieee.org/document/10198233/keywords#keywords

S. Oh and T. Shon, "Cybersecurity Issues in Generative AI," IEEE Xplore, vol. 11, no. 4, pp. 6789-6792, 2023. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10255179/authors#authors

IEEE Standards Association, "Standard for Security and Trustworthiness Requirements in Generative Pretrained Artificial Intelligence (AI) Models," 2023. [Online]. Available: https://standards.ieee.org/ieee/7018/11306/

Y. Liu et al., "Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study," arXiv preprint arXiv:2305.13860, 2023. [Online]. Available: https://arxiv.org/abs/2305.13860

Y. Roh, G. Heo, and S. E. Whang, "A Survey on Data Collection for Machine Learning: a Big Data - AI Integration Perspective," IEEE Xplore, 2018. [Online]. Available: https://arxiv.org/abs/1811.03402

V. Oleshchuk, "Trust-enhanced data integrity model," IEEE 1st International Symposium on Wireless Systems (IDAACS-SWS), 2012. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/6377645

B. Biggio, G. Fumera, and F. Roli, "Security evaluation of pattern classifiers under attack," IEEE Transactions on Knowledge and Data Engineering, vol. 26, no. 4, pp. 984-996, 2014. [Online]. Available: https://ieeexplore.ieee.org/document/6494573

X. Huang, A. D. Joseph, B. Nelson, B. I. P. Rubinstein, and J. D. Tygar, "Adversarial machine learning," Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, 2011. [Online]. Available: https://dl.acm.org/doi/10.1145/2046684.2046692

N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, "Practical black-box attacks against machine learning," Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security. [Online]. Available: https://dl.acm.org/doi/10.1145/3052973.3053009

D. Rosch-Grace and J. Straub, "Considering the Implications of Artificial Intelligence, Quantum Computing, and Cybersecurity," IEEE Conference Publication, 2022. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10216430

D. A. Alber et al., "Medical large language models are vulnerable to data-poisoning attacks," Nature Medicine, 2024. [Online]. Available: https://www.nature.com/articles/s41591-024-03445-1

Downloads

Published

07-02-2025

Issue

Section

Research Articles

How to Cite

Securing the Generative Frontier: A Systematic Analysis of Training Data Poisoning and Prompt Engineering Vulnerabilities in Large Language Models. (2025). International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(1), 1765-1775. https://doi.org/10.32628/CSEIT251112159