Dynamic Scaling of AI-Driven Data Platforms: Resource Management for Generative AI Workloads
DOI:
https://doi.org/10.32628/CSEIT251112112Keywords:
Dynamic Resource Management, Generative AI Workloads, Cloud Computing Optimization, Platform-Specific Scaling, AI Infrastructure ManagementAbstract
This article comprehensively analyzes dynamic scaling mechanisms for AI-driven data platforms, focusing on resource management challenges in generative AI workloads. The article examines the limitations of traditional scaling approaches and proposes a novel algorithm that combines predictive and reactive elements for optimal resource allocation. This article demonstrates significant improvements in resource utilization, cost efficiency, and system performance through an extensive evaluation across multiple cloud platforms, including Databricks and Amazon EMR. The article provides detailed platform-specific optimization strategies and implementation guidelines, offering organizations a robust framework for deploying and managing AI workloads in cloud environments.
Downloads
References
H. Ali, et al., "Global Adoption of Generative AI: What Matters Most?," Journal of Economy and Technology, vol. 12, no. 4, pp. 156-173, Oct. 2024. Available: https://www.sciencedirect.com/science/article/pii/S2949948824000520
K. Randhi and S. R. Bandarapu, "Efficient resource allocation for generative AI workloads in cloud-native infrastructures: A multi-tiered approach," International Journal of Science and Research Archive, vol. 13, no. 2, pp. 826-839, Nov. 2024. Available: https://ijsra.net/sites/default/files/IJSRA-2024-2208.pdf
P. Murthy and S. Bobba, "AI-Powered Predictive Scaling in Cloud Computing: Enhancing Efficiency through Real-Time Workload Forecasting," International Research Journal of Engineering and Technology, vol. 5, no. 4, Oct. 2021. Available: https://www.irejournals.com/formatedpaper/17029432.pdf
K. Chouhan et al., "Comprehensive Analysis of Artificial Intelligence with Human Resources Management," ResearchGate, Mar. 2021. Available: https://www.researchgate.net/publication/353807927_Comprehensive_Analysis_of_Artificial_Intelligence_with_Human_Resources_Management
Aquasec, "What Are AI Workloads?," Cloud Native Academy, Technical Report, pp. 1-28, 2024. Available: https://www.aquasec.com/cloud-native-academy/cspm/ai-workloads/
S. R. Mallreddy, "AI-Driven Orchestration: Enhancing Software Deployment Through Intelligent Automation And Machine Learning," ResearchGate Technical Report, pp. 1-45, Jan. 2021. Available: https://www.researchgate.net/publication/387223673_Ai-Driven_Orchestration_Enhancing_Software_Deployment_Through_Intelligent_Automation_And_Machine_Learning
P. M. Dhulavvagol, V. H. Bhoyar, and S. Shastri, "Performance Analysis of Distributed Processing System using Shard Selection Techniques on Elasticsearch," Procedia Computer Science, vol. 167, pp. 1626-1635, 2020. Available: https://www.sciencedirect.com/science/article/pii/S1877050920308395
S. Henning, "Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices," Doctoral Dissertation, University of Kiel, 2023. Available: https://oceanrep.geomar.de/id/eprint/58268/1/Dissertation_Soeren_Henning.pdf
S. Eeti, P. Kumar, and R. Singh, "Scalability And Performance Optimization In Distributed Systems: Exploring Techniques To Enhance The Scalability And Performance Of Distributed Computing Systems," International Journal of Creative Research Thoughts, vol. 11, no. 5, pp. 234-249, May 2023. Available: https://www.ijcrt.org/papers/IJCRT23A5530.pdf
Shantanu Kumar et al., "Resource Management in AI-Enabled Cloud Native Databases: A Systematic Literature Review Study," ResearchGate Technical Report, pp. 1-42, 2024. Available: https://www.researchgate.net/publication/381480037_Resource_Management_in_AI-Enabled_Cloud_Native_Databases_A_Systematic_Literature_Review_Study
L. Tucci, "What is enterprise AI? A complete guide for businesses," TechTarget Enterprise AI Guide, Oct. 2024. Available: https://www.techtarget.com/searchenterpriseai/Ultimate-guide-to-artificial-intelligence-in-the-enterprise
L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization Methods for Large-Scale Machine Learning," SIAM Review, vol. 60, no. 2, pp. 223-311, 2018. Available: https://epubs.siam.org/doi/abs/10.1137/16M1080173?journalCode=siread
Downloads
Published
Issue
Section
License
Copyright (c) 2025 International Journal of Scientific Research in Computer Science, Engineering and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.