Dynamic Scaling of AI-Driven Data Platforms: Resource Management for Generative AI Workloads

Authors

  • Mouna Reddy Mekala Cloudwick, USA Author

DOI:

https://doi.org/10.32628/CSEIT251112112

Keywords:

Dynamic Resource Management, Generative AI Workloads, Cloud Computing Optimization, Platform-Specific Scaling, AI Infrastructure Management

Abstract

This article comprehensively analyzes dynamic scaling mechanisms for AI-driven data platforms, focusing on resource management challenges in generative AI workloads. The article examines the limitations of traditional scaling approaches and proposes a novel algorithm that combines predictive and reactive elements for optimal resource allocation. This article demonstrates significant improvements in resource utilization, cost efficiency, and system performance through an extensive evaluation across multiple cloud platforms, including Databricks and Amazon EMR. The article provides detailed platform-specific optimization strategies and implementation guidelines, offering organizations a robust framework for deploying and managing AI workloads in cloud environments.

Downloads

Download data is not yet available.

References

H. Ali, et al., "Global Adoption of Generative AI: What Matters Most?," Journal of Economy and Technology, vol. 12, no. 4, pp. 156-173, Oct. 2024. Available: https://www.sciencedirect.com/science/article/pii/S2949948824000520

K. Randhi and S. R. Bandarapu, "Efficient resource allocation for generative AI workloads in cloud-native infrastructures: A multi-tiered approach," International Journal of Science and Research Archive, vol. 13, no. 2, pp. 826-839, Nov. 2024. Available: https://ijsra.net/sites/default/files/IJSRA-2024-2208.pdf

P. Murthy and S. Bobba, "AI-Powered Predictive Scaling in Cloud Computing: Enhancing Efficiency through Real-Time Workload Forecasting," International Research Journal of Engineering and Technology, vol. 5, no. 4, Oct. 2021. Available: https://www.irejournals.com/formatedpaper/17029432.pdf

K. Chouhan et al., "Comprehensive Analysis of Artificial Intelligence with Human Resources Management," ResearchGate, Mar. 2021. Available: https://www.researchgate.net/publication/353807927_Comprehensive_Analysis_of_Artificial_Intelligence_with_Human_Resources_Management

Aquasec, "What Are AI Workloads?," Cloud Native Academy, Technical Report, pp. 1-28, 2024. Available: https://www.aquasec.com/cloud-native-academy/cspm/ai-workloads/

S. R. Mallreddy, "AI-Driven Orchestration: Enhancing Software Deployment Through Intelligent Automation And Machine Learning," ResearchGate Technical Report, pp. 1-45, Jan. 2021. Available: https://www.researchgate.net/publication/387223673_Ai-Driven_Orchestration_Enhancing_Software_Deployment_Through_Intelligent_Automation_And_Machine_Learning

P. M. Dhulavvagol, V. H. Bhoyar, and S. Shastri, "Performance Analysis of Distributed Processing System using Shard Selection Techniques on Elasticsearch," Procedia Computer Science, vol. 167, pp. 1626-1635, 2020. Available: https://www.sciencedirect.com/science/article/pii/S1877050920308395

S. Henning, "Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices," Doctoral Dissertation, University of Kiel, 2023. Available: https://oceanrep.geomar.de/id/eprint/58268/1/Dissertation_Soeren_Henning.pdf

S. Eeti, P. Kumar, and R. Singh, "Scalability And Performance Optimization In Distributed Systems: Exploring Techniques To Enhance The Scalability And Performance Of Distributed Computing Systems," International Journal of Creative Research Thoughts, vol. 11, no. 5, pp. 234-249, May 2023. Available: https://www.ijcrt.org/papers/IJCRT23A5530.pdf

Shantanu Kumar et al., "Resource Management in AI-Enabled Cloud Native Databases: A Systematic Literature Review Study," ResearchGate Technical Report, pp. 1-42, 2024. Available: https://www.researchgate.net/publication/381480037_Resource_Management_in_AI-Enabled_Cloud_Native_Databases_A_Systematic_Literature_Review_Study

L. Tucci, "What is enterprise AI? A complete guide for businesses," TechTarget Enterprise AI Guide, Oct. 2024. Available: https://www.techtarget.com/searchenterpriseai/Ultimate-guide-to-artificial-intelligence-in-the-enterprise

L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization Methods for Large-Scale Machine Learning," SIAM Review, vol. 60, no. 2, pp. 223-311, 2018. Available: https://epubs.siam.org/doi/abs/10.1137/16M1080173?journalCode=siread

Downloads

Published

26-01-2025

Issue

Section

Research Articles

How to Cite

Dynamic Scaling of AI-Driven Data Platforms: Resource Management for Generative AI Workloads. (2025). International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(1), 1147-1157. https://doi.org/10.32628/CSEIT251112112