Real-Time AI: Building Intelligent Stream Processing Pipelines

Authors

  • Himanshu Adhwaryu George Washington University, USA Author

DOI:

https://doi.org/10.32628/CSEIT251112313

Keywords:

Artificial Intelligence, Machine Learning, Real-time Processing, Stream Computing, System Architecture

Abstract

This comprehensive technical article explores the evolution and implementation of real-time AI systems through stream processing pipelines. It explores the transformation from traditional batch processing to dynamic, real-time intelligence, highlighting the architectural components, implementation patterns, and industry applications. The article discusses critical aspects of stream processing foundations, AI/ML integration layers, and real-time inference engines while addressing challenges in feature engineering, model deployment, and system monitoring. Through detailed analysis of applications across financial services, healthcare, and insurance sectors, the article demonstrates how organizations leverage real-time AI to achieve significant improvements in operational efficiency, decision-making capabilities, and customer service delivery. The article also explores emerging trends, including edge computing integration, federated learning, and quantum computing applications, providing insights into the future direction of real-time AI systems.

Downloads

Download data is not yet available.

References

Erum Mehmood, et al., "Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review," IEEE Internet of Things Journal, vol. 7, no. 9, pp. 8132-8147 2020. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9126812

Wazir Zada Khan, et al., "Edge computing: A survey," Future Generation Computer Systems,Volume 97, August 2019, Pages 219-235. Available: https://www.sciencedirect.com/science/article/abs/pii/S0167739X18319903

Muneer Ahmed Salamkar, et al., "Batch vs. Stream Processing: In-depth Comparison of Technologies, with Insights on Selecting the Right Approach for Specific Use Cases," Distributed Learning and Broad Applications in Scientific Research, 2020. Available: https://dlabi.org/index.php/journal/article/view/245/237

Han Wu, et al., "A Reactive Batching Strategy of Apache Kafka for Reliable Stream Processing in Real-time," IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), 2020. Available: https://ieeexplore.ieee.org/abstract/document/9251089

Sagar Vishnubhai Sheta, et al.,, "Developing Efficient Server Monitoring Systems Using Ai For Real-Time Data Processing," International Journal of Engineering and Technology Research (IJETR), Volume 8, Issue 1, January-December 2023. Available: https://iaeme.com/MasterAdmin/Journal_uploads/IJETR/VOLUME_8_ISSUE_1/IJETR_08_01_002.pdf

Kennedy Chinedu Okafor, et al., "Cyber-physical network architecture for data stream provisioning in complex ecosystems," Transactions on Emerging Telecommunications Technologies published by John Wiley & Sons, Ltd., 2021. Available: https://onlinelibrary.wiley.com/doi/full/10.1002/ett.4407

Alhassan Mumuni, et al., "Automated data processing and feature engineering for deep learning and big data applications: A survey," Journal of Information and Intelligence, Available online 8 January 2024. Available: https://www.sciencedirect.com/science/article/pii/S2949715924000027

Bernhard Klöter, et al., "Application of machine learning for production optimization," IEEE 7th World Conference on Photovoltaic Energy Conversion (WCPEC) (A Joint Conference of 45th IEEE PVSC, 28th PVSEC & 34th EU PVSEC), 2018. Available: https://ieeexplore.ieee.org/document/8547467

Mahmoud Nasr, et al., "Smart Healthcare in the Age of AI: Recent Advances, Challenges, and Future Prospects," IEEE Access ( Volume: 9), 2021. Available: https://ieeexplore.ieee.org/abstract/document/9565155

Paresh Vaak, et al., "AI-driven Transformation in the Insurance Industry," LTIMindtree Technical Report, pp. 1-28, 2019. Available: https://www.ltimindtree.com/wp-content/uploads/2019/01/AI-driven-Transformation-in-the-Insurance-Industry.pdf

Christian Jacobi, et al., "Real-time AI for Enterprise Workloads: the IBM Telum Processor," IEEE Hot Chips 33 Symposium (HCS), 2021. Available: https://ieeexplore.ieee.org/document/9567422

Olayiwola Blessing Akinnagbe, "The Future of Artificial Intelligence: Trends and Predictions,"MJEAI Journal, 2024. Available: https://www.researchgate.net/publication/385890167_The_Future_of_Artificial_Intelligence_Trends_and_Predictions

Downloads

Published

10-02-2025

Issue

Section

Research Articles

How to Cite

Real-Time AI: Building Intelligent Stream Processing Pipelines. (2025). International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(1), 3042-3050. https://doi.org/10.32628/CSEIT251112313