Real-Time Data Processing: Challenges and Innovations

Authors

  • N V Rama Sai Chalapathi Gupta Lakkimsetty   Independent Researcher, USA

Keywords:

Data-Critical Applications, Wireless HART Networks, Managing Workloads, Reliable Performance, Adaptive Resource Allocation, Pipelines, Real-Time Task, Resource Consumption, Dynamic Scheduling, Algorithms, Generation Clouds, Mechanisms.

Abstract

Effective processing of information pipelines are now required in complicated contemporary cloud systems due to the growth of data-critical applications. These pipelines serve as the foundation for processing enormous amounts of data, enabling big businesses to make decisions in real time and analyse data. The techniques for addressing the issues of latency, cost, and consumption of resources brought about by big data are discussed in this study. This field focusses on methodical architectural enhancements that increase efficiency, such parallelism, proactive self-scheduling, and AI's capacity to adapt to changing workloads. A study of the literature looks at these models and shows how inadequate they are at handling workload demands in clouds of the present generation, which is why a change is required. Because industrial settings are dynamic and have strict timing constraints, real-time job scheduling in Wired HART networks is very difficult. In order to guarantee dependable performance and effective operation in Wireless HART networks, this research explores the optimisation of real-time job scheduling. Key obstacles such resource limitations, fluctuating network circumstances, or the need for rapid job execution are identified in the research. A number of solutions are put forward, such as adaptive resource allocation techniques and dynamic scheduling algorithms. Performance assessment shows how well these strategies work to satisfy demands in real time while making the best use of available resources. The results further the development and deployment of reliable scheduling systems for industrial Internet of things applications across Wireless HART networks.

References

  1. X. Lian and L. Chen, ‘‘Similarity joins processing on uncertain data streams,’’ IEEE Trans. Knowl. Data Eng., vol. 23, no. 11, pp. 1718–1734, Nov. 2011.
  2. Y.-H. Jeon, K.-H. Lee, and H.-J. Kim, ‘‘Distributed join processing between streaming and stored big data under the micro-batch model,’’ IEEE Access, vol. 7, pp. 34583–34598, 2019.
  3. S. Dolev, P. Florissi, E. Gudes, S. Sharma, and I. Singer, ‘‘A survey on geographically distributed big-data processing using MapReduce,’’ IEEE Trans. Big Data, vol. 5, no. 1, pp. 60–80, Mar. 2019.
  4. E. Mehmood and M. A. Naeem, ‘‘Optimization of cache-based semistream joins,’’ in Proc. IEEE 2nd Int. Conf. Cloud Comput. Big Data Anal. (ICCCBDA), Apr. 2017, pp. 76–81.
  5. M. A. Naeem, I. S. Bajwa, and N. Jamil, ‘‘A cached-based approach to enrich stream data with master data,’’ in Proc. 10th Int. Conf. Digit. Inf. Manage. (ICDIM), Oct. 2015, pp. 57–62.
  6. E. Mehmood and T. Anees, ‘‘Performance analysis of not only SQL semi-stream join using MongoDB for real-time data warehousing,’’ IEEE Access, vol. 7, pp. 134215–134225, 2019.
  7. A. Wibowo, ‘‘Problems and available solutions on the stage of extract, transform, and loading in near real-time data warehousing (a literature study),’’ in Proc. Int. Seminar Intell. Technol. Appl. (ISITIA), May 2015, pp. 345–350.
  8. M. A. Naeem, ‘‘A robust join operator to process streaming data in real-time data warehousing,’’ in Proc. 8th Int. Conf. Digit. Inf. Manage. (ICDIM), Sep. 2013, pp. 119–124.
  9. F. Majeed, S. Mahmood, S. Ubaid, N. Khalil, S. Siddiqi, and F. Ashraf, ‘‘A burst resolution technique for data streams management in the realtime data warehouse,’’ in Proc. 7th Int. Conf. Emerg. Technol., Sep. 2011, pp. 1–5.
  10. R. J. Santos, J. Bernardino, and M. Vieira, ‘‘24/7 real-time data warehousing: A tool for continuous actionable knowledge,’’ in Proc. IEEE 35th Annu. Comput. Softw. Appl. Conf., Jul. 2011, pp. 279–288.
  11. A. R. Ali, ‘‘Real-time big data warehousing and analysis framework,’’ in Proc. IEEE 3rd Int. Conf. Big Data Anal. (ICBDA), Mar. 2018, pp. 43–49.
  12. R. Mukherjee and P. Kar, ‘‘A comparative review of data warehousing ETL tools with new trends and industry insight,’’ in Proc. IEEE 7th Int. Advance Comput. Conf. (IACC), Jan. 2017, pp. 943–948.
  13. Hamdi, E. Bouazizi, S. Alshomrani, and J. Feki, ‘‘2LPA-RTDW: A two-level data partitioning approach for real-time data warehouse,’’ in Proc. IEEE/ACIS 14th Int. Conf. Comput. Inf. Sci. (ICIS), Jun. 2015, pp. 632–638.
  14. Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Khan, S. U. (2015). The rise of “big data” on cloud computing: Review and open research issues. Information systems, 47, 98-115.
  15. Dobre, C., & Xhafa, F. (2014). Parallel programming paradigms and frameworks in big data era. International Journal of Parallel Programming, 42(5), 710-738.
  16. Zhang, W., Tooker, N. B., & Mueller, A. V. (2020). Enabling wastewater treatment process automation: leveraging innovations in real-time sensing, data analysis, and online controls. Environmental Science: Water Research & Technology, 6(11), 2973-2992.
  17. Naphade, M., Banavar, G., Harrison, C., Paraszczak, J., & Morris, R. (2011). Smarter cities and their innovation challenges. Computer, 44(6), 32-39.
  18. Baillieul, J., & Antsaklis, P. J. (2007). Control and communication challenges in networked real-time systems. Proceedings of the IEEE, 95(1), 9-28.
  19. Pigni, F., Piccoli, G., & Watson, R. (2016). Digital data streams: Creating value from the real-time flow of big data. California Management Review, 58(3), 5-25.
  20. Haße, H., Li, B., Weißenberg, N., Cirullies, J., & Otto, B. (2019). Digital twin for real-time data processing in logistics. In Artificial Intelligence and Digital Transformation in Supply Chain Management: Innovative Approaches for Supply Chains. Proceedings of the Hamburg International Conference of Logistics (HICL), Vol. 27 (pp. 4-28). Berlin: epubli GmbH.
  21. Kambatla, K., Kollias, G., Kumar, V., & Grama, A. (2014). Trends in big data analytics. Journal of parallel and distributed computing, 74(7), 2561-2573.
  22. Demchenko, Y., Turkmen, F., de Laat, C., Hsu, C. H., Blanchet, C., & Loomis, C. (2017). Cloud computing infrastructure for data intensive applications. In Big Data Analytics for Sensor-Network Collected Intelligence (pp. 21-62). Academic Press.
  23. Devan, M., Shanmugam, L., & Tomar, M. (2021). AI-powered data migration strategies for cloud environments: Techniques, frameworks, and real-world applications. Australian Journal of Machine Learning Research & Applications, 1(2), 79-111.
  24. Kimovski, D., Bauer, C., Mehran, N., & Prodan, R. (2022, June). Big Data Pipeline Scheduling and Adaptation on the Computing Continuum. In 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC) (pp. 1153-1158). IEEE.
  25. Selvarajan, G. P. Harnessing AI-Driven Data Mining for Predictive Insights: A Framework for Enhancing Decision-Making in Dynamic Data Environments.
  26. Dalsaniya, N. A. (2022). Cognitive Robotic Process Automation (RPA) for Processing Unstructured Data. International Journal of Science and Research Archive, 7(2), 639-643.
  27. Gayam, R. R. (2021). Optimizing Supply Chain Management through Artificial Intelligence: Techniques for Predictive Maintenance, Demand Forecasting, and Inventory Optimization. Journal of AI-Assisted Scientific Discovery, 1(1), 129-144.
  28. Pureti, N. (2022). The Art of Social Engineering: How Hackers Manipulate Human Behavior. International Journal of Machine Learning Research in Cybersecurity and Artificial Intelligence, 13(1), 19-34.
  29. Gayam, R. R. (2021). Artificial Intelligence in Healthcare: Advanced Algorithms for Predictive Diagnosis, Personalized Treatment, and Outcome Prediction. Australian Journal of Machine Learning Research & Applications, 1(1), 113-131.
  30. Maruthi, S., Dodda, S. B., Yellu, R. R., Thuniki, P., & Reddy, S. R. B. (2021). Deconstructing the Semantics of Human-Centric AI: A Linguistic Analysis. Journal of Artificial Intelligence Research and Applications, 1(1), 11-30.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2022-11-30

Issue

Volume 8, Issue 6, November-December-2022

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
N V Rama Sai Chalapathi Gupta Lakkimsetty , " Real-Time Data Processing: Challenges and Innovations" International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 8, Issue 6, pp.716-724, November-December-2022.