Integration of Business Intelligence in Data Lake Solutions
DOI:
https://doi.org/10.32628/CSEIT2410612412Keywords:
Business Intelligence, Big Data, Distributed Systems, Cloud EnvironmentAbstract
Data lakes with BI allows organizations to effectively navigate the advantages of unstructured, semi unstructured and structured data. This paper therefore focuses on BI technologies in data lakes, with especial consideration to the challenges, integration techniques, and the technologies that enable appropriate interoperation. Successful BI in data lakes: Hadoop and Spark as distributed computing frameworks; cloud platforms; and data integration tools. However, the need to develop suitable solutions for integrating such enterprise applications is still a work in progress because of issues like data quality, data governance, and data accessibility on the same.
Downloads
References
Deekshith, A. (2022). Cross-Disciplinary Approaches: The Role of Data Science in Developing AI-Driven Solutions for Business Intelligence. International Machine learning journal and Computer Engineering, 5(5). https://mljce.in/index.php/Imljce/article/view/37
Derakhshannia, M., Gervet, C., Hajj-Hassan, H., Laurent, A., & Martin, A. (2020). Data lake governance: Towards a systemic and natural ecosystem analogy. Future internet, 12(8), 126. https://doi.org/10.3390/fi12080126
Hai, R., Koutras, C., Quix, C., & Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12571-12590. 10.1109/TKDE.2023.3270101
Hosen, M. S., Islam, R., Naeem, Z., Folorunso, E. O., Chu, T. S., Al Mamun, M. A., & Orunbon, N. O. (2024). Data-Driven Decision Making: Advanced Database Systems for Business Intelligence. Nanotechnology Perceptions, 20(3), 687-704. https://www.researchgate.net/profile/Esther-Folorunso-3/publication/382695215_Data-Driven_Decision_Making_Advanced_Database_Systems_for_Business_Intelligence/links/66a9e30775fcd863e5eae75d/Data-Driven-Decision-Making-Advanced-Database-Systems-for-Business-Intelligence.pdf
Kachaoui, J., Larioui, J., & Belangour, A. (2020). Towards an ontology proposal model in data lake for real-time COVID-19 cases prevention. https://www.learntechlib.org/p/217989/.
Katari, A. (2022). Data lakes and Optimizing Query. Available at SSRN. https://ssrn.com/abstract=
Katari, A., & Kalla, D. (2021). Cost Optimization in Cloud-Based Financial Data Lakes: Techniques and Case Studies. ESP Journal of Engineering & Technology Advancements (ESP-JETA), 1(1), 150-157. https://doi.org/10.56472/25832646/ESP-V1I1P116
Manchana, R. (2022). The Power of Cloud-Native Solutions for Descriptive Analytics: Unveiling Insights from Data. Journal of Artificial Intelligence & Cloud Computing. SRC/JAICCE139. DOI: https://doi.org/10.47363/JAICC/2022
Manchana, R. (2023). Building a Modern Data Foundation in the Cloud: Data Lakes and Data Lakehouses as Key Enablers. J Artif Intell Mach Learn & Data Sci, 1(1), 1098-1108. https://doi.org/10.51219/JAIMLD/Ramakrishna-manchana/260
Michael, C. I., Ipede, O. J., Adejumo, A. D., Adenekan, I. O., Adebayo Damilola, O. A., & Ayodele, P. A. (2024). Data-driven decision making in IT: Leveraging AI and data science for business intelligence. World Journal of Advanced Research and Reviews, 23(01), 432-439. https://doi.org/10.30574/wjarr.2024.23.1.2010
Nuthalapati, A. (2024). Architecting data lake-houses in the cloud: Best practices and future directions. Int. J. Sci. Res. Arch, 12(2), 1902-1909. https://doi.org/10.30574/ijsra.2024.12.2.1466
Naveen Bagam, International Journal of Computer Science and Mobile Computing, Vol.13 Issue.11, November- 2024, pg. 6-27
Naveen Bagam. (2024). Optimization of Data Engineering Processes Using AI. International Journal of Research Radicals in Multidisciplinary Fields, ISSN: 2960-043X, 3(1), 20–34. Retrieved from https://www.researchradicals.com/index.php/rr/article/view/138
Naveen Bagam. (2024). Machine Learning Models for Customer Segmentation in Telecom. Journal of Sustainable Solutions, 1(4), 101–115. https://doi.org/10.36676/j.sust.sol.v1.i4.42
Bagam, N. (2023). Implementing Scalable Data Architecture for Financial Institutions. Stallion Journal for Multidisciplinary Associated Research Studies, 2(3), 27
Bagam, N. (2021). Advanced Techniques in Predictive Analytics for Financial Services. Integrated Journal for Research in Arts and Humanities, 1(1), 117–126. https://doi.org/10.55544/ijrah.1.1.16
Enhancing Data Pipeline Efficiency in Large-Scale Data Engineering Projects. (2019). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 7(2), 44- Sai Krishna Shiramshetty. (2024). Enhancing SQL Performance for Real-Time Business Intelligence Applications. International Journal of Multidisciplinary Innovation and Research Methodology, ISSN: 2960-2068, 3(3), 282–297. Retrieved from https://ijmirm.com/index.php/ijmirm/article/view/138
Sai Krishna Shiramshetty, "Big Data Analytics in Civil Engineering : Use Cases and Techniques", International Journal of Scientific Research in Civil Engineering (IJSRCE), ISSN : 2456-6667, Volume 3, Issue 1, pp.39-46, January-February.2019
URL : https://ijsrce.com/IJSRCE19318
Sai Krishna Shiramshetty, " Data Integration Techniques for Cross-Platform Analytics, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 6, Issue 4, pp.593-599, July-August-2020. Available at doi : https://doi.org/10.32628/CSEIT2064139
Shiramshetty, S. K. (2021). SQL BI Optimization Strategies in Finance and Banking. Integrated Journal for Research in Arts and Humanities, 1(1), 106–116. https://doi.org/10.55544/ijrah.1.1.15
Sai Krishna Shiramshetty. (2022). Predictive Analytics Using SQL for Operations Management. Eduzone: International Peer Reviewed/Refereed Multidisciplinary Journal, 11(2), 433–448. Retrieved from https://eduzonejournal.com/index.php/eiprmj/article/view/693
Shiramshetty, S. K. (2023). Data warehousing solutions for business intelligence. International Journal of Computer Science and Mobile Computing, 12(3), 49–62. https://ijcsmc.com/index.php/volume-12-issue-3-march-2023/
Sai Krishna Shiramshetty. (2024). Comparative Study of BI Tools for Real-Time Analytics. International Journal of Research and Review Techniques, 3(3), 1–13. Retrieved from https://ijrrt.com/index.php/ijrrt/article/view/210
Sai Krishna Shiramshetty "Leveraging BI Development for Decision-Making in Large Enterprises" Iconic Research And Engineering Journals Volume 8 Issue 5 2024 Page 548-560
Sai Krishna Shiramshetty "Integrating SQL with Machine Learning for Predictive Insights" Iconic Research And Engineering Journals Volume 1 Issue 10 2018 Page 287-292
Shiramshetty, S. K. (2023). Advanced SQL Query Techniques for Data Analysis in Healthcare. Journal for Research in Applied Sciences and Biotechnology, 2(4), 248–258. https://doi.org/10.55544/jrasb.2.4.3357. https://ijope.com/index.php/home/article/view/166
Kola, H. G. (2024). Optimizing ETL Processes for Big Data Applications. International Journal of Engineering and Management Research, 14(5), 99–112. https://doi.org/10.5281/zenodo.14184235
SQL in Data Engineering: Techniques for Large Datasets. (2023). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 11(2), 36-51. https://ijope.com/index.php/home/article/view/165
Data Integration Strategies in Cloud-Based ETL Systems. (2023). International Journal of Transcontinental Discoveries, ISSN: 3006-628X, 10(1), 48-62. https://internationaljournals.org/index.php/ijtd/article/view/116
Harish Goud Kola. (2024). Real-Time Data Engineering in the Financial Sector. International Journal of Multidisciplinary Innovation and Research Methodology, ISSN: 2960-2068, 3(3), 382–396. Retrieved from https://ijmirm.com/index.php/ijmirm/article/view/143
Harish Goud Kola. (2022). Best Practices for Data Transformation in Healthcare ETL. Edu Journal of International Affairs and Research, ISSN: 2583-9993, 1(1), 57–73. Retrieved from https://edupublications.com/index.php/ejiar/article/view/106
Kola, H. G. (2018). Data warehousing solutions for scalable ETL pipelines. International Journal of Scientific Research in Science, Engineering and Technology, 4(8), 762. https://doi.org/10.1.1.123.4567
Harish Goud Kola, " Building Robust ETL Systems for Data Analytics in Telecom , IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 5, Issue 3, pp.694-700, May-June-2019. Available at doi : https://doi.org/10.32628/CSEIT1952292
Kola, H. G. (2022). Data security in ETL processes for financial applications. International Journal of Enhanced Research in Science, Technology & Engineering, 11(9), 55. https://ijsrcseit.com/CSEIT1952292.
Santhosh Bussa, "Advancements in Automated ETL Testing for Financial Applications", IJRAR - International Journal of Research and Analytical Reviews (IJRAR), E-ISSN 2348-1269, P- ISSN 2349-5138, Volume.7, Issue 4, Page No pp.426-443, November 2020, Available at : http://www.ijrar.org/IJRAR2AA1744.pdf
Bussa, S. (2023). Artificial Intelligence in Quality Assurance for Software Systems. Stallion Journal for Multidisciplinary Associated Research Studies, 2(2), 15–26. https://doi.org/10.55544/sjmars.2.2.2.
Bussa, S. (2021). Challenges and solutions in optimizing data pipelines. International Journal for Innovative Engineering and Management Research, 10(12), 325–341. https://sjmars.com/index.php/sjmars/article/view/116
Bussa, S. (2022). Machine Learning in Predictive Quality Assurance. Stallion Journal for Multidisciplinary Associated Research Studies, 1(6), 54–66. https://doi.org/10.55544/sjmars.1.6.8
Bussa, S. (2022). Emerging trends in QA testing for AI-driven software. International Journal of All Research Education and Scientific Methods (IJARESM, 10(11), 1712. Retrieved from http://www.ijaresm.com
Santhosh Bussa. (2024). Evolution of Data Engineering in Modern Software Development. Journal of Sustainable Solutions, 1(4), 116–130. https://doi.org/10.36676/j.sust.sol.v1.i4.43
Santhosh Bussa. (2024). Big Data Analytics in Financial Systems Testing. International Journal of Multidisciplinary Innovation and Research Methodology, ISSN: 2960-2068, 3(3), 506–521. Retrieved from https://ijmirm.com/index.php/ijmirm/article/view/150
Bussa, S. (2019). AI-driven test automation frameworks. International Journal for Innovative Engineering and Management Research, 8(10), 68–87. Retrieved from https://www.ijiemr.org/public/uploads/paper/427801732865437.pdf
Santhosh Bussa. (2023). Role of Data Science in Improving Software Reliability and Performance. Edu Journal of International Affairs and Research, ISSN: 2583-9993, 2(4), 95–111. Retrieved from https://edupublications.com/index.php/ejiar/article/view/111
Bussa, S. (2023). Enhancing BI tools for improved data visualization and insights. International Journal of Computer Science and Mobile Computing, 12(2), 70–92. https://doi.org/10.47760/ijcsmc.2023.v12i02.005
Annam, S. N. (2020). Innovation in IT project management for banking systems. International Journal of Enhanced Research in Science, Technology & Engineering, 9(10), 19. https://www.erpublications.com/uploaded_files/download/sri-nikhil-annam_gBNPz.pdf
Annam, S. N. (2018). Emerging trends in IT management for large corporations. International Journal of Scientific Research in Science, Engineering and Technology, 4(8), 770. https://ijsrset.com/paper/12213.pdf
Sri Nikhil Annam, " IT Leadership Strategies for High-Performance Teams, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 7, Issue 1, pp.302-317, January-February-2021. Available at doi : https://doi.org/10.32628/CSEIT228127
Annam, S. N. (2024). Comparative Analysis of IT Management Tools in Healthcare. Stallion Journal for Multidisciplinary Associated Research Studies, 3(5), 72–86. https://doi.org/10.55544/sjmars.3.5.9.
Annam, N. (2024). AI-Driven Solutions for IT Resource Management. International Journal of Engineering and Management Research, 14(6), 15–30. https://doi.org/10.31033/ijemr.14.6.15-30
Annam, S. N. (2022). Optimizing IT Infrastructure for Business Continuity. Stallion Journal for Multidisciplinary Associated Research Studies, 1(5), 31–42. https://doi.org/10.55544/sjmars.1.5.7
Sri Nikhil Annam , " Managing IT Operations in a Remote Work Environment, IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 8, Issue 5, pp.353-368, September-October-2022. https://ijsrcseit.com/paper/CSEIT23902179.pdf
Annam, S. (2023). Data security protocols in telecommunication systems. International Journal for Innovative Engineering and Management Research, 8(10), 88–106. https://www.ijiemr.org/downloads/paper/Volume-8/data-security-protocols-in-telecommunication-systems
Annam, S. N. (2023). Enhancing IT support for enterprise-scale applications. International Journal of Enhanced Research in Science, Technology & Engineering, 12(3), 205. https://www.erpublications.com/uploaded_files/download/sri-nikhil-annam_urfNc.pdf
Kola, H. G. (2024). Optimizing ETL Processes for Big Data Applications. International Journal of Engineering and Management Research, 14(5), 99–112. https://doi.org/10.5281/zenodo.14184235
SQL in Data Engineering: Techniques for Large Datasets. (2023). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 11(2), 36-51. https://ijope.com/index.php/home/article/view/165
Data Integration Strategies in Cloud-Based ETL Systems. (2023). International Journal of Transcontinental Discoveries, ISSN: 3006-628X, 10(1), 48-62. https://internationaljournals.org/index.php/ijtd/article/view/116
Harish Goud Kola. (2024). Real-Time Data Engineering in the Financial Sector. International Journal of Multidisciplinary Innovation and Research Methodology, ISSN: 2960-2068, 3(3), 382–396. Retrieved from https://ijmirm.com/index.php/ijmirm/article/view/143
Harish Goud Kola. (2022). Best Practices for Data Transformation in Healthcare ETL. Edu Journal of International Affairs and Research, ISSN: 2583-9993, 1(1), 57–73. Retrieved from https://edupublications.com/index.php/ejiar/article/view/106
Kola, H. G. (2018). Data warehousing solutions for scalable ETL pipelines. International Journal of Scientific Research in Science, Engineering and Technology, 4(8), 762. https://doi.org/10.1.1.123.4567
Harish Goud Kola, " Building Robust ETL Systems for Data Analytics in Telecom , IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 5, Issue 3, pp.694-700, May-June-2019. Available at doi : https://doi.org/10.32628/CSEIT1952292
Kola, H. G. (2022). Data security in ETL processes for financial applications. International Journal of Enhanced Research in Science, Technology & Engineering, 11(9), 55. https://ijsrcseit.com/CSEIT1952292.
Naveen Bagam. (2024). Data Integration Across Platforms: A Comprehensive Analysis of Techniques, Challenges, and Future Directions. International Journal of Intelligent Systems and Applications in Engineering, 12(23s), 902–919. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/7062
Naveen Bagam, Sai Krishna Shiramshetty, Mouna Mothey, Harish Goud Kola, Sri Nikhil Annam, & Santhosh Bussa. (2024). Advancements in Quality Assurance and Testing in Data Analytics. Journal of Computational Analysis and Applications (JoCAAA), 33(08), 860–878. Retrieved from https://eudoxuspress.com/index.php/pub/article/view/1487
Bagam, N., Shiramshetty, S. K., Mothey, M., Kola, H. G., Annam, S. N., & Bussa, S. (2024). Optimizing SQL for BI in diverse engineering fields. International Journal of Communication Networks and Information Security, 16(5). https://ijcnis.org/
Bagam, N., Shiramshetty, S. K., Mothey, M., Annam, S. N., & Bussa, S. (2024). Machine Learning Applications in Telecom and Banking. Integrated Journal for Research in Arts and Humanities, 4(6), 57–69. https://doi.org/10.55544/ijrah.4.6.8
Bagam, N., Shiramshetty, S. K., Mothey, M., Kola, H. G., Annam, S. N., & Bussa, S. (2024). Collaborative approaches in data engineering and analytics. International Journal of Communication Networks and Information Security, 16(5). https://ijcnis.org/
Kulkarni, A. (2024). Natural Language Processing for Text Analytics in SAP HANA. International Journal of Multidisciplinary Innovation and Research Methodology (IJMIRM), ISSN, 2960-2068. https://scholar.google.com/scholar?oi=bibs&cluster=15918532763612424504&btnI=1&hl=en
Kulkarni, A. (2024). Digital Transformation with SAP Hana. International Journal on Recent and Innovation Trends in Computing and Communication ISSN, 2321-8169. https://scholar.google.com/scholar?cluster=12193741245105822786&hl=en&as_sdt=2005
Kulkarni, A. (2024). Enhancing Customer Experience with AI-Powered Recommendations in SAP HANA. International Journal of Business Management and Visuals, ISSN, 3006-2705. https://scholar.google.com/scholar?cluster=8922856457601624723&hl=en&as_sdt=2005&as_ylo=2024&as_yhi=2024
Kulkarni, A. (2024). Generative AI-Driven for SAP Hana Analytics. International Journal on Recent and Innovation Trends in Computing and Communication, 12(2), 438-444. https://scholar.google.com/scholar?cluster=10311553701865565222&hl=en&as_sdt=2005
S. Dodda, "Exploring Variational Autoencoders and Generative Latent Time-Series Models for Synthetic Data Generation and Forecasting," 2024 Control Instrumentation System Conference (CISCON), Manipal, India, 2024, pp. 1-6, doi: 10.1109/CISCON62171.2024.10696588.
S. Dodda, "Enhancing Foreground-Background Segmentation for Indoor Autonomous Navigation using Superpixels and Decision Trees," 2024 Control Instrumentation System Conference (CISCON), Manipal, India, 2024, pp. 1-7, doi: 10.1109/CISCON62171.2024.10696719.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 International Journal of Scientific Research in Computer Science, Engineering and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.