Lakehouse Architecture: Bridging On-Prem Big Data and Modern Data Platforms

Authors

  • Sankara Reddy Thamma   Deloitte Consulting LLP, USA

DOI:

https://doi.org/10.32628/CSEIT238325

Keywords:

Lakehouse Architecture, Data Management, Big Data, Analytics and AI, Scalability and Flexibility

Abstract

The evolving data landscape demands a scalable, flexible, and cost-effective solution for efficient data management. The Lakehouse architecture merges the benefits of data lakes and data warehouses, providing a unified data platform for analytics and AI. This paper explores the transition from traditional on-premises big data environments to Lakehouse architecture, highlighting its unique capabilities, benefits, and real-world implementations.

References

  1. Armbrust, M., et al., "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores," Proceedings of the VLDB Endowment, 2019.
  2. Xin, R., et al., "Unified Analytics Infrastructure for Big Data and AI," Databricks Engineering Blog, 2020.
  3. Ghodsi, A., et al., "The Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics," Databricks Whitepaper, 2020.
  4. Chen, J., et al., "Apache Iceberg: A Table Format for Huge Analytic Datasets," Proceedings of the VLDB Endowment, 2020.
  5. Lin, J., and Dyer, C., "Data-Intensive Text Processing with MapReduce," Morgan & Claypool Publishers, 2010.
  6. Zaharia, M., et al., "Apache Spark: A Unified Engine for Big Data Processing," Communications of the ACM, 2016.
  7. Malik, R., et al., "Migrating On-Premises Data Pipelines to Cloud-Native Architectures," IEEE Cloud Computing, vol. 7, no. 1, 2020.
  8. Krishnan, S., et al., "Data Lake Management: Challenges and Solutions," IEEE Data Engineering Bulletin, vol. 42, no. 2, 2019.
  9. Vartak, M., et al., "ModelDB: A System for Machine Learning Model Management," Proceedings of the ACM Symposium on Cloud Computing, 2016.
  10. Halevy, A., et al., "Data Integration: The Teenage Years," Proceedings of the VLDB Endowment, 2006.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2023-08-30

Issue

Volume 9, Issue 4, July-August-2023

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
Sankara Reddy Thamma , " Lakehouse Architecture: Bridging On-Prem Big Data and Modern Data Platforms" International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 9, Issue 4, pp.495-499, July-August-2023. Available at doi : https://doi.org/10.32628/CSEIT238325