CICD Lakehouse Real-Time Analytics on Azure Databricks
DOI:
https://doi.org/10.5281/zenodo.22874645Keywords:
CI/CD for Data Engineering, DataOps Frameworks, MLOps Pipelines, DevSecOps Integration, Infrastructure as Code (IaC), Data Lakehouse Architecture, Data Mesh Adoption, Azure Databricks Pipelines, ETL Pipeline Automation, Data Pipeline Orchestration, Near Real-Time Analytics, Streaming Data Processing, Data Availability Optimization, Data Quality Governance, Secure Data Engineering, Observability in Data Systems, Cloud Data Platforms, Automated Data Deployment, Enterprise Data Engineering, Scalable Analytics Systems.Abstract
This paper presents an end-to-end Continuous Integration/Continuous Deployment (CI/CD) framework for data engineering, automating the full lifecycle from source control through development, testing, deployment, and monitoring. By incorporating Infrastructure as Code (IaC), DataOps, and MLOps practices, the framework improves collaboration, simplifies provisioning, and strengthens observability, while maintaining consistent quality, security, and compliance standards.
The proposed architecture underpins a production-grade data engineering pipeline for Zoomcar on Azure Databricks, adopting a demand-based approach that prioritizes data availability over access latency in support of Data Mesh principles. Cloud-based data and analytics platforms enable advanced analytics, AI, and machine learning on near-real-time streaming data, driving enterprise decision-making.
While CI/CD with DevSecOps is now standard in software and application development, extending these practices to ETL pipelines for MLOps and DataOps enables faster, more reliable delivery of production-ready code in cloud-based data engineering environments. This shift reflects a broader principle: on-demand data availability increasingly outweighs strict real-time delivery. A well-governed Data Lakehouse, structured around Data Mesh architecture and prioritizing availability, quality, and security over streaming latency, provides the foundation needed to realize a truly data-driven enterprise.
References
1. Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., Torres, J., van Hovell, H., Ionescu, A., Łuszczak, A., Świtakowski, M., Szafrański, M., Li, X., Ueshin, T., Mokhtar, M., Boncz, P., Ghodsi, A., Paranjpye, S., Senster, P., Xin, R., & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
2. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In Conference on Innovative Data Systems Research (CIDR).
3. Yandamuri, U. S. (2022). Cloud-Based Data Integration Architectures for Scalable Enterprise Analytics. International Journal of Intelligent Systems and Applications in Engineering, 10, 472-483.
4. Chen, A., Chow, A., Davidson, A., DCunha, A., Ghodsi, A., Hong, S. A., Konwinski, A., Mewald, C., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., Singh, A., Xie, F., Zaharia, M., Zang, R., Zheng, J., & Zumar, C. (2020). Developments in MLflow: A system to accelerate the machine learning lifecycle. In Proceedings of the 4th International Workshop on Data Management for End-to-End Machine Learning.
5. Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., & Xin, R. (2021). Accelerating data and AI workflows with the lakehouse architecture. Communications of the ACM, 64(9), 44–52.
6. Kleppmann, M. (2021). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O’Reilly Media.
7. Mattaparthi, R. (2021). Unified Data Lineage and Quality Governance Framework for Multi-Source Sensor Streams in Heavy-Duty Powertrain Manufacturing. Online Journal of Mechanical Engineering, 1(1), 1-15.
8. L’Esteve, R. (2022). The Azure data lakehouse toolkit: Building and scaling data lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake. Apress.
9. Koppula, R. S. (2022). Implementing data lakes with Databricks for advanced analytics. North American Journal of Engineering Research, 3(2).
10. Akidau, T., Chernyak, S., & Lax, R. (2020). Streaming systems: The what, where, when, and how of large-scale data processing. O’Reilly Media.
11. Reddy, V. A. R. (2021). Challenges in Standardizing Member Eligibility Data Across Multi-Payer Healthcare Ecosystems. International Journal of Medical Toxicology and Legal Medicine, 24(3), 1-19.
12. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2020). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 43(1), 28–38.
13. Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2020). Discretized streams: Fault-tolerant streaming computation at scale. Communications of the ACM, 63(9), 56–65.
14. Ghodsi, A., Zaharia, M., Xin, R., Armbrust, M., & others. (2020). Lakehouse systems for modern data and analytics workloads. Proceedings of the VLDB Endowment.
15. Mangala, N. (2021). Optimizing Large-Scale ETL Pipelines Using Medallion Architecture on Azure Data Lake. Journal of Artificial Intelligence and Big Data, 1(1), 1-20.
16. Abadi, D. J., Boncz, P., Harizopoulos, S., Idreos, S., & Madden, S. (2020). The design and implementation of modern analytical data systems. Foundations and Trends in Databases, 9(2), 73–124.
17. Armbrust, M., Xin, R., Lian, C., Huai, Y., Liu, D., Bradley, J. K., Meng, X., Kaftan, T., Franklin, M. J., Ghodsi, A., & Zaharia, M. (2020). Spark SQL: Relational data processing in Spark. Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, 1383–1394.
18. Mangalampalli, B. M. (2022). Automated Invoice Validation Systems Using Advanced SQL Analytics in Healthcare Insurance. Front Health Inform, 11.
19. Dean, J., & Ghemawat, S. (2020). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 63(1), 40–44.
20. Kreps, J., Narkhede, N., & Rao, J. (2020). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 1–7.
21. Eismann, S., Grohmann, J., Herbst, N. R., Kounev, S., & Abad, C. L. (2021). A review of serverless computing for cloud applications. ACM Computing Surveys, 54(2), 1–36.
22. Loganathan, R. (2021). Integrated Risk and Compliance Frameworks for Global Data Center Operations: A Governance-Centric Approach. Universal Journal of Computer Sciences and Communications, 1(1), 1-26.
23. Buyya, R., Srirama, S. N., Casale, G., Calheiros, R. N., Simmhan, Y., Varghese, B., Gelenbe, E., Javadi, B., Vaquero, L. M., Netto, M. A. S., Toosi, A. N., Rodríguez, M. A., Llorente, I. M., & Bahsoon, R. (2020). A manifesto for cloud-native applications. ACM Computing Surveys, 53(5), 1–32.
24. Jamshidi, P., Pahl, C., Chinenyeze, S., & Liu, X. (2020). Cloud migration patterns: A multi-cloud perspective. IEEE Cloud Computing, 7(5), 70–79.
25. Mangala, N. (2021). CI/CD Pipeline Automation for Enterprise Data Artifacts Using Azure DevOps. Universal Journal of Business and Management, 1(1), 1-18.
26. Erich, F., Amrit, C., & Daneva, M. (2020). A qualitative study of DevOps usage in practice. Journal of Software: Evolution and Process, 32(3), 1–20.
27. Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P. (2020). A survey of DevOps concepts and challenges. ACM Computing Surveys, 52(6), 1–35.
28. Shahin, M., Babar, M. A., & Zhu, L. (2020). Continuous integration, delivery and deployment: A systematic review on approaches, tools, challenges and practices. IEEE Access, 8, 3909–3942.
29. Mangalampalli, B. M. (2021). Scalable Data Warehouse Architecture for Population Health Management and Predictive Analytics. World Journal of Clinical Medicine Research, 1(1), 1-18.
30. Chen, L. (2020). Continuous delivery: Huge benefits, but challenges too. IEEE Software, 37(1), 50–54.
31. Sato, D., Wider, A., & Windheuser, C. (2021). Continuous delivery and DevOps practices in cloud-based data platforms. IEEE Software, 38(4), 31–38.
32. Madera, C., & Laurent, A. (2021). Data lakes: A survey of functions and systems. ACM Computing Surveys, 54(2), 1–34.
33. Alrehamy, M., & Walker, C. (2022). Data lake architecture: A systematic review of data management and analytics approaches. Journal of Big Data, 9, 1–24.
34. Yandamuri, U. S. (2021). A Comparative Study of Traditional Reporting Systems versus Real-Time Analytics Dashboards in Enterprise Operations. Universal Journal of Business and Management, 1(1), 1-13.
Additional Files
Published
Data Availability Statement
None
Issue
Section
License
Copyright (c) 2023 Elena Petrova (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons
Attribution 4.0 International License (CC BY 4.0).
You are free to:
- Share: copy and redistribute the material
- Adapt: remix, transform, and build upon the material
for any purpose, even commercially.
Under the following terms:
Attribution — You must give appropriate credit to
the original author(s) and source.
Full license text: https://creativecommons.org/licenses/by/4.0/
