Dependable Data In, Operational Intelligence Out
DOI:
https://doi.org/10.5281/gz4d2076Keywords:
Analytics Pipelines, Data Engineering, Data Lakehouse, Data Observability, Data Pipelines, Data Quality, Enterprise Analytics, Enterprise Data Architecture, Lakehouse Architecture, Operational Data Integrity, Operational Intelligence, Pipeline Reliability, Resilient Data Systems, Service-Level Agreements (SLAs).Abstract
A uniform lakehouse architecture supports a consistent approach to designing, building, and maintaining analytics pipelines while preserving operational data integrity. Although “lakehouse” broadly describes a hybrid architecture that blends elements of data warehouse and data lake design, a precise, agreed-upon definition remains elusive, and implementations vary considerably across the industry. This work examines the motivation for a unified lakehouse approach and the rationale for allocating processing power equally between analytics and operational-intelligence pipelines. It also presents design requirements for reliability and observability, along with an architectural model capable of meeting them.
Viewing the hybrid architecture as distinct yet intertwined domains enables a shared design, build, and maintenance approach across two pipeline types: those driving enterprise business intelligence, exploration, and reporting, and those operationalizing enterprise data within transactional systems, third-party applications, and enterprise products. This symmetry allows engineering principles refined in the analytics context to be carried over effectively to operational pipelines. Increasingly rigorous SLAs for accurate analytics delivery demand formal monitoring, alerting, testing, and resilience capabilities—while growing emphasis on observability in operational pipelines drives a parallel need for data quality assurance, recovery procedures, and failure mode analysis.
References
1. Renggli, C., Rimanic, L., Gürel, N. M., Karlas, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 44(1), 11–23.
2. Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11, 31866–31879.
3. Diaz-de-Arcaya, J., Torre-Bastida, A. I., Zárate, G., Miñón, R., & Almeida, A. (2024). A joint study of the challenges, opportunities, and roadmap of MLOps and AIOps: A systematic survey. ACM Computing Surveys, 56(4), Article 84.
4. Manukonda, K. R. R., Swamy, H., Sankar, R., & CM, C. (2024, August). Application of Explicit Group Sparse Projection in Optimizing 5G Network Resource Allocation and Topology Design. In 2024 4th Asian Conference on Innovation in Technology (ASIANCON) (pp. 1-6). IEEE.
5. Priestley, M., O’Donnell, F., & Simperl, E. (2023). A survey of data quality requirements that matter in ML development pipelines. ACM Journal of Data and Information Quality, 15(2), Article 11.
6. Heck, P. (2024). What about the data? A mapping study on data engineering for AI systems. In Proceedings of the 3rd ACM International Conference on AI Engineering—Software Engineering for AI (CAIN 2024).
7. Yandamuri, U. S. (2024). AI-Driven Decision Support Systems for Operational Optimization in Hospitality Technology. Metallurgical and Materials Engineering.
8. Shankar, S., Li, H., Asawa, P., Hulsebos, M., Lin, Y., Zamfirescu-Pereira, J. D., Chase, H., Fu-Hinthorn, W., & Parameswaran, A. G., et al. (2024). SPADE: Synthesizing data quality assertions for large language model pipelines. Proceedings of the VLDB Endowment, 17(12), 4173–4186.
9. Chatterjee, A., Ahmed, B. S., Hallin, E., & Engman, A. (2022). Quality assurance in MLOps setting: An industrial perspective. arXiv preprint.
10. Bandi, V. D. V. K. (2023). MLOps frameworks for reliable model deployment in cloud data platforms. Journal of Artificial Intelligence and Big Data, 3(1), 84–101.
11. KOLLA, S. K. (2024). DEEP LEARNING AND CLINICAL NLP FRAMEWORK FOR AUTOMATED KNOWLEDGE EXTRACTION FROM MULTIMODAL HEALTH . TPM – Testing, Psychometrics, Methodology in Applied Psychology, 31(S2), 134–140. Retrieved from https://tpmap.org/submission/index.php/tpm/article/view/4748
12. Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers. (2024). Journal of Systems and Software, 207, Article 111855.
13. A survey on dataset quality in machine learning. (2023). Information and Software Technology, 162, Article 107268.
14. Next-generation challenges of responsible data integration. (2023). In Proceedings of the 16th ACM International Conference on Web Search and Data Mining (pp. 1256–1259).
15. Mangalampalli, B. M. Generative AI Applications In Healthcare Data Mart Design And Optimization.
16. Supporting better insights of data science pipelines with fine-grained provenance. (2024). ACM Transactions on Database Systems.
17. Active learning for data quality control: A survey. (2024). ACM Journal of Data and Information Quality, 16(2), Article 11.
18. Machine learning operations: A mapping study. (2024). arXiv preprint.
19. Rohit Gorle. (2024). Deep Learning Approaches for Threat Detection in Multi-Cloud Environments. Journal of Computational Analysis and Applications (JoCAAA), 33(07), 3857–3864. Retrieved from https://eudoxuspress.com/index.php/pub/article/view/5879
20. Machine learning operations: A survey on MLOps tool support. (2022).
21. Who needs MLOps: What data scientists seek to accomplish and how can MLOps help? (2021).
22. Demystifying MLOps and presenting a recipe for the selection of open-source tools. (2021).
23. MLOps challenges in multi-organization setup: Experiences from two real-world cases. (2021).
24. Mashetty, S. (2024). Leveraging Deep Learning, Neural Networks, and Data Engineering for Intelligent Mortgage Loan Validation: A Data-Driven Approach to Automating Borrower Income, Employment, and Asset Verification. Neural Networks, and Data Engineering for Intelligent Mortgage Loan Validation: A Data-Driven Approach to Automating Borrower Income, Employment, and Asset Verification (December 21, 2024).
25. The machine learning life cycle and the cloud: Implications for drug discovery. (2021).
26. Application of machine learning in Industry 4.0. (2021).
27. Detecting concept drift with neural network model uncertainty. (2021).
28. Mahadevan, S., Desani, N. R., Yadav, P. S., Joseph, T., & Malcom, A. (2024). FutureTech Innovations. Cari Journals USA LLC.
29. Efficient large-scale NLP feature engineering with Apache Spark. (2022).
30. MLOps in the financial industry: Philosophy, practices, and tools. (2022).
31. Yandamuri, U. S. (2022). Cloud-Based Data Integration Architectures for Scalable Enterprise Analytics. International Journal of Intelligent Systems and Applications in Engineering, 10, 472-483.
32. A dynamic web data extraction from SRLDC (Southern Regional Load Dispatch Centre) and feature engineering using ETL tool. (2022).
33. A joint study of the challenges, opportunities, and roadmap of MLOps and AIOps: A systematic survey. (2023). ACM Computing Surveys, 56(4), Article 84.
34. Mahadevan, S. (2024). Empowering Manufacturing: Generative AI Revolutionizes ERP Application. Int. J. Innov. Sci. Res, 9, 593-595.
35. A survey of data quality requirements that matter in ML development pipelines. (2023). ACM Journal of Data and Information Quality, 15(2), Article 11.
36. A survey on dataset quality in machine learning. (2023). Information and Software Technology, 162, Article 107268.
37. MLOps frameworks for reliable model deployment in cloud data platforms. (2023). Journal of Artificial Intelligence and Big Data, 3(1), 84–101.
38. Machine learning operations (MLOps): Overview, definition, and architecture. (2023). IEEE Access, 11, 31866–31879.
39. What about the data? A mapping study on data engineering for AI systems. (2024). In Proceedings of the 3rd ACM International Conference on AI Engineering—Software Engineering for AI.
40. Manukonda, K. R. R. (2024). Chapter-5 Enhancing Test Automation Efficiency Through AI-driven Reporting: An Empirical Study on Defect Detection and Resolution Time. ADVANCES IN, 119, 107.
41. SPADE: Synthesizing data quality assertions for large language model pipelines. (2024). Proceedings of the VLDB Endowment, 17(12), 4173–4186.
42. Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers. (2024). Journal of Systems and Software, 207, Article 111855.
43. Mangalampalli, B. M. Intelligent Data Profiling for Healthcare Data Lakes Using AI-Enhanced Analytics.
44. Supporting better insights of data science pipelines with fine-grained provenance. (2024). ACM Transactions on Database Systems.
45. Active learning for data quality control: A survey. (2024). ACM Journal of Data and Information Quality, 16(2), Article 11.
46. Movva, V. R. (2024). Cloud-Native Platform Engineering for High-Scale Retail Systems: Designing Resilient Microservices and Intelligent Infrastructure. Available at SSRN 7449480.
47. Renggli, C., Rimanic, L., Gürel, N. M., Karlas, B., Wu, W., & Zhang, C. (2021). A data quality-driven view of MLOps. IEEE Data Engineering Bulletin, 44(1), 11–23.
Additional Files
Published
Data Availability Statement
None
Issue
Section
License
Copyright (c) 2025 Shakir Syed, Carlos Fernnadez (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons
Attribution 4.0 International License (CC BY 4.0).
You are free to:
- Share: copy and redistribute the material
- Adapt: remix, transform, and build upon the material
for any purpose, even commercially.
Under the following terms:
Attribution — You must give appropriate credit to
the original author(s) and source.
Full license text: https://creativecommons.org/licenses/by/4.0/
