MLOps and Data Engineering: Integrating ETL into the ML Lifecycle
DOI:
https://doi.org/10.56830/IJSIE202503Keywords:
MLOps, Data Engineering, ETL Integration, Feature Pipeline Automation IntroductionAbstract
In this paper, MLOps and data engineering are discussed with a focus on the role of ETLs throughout the ML pipeline. With the growing adoption of AI solutions in organizations, the—to be used—integration of sound data management has become a crucial success factor influencing its effectiveness, reliability, and value. The paper overviews the architectural strategies for integrating ETL into MLOps methodologies, introduces methods of automated feature engineering, and discusses main issues like data drift detection and versioning. Through the analysis of the current trends and technologies, this paper outlines how integrated ETL is in the process of moving traditional ML projects from proving grounds to scalable production systems that are defined to deliver tangible business value.
References
Büyükkeçeci̇, M., & Okur, M. C. (2023). A comprehensive review of feature selection and feature selection stability in machine learning. Gazi University Journal of Science, 36(4), 1506–1520. https://doi.org/10.35378/gujs.993763. DOI: https://doi.org/10.35378/gujs.993763
Gartner. (2020). Gartner Identifies the Top Strategic Technology Trends for 2021. Gartner, https://www.gartner.com/en/newsroom/press-releases/2020-10-19gartner-identifies-the-top-strategic-technology-trends-for-
2021#:~:text=Gartner%20research%20shows%20only%2053,a%20producti on%2Dgrade%20AI%20pipeline.
Idowu, S., Osman, O., Strüber, D., & Berger, T. (2024). Machine learning experiment management tools: A mixed-methods empirical study. Empirical Software Engineering, 29(4). https://doi.org/10.1007/s10664-024-10444-w. DOI: https://doi.org/10.1007/s10664-024-10444-w
Intelligence, S. U.-C. (2024). The 2024 AI Index Report. Stanford HAI, https://hai.stanford.edu/ai-index/2024-ai-index-report|.
Muttakin, F., Wang, J.-T., Mulyanto, M., & Leu, J.-S. (2021). Evaluation of Feature Selection Methods on Psychosocial Education Data Using Additive Ratio Assessment. Electronics, 11(1). https://doi.org/10.3390/electronics11010114. DOI: https://doi.org/10.3390/electronics11010114
Ransbotham, S., Gerbert, P., Reeves, M., Kiron, D., & Spira, M. (2023). Artificial Intelligence in Business Gets Real. MIT Sloan Management Review.
