Delta Lake for Collaborative AI: A ServiceNow Case Study
DOI:
https://doi.org/10.56830/IJSIE202506Keywords:
Delta Lake, Collaborative AI, ServiceNow, Real-Time Data Streaming, Delta SharingAbstract
This study examines how the concept of Delta Lake can be connected with the business value of ServiceNow to accelerate the collaborative work with AI, especially in terms of applying key Delta Lake functionality such as ACID transactions, automatic data streaming, and governance status. The study highlights the potential offered by the Delta Lake to allow real-time data ingestion and an uninterrupted teamwork of AI teams and IT operations to result in quantifiable enhancements of AI model performance and operational measures. Delta Lake provides consistency in AI models trained on ServiceNow data via time travel as a version control feature and continuous updates to keep these models relevant and generate better predictions and decisions, using Change Data Capture (CDC). The study also emphasizes how Unity Catalog facilitated data governance and how Delta Sharing helped to secure the collaboration of data among teams to make the AI workflow more productive. The author reports noteworthy findings with an 80% reduction in the time taken to create a resolution note, 14% lower deflection rates on a self-service level, as well as 2-minute time period savings of an AI-driven search. These findings also emphasize the use of Delta Lake to remove time delays in implementing AI models and their effectiveness, as well as the development of the field of Data governance, and thus, organizations have the keys to unlock the full value of their Data in ServiceNow to implement new AI models jointly.
References
Adepoju, A. H., Austin-Gabriel, B. L., Eweje, A. D., & Collins, A. N. (2022). Framework for automating multi-team workflows to maximize operational efficiency and minimize redundant data han. IRE Journal, 5(9), 663-664.
Begoli, E., Goethert, I., & Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. In 2021 IEEE International Conference on Big Data (Big Data), (pp. 4643-4651). IEEE. DOI: https://doi.org/10.1109/BigData52589.2021.9671534
Berkowitz, D. (2019). Reporters and their sources. In The handbook of journalism studies , (pp. 165-179). Routledge. DOI: https://doi.org/10.4324/9781315167497-11
Bonthu, C., Kumar, A., & Goel, G. (2025). Impact of AI and machine learning on master data management. Journal of Information Systems Engineering and
Management, https://www.jisemjournal.com/index.php/journal/article/view/5186.
Bonthu, C.; Goel, G. (2025). Autonomous supplier evaluation and data stewardship with AI: Building transparent and resilient supply chains. International Journal of Computational and Experimental Science and Engineering, https://ijcesen.com/index.php/ijcesen/article/view/3854/1154. DOI: https://doi.org/10.22399/ijcesen.3854
Chadha, K. S. (2025). Edge AI for real-time ICU alarm fatigue reduction: Federated anomaly detection on wearable streams. Utilitas Mathematica, 122(2), 291– 308. https://utilitasmathematica.com/index.php/Index/article/view/2708.
Chavan, A. (2023). Managing scalability and cost in microservices architecture:
Balancing infinite scalability with financial constraints. Journal of Artificial Intelligence & Cloud Computing, 2, E264. http://doi.org/10.47363/JAICC/2023(2)E264. DOI: https://doi.org/10.47363/JAICC/2023(2)E264
Chavan, A. (2025). The role of domain-driven design in successful microservices migration strategies. Journal of Information Systems Engineering and
Management, https://www.jisemjournal.com/index.php/journal/article/view/8888.
Chen, T., Guo, W., Gao, X., & Liang, Z. (2021). AI-based self-service technology in public service delivery: User experience and influencing factors. DOI: https://doi.org/10.1016/j.giq.2020.101520
Government Information Quarterly, 38(4), 101520.
Fabbian, A. (2023). Perception and Approaches to New Information Systems Acceptance in Digital Transformation: a ServiceNow Case Study Analysis.
Gandhi, S. T. (2023). AI-Driven Compliance Audits: Enhancing Regulatory Adherence in Financial and| Legal Sectors. International Journal of Advanced Research in Computer Science & Technology (IJARCST), 6(5), 8981-8988. DOI: https://doi.org/10.15662/IJARCST.2023.0605004
Goel, G., & Bhramhabhatt, R. (2024). Dual sourcing strategies. International Journal of Science and Research Archive, 13(2), 2155. https://doi.org/10.30574/ijsra.2024.13.2.2155. DOI: https://doi.org/10.30574/ijsra.2024.13.2.2155
Gordon, I., & Thompson, N. (2024). Radical Technologies. In Data and the Built Environment: A Practical Guide to Building a Better World Using Data, (pp. 239-337). Cham: Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-031-51008-3_6
Gupta, N., & Yip, J. (2024). Spark Structured Streaming: A Comprehensive Guide. DOI: https://doi.org/10.1007/979-8-8688-0444-1_18
In Databricks Data Intelligence Platform, Unlocking the GenAI Revolution (pp. 409-429). Berkeley, CA: Apress.
Karwa, K. (2023). AI-powered career coaching: Evaluating feedback tools for design students. Indian Journal of Economics & Business, https://www.ashwinanokha.com/ijeb-v22-4-2023.php.
Koneru, N. M. (2025). Containerization best practices: Using Docker and Kubernetes for enterprise applications. Journal of Information Systems
Engineering and Management, https://www.jisemjournal.com/index.php/journal/article/view/8905.
Koneru, N. M. K. (2025). Leveraging AWS CloudWatch, Nagios, and Splunk for real-time cloud observability. IJCESEN, Advance online publication. https://ijcesen.com/index.php/ijcesen/article/view/3781.
Latifian, A. (2022). How does cloud computing help businesses to manage big data issues. Kybernetes, 51(6), 1917-1948. DOI: https://doi.org/10.1108/K-05-2021-0432
Lu, Y. (2019). Artificial intelligence: a survey on evolution, models, applications and future trends. Journal of management analytics, 6(1), 1-29. DOI: https://doi.org/10.1080/23270012.2019.1570365
Mahapatra, A., & May, D. (2022). Simplifying Data Engineering and Analytics with Delta: Create analytics-ready data that fuels artificial intelligence and business intelligence. Packt Publishing Ltd.
Malik, G. (2025). Business continuity & incident response. Journal of Information
Systems Engineering and Management, https://www.jisemjournal.com/index.php/journal/article/view/8891.
Malik, G., Brahmbhatt, R., & Prashasti. (2025). AI-driven security and inventory optimization: Automating vulnerability management and demand forecasting DOI: https://doi.org/10.22399/ijcesen.3855
in CI/CD-powered retail systems. . International Journal of Computational and Experimental Science and Engineering .
Manner, J. (2024). A Simulation Framework for Function as a Service. University of Bamberg Press, (Vol. 43). DOI: https://doi.org/10.20378/irb-92913
Nyati, S. (2018). Transforming telematics in fleet management: Innovations in asset tracking, efficiency, and communication. International Journal of Science and Research (IJSR), 7(10), 1804-1810. Retrieved from https://www.ijsr.net/getabstract.php?paperid=SR24203184230. DOI: https://doi.org/10.21275/SR24203184230
Pinnapareddy, N. R. (2025). Cloud cost optimization and sustainability in Kubernetes. Journal of Information Systems Engineering and Management, https://www.jisem-journal.com/index.php/journal/article/view/8895.
Raju, R. K. (2017). Dynamic memory inference network for natural language inference. International Journal of Science and Research (IJSR), 6(2). https://www.ijsr.net/archive/v6i2/SR24926091431.pdf. DOI: https://doi.org/10.21275/SR24926091431
Rivera, S., Prokaieva, A., & Baker, A. (2024). Databricks ML in Action.
Rupprecht, L., Davis, J. C., Arnold, C., Gur, Y., & Bhagwat, D. (2020). Improving reproducibility of data science pipelines through transparent provenance capture. Proceedings of the VLDB Endowment, 13(12), 3354-3368. DOI: https://doi.org/10.14778/3415478.3415556
Sardana, J. (2025). Automating global trade compliance through product classification systems. The American Journal of Management and Economics Innovations, 7(4). https://doi.org/10.37547/tajmei/Volume07Issue04-04. DOI: https://doi.org/10.37547/tajmei/Volume07Issue04-04
Sidikova, M., Martinek, R., Kawala-Sterniuk, A., Ladrova, M., Jaros, R., Danys, L., et al. (2020). Vital sign monitoring in car seats based on electrocardiography, ballistocardiography and seismocardiography: A review. Sensors, 20(19), 5699. DOI: https://doi.org/10.3390/s20195699
Singh, V. (2022). Advanced generative models for 3D multi-object scene generation: Exploring the use of cutting-edge generative models like diffusion models to synthesize complex 3D environments. https://doi.org/10.47363/JAICC/2022(1)E224. DOI: https://doi.org/10.47363/JAICC/2022(1)E224
Sousa, T. B., Ferreira, H. S., & Correia, F. F. (2021). A Survey on the Adoption of Patterns for Engineering Software for the Cloud. IEEE Transactions on Software Engineering, 48(6), 2128-2140. DOI: https://doi.org/10.1109/TSE.2021.3052177
Subham, K. (2025). Scalable SaaS implementation governance for enterprise sales operations. International Journal of Computational and Experimental Science and Engineering, https://ijcesen.com/index.php/ijcesen/article/view/3782.
Tan, Z., E, H., & Song, M. (2020). A Column-Level Data Lineage Processing System Based on Hive. In Proceedings of the 3rd International Conference on Big Data Technologies, (pp. 47-52). DOI: https://doi.org/10.1145/3422713.3422719
Villegas-Ch, W., García-Ortiz, J., & Sánchez-Viteri, S. (2024). Toward intelligent monitoring in IoT: AI applications for real-time analysis and prediction. IEEE Access, 12, 40368-40386. DOI: https://doi.org/10.1109/ACCESS.2024.3376707
Zhang, W., Yang, D., & Wang, H. (2019). Data-driven methods for predictive maintenance of industrial equipment: A survey. IEEE systems journal, 13(3), 2213-2227. DOI: https://doi.org/10.1109/JSYST.2019.2905565
