##common.pageHeaderLogo.altText##

AI-Powered Automation of Data Pipelines: Bridging Data Engineering and Intelligent Systems

International Journal of Emerging Trends in Science and Technology, , 14 April 2022 , Page 1-17
https://doi.org/10.18535/ijetst/v9i3.03

Abstract

The presence of AI and advanced data engineering has transformed ITS and physical infrastructure and resolved main issues to urban mobility. This article is aimed at revealing the role of AI solutions in improving traffic flow, safety levels and emissions. Smart traffic signals, big data technology, and predictive algorithms have gone a long way to negate traffic jams, as well as emissions and enhance travel time and security. Real examples illustrate the applicability of AI when it comes to traffic control in cities and in self-driving cars.


The modern nature of the business requires openness and connectivity, making it difficult to manage ever-increasing volumes of data in enterprises. Application of Artificial Intelligence in automation of data pipelines is a game changer in data engineering that seeks to close data to intelligence chasm. This gives AI the ability to co-ordinate machine learning, natural language processing and advanced orchestration tools to create complex and sustainable pipelines which allows for real-time data ingestion, transformation and delivery.


This article aims to demonstrate the change that AI brings into organizations’ daily vocabulary by automating mostly clerklike tasks, making use of various methodologies like, Anomaly detection, Predictive analytics and Dynamic resource allocation. Through case studies, large scale implementation of AI DP in areas of financial fraud detection, IoT based smart manufacturing, and smart retail experiences are shown.


The foremost advantage is that, using MEAN, it becomes possible to decrease latency, improve scalability, and increase the overall operational efficiency as a result of proper integration and better inbuilt functions of corresponding supplying tools and technologies Nevertheless, there are still crucial problems such as data security, ethical approaches to the usage of AI, and integration of MEAN with legacy systems. The article also expands to future possibilities such as linking of quantum computing and generative AI to create optimizing pipelines. In doing so, this research also draws focus on the future of data engineering powered by AI automation to promote intelligent decision making and innovation in the industries.

How to Cite

Devarasetty, N. (2022). AI-Powered Automation of Data Pipelines: Bridging Data Engineering and Intelligent Systems. International Journal of Emerging Trends in Science and Technology, 9(03), 1–17. https://doi.org/10.18535/ijetst/v9i3.03

Download Citation

References

  • Malhotra, I., Gopinath, S., Janga, K. C., Greenberg, S., Sharma, S. K., & Tarkovsky, R. (2014). Unpredictable nature of tolvaptan in treatment of hypervolemic hyponatremia: case review on role of vaptans. Case reports in endocrinology, 2014(1), 807054.
  • Shati, Z. R. K., Mulakhudair, A. R., & Khalaf, M. N. Studying the effect of Anethum Graveolens extract on parameters of lipid metabolism in white rat males.
  • Karakolias, S., Kastanioti, C., Theodorou, M., & Polyzos, N. (2017). Primary care doctors’ assessment of and preferences on their remuneration: Evidence from Greek public sector. INQUIRY: The Journal of Health Care Organization, Provision, and Financing, 54, 0046958017692274.
  • Singh, V. K., Mishra, A., Gupta, K. K., Misra, R., & Patel, M. L. (2015). Reduction of microalbuminuria in type-2 diabetes mellitus with angiotensin-converting enzyme inhibitor alone and with cilnidipine. Indian Journal of Nephrology, 25(6), 334-339.
  • Karakolias, S. E., & Polyzos, N. M. (2014). The newly established unified healthcare fund (EOPYY): current situation and proposed structural changes, towards an upgraded model of primary health care, in Greece. Health, 2014.
  • Shilpa, Lalitha, Prakash, A., & Rao, S. (2009). BFHI in a tertiary care hospital: Does being Baby friendly affect lactation success?. The Indian Journal of Pediatrics, 76, 655-657.
  • Polyzos, N. (2015). Current and future insight into human resources for health in Greece. Open Journal of Social Sciences, 3(05), 5.
  • Gopinath, S., Janga, K. C., Greenberg, S., & Sharma, S. K. (2013). Tolvaptan in the treatment of acute hyponatremia associated with acute kidney injury. Case reports in nephrology, 2013(1), 801575.
  • Gopinath, S., Giambarberi, L., Patil, S., & Chamberlain, R. S. (2016). Characteristics and survival of patients with eccrine carcinoma: a cohort study. Journal of the American Academy of Dermatology, 75(1), 215-217.
  • Shakibaie-M, B. (2013). Comparison of the effectiveness of two different bone substitute materials for socket preservation after tooth extraction: a controlled clinical study. International Journal of Periodontics & Restorative Dentistry, 33(2).
  • Swarnagowri, B. N., & Gopinath, S. (2013). Ambiguity in diagnosing esthesioneuroblastoma--a case report. Journal of Evolution of Medical and Dental Sciences, 2(43), 8251-8255.
  • Shilpa, Lalitha, Prakash, A., & Rao, S. (2009). BFHI in a tertiary care hospital: Does being Baby friendly affect lactation success?. The Indian Journal of Pediatrics, 76, 655-657.
  • Gopinath, S., Janga, K. C., Greenberg, S., & Sharma, S. K. (2013). Tolvaptan in the treatment of acute hyponatremia associated with acute kidney injury. Case reports in nephrology, 2013(1), 801575.
  • Swarnagowri, B. N., & Gopinath, S. (2013). Pelvic Actinomycosis Mimicking Malignancy: A Case Report. tuberculosis, 14, 15.
  • Gopinath, S., Giambarberi, L., Patil, S., & Chamberlain, R. S. (2016). Characteristics and survival of patients with eccrine carcinoma: a cohort study. Journal of the American Academy of Dermatology, 75(1), 215-217.
  • Swarnagowri, B. N., & Gopinath, S. (2013). Ambiguity in diagnosing esthesioneuroblastoma--a case report. Journal of Evolution of Medical and Dental Sciences, 2(43), 8251-8255.
  • Malhotra, I., Gopinath, S., Janga, K. C., Greenberg, S., Sharma, S. K., & Tarkovsky, R. (2014). Unpredictable nature of tolvaptan in treatment of hypervolemic hyponatremia: case review on role of vaptans. Case reports in endocrinology, 2014(1), 807054.
  • Swarnagowri, B. N., & Gopinath, S. (2013). Pelvic Actinomycosis Mimicking Malignancy: A Case Report. tuberculosis, 14, 15.
  • Papakonstantinidis, S., Poulis, A., & Theodoridis, P. (2016). RU# SoLoMo ready?: Consumers and brands in the digital era. Business Expert Press.
  • Poulis, A., Panigyrakis, G., & Panos Panopoulos, A. (2013). Antecedents and consequents of brand managers’ role. Marketing Intelligence & Planning, 31(6), 654-673.
  • Poulis, A., & Wisker, Z. (2016). Modeling employee-based brand equity (EBBE) and perceived environmental uncertainty (PEU) on a firm’s performance. Journal of Product & Brand Management, 25(5), 490-503.
  • Damacharla, P., Javaid, A. Y., Gallimore, J. J., & Devabhaktuni, V. K. (2018). Common metrics to benchmark human-machine teams (HMT): A review. IEEE Access, 6, 38637-38655.
  • Mulakhudair, A. R., Hanotu, J., & Zimmerman, W. (2017). Exploiting ozonolysis-microbe synergy for biomass processing: Application in lignocellulosic biomass pretreatment. Biomass and bioenergy, 105, 147-154.
  • Mulakhudair, A. R., Hanotu, J., & Zimmerman, W. (2016). Exploiting microbubble-microbe synergy for biomass processing: application in lignocellulosic biomass pretreatment. Biomass and Bioenergy, 93, 187-193.
  • Mulakhudair, A. R., Al‐Mashhadani, M., Hanotu, J., & Zimmerman, W. (2017). Inactivation combined with cell lysis of Pseudomonas putida using a low pressure carbon dioxide microbubble technology. Journal of Chemical Technology & Biotechnology, 92(8), 1961-1969.
  • Ashraf, S., Aggarwal, P., Damacharla, P., Wang, H., Javaid, A. Y., & Devabhaktuni, V. (2018). A low-cost solution for unmanned aerial vehicle navigation in a global positioning system–denied environment. International Journal of Distributed Sensor Networks, 14(6), 1550147718781750.
  • Karakolias, S., Kastanioti, C., Theodorou, M., & Polyzos, N. (2017). Primary care doctors’ assessment of and preferences on their remuneration: Evidence from Greek public sector. INQUIRY: The Journal of Health Care Organization, Provision, and Financing, 54, 0046958017692274.
  • Karakolias, S. E., & Polyzos, N. M. (2014). The newly established unified healthcare fund (EOPYY): current situation and proposed structural changes, towards an upgraded model of primary health care, in Greece. Health, 2014.
  • Polyzos, N., Kastanioti, C., Zilidis, C., Mavridoglou, G., Karakolias, S., Litsa, P., ... & Kani, C. (2016). Greek national e-prescribing system: Preliminary results of a tool for rationalizing pharmaceutical use and cost. Glob J Health Sci, 8(10), 55711.
  • Alam, K., Mostakim, M. A., & Khan, M. S. I. (2017). Design and Optimization of MicroSolar Grid for Off-Grid Rural Communities. Distributed Learning and Broad Applications in Scientific Research, 3.
  • Mahmud, U., Alam, K., Mostakim, M. A., & Khan, M. S. I. (2018). AI-driven micro solar power grid systems for remote communities: Enhancing renewable energy efficiency and reducing carbon emissions. Distributed Learning and Broad Applications in Scientific Research, 4.
  • Manoharan, A., & Nagar, G. MAXIMIZING LEARNING TRAJECTORIES: AN INVESTIGATION INTO AI-DRIVEN NATURAL LANGUAGE PROCESSING INTEGRATION IN ONLINE EDUCATIONAL PLATFORMS.
  • Arefin, S. Mental Strength and Inclusive Leadership: Strategies for Workplace Well-being.
  • Nagar, G. (2018). Leveraging Artificial Intelligence to Automate and Enhance Security Operations: Balancing Efficiency and Human Oversight. Valley International Journal Digital Library, 78-94.
  • Nagar, G. The Evolution of Security Operations Centers (SOCs): Shifting from Reactive to Proactive Cybersecurity Strategies
  • Zhu, L., Yu, F. R., Wang, Y., Ning, B., & Tang, T. (2018). Big data analytics in intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 20(1), 383-398.
  • Khan, S. M., Rahman, M., Apon, A., & Chowdhury, M. (2017). Characteristics of intelligent transportation systems and its relationship with data analytics. In Data analytics for intelligent transportation systems (pp. 1-29). Elsevier.
  • Kibria, M. G., Nguyen, K., Villardi, G. P., Zhao, O., Ishizu, K., & Kojima, F. (2018). Big data analytics, machine learning, and artificial intelligence in next-generation wireless networks. IEEE access, 6, 32328-32338.
  • Schmidtke, H. R. (2018). A survey on verification strategies for intelligent transportation systems. Journal of Reliable Intelligent Environments, 4(4), 211-224.
  • Hamza-Lup, G. L., Hua, K. A., & Peng, R. (2007). Leveraging e-transportation in real-time traffic evacuation management. Electronic Commerce Research and Applications, 6(4), 413-424.
  • Kumar, S. A., Madhumathi, R., Chelliah, P. R., Tao, L., & Wang, S. (2018). A novel digital twin-centric approach for driver intention prediction and traffic congestion avoidance. Journal of Reliable Intelligent Environments, 4(4), 199-209.
  • Davenport, T. H. (2018). From analytics to artificial intelligence. Journal of Business Analytics, 1(2), 73-80.
  • Demirkan, H., & Delen, D. (2013). Leveraging the capabilities of service-oriented decision support systems: Putting analytics and big data in cloud. Decision Support Systems, 55(1), 412-421.
  • Rabah, K. (2018). Convergence of AI, IoT, big data and blockchain: a review. The lake institute Journal, 1(1), 1-18.
  • Torre‐Bastida, A. I., Del Ser, J., Laña, I., Ilardia, M., Bilbao, M. N., & Campos‐Cordobés, S. (2018). Big Data for transportation and mobility: recent advances, trends and challenges. IET Intelligent Transport Systems, 12(8), 742-755.
  • Wang, C., Li, X., Zhou, X., Wang, A., & Nedjah, N. (2016). Soft computing in big data intelligent transportation systems. Applied Soft Computing, 38, 1099-1108.
  • Nagar, G. (2018). Leveraging Artificial Intelligence to Automate and Enhance Security Operations: Balancing Efficiency and Human Oversight. Valley International Journal Digital Library, 78-94.
  • Lana, I., Del Ser, J., Velez, M., & Vlahogianni, E. I. (2018). Road traffic forecasting: Recent advances and new challenges. IEEE Intelligent Transportation Systems Magazine, 10(2), 93-109.
  • Xiong, G., Zhu, F., Liu, X., Dong, X., Huang, W., Chen, S., & Zhao, K. (2015). Cyber-physical-social system in intelligent transportation. IEEE/CAA Journal of Automatica Sinica, 2(3), 320-333.
  • Cottrill, C. D., & Derrible, S. (2015). Leveraging big data for the development of transport sustainability indicators. Journal of Urban Technology, 22(1), 45-64.
  • Edge, D., Larson, J., & White, C. (2018, April). Bringing AI to BI: enabling visual analytics of unstructured data in a modern Business Intelligence platform. In Extended abstracts of the 2018 CHI conference on human factors in computing systems (pp. 1-9).
  • Johnsen, M. (2017). The future of Artificial Intelligence in Digital Marketing: The next big technological break. Maria Johnsen.
  • Ford, M. (2018). Architects of Intelligence: The truth about AI from the people building it. Packt Publishing Ltd.
  • Kumar, A., Bansal, P., Kumar, A., Gupta, A. K., & Choudhari, A. Digital Manufacturing: Artificial Intelligence in Industry 5.0. In Artificial Intelligence and Communication Techniques in Industry 5.0 (pp. 1-25). CRC Press.
  • Bughin, J., Hazan, E., Sree Ramaswamy, P., DC, W., & Chu, M. (2017). Artificial intelligence the next digital frontier.
  • Voth, D. (2004). Holonics in manufacturing: Bringing intelligence closer to the machine. IEEE Intelligent Systems, 19(6), 4-6.
  • Syam, N., & Sharma, A. (2018). Waiting for a sales renaissance in the fourth industrial revolution: Machine learning and artificial intelligence in sales research and practice. Industrial marketing management, 69, 135-146.
  • Chui, M., & Francisco, S. (2017). Artificial intelligence the next digital frontier. McKinsey and Company Global Institute, 47(3.6), 6-8.
  • Abbas, Z., & Hussain, N. (2017). Enterprise Integration in Modern Cloud Ecosystems: Patterns, Strategies, and Tools.
  • Kaur, A. (2015). Examining The Future of AI in Talent Acquisition. e-IIEMS J, 32.
    • Article Viewed: 20 Total Download

    Downloads

    ##plugins.themes.ojsPlusA.frontend.article.downloadstatastics##

    Make a Submission

    Make a Submission

    Current Issue

    Information

    Keywords