Event Driven System Design with High Availability
DOI:
https://doi.org/10.32628/CSEIT251112158Keywords:
Kafka, AWS MSK, High Availability, Idempotent Consumers, Fault ToleranceAbstract
Modern event-driven architectures rely on frameworks like Apache Kafka to enable real-time asynchronous processing. However, ensuring high availability across distributed systems, particularly in a multi-region environment, poses challenges such as avoiding duplication of data and achieving seamless failover with minimal manual intervention. This paper proposes a robust solution that leverages AWS Managed Streaming for Apache Kafka (MSK) to design an active-active highly available system. By implementing a smart consumer-based idempotent mechanism with a globally shared database and message-level UUIDs, the system ensures zero message duplication while maintaining fault tolerance. This design effectively minimizes latency, supports system resilience, and addresses the challenges of distributed event streaming systems. Practical considerations, including scalability, failure recovery, and latency optimization, are also discussed to provide a comprehensive guide to the deployment of resilient event-driven architectures in cloud-based infrastructures.
Downloads
References
V. Addala, D. Goswami, and M. H. Hsu, “Asynchronous communication in web applications,” in Proceedings of the International Conference on Web Applications, Sep. 2011
G. Rankovski and I. Chorbev, “Improving scalability of web applications by utilizing asynchronous i/o,” 2016. DOI: 10.1007/978-3-319-68855-8_21 DOI: https://doi.org/10.1007/978-3-319-68855-8_21
U. Zdun, M. Voelter, and M. Kircher, “Design and im-plementation of an asynchronous invocation framework for web services,” 2003. DOI: 10.1007/978-3-540-39872-1_6 DOI: https://doi.org/10.1007/978-3-540-39872-1_6
U. Praphamontripong, S. S. Gokhale, A. Gokhale, and J. Gray, “Performance analysis of an asynchronous web server,” in Computer Software and Applications Conference, Sep. 2006. DOI: 10.1109/COMPSAC.2006.148 DOI: https://doi.org/10.1109/COMPSAC.2006.148
S. Pradhan and M. Ray, “Asynchronous testing in web applications,” 2021. DOI: 10.1007/978-981-15-5971-6_39 DOI: https://doi.org/10.1007/978-981-15-5971-6_39
Amazon Web Services. “Amazon ec2 documentation.” Accessed: 2024-10-15. (2024), [Online]. Available: https://docs.aws.amazon.com/ec2/?nc2=h_ql_doc_ec2
Amazon Web Services. “Getting started with amazon msk.” Accessed: 2024-10-15. (2024), [Online]. Available: https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html
D. Vohra, “Using apache kafka,” 2016. DOI: 10.1007/978-1-4842-1830-3_12 DOI: https://doi.org/10.1007/978-1-4842-1830-3_12
Amazon Web Services. “Amazon msk replicator.” Accessed: 2024-10-15. (2024), [Online]. Available: https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator.html
Amazon Web Services. “Amazon aurora global databases.” Accessed: 2024-10-15. (2024), [Online]. Available: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html
Wikipedia Contributors. “Universally unique identifier.” Accessed: 2024-10-15. (2024), [Online]. Available: https://en.wikipedia.org/wiki/Universally_unique_identifier
J. Rodrigues. “Efficient job scheduling and locking in spring boot applications: Quartz vs shedlock.” Accessed: 2024-10-15. (2024), [Online]. Available: https://jetherrodrigues.dev.br/efficient-job-scheduling-and-locking-in-spring-boot-applications-quartz-vs-shedlock-b3ccca18cfe5
T. P. Raptis and A. Passarella, “On efficiently partitioning a topic in apache kafka,” in Proceedings of the International Conference on Computer, Information and Telecommunication Systems, May 2022. DOI: 10.1109/CITS55221.2022.9832981 DOI: https://doi.org/10.1109/CITS55221.2022.9832981
S. A. Noghabi and et al., “Stateful scalable stream processing at linkedin,” in Proceedings of Very Large Data Bases, Jan. 2017.
Z. Zhou, “Research on change-data-capture technology,” Journal of Wuhan University of Technology, Jan.2009
References
Behera, R., et al. (2020). Optimizing Energy Consumption in Data Centers Using Predictive Analytics. Journal of Sustainable Computing, 15(4), 23-35.
Pinarayi, T., & Thomas, J. (2021). AI-Driven Cooling Systems in Large-Scale Data Centers. International Journal of Green Technology, 9(3), 187-203.
Fusi, G., et al. (2019). Reducing Carbon Emissions in IT through AI-Based Compression Techniques. Energy Informatics, 12(2), 89-97.
Chakrabarti, D., & Saha, P. (2020). Content Delivery Optimization Using AI Models. Computing and Sustainability, 7(1), 45-61.
Zhang, Y., et al. (2021). AI-Enabled Network Optimization for Sustainable IT Infrastructure. IEEE Transactions on Sustainable Computing, 8(1), 56-71.
Sridhar, A., & Kumar, R. (2022). Edge Computing and AI: Enhancing Network Sustainability. Communications in Sustainable Development, 14(5), 32-44.
Greco, P., et al. (2020). AI in Sustainable Software Development. Advances in Green Software Engineering, 10(2), 98-112.
Martinez, L., & Gupta, A. (2022). Renewable Energy Integration in IT Using AI Technologies. Renewable Energy Research, 18(3), 67-79.
Osei, K., et al. (2021). Challenges in Implementing AI for Sustainability. Journal of Applied AI Ethics, 4(2), 101-114.
Li, X., et al. (2023). The Cost and Ethical Implications of AI in Sustainable IT Practices. Global AI & Sustainability, 6(1), 123-136.
Bolón-Canedo, V., Morán-Fernández, L., Cancela, B., & Alonso-Betanzos, A. (2024). A review of green artificial intelligence: Towards a more sustainable future. Neurocomputing,599,128096–128096. https://doi.org/10.1016/j.neucom.2024.128096 DOI: https://doi.org/10.1016/j.neucom.2024.128096
The Environmental Footprint of LLMs: A Reflection on a Warm Fall Day https://www.linkedin.com/pulse/environmental-footprint-llms-reflection-warm-fall-day-singh-phd/
A.S. Luccioni, A. Hernandez-Garcia, Counting carbon: A survey of factors influencing the emissions of machine learning, 2023, arXiv preprint arXiv: 2302.08476.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 International Journal of Scientific Research in Computer Science, Engineering and Information Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
