Techniques for Data Integrity across Distributed Resource Planning Systems

Authors

  • Gnana Teja Reddy  Software Engineer, Google, USA
  • Nelavoy Rajendra  San Francisco Bay Area, USA

Keywords:

Data Integrity, Distributed Systems, Resource Planning, Consistency, Replication, CAP Theorem, Consensus Algorithms, Eventual Consistency.

Abstract

Distributed Resource Planning Systems (DRPS) allow the scheduling, resource management, and co-ordinate and synchronization of key operations in enterprises with different branches in different locations. However, maintaining integrity in such systems is difficult because updates, network delays, and computer hardware differences make the data conflict or stale. This paper highlights these challenges by capturing, evaluating, and discussing superior strategies for data consistency, reliability, and accuracy in DRPS. It starts by situating the CAP theorem and comparing consistency, availability, and partition tolerance for real-world resource acquisition purposes. The empirical paper examines different approaches, such as leader-based replication, CQRS, and CRDTs, using their perspectives on how they are useful in building strong consistency, scalability, and conflict resolution. Further, it looks at transactional consistency with distributed protocols such as Two-Phase Commit and the part played by Multi-Version Concurrency Control (MVCC) in concurrent operations. Event sourcing is proposed to make data more traceable and recover from faults. The performance of the consensus algorithms, such as Paxos and Raft, is assessed in terms of providing synchronous views. The paper also suggests an equal blend of methodologies, which, if adopted, will maximize performance, scalability, and data consistency for smooth functioning across the distributed architecture. This work equips the practitioners with the knowledge that helps design systems that can withstand the test of time and ensure quality data is maintained, a crucial factor for success in complex, dynamic, and resource-intensive environments.

References

  1. Abadi, D. J. (2009). Data management in the cloud: Limitations and opportunities. IEEE Data Eng. Bull., 33(1), 3-12.
  2. Arnold, K. A., & Loughlin, C. (2013). Integrating transformational and participative versus directive leadership theories: Examining intellectual stimulation in male and female leaders across three contexts. Leadership & Organization Development Journal, 34(1), 67-84.
  3. Bailis, P., Davidson, A., Fekete, A., Ghodsi, A., Hellerstein, J. M., & Stoica, I. (2013). Highly available transactions: virtues and limitations (extended version). arXiv preprint arXiv:1302.0309.
  4. Baker, J., Bond, C., Corbett, J. C., Furman, J., Khorlin, A., Larson, J,. & Yushprakh, V. (2011). Megastore: Providing scalable, highly available storage for interactive services. CIDR, 6, 223-234.
  5. Bannour, F., Souihi, S., & Mellouk, A. (2017). Distributed SDN control: Survey, taxonomy, and challenges. IEEE Communications Surveys & Tutorials, 20(1), 333-354.
  6. Bernstein, P. A., & Goodman, N. (1981). Concurrency control in distributed database systems. ACM Computing Surveys (CSUR), 13(2), 185–221.
  7. Bernstein, P. A., Hadzilacos, V., & Goodman, N. (1987). Concurrency control and recovery in database systems. Addison-Wesley.
  8. Brewer, E. (2012). CAP twelve years later: How the "rules" have changed. Computer, 45(2), 23–29.
  9. Campbell, L., & Majors, C. (2017). Database reliability engineering: designing and operating resilient database systems. " O'Reilly Media, Inc.".
  10. Carpineto, C., & Romano, G. (2012). A survey of automatic query expansion in information retrieval. Acm Computing Surveys (CSUR), 44(1), 1-50.
  11. Castro, M., & Liskov, B. (1999). Practical Byzantine fault tolerance. In OSDI (Vol. 99, pp. 173-186).
  12. Chandra, T. D., & Toueg, S. (1996). Unreliable failure detectors for reliable distributed systems. Journal of the ACM (JACM), 43(2), 225-267.
  13. Codd, E. F. (1970). A Relational Model of Data for Large Shared Data Banks. Communications of the ACM, 13(6), 377–387.
  14. Coulouris, G., Dollimore, J., & Kindberg, T. (2011). Distributed Systems: Concepts and Design (5th ed.). Addison-Wesley.
  15. DeCandia, G., Hastorun, D., Jampani, M., & Kakulapati, G. (2007). Dynamo: Amazon's highly available key-value store. ACM SIGOPS Operating Systems Review, 41(6), 205-220.
  16. Ducharme, D., & Brightman, H. (2011). Maritime Stability Operations Game'11.
  17. Evans, E. (2004). Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley.
  18. Fowler, M. (2012). Patterns of enterprise application architecture. Addison-Wesley.
  19. Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51-59.
  20. Gray, J. (1981). The transaction concept: Virtues and limitations. In Proceedings of the seventh international conference on very large data bases (pp. 144–154).
  21. Gray, J. (1981). The transaction concept: Virtues and limitations. In VLDB (Vol. 81, pp. 144-154).
  22. Gray, J., & Lamport, L. (2006). Consensus on transaction commit. ACM Transactions on Database Systems, 31(1), 133–160.
  23. Gray, J., & Reuter, A. (1992). Transaction Processing: Concepts and Techniques. Morgan Kaufmann.
  24. Hajro, A., Gibson, C. B., & Pudelko, M. (2017). Knowledge exchange processes in multicultural teams: Linking organizational diversity climates to teams’ effectiveness. Academy of Management Journal, 60(1), 345-372.
  25. Helland, P. (2015). Immutability Changes Everything. Communications of the ACM, 59(1), 64-70.
  26. Helland, P., & Campbell, C. (2009). Building on quicksand. In Proceedings of the 3rd Biennial Conference on Innovative Data Systems Research (CIDR’09) (pp. 218–231).
  27. Hohpe, G., & Woolf, B. (2012). Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley.
  28. Holt, B., Bornholt, J., Zhang, I., Ports, D., Oskin, M., & Ceze, L. (2016, October). Disciplined inconsistency with consistency types. In Proceedings of the Seventh ACM Symposium on Cloud Computing (pp. 279-293).
  29. Kasheff, Z., & Walsh, L. (2014). Ark: a real-world consensus implementation. arXiv preprint arXiv:1407.4765.
  30. Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas behind Reliable, Scalable, and Maintainable Systems. O’Reilly Media.
  31. Kung, H. T., & Robinson, J. T. (1981). On optimistic methods for concurrency control. ACM Transactions on Database Systems (TODS), 6(2), 213–226.
  32. Lamport, L. (1998). The part-time parliament. ACM Transactions on Computer Systems, 16(2), 133–169.
  33. Lamport, L. (2001). Paxos made simple. ACM SIGACT News, 32(4), 18–25.
  34. Lin, M. (2009). Distributed database systems: Transaction processing and concurrency control. Journal of Systems and Software, 82(3), 482-490.
  35. Nadareishvili, I., Mitra, R., McLarty, M., & Amundsen, M. (2016). Microservice architecture: aligning principles, practices, and culture. " O'Reilly Media, Inc.".
  36. O’Neil, P. (1993). The LRU-K page replacement algorithm for database disk buffering. ACM SIGMOD Record, 22(2), 297–306.
  37. Ongaro, D., & Ousterhout, J. (2014). In search of an understandable consensus algorithm. In USENIX Annual Technical Conference (Vol. 2014).
  38. Papadimitriou, C. H. (1986). The theory of database concurrency control. Computer Science Press.
  39. Patni, M., & Elsayed, A. (2015). A comparative study of distributed transaction protocols. IEEE Transactions on Computers, 64(2), 542-554.
  40. Pease, M., Shostak, R., & Lamport, L. (1980). Reaching agreement in the presence of faults. Journal of the ACM, 27(2), 228-234.
  41. Rahimi, S., & Haug, G. (2010). Database concurrency control. International Journal of Computer Science, 8(1), 47-59.
  42. Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). Conflict-free replicated data types. In Proceedings of the 13th International Symposium on Stabilization, Safety, and Security of Distributed Systems (pp. 386–400). Springer.
  43. Slovic, P., & Weber, E. U. (2013). Perception of risk posed by extreme events. Regulation of Toxic Substances and Hazardous Waste (2nd edition)(Applegate, Gabba, Laitos, and Sachs, Editors), Foundation Press, Forthcoming.
  44. Stonebraker, M. (1979). Concurrency control and consistency of multiple copies in distributed Ingres. IEEE Transactions on Software Engineering, 3, 188–194.
  45. Stonebraker, M. (1986). The case for shared nothing. IEEE Database Engineering Bulletin, 25(3), 4–9.
  46. Stonebraker, M., & Cattell, R. (2011). 10 rules for scalable performance in ‘simple operation’ datastores. Communications of the ACM, 54(6), 72-80.
  47. Vogels, W. (2009). Eventually consistent. Communications of the ACM, 52(1), 40-44.
  48. Wada, H., Fekete, A., Zhao, L., Lee, K., & Liu, A. (2011). Data consistency trade-offs in distributed database systems: CAP is only part of the story. IEEE Internet Computing, 15(2), 14-20.
  49. Weikum, G., & Vossen, G. (2001). Transactional Information Systems: Theory, Algorithms, and the Practice of Concurrency Control and Recovery. Morgan Kaufmann.
  50. Zhang, Q., Chen, Z., & Li, C. (2013). The design of a distributed database system for reliability. Journal of Systems Architecture, 59(10), 1349-1362.
  51. Zhuravlev, S., Saez, J. C., Blagodurov, S., Fedorova, A., & Prieto, M. (2012). Survey of scheduling techniques for addressing shared resources in multicore processors. ACM Computing Surveys (CSUR), 45(1), 1-28.

IInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology

Downloads

Published

2021-06-30

Issue

Volume 3, Issue 7, September-October-2018

Section

Research Articles

License

Copyright (c) IJSRCSEIT

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

[1]
Gnana Teja Reddy, Nelavoy Rajendra, " Techniques for Data Integrity across Distributed Resource Planning Systems " International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 3, Issue 7, pp.527-550, September-October-2018.