Shadow Testing in Autonomous Vehicles : A Novel Approach to Validating Full Self-Driving AI Systems

Revanth Pathuri

doi:10.32628/CSEIT24106165

Authors

Revanth Pathuri North Carolina State University, USA Author

DOI:

https://doi.org/10.32628/CSEIT24106165

Keywords:

Autonomous Vehicles, Shadow Testing, Full Self-Driving AI, Safety Validation, Machine Learning

Abstract

As autonomous vehicles progress towards widespread deployment, ensuring the reliability and safety of Full Self-Driving (FSD) AI systems remains a critical challenge. This article presents a comprehensive analysis of shadow testing, an innovative approach to validating new FSD AI models in real-world conditions without compromising user safety. We examine the methodology of deploying new AI models in "shadow mode," where they process live sensory data alongside the operational system but do not control the vehicle. This approach enables the collection of valuable performance metrics and the identification of edge cases while mitigating risks associated with direct deployment. Our study demonstrates how shadow testing facilitates a continuous feedback loop for iterative improvement, enhancing model reliability through data-driven refinement. We further explore the challenges of implementing shadow testing at scale, including data processing requirements and ethical considerations. Our findings suggest that shadow testing not only accelerates the development of robust FSD systems but also plays a crucial role in building public trust in autonomous vehicle technology. This article contributes to the growing body of knowledge on AI safety validation techniques and provides insights for standardizing testing protocols in the autonomous vehicle industry.

Downloads

Download data is not yet available.

References

P. Koopman and M. Wagner, "Challenges in Autonomous Vehicle Testing and Validation," SAE International Journal of Transportation Safety, vol. 4, no. 1, pp. 15-24, 2016. [Online]. Available: https://doi.org/10.4271/2016-01-0128 DOI: https://doi.org/10.4271/2016-01-0128

A. Corso, P. Du, K. Driggs-Campbell, and M. J. Kochenderfer, "Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation," in 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 2019, pp. 163-168. [Online]. Available: https://arxiv.org/abs/1908.01046 DOI: https://doi.org/10.1109/ITSC.2019.8917242

L. Tang, J. Mars, N. Vachharajani, R. Hundt, and M. L. Soffa, "The impact of memory subsystem resource sharing on datacenter applications," in Proceedings of the 38th annual international symposium on Computer architecture (ISCA '11), 2011, pp. 283-294. [Online]. Available: https://doi.org/10.1145/2000064.2000099 DOI: https://doi.org/10.1145/2000064.2000099

P. Koopman and M. Wagner, "Toward a Framework for Highly Automated Vehicle Safety Validation," SAE Technical Paper 2018-01-1071, 2018. [Online]. Available: https://doi.org/10.4271/2018-01-1071 DOI: https://doi.org/10.4271/2018-01-1071

A. Gambi, M. Mueller, and G. Fraser, "Automatically Testing Self-Driving Cars with Search-Based Procedural Content Generation," in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019), 2019, pp. 318-328. [Online]. Available: https://doi.org/10.1145/3293882.3330566 DOI: https://doi.org/10.1145/3293882.3330566

S. Liu, L. Li, J. Tang, S. Wu, and J. L. Gaudiot, "Creating Autonomous Vehicle Systems," Synthesis Lectures on Computer Science, vol. 6, no. 1, pp. 1-186, 2017. [Online]. Available: https://doi.org/10.2200/S00787ED1V01Y201707CSL009 DOI: https://doi.org/10.2200/S00787ED1V01Y201707CSL009

W. Wang, et al., "A Survey of Deep Learning Techniques for Autonomous Driving," Journal of Field Robotics, vol. 38, no. 2, pp. 137-191, 2021. [Online]. Available: https://doi.org/10.1002/rob.21918 DOI: https://doi.org/10.1002/rob.21918

M. Hallerbach, Y. Xia, U. Eberle, and F. Koester, "Simulation-Based Identification of Critical Scenarios for Cooperative and Automated Vehicles," SAE International Journal of Connected and Automated Vehicles, vol. 1, no. 2, pp. 93-106, 2018. [Online]. Available: https://doi.org/10.4271/2018-01-1066 DOI: https://doi.org/10.4271/2018-01-1066

J. M. Anderson, K. Nidhi, K. D. Stanley, P. Sorensen, C. Samaras, and O. A. Oluwatola, "Autonomous Vehicle Technology: A Guide for Policymakers," RAND Corporation, 2016. [Online]. Available: https://www.rand.org/pubs/research_reports/RR443-2.html DOI: https://doi.org/10.7249/RR443-2

A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. M. Allen, V. D. Lam, A. Bewley, and A. Shah, "Learning to Drive in a Day," in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8248-8254. [Online]. Available: https://doi.org/10.1109/ICRA.2019.8793742 DOI: https://doi.org/10.1109/ICRA.2019.8793742

S. S. Banerjee, S. Jha, J. Cyriac, Z. T. Kalbarczyk, and R. K. Iyer, "Hands Off the Wheel in Autonomous Vehicles?: A Systems Perspective on over a Million Miles of Field Data," in 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2018, pp. 586-597. [Online]. Available: https://doi.org/10.1109/DSN.2018.00066 DOI: https://doi.org/10.1109/DSN.2018.00066

W. Schwarting, J. Alonso-Mora, and D. Rus, "Planning and Decision-Making for Autonomous Vehicles," Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 187-210, 2018. [Online]. Available: https://doi.org/10.1146/annurev-control-060117-105157 DOI: https://doi.org/10.1146/annurev-control-060117-105157

D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané, "Concrete Problems in AI Safety," arXiv preprint arXiv:1606.06565, 2016. [Online]. Available: https://arxiv.org/abs/1606.06565