Model Validation and Testing in Simulation: a Literature Review

Authors Naoum Tsioptsias, Antuela Tako, Stewart Robinson



PDF
Thumbnail PDF

File

OASIcs.SCOR.2016.6.pdf
  • Filesize: 407 kB
  • 11 pages

Document Identifiers

Author Details

Naoum Tsioptsias
Antuela Tako
Stewart Robinson

Cite AsGet BibTex

Naoum Tsioptsias, Antuela Tako, and Stewart Robinson. Model Validation and Testing in Simulation: a Literature Review. In 5th Student Conference on Operational Research (SCOR 2016). Open Access Series in Informatics (OASIcs), Volume 50, pp. 6:1-6:11, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2016)
https://doi.org/10.4230/OASIcs.SCOR.2016.6

Abstract

Model validation is a key activity undertaken during the model development process in simulation. There is a large body of literature on model validation, albeit there exists little convergence in terms of the definitions, types of validity, and tests used. Yet it is not clear what standards should be taken into consideration to avoid developing what could be considered to be invalid or wrong models. In this paper we examine existing literature on model validation with the view to identifying the existing validation approaches and types of tests used to assess model validity. In this review we focus our attention on three domains that usually overlap in methods and techniques: general Operational Research (OR), Modelling & Simulation (M&S) and Computer Science (CS). We analyze each field to identify the aspects of validity considered including the tests used, the validation approach taken, i.e. the suggested level of validity achieved (if this applies) and the reported outcome. The analysis shows that there are common validation practices used in all three fields as well as new ideas that could be adopted in discrete event simulation. Some main points of concurrence include the lack of universal validation, the continuous need for validation, and, the indispensable need for modelers and users to work closely together during the model validation process. This review provides an initial categorization of literature on model validation which can in turn be used as a basis for future work in investigating how and to what extent models are considered sufficiently valid.
Keywords
  • Validation
  • Simulation
  • Literature review
  • Types of validity
  • Field Comparisons

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads

References

  1. A. D. Athanassopoulos. Decision Support for Target-Based Resource Allocation of Public Services in Multiunit and Multilevel Systems. Management Science, 44(2):173-187, 1998. URL: http://dx.doi.org/10.1287/mnsc.44.2.173.
  2. O. Balci. Validation, verification, and testing techniques throughout the life cycle of a simulation study. Annals of Operations Research, 53:121-173, 1994. URL: http://dx.doi.org/10.1109/WSC.1994.717129.
  3. O. Balci. Principles and techniques of simulation validation, verification, and testing. In Winter Simulation Conference Proceedings, 1995., pages 147-154, 1995. URL: http://dx.doi.org/10.1109/WSC.1995.478717.
  4. Y. Barlas. Multiple tests for validation of system dynamics type of simulation models. European Journal of Operational Research, 42(1):59-87, 1989. URL: http://dx.doi.org/10.1016/0377-2217(89)90059-3.
  5. M. J. Bayarri, J. O. Berger, R. Paulo, J. Sacks, J. A. Cafeo, J. Cavendish, C.-H. Lin, and J. Tu. A Framework for Validation of Computer Models. Technometrics, 49(2):138-154, 2007. URL: http://dx.doi.org/10.1198/004017007000000092.
  6. A. J. Bennett, T. Field, and P. Harrison. Modelling and validation of shared memory coherency protocols. Performance evaluation, 27&28:541-563, 1996. Google Scholar
  7. G. E. P. Box and N. R. Draper. Empirical model-building and response surfaces. Wiley series in probability and mathematical statistics: Applied probability and statistics. Wiley, 1987. Google Scholar
  8. R. J. Brooks and A. M. Tobias. Choosing the best model: Level of detail, complexity, and model performance. Mathematical and Computer Modelling, 24(4):1-14, 1996. URL: http://dx.doi.org/10.1016/0895-7177(96)00103-3.
  9. S. I. Gass. Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis. Operations Research, 31(4):603-631, 1983. URL: http://dx.doi.org/10.1287/opre.31.4.603.
  10. S. I. Gass. Model accreditation: A rationale and process for determining a numerical rating. European Journal of Operational Research, 66(2):250-258, 1993. URL: http://dx.doi.org/10.1016/0377-2217(93)90316-F.
  11. S. Groesser and M. Schwaninger. Contributions to model validation: hierarchy, process, and cessation. System Dynamics Review, 28(2):157-181, 2012. URL: http://dx.doi.org/10.1002/sdr.
  12. H. Gull, S. Alrashed, and S. Z. Iqbal. Validation of Usability Driven Web based Software Process Model using Simulation. Procedia Computer Science, 62:487-494, 2015. URL: http://dx.doi.org/10.1016/j.procs.2015.08.520.
  13. H. A. Hahn. The conundrum of verification and validation of social science-based models. Procedia Computer Science, 16:878-887, 2013. URL: http://dx.doi.org/10.1016/j.procs.2013.01.092.
  14. M. Landry, J.-L. Malouin, and M. Oral. Model validation in operations research. European Journal of Operational Research, 14(3):207-220, 1983. URL: http://dx.doi.org/10.1016/0377-2217(83)90257-6.
  15. A. M. Law and D. M. Kelton. Simulation Modeling and Analysis. McGraw-Hill Higher Education, 3rd edition, 1999. Google Scholar
  16. M. Le Goc, C. Frydman, and L. Torres. Verification and validation of the SACHEM conceptual model. International Journal of Human-Computer Studies, 56(2):199-223, 2002. URL: http://dx.doi.org/10.1006/ijhc.2001.0521.
  17. A. A. Longaray, L. Ensslin, S. R. Ensslin, and I. O. da Rosa. Assessment of a Brazilian public hospital’s performance for management purposes: A soft operations research case in action. Operations Research for Health Care, 5:28-48, 2015. URL: http://dx.doi.org/10.1016/j.orhc.2015.05.001.
  18. R. E. Nance and J. D. Arthur. Software Requirements Engineering: Exploring the Role in Simulation Model Development. In Proceedings of the Third Operational Research Society Simulation Workshop (SW06), pages 117-127, 2006. Google Scholar
  19. M. Oral and O. Kettani. The facets of the modeling and validation process in operations research. European Journal of Operational Research, 66(2):216-234, 1993. URL: http://dx.doi.org/10.1016/0377-2217(93)90314-D.
  20. D. K. Pace. Modeling and simulation verification and validation challenges. Johns Hopkins APL Technical Digest (Applied Physics Laboratory), 25(2):163-172, 2004. Google Scholar
  21. Ö. Pala, J. A. M. Vennix, and J. P. C. Kleijnen. Validation in Soft OR, Hard OR and System Dynamics: A Critical Comparison and Contribution to the Debate. In The 17th International Conference of The System Dynamics Society and the 5th Australian & New Zealand Systems Conference, 1999. URL: http://www.systemdynamics.org/conferences/1999/PAPERS/PARA199.PDF.
  22. K. Poolla, P. Khargonekar, A. Tikku, J. Krause, and K. Nagpal. A time-domain approach to model validation. Transactions on Automatic Control, 39(5):951-959, 1994. URL: http://dx.doi.org/10.1109/9.284871.
  23. S. Robinson. Simulation model verification and validation: Increasing the users' confidence. In Proceedings of the 1997 Winter Simulation Conference, pages 53-59, 1997. URL: http://dx.doi.org/10.1145/268437.268460.
  24. S. Robinson. Simulation: The Practice of Model Development and Use. Palgrave Macmillan, 2nd edition, 2014. Google Scholar
  25. S. Robinson, C. Worthington, N. Burgess, and Z. J. Radnor. Facilitated modelling with discrete-event simulation: Reality or myth? European Journal of Operational Research, 234(1):231-240, 2014. URL: http://dx.doi.org/10.1016/j.ejor.2012.12.024.
  26. N. Ronald, T. Arentze, and H. Timmermans. Towards process validation for complex transport models: A sensitivity analysis of a social network-enhanced activity-travel model. Computers, Environment and Urban Systems, 55:24-32, 2016. URL: http://dx.doi.org/10.1016/j.compenvurbsys.2015.09.005.
  27. C. J. Roy and W. L. Oberkampf. A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Computer Methods in Applied Mechanics and Engineering, 200(25-28):2131-2144, 2011. URL: http://dx.doi.org/10.1016/j.cma.2011.03.016.
  28. R. G. Sargent. Some approaches and paradigms for verifying and validating simulation models. In Proceedings of the 2001 Winter Simulation Conference, pages 106-114, 2001. URL: http://dx.doi.org/10.1109/WSC.2001.977367.
  29. R. G Sargent. Verification and validation of simulation models. Journal of Simulation, 7(1):12-24, 2012. http://arxiv.org/abs/10, URL: http://dx.doi.org/10.1057/jos.2012.20.
  30. J. H. Smith. Modeling muddles: Validation beyond the numbers. European Journal of Operational Research, 66(2):235-249, 1993. URL: http://dx.doi.org/10.1016/0377-2217(93)90315-E.
  31. A. A. Tako and K. Kotiadis. PartiSim: A multi-methodology framework to support facilitated simulation modelling in healthcare. European Journal of Operational Research, 244(2):555-564, 2015. URL: http://dx.doi.org/10.1016/j.ejor.2015.01.046.
  32. B. H. Thacker, S. W. Doebling, F. M. Hemez, M. C. Anderson, J. E. Pepin, and E. A. Rodriguez. Concepts of Model Verification and Validation. Technical report, Los Alamos National Laboratory, University of California, 2004. URL: http://dx.doi.org/10.2172/835920.
  33. U.S. G.A.O. Guidelines for Model Evaluation. Technical Report January, National Criminal Justice Reference Service (NCJRS), 1979. URL: https://www.ncjrs.gov/App/abstractdb/AbstractDBDetails.aspx?id=84469.
  34. A. S. White and R. Sinclair. Quantitative validation techniques a database. (I). Simple examples. Simulation Modelling Practice and Theory, 12(6):451-473, 2004. URL: http://dx.doi.org/10.1016/j.simpat.2004.06.001.
  35. T. R. Willemain. Insights on Modeling from a Dozen Experts. Operations Research, 42(2):213-222, 1994. URL: http://dx.doi.org/10.1287/opre.42.2.213.
  36. M. V. Zelkowitz and D. R. Wallace. Experimental model for validating technology. IEEE Computer, 31(5):23-31, 1998. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail