Document Open Access Logo

Assessing the Construct Validity of Object-Oriented, Class-Level Code Quality Metrics

Authors Hera Arif , Miikka Kuutila , Paul Ralph

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.10.pdf
  • Filesize: 1.18 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Empirical software validation
Keywords
  • Code quality metrics
  • Factor analysis
  • Exploratory factor analysis
  • Confirmatory factor analysis
  • Software quality
  • Size
  • Inheritance
  • Coupling
  • Cohesion

Metrics

Abstract

Background. Code quality metrics are intended to measure latent properties of software source code. Although numerous code metrics have been proposed and used, their construct validity is rarely evaluated. Thus, the extent to which code metrics actually measure what they claim to measure is often unclear.

Aim. Drawing from modern measurement theory, we investigate the construct validity of common class-level, object-oriented code quality metrics. 

Method. As code quality metrics are intended to reflect latent attributes, such as cohesion and coupling, we identified the factor structure of code quality metrics using Exploratory Factor Analysis (EFA). The metrics were extracted from the Apache Maven project by three software tools: Designite, JHawk, and Understand. The factor structure was later verified using Confirmatory Factor Analysis (CFA) on 22 randomly selected open source projects meeting a predetermined eligibility criteria.

Results. 24 code quality metrics that correspond to six constructs: Cohesion, In-Coupling, Out-Coupling, Size, Sub-Inheritance (related to subclasses), and Sup-Inheritance (related to superclasses) were revealed in the underlying factor structure. Ten metrics did not correspond to any known dimension of software quality and were removed in the exploratory analysis. Ten additional metrics exhibited low loadings in the confirmatory analysis, suggesting their removal from the final measurement model. Size, Cohesion, Inheritance, and Coupling were the constructs retained, with subcategories identified for Inheritance and Coupling.

Conclusions. Our results strongly support the construct validity of 24 code quality metrics. Coupling and Inheritance are revealed as multidimensional constructs, since they require measuring two different concepts, revealed as sub-categories in our analysis, and Complexity may be better explored in a multilevel model. Our results also corroborate the relationship between Cohesion, Size, and Out-Coupling which can be further explored in a structural model. Some metrics from the Chidamber & Kemerer metrics suite are found to perhaps be measuring different constructs than intended. Our results reveal the need for creating or integrating metrics that reflect existing constructs but measure fundamentally different properties to improve the content validity of the measurement model. Additionally, we provide useful recommendations for researchers, developers, and tool providers which stem from theoretical and empirical justification. Overall, our study demonstrates the value of applying modern measurement theory and latent variable modeling in validating software code quality metrics.

Cite As Get BibTex

Hera Arif, Miikka Kuutila, and Paul Ralph. Assessing the Construct Validity of Object-Oriented, Class-Level Code Quality Metrics. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 10:1-10:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.10

Author Details

Hera Arif
  • Dalhousie University, Halifax, Canada
Miikka Kuutila
  • LUT University, Lahti, Finland
Paul Ralph
  • Dalhousie University, Halifax, Canada

Funding

  • Ralph, Paul: NSERC Discovery Grant RGPIN-2020-05001

Supplementary Materials

References

  1. Fernando Brito Abreu and Rogério Carapuça. Object-oriented software engineering: Measuring and controlling the development process. In Proc. of the 4th Int. Conf. on Soft. Quality, volume 186, 1994. Google Scholar
  2. Luca Ardito, Riccardo Coppola, Luca Barbato, and Diego Verga. A tool-based perspective on software code maintainability metrics: A systematic literature review. Scientific Programming, 2020(1):8840389, 2020. URL: https://doi.org/10.1155/2020/8840389.
  3. V.R. Basili, L.C. Briand, and W.L. Melo. A validation of object-oriented design metrics as quality indicators. IEEE Trans. on Soft. Eng., 22(10):751-761, 1996. URL: https://doi.org/10.1109/32.544352.
  4. Eric Bouwers, Joost Visser, and Arie Deursen. Getting what you measure. Communications of The ACM - CACM, 55:54-59, May 2012. URL: https://doi.org/10.1145/2209249.2209266.
  5. L. C. Briand, J. W. Daly, and J. Wust. A unified framework for cohesion measurement in object-oriented systems. In Proc. Fourth Int. Soft. Metrics Symp., pages 43-53, 1997. URL: https://doi.org/10.1109/METRIC.1997.637164.
  6. L. C. Briand, J. W. Daly, and J. K. Wust. A unified framework for coupling measurement in object-oriented systems. IEEE Transactions on Software Engineering, 25(1):91-121, 1999. URL: https://doi.org/10.1109/32.748920.
  7. L. C. Briand, S. Morasca, and V. R. Basili. Property-based software engineering measurement. IEEE Trans. on Soft. Eng., 22(1):68-86, 1996. URL: https://doi.org/10.1109/32.481535.
  8. Lionel C. Briand, Jürgen Wüst, John W. Daly, and D. Victor Porter. Exploring the relationships between design measures and software quality in object-oriented systems. J. of Sys. & Soft., 51(3):245-273, 2000. URL: https://doi.org/10.1016/S0164-1212(99)00102-8.
  9. T. A. Brown. Confirmatory Factor Analysis for Applied Research. Guilford Publications, 2015. Google Scholar
  10. Rachel Burrows, Alessandro Garcia, and François Taïani. Coupling metrics for aspect-oriented programming: A systematic review of maintainability studies. In Leszek A. Maciaszek, César González-Pérez, and Stefan Jablonski, editors, Eval. of Novel Approaches to Soft. Eng., pages 277-290. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. Google Scholar
  11. N.R. Campbell. Physics: The Elements. Cambridge University Press, 2013. Google Scholar
  12. R. B. Cattell. The scree test for the number of factors. Multivariate Behavioral Research, 1(2):245-276, 1966. Google Scholar
  13. J. C. Cherniavsky and C. H. Smith. On weyuker’s axioms for software complexity measures. IEEE Trans. on Soft. Eng., 17, June 1991. URL: https://doi.org/10.1109/32.87287.
  14. S. R. Chidamber and C. F. Kemerer. A metrics suite for object oriented design. IEEE Trans. Softw. Eng., 20(6):476-493, June 1994. URL: https://doi.org/10.1109/32.295895.
  15. Shyam R. Chidamber and Chris F. Kemerer. Towards a metrics suite for object oriented design. In Proc. on OOPLSA, pages 197-211. ACM, New York, USA, 1991. URL: https://doi.org/10.1145/117954.117970.
  16. Shaiful Chowdhury, Reid Holmes, Andy Zaidman, and Rick Kazman. Revisiting the debate: Are code metrics useful for measuring maintenance effort? Empirical Softw. Eng., 27(6), November 2022. URL: https://doi.org/10.1007/s10664-022-10193-8.
  17. Curtis R. Cook and Andreas Roesch. Real-time software metrics. J. of Sys. & Soft., 24(3):223-237, 1994. Oregon Workshop on Software Metrics. URL: https://doi.org/10.1016/0164-1212(94)90065-5.
  18. A. B. Costello and Jason Osborne. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10:1-9, January 2005. Google Scholar
  19. Lee J. Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16:297-334, 1951. URL: https://doi.org/10.1007/BF02310555.
  20. K. El Emam, S. Benlarbi, N. Goel, and S.N. Rai. The confounding effect of class size on the validity of object-oriented metrics. IEEE Trans. on Soft. Eng., 27(7):630-650, 2001. URL: https://doi.org/10.1109/32.935855.
  21. Norman Fenton. Software measurement: A necessary scientific basis. IEEE Trans. on Soft. Eng., 20:199-206, April 1994. URL: https://doi.org/10.1109/32.268921.
  22. A Field. Discovering Statistics Using IBM SPSS Statistics. Sage, 5th edition, 2017. Google Scholar
  23. Enrico Fregnan, Tobias Baum, Fabio Palomba, and Alberto Bacchelli. A survey on software coupling relations and tools. Inf. & Soft. Technol., 107:159-178, 2019. URL: https://doi.org/10.1016/j.infsof.2018.11.008.
  24. Joseph Gil and Gal Lalouche. On the correlation between size and metric validity. Empirical Soft. Eng., 22:2585-2611, 2017. URL: https://doi.org/10.1007/S10664-017-9513-5.
  25. Oswaldo Gómez, Hanna Oktaba, Mario Piattini, and Félix García. A systematic review measurement in software engineering: State-of-the-art in measures. In Soft. & Data Technol., pages 165-176. Springer, Berlin, Heidelberg, 2008. Google Scholar
  26. T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy. Predicting fault incidence using software change history. IEEE Trans. on Soft. Eng., 26(7):653-661, 2000. URL: https://doi.org/10.1109/32.859533.
  27. Daniel Graziotin, Per Lenberg, Robert Feldt, and Stefan Wagner. Psychometrics in behavioral software engineering: A methodological introduction with guidelines. ACM Trans. on Soft. Eng. & Methodol., 31:7, January 2022. URL: https://doi.org/10.1145/3469888.
  28. J. F. Hair, W. C. Black, B. J. Babin, and R. E. Anderson. Multivariate Data Analysis. Always learning. Pearson Education Limited, 2013. Google Scholar
  29. J. F. Hair, W. C. Black, B. J. Babin, and R. E. Anderson. Multivariate Data Analysis. Cengage, 2018. Google Scholar
  30. Maurice H. Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., USA, 1977. Google Scholar
  31. Rachel Harrison, Steve J. Counsell, and Reuben V. Nithi. An evaluation of the mood set of object-oriented software metrics. IEEE Trans. on Soft. Eng., 24(6):491-496, 1998. URL: https://doi.org/10.1109/32.689404.
  32. Brian Henderson-Sellers, Larry L Constantine, and Ian M Graham. Coupling and cohesion (towards a valid metrics suite for object-oriented analysis and design). Object oriented systems, 3(3):143-158, 1996. Google Scholar
  33. J. Henseler. Composite-Based Structural Equation Modeling: Analyzing Latent and Emergent Variables. Methodology in the Social Sciences Series. Guilford Publications, 2020. Google Scholar
  34. J. L. Horn. A rationale and test for the number of factors in factor analysis. Psychometrika, 30:179-185, 1965. Google Scholar
  35. Baskhad Idrisov and Tim Schlippe. Program code generation with generative ais. Algorithms, 17:62, January 2024. URL: https://doi.org/10.3390/a17020062.
  36. Karl G. Jöreskog. Structural equation modeling with ordinal variables. Lecture Notes-Monograph Series, 24:297-310, 1994. URL: http://www.jstor.org/stable/4355811.
  37. H. F. Kaiser and J. Rice. Little jiffy, mark iv. Educational & Psychological Measurement, 34(1):111-117, 1974. Google Scholar
  38. Henry F. Kaiser. The application of electronic computers to factor analysis. Educational & Psychological Measurement, 20(1):141-151, 1960. Google Scholar
  39. B. Kitchenham, S.L. Pfleeger, and N. Fenton. Towards a framework for software measurement validation. IEEE Trans. on Soft. Eng., 21(12):929-944, 1995. URL: https://doi.org/10.1109/32.489070.
  40. Barbara Kitchenham. What’s up with software metrics? – A preliminary mapping study. J. of Sys. & Soft., 83(1):37-51, 2010. URL: https://doi.org/10.1016/j.jss.2009.06.041.
  41. Al Lake and Curtis R. Cook. Use of factor analysis to develop oop software complexity metrics, 1994. Google Scholar
  42. Mark Lorenz and Jeff Kidd. Object-oriented software metrics: a practical guide. Prentice-Hall, Inc., 1994. Google Scholar
  43. U. A. Mannan, I. Ahmed, and A. Sarma. Towards understanding code readability and its impact on design quality. In Proc. of the 4th ACM SIGSOFT Int. Workshop on NLP for Software Engineering, NL4SE 2018, pages 18-21, NY, USA, 2018. ACM. URL: https://doi.org/10.1145/3283812.3283820.
  44. Robert Cecil Martin. Designing object-oriented C++ applications. Prentice Hall, 1995. Google Scholar
  45. T.J. McCabe. A complexity measure. IEEE Transactions on Software Engineering, SE-2(4):308-320, 1976. URL: https://doi.org/10.1109/TSE.1976.233837.
  46. A. Meneely, B. Smith, and L Williams. Validating software metrics: A spectrum of philosophies. ACM Trans. Softw. Eng. Methodol., 21(4), February 2013. URL: https://doi.org/10.1145/2377656.2377661.
  47. J.C. Munson and T.M. Khoshgoftaar. Measuring dynamic program complexity. IEEE Soft., 9(6):48-55, 1992. URL: https://doi.org/10.1109/52.168858.
  48. Sirisha Murthy, Lohith Nagaraja, Karan Wale, and William Andreopoulos. Evaluating ai code generation quality with software metrics. SSRN 6087362, January 2026. URL: https://doi.org/10.2139/ssrn.7076299.
  49. Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot’s code suggestions. In Conf. on Mining Soft. Repositories, pages 1-5, 2022. URL: https://doi.org/10.1145/3524842.3528470.
  50. Paul Ralph, Nauman bin Ali, Sebastian Baltes, Domenico Bianculli, Jessica Diaz, Yvonne Dittrich, Neil Ernst, Michael Felderer, Robert Feldt, Antonio Filieri, et al. Empirical standards for software engineering research. arXiv preprint, 2020. URL: https://arxiv.org/abs/2010.03525.
  51. Paul Ralph, Miikka Kuutila, Hera Arif, and Bimpe Ayoola. Teaching Software Metrology: The Science of Measurement for Software Engineering, pages 101-154. Springer, Cham, 2024. URL: https://doi.org/10.1007/978-3-031-71769-7_5.
  52. Paul Ralph and Ewan Tempero. Construct validity in software engineering research and software metrics. In Proc. of EASE 2018, EASE'18, pages 13-23. ACM, New York, NY, USA, 2018. URL: https://doi.org/10.1145/3210459.3210461.
  53. William Revelle. psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois, 2026. R package version 2.6.4. URL: https://CRAN.R-project.org/package=psych.
  54. Yves Rosseel. lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2):1-36, 2012. URL: https://doi.org/10.18637/jss.v048.i02.
  55. Yves Rosseel, Terrence D. Jorgensen, and Luc De Wilde. lavaan: Latent Variable Analysis, 2025. R package version 0.6-20. URL: https://doi.org/10.32614/CRAN.package.lavaan.
  56. J. Saraiva, E. Barreiros, A. Almeida, F. Lima, A. Alencar, G. Lima, S. Soares, and F. Castor. Aspect-oriented software maintenance metrics: A systematic mapping study. In Conf. on Evaluation & Assessment in Soft. Eng. (EASE 2012), pages 253-262. IET, 2012. Google Scholar
  57. Tushar Sharma and Diomidis Spinellis. A survey on software smells. J. of Sys. & Soft., 138:158-173, 2018. URL: https://doi.org/10.1016/j.jss.2017.12.034.
  58. Tushar Sharma and Diomidis Spinellis. Do we need improved code quality metrics?, 2020. URL: https://arxiv.org/abs/2012.12324.
  59. Dag I. K. Sjøberg, Bente Anda, and Audris Mockus. Questioning software maintenance metrics: A comparative case study. In Proc. of the Int. Symposium on Empirical Soft. Eng. & Measurement, pages 107-110, 2012. URL: https://doi.org/10.1145/2372251.2372269.
  60. Dag I. K. Sjøberg and Gunnar Rye Bergersen. Construct validity in software engineering. IEEE Trans. on Soft. Eng., 49(3):1374-1396, 2023. URL: https://doi.org/10.1109/TSE.2022.3176725.
  61. Srdjan Stevanetic and Uwe Zdun. Software metrics for measuring the understandability of architectural structures: a systematic mapping study. In Proc. of EASE, EASE '15. ACM, New York, NY, USA, 2015. URL: https://doi.org/10.1145/2745802.2745822.
  62. Ewan Tempero and Paul Ralph. A framework for defining coupling metrics. Science of Computer Programming, 166:214-230, 2018. URL: https://doi.org/10.1016/j.scico.2018.02.004.
  63. Ewan Tempero and Paul Ralph. Making software metrics useful. arXiv, 2026. URL: https://doi.org/10.48550/arXiv.2603.16012.
  64. William Trochim, James P. Donnelly, and Arora Kanika. The Research Methods Knowledge Base. Wadsworth Publishing, 2015. Google Scholar
  65. E.J. Weyuker. Evaluating software complexity measures. IEEE Trans. on Soft. Eng., 14(9):1357-1365, 1988. URL: https://doi.org/10.1109/32.6178.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact