Skip to main navigation menu Skip to main content Skip to site footer

Metrics Model to Complement the Evaluation of DevOps in Software Companies


This article presents a model to complement the evaluation of DevOps in software companies. It was designed by harmonizing the elements of the DevOps process identified through a systematic mapping of the literature and aimed to know the state of the art of methodological solutions and tools to evaluate DevOps in the industry. The process elements were identified, compared, and integrated into a common process structure that was used to establish a total of 11 metrics using the Goal-Question-Metric approach. The model was evaluated by a focus group of expert DevOps professionals. They determined that the model is clear, easy to apply, and provides valuable information to companies to improve their DevOps practices.


assessment, DevOps, GQM, Metrics


Author Biography

Carlos-Eduardo Orozco-Garcés

Roles: Investigation, Formal Analysis, Methodology, Writing-original draft.

César-Jesús Pardo-Calvache

Roles: Supervision, Methodology, Validation, Writing-Review and Editing.

Elizabeth Suescún-Monsalve

Roles: Supervision, Methodology, Validation.


  1. H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. DOI:
  2. CMMI Institute, Capability maturity model integration for development, 2018.
  3. Rational Software, “Rational Unified Process,” Best Practices for Software Development Teams, 2020.
  4. W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings of 9th International Conference on Software Engineering, 1987, pp. 328–338.
  5. B. W. Boehm, “A spiral model of software development and enhancement,” Computer (Long Beach Calif), vol. 21, no. 5, pp. 61–72, 1988. DOI:
  6. J. Martin, Rapid application development. Macmillan Publishing Co., Inc., 1991.
  7. K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017.
  8. M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003.
  9. K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002.
  10. K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000.
  11. J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in JIISIC 2019, 2019.
  12. A. Cockburn, Crystal clear: A human-powered methodology for small teams: A human-powered methodology for small teams, Pearson Education, 2004.
  13. J. Highsmith, Adaptive software development: a collaborative approach to managing complex systems, Addison-Wesley, 2013.
  14. J. Stapleton, DSDM, dynamic systems development method: the method in practice, Cambridge University Press, 1997.
  15. H. Kniberg, Scrum and XP from the Trenches, 2015.
  16. C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009.
  17. J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Proceedings of the 41st Annual Hawaii International Conference on System Sciences, 2008, pp. 466–466. DOI:
  18. B. de França, H. Jeronimo, G. Travassos, Characterizing DevOps by Hearing Multiple Voices. New York, NY: Association for Computing Machinery, 2016. DOI:
  19. M. Sanchez Gordon, R. Colomo Palacios, “Characterizing DevOps Culture: A Systematic Literature Review,” in International Conference on Software Process Improvement and Capability Determination. SPICE 2018, 2018, pp. 3–15. DOI:
  20. A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” EEE, pp. 704-710, 2005. DOI:
  21. J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” HICSS, 2014.
  22. ISO/IEC, Calidad de los servicios TI, 2019.
  23. ISO/IEC, Sistemas de Gestión la Seguridad de la Información, 2013.
  24. P. Debois, Devopsdays - Organizing Guide, 2009.
  25. M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in International Conference on Innovation and Computational Technology, 2015, pp. 78–82. DOI:
  26. S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, Sep. 2017, pp. 1–7. DOI:
  27. M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017. DOI:
  28. C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, vol. 2020, no. E32, 2020.
  29. J. Michelsen, “Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps,” CA Technologies, p. 26, 2014.
  30. E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in 11th International Conference on Global Software Engineering, 2016, pp. 24–28. DOI:
  31. M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computational in Emerging Markets, 2015, pp. 85–89. DOI:
  32., 15th Annual State of Agile Report, 2021.
  33. J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65. DOI:
  34. F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014.
  35. C.-E. Orozco-Garcés, C.-J. Pardo-Calvache, Y.-H. Salazar-Mondragón, “What is There About DevOps Assessment? A Systematic Mapping,” Revista Facultad de Ingeniería, vol. 31, no. 59, pp. e13896, 2022. DOI:
  36. J. Guerrero, K. Zúniga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Journal de Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. DOI:
  37. M. Dini, N. Gligo, A. Patiño, Transformación digital de las mipymes: elementos para el diseño de políticas, 2021.
  38. V. R. Basili, Software modeling and measurement: the Goal/Question/Metric paradigm, 1992.
  39. M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in International Conference on Software Process Improvement and Capability Determination, 2016, pp. 123–134. DOI:
  40. A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020. DOI:
  41. G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceeding of International Conference on Software System, 2016, pp. 76–85. DOI:
  42. M. Gasparaite, S. Ragaisis, Comparison of devops maturity models, 2019.
  43. M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019. DOI:
  44. C. Marnewick, J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE Software, vol. 38, no. 5, pp. 48-55, 2021. DOI:
  45. R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, “Towards the adoption of DevOps in software product organizations: A Maturity Model Approach,” Technical Report Series, no. UU-CS-2017-009. UU BETA ICS Departement Informatica, 2017.
  46. L. König, A. Steffens, “Towards a quality model for devops,” in Continuous Software Engineering & Full-scale Software Engineering, vol. 37, pp. 37-42, 2018.
  47. S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, University of Technology, Eindhoven, 2014.
  48. L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in EuroSymposium on Systems Analysis and Design, 2019, pp. 77–90. DOI:
  49. P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760. DOI:
  50. P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Conference on Software Process Improvement, 2019, pp. 111–122. DOI:
  51. T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, 2018, pp. 279–285. DOI:
  52. J. M. Radstaak, “Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations,” Master Thesis, University of Twente, 2019.
  53. D. Teixeira, R. Pereira, T. Henriques, M. M. da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020. DOI:
  54. T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020. DOI:
  55. R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information Systems Modeling, 2018, pp. 244–259. DOI:
  56. T. Seppä-Lassila, A. Järvi, S. Hyrynsalmi, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, 2017.
  57. R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Workshop on Agile Methods, 2016, pp. 39–49. DOI:
  58. M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in Proceedings of the 11th International Conference on Management of Digital EcoSystems, 2019, pp. 205–212. DOI:
  59. N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference on Cyber Security and Protection of Digital Services (Cyber Security), 2019, pp. 1–8. DOI:
  60. IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Std 2675-2021, vol. 1, pp. 1–91, 2021. DOI:
  61. O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” in International Conference on Software Process Improvement and Capability Determination., 2016, pp. 135–149. DOI:
  62. ATOS, DevOps Maturity Assessment, 2020.
  63. Microsoft, Microsoft DevOps Self-Assessment, 2021.
  64. IBM, IBM DevOps Practice Self Assesment, 2021.
  65. IVI, IVI’s DevOps Assessment, 2021.
  66. B. Kitchenham, S. Linkman, D. Law, “DESMET: a methodology for evaluating software engineering methods and tools,” Computing & Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997. DOI:
  67. Infostretch, Infostretch DevOps Self-Assessment, 2020.
  68. InCycle, InCycle Evaluacion de devops, 2020.
  69. Xmatters, DevOps Maturity Survey Report, 2021.
  70. Atlassian, DevOps Maturity model, 2021.
  71. Veritis, Veritis, 2021.
  72. Boxboat, Boxboat, 2021.
  73. Humanitec, DevOps Assessment, 2021.
  74. Atlassian, DevOps Assessment, 2021.
  75. Eficode, Eficode DevOps Assesment, 2021.
  76. C. Pardo, F. J. Pino, F. García, M. Piattini, M. T. Baldassarre, “A process for driving the harmonization of models,” in Proceedings of the 11th International Conference on Product Focused Software, 2010, pp. 51–54. DOI:
  77. C. Pardo, F. J. Pino, F. Garcia, “Towards an integrated management system (IMS), harmonizing the ISO/IEC 27001 and ISO/IEC 20000-2 standards,” International Journal of Software Engineering and Its Applications, vol. 10, no. 9, pp. 217–230, 2016. DOI:
  78. C. Pardo, F. García, F. J. Pino, M. Piattini, M. T. Baldassarre, “PrMO: An Ontology of Process-reference Models,” in XVII Jornadas de Ingeniería del Software y Bases de Datos, 2012.
  79. F. J. Pino, M. T. Baldassarre, M. Piattini, G. Visaggio, “Harmonizing maturity levels from CMMI‐DEV and ISO/IEC 15504,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 22, no. 4, pp. 279–296, 2010. DOI:
  80. J. Guerrero, DevOps Model - Modelo de referencia para la adopción de DevOps en empresas de desarrollo de software, Grade Thesis, Universidad del Cauca, Colombia, 2021.
  81. C. Pardo, F. Garcia, M. Piattini, F. J. Pino, S. Lemus, M. T. Baldassarre, “Integrating multiple models for definition of IT governance model for banking ITGSM,” International Business Management, vol. 10, no. 19, pp. 4644–4653, 2016.
  82. N. Fenton, J. Bieman, Software metrics: a rigorous and practical approach. CRC press, 2014. DOI:
  83. Universidad de Cantabria, Tema 2. Métodos de Valoración: Cuestionarios, 2022,
  84. M. F. Triola, Probabilidad y estadística. Pearson educación, 2004.
  85. J. Kontio, J. Bragge, L. Lehtola, “The focus group method as an empirical tool in software engineering,” in Guide to advanced empirical software engineering, Springer, 2008, pp. 93–116. DOI:
  86. R. Likert, “A technique for the measurement of attitudes,” Archives of psychology, vol. 22 no. 140, pp. 55, 1932.


Download data is not yet available.

Most read articles by the same author(s)