Skip to main navigation menu Skip to main content Skip to site footer

Vol. 31 No. 62 (2022)
October-December 2022 (Continuous Publication)

Papers

Metrics Model to Complement the Evaluation of DevOps in Software Companies

https://doi.org/10.19053/01211129.v31.n62.2022.14766

Published 2022-10-04

Carlos-Eduardo Orozco-Garcés
Universidad del Cauca

César-Jesús Pardo-Calvache
Universidad del Cauca

Elizabeth Suescún-Monsalve
Universidad EAFIT

Abstract

This article presents a model to complement the evaluation of DevOps in software companies. It was designed by harmonizing the elements of the DevOps process identified through a systematic mapping of the literature and aimed to know the state of the art of methodological solutions and tools to evaluate DevOps in the industry. The process elements were identified, compared, and integrated into a common process structure that was used to establish a total of 11 metrics using the Goal-Question-Metric approach. The model was evaluated by a focus group of expert DevOps professionals. They determined that the model is clear, easy to apply, and provides valuable information to companies to improve their DevOps practices.

Keywords

assessment, DevOps, GQM, Metrics

XML PDF

Author Biography

Carlos-Eduardo Orozco-Garcés

Roles: Investigation, Formal Analysis, Methodology, Writing-original draft.

César-Jesús Pardo-Calvache

Roles: Supervision, Methodology, Validation, Writing-Review and Editing.

Elizabeth Suescún-Monsalve

Roles: Supervision, Methodology, Validation.

References

  1. H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295 DOI: https://doi.org/10.1109/MS.2002.1020295
  2. CMMI Institute, Capability maturity model integration for development, 2018.
  3. Rational Software, “Rational Unified Process,” Best Practices for Software Development Teams, 2020.
  4. W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings of 9th International Conference on Software Engineering, 1987, pp. 328–338.
  5. B. W. Boehm, “A spiral model of software development and enhancement,” Computer (Long Beach Calif), vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59 DOI: https://doi.org/10.1109/2.59
  6. J. Martin, Rapid application development. Macmillan Publishing Co., Inc., 1991.
  7. K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017.
  8. M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003.
  9. K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002.
  10. K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000.
  11. J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in JIISIC 2019, 2019.
  12. A. Cockburn, Crystal clear: A human-powered methodology for small teams: A human-powered methodology for small teams, Pearson Education, 2004.
  13. J. Highsmith, Adaptive software development: a collaborative approach to managing complex systems, Addison-Wesley, 2013.
  14. J. Stapleton, DSDM, dynamic systems development method: the method in practice, Cambridge University Press, 1997.
  15. H. Kniberg, Scrum and XP from the Trenches, 2015.
  16. C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009.
  17. J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Proceedings of the 41st Annual Hawaii International Conference on System Sciences, 2008, pp. 466–466. https://doi.org/10.1109/hicss.2008.384 DOI: https://doi.org/10.1109/HICSS.2008.384
  18. B. de França, H. Jeronimo, G. Travassos, Characterizing DevOps by Hearing Multiple Voices. New York, NY: Association for Computing Machinery, 2016. https://doi.org/10.1145/2973839.2973845 DOI: https://doi.org/10.1145/2973839.2973845
  19. M. Sanchez Gordon, R. Colomo Palacios, “Characterizing DevOps Culture: A Systematic Literature Review,” in International Conference on Software Process Improvement and Capability Determination. SPICE 2018, 2018, pp. 3–15. https://doi.org/10.1007/978-3-030-00623-5_1 DOI: https://doi.org/10.1007/978-3-030-00623-5_1
  20. A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” EEE, pp. 704-710, 2005. https://doi.org/10.1109/eee.2005.86 DOI: https://doi.org/10.1109/EEE.2005.86
  21. J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” HICSS, 2014.
  22. ISO/IEC, Calidad de los servicios TI, 2019.
  23. ISO/IEC, Sistemas de Gestión la Seguridad de la Información, 2013.
  24. P. Debois, Devopsdays - Organizing Guide, 2009.
  25. M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in International Conference on Innovation and Computational Technology, 2015, pp. 78–82. https://doi.org/10.1109/intech.2015.7173368 DOI: https://doi.org/10.1109/INTECH.2015.7173368
  26. S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, Sep. 2017, pp. 1–7. https://doi.org/10.1109/icter.2017.8257808 DOI: https://doi.org/10.1109/ICTER.2017.8257808
  27. M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017. https://doi.org/10.1109/access.2017.2685629 DOI: https://doi.org/10.1109/ACCESS.2017.2685629
  28. C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, vol. 2020, no. E32, 2020.
  29. J. Michelsen, “Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps,” CA Technologies, p. 26, 2014.
  30. E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in 11th International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28 DOI: https://doi.org/10.1109/ICGSE.2016.28
  31. M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computational in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29 DOI: https://doi.org/10.1109/CCEM.2015.29
  32. Digital.ai, 15th Annual State of Agile Report, 2021. https://bit.ly/3Lgw4KE
  33. J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65. DOI: https://doi.org/10.1109/IC2E.2015.23
  34. F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
  35. C.-E. Orozco-Garcés, C.-J. Pardo-Calvache, Y.-H. Salazar-Mondragón, “What is There About DevOps Assessment? A Systematic Mapping,” Revista Facultad de Ingeniería, vol. 31, no. 59, pp. e13896, 2022. DOI: https://doi.org/10.19053/01211129.v31.n59.2022.13896
  36. J. Guerrero, K. Zúniga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Journal de Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/jci.2020.1.5 DOI: https://doi.org/10.46571/JCI.2020.1.5
  37. M. Dini, N. Gligo, A. Patiño, Transformación digital de las mipymes: elementos para el diseño de políticas, 2021.
  38. V. R. Basili, Software modeling and measurement: the Goal/Question/Metric paradigm, 1992.
  39. M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in International Conference on Software Process Improvement and Capability Determination, 2016, pp. 123–134. https://doi.org/10.1007/978-3-319-38980-6_10 DOI: https://doi.org/10.1007/978-3-319-38980-6_10
  40. A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020. https://doi.org/10.1016/j.cosrev.2020.100308 DOI: https://doi.org/10.1016/j.cosrev.2020.100308
  41. G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceeding of International Conference on Software System, 2016, pp. 76–85. https://doi.org/10.1145/2904354.2904372 DOI: https://doi.org/10.1145/2904354.2904372
  42. M. Gasparaite, S. Ragaisis, Comparison of devops maturity models, 2019.
  43. M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019. DOI: https://doi.org/10.35940/ijrte.C6888.098319
  44. C. Marnewick, J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE Software, vol. 38, no. 5, pp. 48-55, 2021. https://doi.org/10.1109/MS.2020.3023298 DOI: https://doi.org/10.1109/MS.2020.3023298
  45. R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, “Towards the adoption of DevOps in software product organizations: A Maturity Model Approach,” Technical Report Series, no. UU-CS-2017-009. UU BETA ICS Departement Informatica, 2017.
  46. L. König, A. Steffens, “Towards a quality model for devops,” in Continuous Software Engineering & Full-scale Software Engineering, vol. 37, pp. 37-42, 2018.
  47. S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, University of Technology, Eindhoven, 2014.
  48. L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in EuroSymposium on Systems Analysis and Design, 2019, pp. 77–90. https://doi.org/10.1007/978-3-030-29608-7_7 DOI: https://doi.org/10.1007/978-3-030-29608-7_7
  49. P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760. https://doi.org/10.1109/compe49325.2020.9200149 DOI: https://doi.org/10.1109/ComPE49325.2020.9200149
  50. P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Conference on Software Process Improvement, 2019, pp. 111–122. https://doi.org/10.1007/978-3-030-28005-5_9 DOI: https://doi.org/10.1007/978-3-030-28005-5_9
  51. T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, 2018, pp. 279–285. https://doi.org/10.1145/3278681.3278714 DOI: https://doi.org/10.1145/3278681.3278714
  52. J. M. Radstaak, “Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations,” Master Thesis, University of Twente, 2019.
  53. D. Teixeira, R. Pereira, T. Henriques, M. M. da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020. https://doi.org/10.1504/ijasm.2020.10034553 DOI: https://doi.org/10.1504/IJASM.2020.112343
  54. T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020. DOI: https://doi.org/10.5220/0010177801170129
  55. R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information Systems Modeling, 2018, pp. 244–259. https://doi.org/10.1007/978-3-319-91704-7_16 DOI: https://doi.org/10.1007/978-3-319-91704-7_16
  56. T. Seppä-Lassila, A. Järvi, S. Hyrynsalmi, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, 2017.
  57. R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Workshop on Agile Methods, 2016, pp. 39–49. https://doi.org/10.1007/978-3-319-55907-0_4 DOI: https://doi.org/10.1007/978-3-319-55907-0_4
  58. M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in Proceedings of the 11th International Conference on Management of Digital EcoSystems, 2019, pp. 205–212. https://doi.org/10.1145/3297662.3365827 DOI: https://doi.org/10.1145/3297662.3365827
  59. N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference on Cyber Security and Protection of Digital Services (Cyber Security), 2019, pp. 1–8. https://doi.org/10.1109/cybersecpods.2019.8884935 DOI: https://doi.org/10.1109/CyberSecPODS.2019.8884935
  60. IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Std 2675-2021, vol. 1, pp. 1–91, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476 DOI: https://doi.org/10.1109/IEEESTD.2021.9415476
  61. O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” in International Conference on Software Process Improvement and Capability Determination., 2016, pp. 135–149. https://doi.org/10.1007/978-3-319-38980-6_11 DOI: https://doi.org/10.1007/978-3-319-38980-6_11
  62. ATOS, DevOps Maturity Assessment, 2020. https://bit.ly/3uTbPve
  63. Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
  64. IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
  65. IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
  66. B. Kitchenham, S. Linkman, D. Law, “DESMET: a methodology for evaluating software engineering methods and tools,” Computing & Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997. https://doi.org/10.1049/cce:19970304 DOI: https://doi.org/10.1049/cce:19970304
  67. Infostretch, Infostretch DevOps Self-Assessment, 2020. https://bit.ly/3fh4krm
  68. InCycle, InCycle Evaluacion de devops, 2020. https://bit.ly/2RqYQCl
  69. Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
  70. Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
  71. Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
  72. Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
  73. Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
  74. Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
  75. Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
  76. C. Pardo, F. J. Pino, F. García, M. Piattini, M. T. Baldassarre, “A process for driving the harmonization of models,” in Proceedings of the 11th International Conference on Product Focused Software, 2010, pp. 51–54. https://doi.org/10.1145/1961258.1961271 DOI: https://doi.org/10.1145/1961258.1961271
  77. C. Pardo, F. J. Pino, F. Garcia, “Towards an integrated management system (IMS), harmonizing the ISO/IEC 27001 and ISO/IEC 20000-2 standards,” International Journal of Software Engineering and Its Applications, vol. 10, no. 9, pp. 217–230, 2016. DOI: https://doi.org/10.14257/ijseia.2016.10.9.18
  78. C. Pardo, F. García, F. J. Pino, M. Piattini, M. T. Baldassarre, “PrMO: An Ontology of Process-reference Models,” in XVII Jornadas de Ingeniería del Software y Bases de Datos, 2012.
  79. F. J. Pino, M. T. Baldassarre, M. Piattini, G. Visaggio, “Harmonizing maturity levels from CMMI‐DEV and ISO/IEC 15504,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 22, no. 4, pp. 279–296, 2010. https://doi.org/10.1002/spip.437 DOI: https://doi.org/10.1002/spip.437
  80. J. Guerrero, DevOps Model - Modelo de referencia para la adopción de DevOps en empresas de desarrollo de software, Grade Thesis, Universidad del Cauca, Colombia, 2021.
  81. C. Pardo, F. Garcia, M. Piattini, F. J. Pino, S. Lemus, M. T. Baldassarre, “Integrating multiple models for definition of IT governance model for banking ITGSM,” International Business Management, vol. 10, no. 19, pp. 4644–4653, 2016.
  82. N. Fenton, J. Bieman, Software metrics: a rigorous and practical approach. CRC press, 2014. https://doi.org/10.1201/b17461 DOI: https://doi.org/10.1201/b17461
  83. Universidad de Cantabria, Tema 2. Métodos de Valoración: Cuestionarios, 2022, https://ocw.unican.es/mod/page/view.php?id=498#6
  84. M. F. Triola, Probabilidad y estadística. Pearson educación, 2004.
  85. J. Kontio, J. Bragge, L. Lehtola, “The focus group method as an empirical tool in software engineering,” in Guide to advanced empirical software engineering, Springer, 2008, pp. 93–116. https://doi.org/10.1007/978-1-84800-044-5_4 DOI: https://doi.org/10.1007/978-1-84800-044-5_4
  86. R. Likert, “A technique for the measurement of attitudes,” Archives of psychology, vol. 22 no. 140, pp. 55, 1932.

Downloads

Download data is not yet available.

Most read articles by the same author(s)