Metrics Model to Complement the Evaluation of DevOps in Software Companies
Abstract
This article presents a model to complement the evaluation of DevOps in software companies. It was designed by harmonizing the elements of the DevOps process identified through a systematic mapping of the literature and aimed to know the state of the art of methodological solutions and tools to evaluate DevOps in the industry. The process elements were identified, compared, and integrated into a common process structure that was used to establish a total of 11 metrics using the Goal-Question-Metric approach. The model was evaluated by a focus group of expert DevOps professionals. They determined that the model is clear, easy to apply, and provides valuable information to companies to improve their DevOps practices.
Keywords
assessment, DevOps, GQM, Metrics
Author Biography
Carlos-Eduardo Orozco-Garcés
Roles: Investigation, Formal Analysis, Methodology, Writing-original draft.
César-Jesús Pardo-Calvache
Roles: Supervision, Methodology, Validation, Writing-Review and Editing.
Elizabeth Suescún-Monsalve
Roles: Supervision, Methodology, Validation.
References
- H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295 DOI: https://doi.org/10.1109/MS.2002.1020295
- CMMI Institute, Capability maturity model integration for development, 2018.
- Rational Software, “Rational Unified Process,” Best Practices for Software Development Teams, 2020.
- W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings of 9th International Conference on Software Engineering, 1987, pp. 328–338.
- B. W. Boehm, “A spiral model of software development and enhancement,” Computer (Long Beach Calif), vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59 DOI: https://doi.org/10.1109/2.59
- J. Martin, Rapid application development. Macmillan Publishing Co., Inc., 1991.
- K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017.
- M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003.
- K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002.
- K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000.
- J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in JIISIC 2019, 2019.
- A. Cockburn, Crystal clear: A human-powered methodology for small teams: A human-powered methodology for small teams, Pearson Education, 2004.
- J. Highsmith, Adaptive software development: a collaborative approach to managing complex systems, Addison-Wesley, 2013.
- J. Stapleton, DSDM, dynamic systems development method: the method in practice, Cambridge University Press, 1997.
- H. Kniberg, Scrum and XP from the Trenches, 2015.
- C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009.
- J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Proceedings of the 41st Annual Hawaii International Conference on System Sciences, 2008, pp. 466–466. https://doi.org/10.1109/hicss.2008.384 DOI: https://doi.org/10.1109/HICSS.2008.384
- B. de França, H. Jeronimo, G. Travassos, Characterizing DevOps by Hearing Multiple Voices. New York, NY: Association for Computing Machinery, 2016. https://doi.org/10.1145/2973839.2973845 DOI: https://doi.org/10.1145/2973839.2973845
- M. Sanchez Gordon, R. Colomo Palacios, “Characterizing DevOps Culture: A Systematic Literature Review,” in International Conference on Software Process Improvement and Capability Determination. SPICE 2018, 2018, pp. 3–15. https://doi.org/10.1007/978-3-030-00623-5_1 DOI: https://doi.org/10.1007/978-3-030-00623-5_1
- A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” EEE, pp. 704-710, 2005. https://doi.org/10.1109/eee.2005.86 DOI: https://doi.org/10.1109/EEE.2005.86
- J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” HICSS, 2014.
- ISO/IEC, Calidad de los servicios TI, 2019.
- ISO/IEC, Sistemas de Gestión la Seguridad de la Información, 2013.
- P. Debois, Devopsdays - Organizing Guide, 2009.
- M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in International Conference on Innovation and Computational Technology, 2015, pp. 78–82. https://doi.org/10.1109/intech.2015.7173368 DOI: https://doi.org/10.1109/INTECH.2015.7173368
- S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, Sep. 2017, pp. 1–7. https://doi.org/10.1109/icter.2017.8257808 DOI: https://doi.org/10.1109/ICTER.2017.8257808
- M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017. https://doi.org/10.1109/access.2017.2685629 DOI: https://doi.org/10.1109/ACCESS.2017.2685629
- C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, vol. 2020, no. E32, 2020.
- J. Michelsen, “Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps,” CA Technologies, p. 26, 2014.
- E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in 11th International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28 DOI: https://doi.org/10.1109/ICGSE.2016.28
- M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computational in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29 DOI: https://doi.org/10.1109/CCEM.2015.29
- Digital.ai, 15th Annual State of Agile Report, 2021. https://bit.ly/3Lgw4KE
- J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65. DOI: https://doi.org/10.1109/IC2E.2015.23
- F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
- C.-E. Orozco-Garcés, C.-J. Pardo-Calvache, Y.-H. Salazar-Mondragón, “What is There About DevOps Assessment? A Systematic Mapping,” Revista Facultad de Ingeniería, vol. 31, no. 59, pp. e13896, 2022. DOI: https://doi.org/10.19053/01211129.v31.n59.2022.13896
- J. Guerrero, K. Zúniga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Journal de Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/jci.2020.1.5 DOI: https://doi.org/10.46571/JCI.2020.1.5
- M. Dini, N. Gligo, A. Patiño, Transformación digital de las mipymes: elementos para el diseño de políticas, 2021.
- V. R. Basili, Software modeling and measurement: the Goal/Question/Metric paradigm, 1992.
- M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in International Conference on Software Process Improvement and Capability Determination, 2016, pp. 123–134. https://doi.org/10.1007/978-3-319-38980-6_10 DOI: https://doi.org/10.1007/978-3-319-38980-6_10
- A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020. https://doi.org/10.1016/j.cosrev.2020.100308 DOI: https://doi.org/10.1016/j.cosrev.2020.100308
- G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceeding of International Conference on Software System, 2016, pp. 76–85. https://doi.org/10.1145/2904354.2904372 DOI: https://doi.org/10.1145/2904354.2904372
- M. Gasparaite, S. Ragaisis, Comparison of devops maturity models, 2019.
- M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019. DOI: https://doi.org/10.35940/ijrte.C6888.098319
- C. Marnewick, J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE Software, vol. 38, no. 5, pp. 48-55, 2021. https://doi.org/10.1109/MS.2020.3023298 DOI: https://doi.org/10.1109/MS.2020.3023298
- R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, “Towards the adoption of DevOps in software product organizations: A Maturity Model Approach,” Technical Report Series, no. UU-CS-2017-009. UU BETA ICS Departement Informatica, 2017.
- L. König, A. Steffens, “Towards a quality model for devops,” in Continuous Software Engineering & Full-scale Software Engineering, vol. 37, pp. 37-42, 2018.
- S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, University of Technology, Eindhoven, 2014.
- L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in EuroSymposium on Systems Analysis and Design, 2019, pp. 77–90. https://doi.org/10.1007/978-3-030-29608-7_7 DOI: https://doi.org/10.1007/978-3-030-29608-7_7
- P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760. https://doi.org/10.1109/compe49325.2020.9200149 DOI: https://doi.org/10.1109/ComPE49325.2020.9200149
- P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Conference on Software Process Improvement, 2019, pp. 111–122. https://doi.org/10.1007/978-3-030-28005-5_9 DOI: https://doi.org/10.1007/978-3-030-28005-5_9
- T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, 2018, pp. 279–285. https://doi.org/10.1145/3278681.3278714 DOI: https://doi.org/10.1145/3278681.3278714
- J. M. Radstaak, “Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations,” Master Thesis, University of Twente, 2019.
- D. Teixeira, R. Pereira, T. Henriques, M. M. da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020. https://doi.org/10.1504/ijasm.2020.10034553 DOI: https://doi.org/10.1504/IJASM.2020.112343
- T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020. DOI: https://doi.org/10.5220/0010177801170129
- R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information Systems Modeling, 2018, pp. 244–259. https://doi.org/10.1007/978-3-319-91704-7_16 DOI: https://doi.org/10.1007/978-3-319-91704-7_16
- T. Seppä-Lassila, A. Järvi, S. Hyrynsalmi, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, 2017.
- R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Workshop on Agile Methods, 2016, pp. 39–49. https://doi.org/10.1007/978-3-319-55907-0_4 DOI: https://doi.org/10.1007/978-3-319-55907-0_4
- M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in Proceedings of the 11th International Conference on Management of Digital EcoSystems, 2019, pp. 205–212. https://doi.org/10.1145/3297662.3365827 DOI: https://doi.org/10.1145/3297662.3365827
- N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference on Cyber Security and Protection of Digital Services (Cyber Security), 2019, pp. 1–8. https://doi.org/10.1109/cybersecpods.2019.8884935 DOI: https://doi.org/10.1109/CyberSecPODS.2019.8884935
- IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Std 2675-2021, vol. 1, pp. 1–91, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476 DOI: https://doi.org/10.1109/IEEESTD.2021.9415476
- O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” in International Conference on Software Process Improvement and Capability Determination., 2016, pp. 135–149. https://doi.org/10.1007/978-3-319-38980-6_11 DOI: https://doi.org/10.1007/978-3-319-38980-6_11
- ATOS, DevOps Maturity Assessment, 2020. https://bit.ly/3uTbPve
- Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
- IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
- IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
- B. Kitchenham, S. Linkman, D. Law, “DESMET: a methodology for evaluating software engineering methods and tools,” Computing & Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997. https://doi.org/10.1049/cce:19970304 DOI: https://doi.org/10.1049/cce:19970304
- Infostretch, Infostretch DevOps Self-Assessment, 2020. https://bit.ly/3fh4krm
- InCycle, InCycle Evaluacion de devops, 2020. https://bit.ly/2RqYQCl
- Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
- Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
- Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
- Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
- Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
- Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
- Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
- C. Pardo, F. J. Pino, F. García, M. Piattini, M. T. Baldassarre, “A process for driving the harmonization of models,” in Proceedings of the 11th International Conference on Product Focused Software, 2010, pp. 51–54. https://doi.org/10.1145/1961258.1961271 DOI: https://doi.org/10.1145/1961258.1961271
- C. Pardo, F. J. Pino, F. Garcia, “Towards an integrated management system (IMS), harmonizing the ISO/IEC 27001 and ISO/IEC 20000-2 standards,” International Journal of Software Engineering and Its Applications, vol. 10, no. 9, pp. 217–230, 2016. DOI: https://doi.org/10.14257/ijseia.2016.10.9.18
- C. Pardo, F. García, F. J. Pino, M. Piattini, M. T. Baldassarre, “PrMO: An Ontology of Process-reference Models,” in XVII Jornadas de Ingeniería del Software y Bases de Datos, 2012.
- F. J. Pino, M. T. Baldassarre, M. Piattini, G. Visaggio, “Harmonizing maturity levels from CMMI‐DEV and ISO/IEC 15504,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 22, no. 4, pp. 279–296, 2010. https://doi.org/10.1002/spip.437 DOI: https://doi.org/10.1002/spip.437
- J. Guerrero, DevOps Model - Modelo de referencia para la adopción de DevOps en empresas de desarrollo de software, Grade Thesis, Universidad del Cauca, Colombia, 2021.
- C. Pardo, F. Garcia, M. Piattini, F. J. Pino, S. Lemus, M. T. Baldassarre, “Integrating multiple models for definition of IT governance model for banking ITGSM,” International Business Management, vol. 10, no. 19, pp. 4644–4653, 2016.
- N. Fenton, J. Bieman, Software metrics: a rigorous and practical approach. CRC press, 2014. https://doi.org/10.1201/b17461 DOI: https://doi.org/10.1201/b17461
- Universidad de Cantabria, Tema 2. Métodos de Valoración: Cuestionarios, 2022, https://ocw.unican.es/mod/page/view.php?id=498#6
- M. F. Triola, Probabilidad y estadística. Pearson educación, 2004.
- J. Kontio, J. Bragge, L. Lehtola, “The focus group method as an empirical tool in software engineering,” in Guide to advanced empirical software engineering, Springer, 2008, pp. 93–116. https://doi.org/10.1007/978-1-84800-044-5_4 DOI: https://doi.org/10.1007/978-1-84800-044-5_4
- R. Likert, “A technique for the measurement of attitudes,” Archives of psychology, vol. 22 no. 140, pp. 55, 1932.
Downloads
Download data is not yet available.