Metrics Model to Complement the Evaluation of DevOps in Software Companies
DOI:
https://doi.org/10.19053/01211129.v31.n62.2022.14766Keywords:
assessment, DevOps, GQM, MetricsAbstract
This article presents a model to complement the evaluation of DevOps in software companies. It was designed by harmonizing the elements of the DevOps process identified through a systematic mapping of the literature and aimed to know the state of the art of methodological solutions and tools to evaluate DevOps in the industry. The process elements were identified, compared, and integrated into a common process structure that was used to establish a total of 11 metrics using the Goal-Question-Metric approach. The model was evaluated by a focus group of expert DevOps professionals. They determined that the model is clear, easy to apply, and provides valuable information to companies to improve their DevOps practices.
Downloads
References
H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295
CMMI Institute, Capability maturity model integration for development, 2018.
Rational Software, “Rational Unified Process,” Best Practices for Software Development Teams, 2020.
W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings of 9th International Conference on Software Engineering, 1987, pp. 328–338.
B. W. Boehm, “A spiral model of software development and enhancement,” Computer (Long Beach Calif), vol. 21, no. 5, pp. 61–72, 1988. https://doi.org/10.1109/2.59
J. Martin, Rapid application development. Macmillan Publishing Co., Inc., 1991.
K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017.
M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003.
K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002.
K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000.
J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in JIISIC 2019, 2019.
A. Cockburn, Crystal clear: A human-powered methodology for small teams: A human-powered methodology for small teams, Pearson Education, 2004.
J. Highsmith, Adaptive software development: a collaborative approach to managing complex systems, Addison-Wesley, 2013.
J. Stapleton, DSDM, dynamic systems development method: the method in practice, Cambridge University Press, 1997.
H. Kniberg, Scrum and XP from the Trenches, 2015.
C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009.
J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Proceedings of the 41st Annual Hawaii International Conference on System Sciences, 2008, pp. 466–466. https://doi.org/10.1109/hicss.2008.384
B. de França, H. Jeronimo, G. Travassos, Characterizing DevOps by Hearing Multiple Voices. New York, NY: Association for Computing Machinery, 2016. https://doi.org/10.1145/2973839.2973845
M. Sanchez Gordon, R. Colomo Palacios, “Characterizing DevOps Culture: A Systematic Literature Review,” in International Conference on Software Process Improvement and Capability Determination. SPICE 2018, 2018, pp. 3–15. https://doi.org/10.1007/978-3-030-00623-5_1
A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” EEE, pp. 704-710, 2005. https://doi.org/10.1109/eee.2005.86
J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” HICSS, 2014.
ISO/IEC, Calidad de los servicios TI, 2019.
ISO/IEC, Sistemas de Gestión la Seguridad de la Información, 2013.
P. Debois, Devopsdays - Organizing Guide, 2009.
M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in International Conference on Innovation and Computational Technology, 2015, pp. 78–82. https://doi.org/10.1109/intech.2015.7173368
S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, Sep. 2017, pp. 1–7. https://doi.org/10.1109/icter.2017.8257808
M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017. https://doi.org/10.1109/access.2017.2685629
C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, vol. 2020, no. E32, 2020.
J. Michelsen, “Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps,” CA Technologies, p. 26, 2014.
E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in 11th International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28
M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computational in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29
Digital.ai, 15th Annual State of Agile Report, 2021. https://bit.ly/3Lgw4KE
J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65.
F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
C.-E. Orozco-Garcés, C.-J. Pardo-Calvache, Y.-H. Salazar-Mondragón, “What is There About DevOps Assessment? A Systematic Mapping,” Revista Facultad de Ingeniería, vol. 31, no. 59, pp. e13896, 2022.
J. Guerrero, K. Zúniga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Journal de Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/jci.2020.1.5
M. Dini, N. Gligo, A. Patiño, Transformación digital de las mipymes: elementos para el diseño de políticas, 2021.
V. R. Basili, Software modeling and measurement: the Goal/Question/Metric paradigm, 1992.
M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in International Conference on Software Process Improvement and Capability Determination, 2016, pp. 123–134. https://doi.org/10.1007/978-3-319-38980-6_10
A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020. https://doi.org/10.1016/j.cosrev.2020.100308
G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceeding of International Conference on Software System, 2016, pp. 76–85. https://doi.org/10.1145/2904354.2904372
M. Gasparaite, S. Ragaisis, Comparison of devops maturity models, 2019.
M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019.
C. Marnewick, J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE Software, vol. 38, no. 5, pp. 48-55, 2021. https://doi.org/10.1109/MS.2020.3023298
R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, “Towards the adoption of DevOps in software product organizations: A Maturity Model Approach,” Technical Report Series, no. UU-CS-2017-009. UU BETA ICS Departement Informatica, 2017.
L. König, A. Steffens, “Towards a quality model for devops,” in Continuous Software Engineering & Full-scale Software Engineering, vol. 37, pp. 37-42, 2018.
S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, University of Technology, Eindhoven, 2014.
L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in EuroSymposium on Systems Analysis and Design, 2019, pp. 77–90. https://doi.org/10.1007/978-3-030-29608-7_7
P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760. https://doi.org/10.1109/compe49325.2020.9200149
P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Conference on Software Process Improvement, 2019, pp. 111–122. https://doi.org/10.1007/978-3-030-28005-5_9
T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists, 2018, pp. 279–285. https://doi.org/10.1145/3278681.3278714
J. M. Radstaak, “Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations,” Master Thesis, University of Twente, 2019.
D. Teixeira, R. Pereira, T. Henriques, M. M. da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020. https://doi.org/10.1504/ijasm.2020.10034553
T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020.
R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information Systems Modeling, 2018, pp. 244–259. https://doi.org/10.1007/978-3-319-91704-7_16
T. Seppä-Lassila, A. Järvi, S. Hyrynsalmi, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, 2017.
R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Workshop on Agile Methods, 2016, pp. 39–49. https://doi.org/10.1007/978-3-319-55907-0_4
M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in Proceedings of the 11th International Conference on Management of Digital EcoSystems, 2019, pp. 205–212. https://doi.org/10.1145/3297662.3365827
N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference on Cyber Security and Protection of Digital Services (Cyber Security), 2019, pp. 1–8. https://doi.org/10.1109/cybersecpods.2019.8884935
IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Std 2675-2021, vol. 1, pp. 1–91, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476
O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” in International Conference on Software Process Improvement and Capability Determination., 2016, pp. 135–149. https://doi.org/10.1007/978-3-319-38980-6_11
ATOS, DevOps Maturity Assessment, 2020. https://bit.ly/3uTbPve
Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
B. Kitchenham, S. Linkman, D. Law, “DESMET: a methodology for evaluating software engineering methods and tools,” Computing & Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997. https://doi.org/10.1049/cce:19970304
Infostretch, Infostretch DevOps Self-Assessment, 2020. https://bit.ly/3fh4krm
InCycle, InCycle Evaluacion de devops, 2020. https://bit.ly/2RqYQCl
Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
C. Pardo, F. J. Pino, F. García, M. Piattini, M. T. Baldassarre, “A process for driving the harmonization of models,” in Proceedings of the 11th International Conference on Product Focused Software, 2010, pp. 51–54. https://doi.org/10.1145/1961258.1961271
C. Pardo, F. J. Pino, F. Garcia, “Towards an integrated management system (IMS), harmonizing the ISO/IEC 27001 and ISO/IEC 20000-2 standards,” International Journal of Software Engineering and Its Applications, vol. 10, no. 9, pp. 217–230, 2016.
C. Pardo, F. García, F. J. Pino, M. Piattini, M. T. Baldassarre, “PrMO: An Ontology of Process-reference Models,” in XVII Jornadas de Ingeniería del Software y Bases de Datos, 2012.
F. J. Pino, M. T. Baldassarre, M. Piattini, G. Visaggio, “Harmonizing maturity levels from CMMI‐DEV and ISO/IEC 15504,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 22, no. 4, pp. 279–296, 2010. https://doi.org/10.1002/spip.437
J. Guerrero, DevOps Model - Modelo de referencia para la adopción de DevOps en empresas de desarrollo de software, Grade Thesis, Universidad del Cauca, Colombia, 2021.
C. Pardo, F. Garcia, M. Piattini, F. J. Pino, S. Lemus, M. T. Baldassarre, “Integrating multiple models for definition of IT governance model for banking ITGSM,” International Business Management, vol. 10, no. 19, pp. 4644–4653, 2016.
N. Fenton, J. Bieman, Software metrics: a rigorous and practical approach. CRC press, 2014. https://doi.org/10.1201/b17461
Universidad de Cantabria, Tema 2. Métodos de Valoración: Cuestionarios, 2022, https://ocw.unican.es/mod/page/view.php?id=498#6
M. F. Triola, Probabilidad y estadística. Pearson educación, 2004.
J. Kontio, J. Bragge, L. Lehtola, “The focus group method as an empirical tool in software engineering,” in Guide to advanced empirical software engineering, Springer, 2008, pp. 93–116. https://doi.org/10.1007/978-1-84800-044-5_4
R. Likert, “A technique for the measurement of attitudes,” Archives of psychology, vol. 22 no. 140, pp. 55, 1932.
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Carlos-Eduardo Orozco-Garcés, César-Jesús Pardo-Calvache, Elizabeth Suescún-Monsalve
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles included in the Revista Facultad de Ingeniería are published under the Creative Commons (BY) license.
Authors must complete, sign, and submit the Review and Publication Authorization Form of the manuscript provided by the Journal; this form should contain all the originality and copyright information of the manuscript.
The authors who publish in this Journal accept the following conditions:
a. The authors retain the copyright and transfer the right of the first publication to the journal, with the work registered under the Creative Commons attribution license, which allows third parties to use what is published as long as they mention the authorship of the work and the first publication in this Journal.
b. Authors can make other independent and additional contractual agreements for the non-exclusive distribution of the version of the article published in this journal (eg, include it in an institutional repository or publish it in a book) provided they clearly indicate that the work It was first published in this Journal.
c. Authors are allowed and recommended to publish their work on the Internet (for example on institutional or personal pages) before and during the process.
review and publication, as it can lead to productive exchanges and a greater and faster dissemination of published work.
d. The Journal authorizes the total or partial reproduction of the content of the publication, as long as the source is cited, that is, the name of the Journal, name of the author (s), year, volume, publication number and pages of the article.
e. The ideas and statements issued by the authors are their responsibility and in no case bind the Journal.
