Skip to main navigation menu Skip to main content Skip to site footer

Vol. 31 No. 59 (2022)
January-March 2022 (Continuous Publication)

Papers

What is There About DevOps Assessment? A Systematic Mapping

https://doi.org/10.19053/01211129.v31.n59.2022.13896

Published 2022-03-14

Carlos-Eduardo Orozco-Garcés
Universidad del Cauca

César-Jesús Pardo-Calvache
Universidad del Cauca

Yilber-Hernán Salazar-Mondragón
Universidad del Cauca

Abstract

DevOps has been established as a framework used by software development companies seeking to set mechanisms to automate their development processes. Consequently, over the last decade, many companies have adopted DevOps to support their project’s development process and perform continuous improvement tasks to ensure that it is applied correctly. To achieve this, companies are looking for solutions that allow them to evaluate the degree of implementation of DevOps in their internal processes. In this sense, the objective of this study focuses on identifying, through a systematic mapping of the literature, the mechanisms used to assess DevOps in software development companies. According to the above, the current state of knowledge related to the proposal of processes, models, techniques, tools, and methodological guides is presented to conduct the DevOps assessment. As a result, it is noted that there are multiple methodological solutions that seek to assess DevOps; however, a high degree of heterogeneity was evidenced in the identified solutions, resulting in the need to establish a clear framework that serves as the basis for proposing a generic, structured, and unambiguous DevOps assessment model applicable to software companies.

Keywords

Development and Operations, DevOps, Assessment, Appraisal, SLM

PDF XML

Author Biography

Carlos-Eduardo Orozco-Garcés

Roles: Investigation, Formal Analysis, Methodology, Writing – original draft.

César-Jesús Pardo-Calvache

Roles: Supervision, Methodology, Validation, Writing-Revision and Edition.

Yilber-Hernán Salazar-Mondragón

Roles: Supervision, Methodology, Validation.

References

  • H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295 DOI: https://doi.org/10.1109/MS.2002.1020295
  • CMMI Institute, Capability maturity model integration for development, 2018.
  • Rational Software, “Rational Unified Process,” in Best Practices in Software Development Teams, 2020
  • W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings 9th International Conference in Software Engineering, 1987, pp. 328–338
  • K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017
  • M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003
  • K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002
  • K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000
  • H. Kniberg, Scrum and XP from the Trenches, 2015
  • C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009
  • J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Hawaii International Conference on System Sciences, 2008, pp. 466–466 DOI: https://doi.org/10.1109/HICSS.2008.384
  • A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” in Conference on Electrical and Electronic Engineering, 2005, pp. 704-710
  • J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” in Hawaii International Conference on System Sciences, 2014
  • ISO/IEC, Calidad de los servicios TI,” 2019
  • M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in The International Conference on Information and Computer Technologies, 2015, pp. 78–82 DOI: https://doi.org/10.1109/INTECH.2015.7173368
  • S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, 2017, pp. 1–7 DOI: https://doi.org/10.1109/ICTER.2017.8257808
  • P. Debois, Devopsdays - Organizing Guide, 2009
  • S. Nagpal, A. Shadab, Literature Review: Promises and Challenges of DevOps, 2017
  • M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017 DOI: https://doi.org/10.1109/ACCESS.2017.2685629
  • C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI, vol. 2020, no. E32, 2020
  • J. Michelsen, Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps, 2014
  • E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28 DOI: https://doi.org/10.1109/ICGSE.2016.28
  • M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computing in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29 DOI: https://doi.org/10.1109/CCEM.2015.29
  • J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65 DOI: https://doi.org/10.1109/IC2E.2015.23
  • F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
  • J. D. Patón-Romero, M. Piattini, “Green IT maturity models: a systematic mapping study,” in 12th Iberian Conference on Information Systems and Technologies, 2017, pp. 1–6 DOI: https://doi.org/10.23919/CISTI.2017.7975693
  • K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, “Systematic mapping studies in software engineering,” in International Congress on Engineering and Sustainability, 2008, pp. 1–10 DOI: https://doi.org/10.14236/ewic/EASE2008.8
  • J. Biolchini, P. G. Mian, A. C. Natali, G. H. Travassos, “Systematic review in software engineering,” in System Engineering and Computer Science Department, 2005
  • D. Budgen, M. Turner, P. Brereton, B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Psychology of Programming Interest Group, 2008, pp. 195–204
  • M. Genero, L. Cruz, M. Piattini, Métodos de investigación en ingeniería de software. Bogota, DC: Ra-ma, 2014
  • N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference On Cyber Security And Protection Of Digital Services, 2019, pp. 1–8 DOI: https://doi.org/10.1109/CyberSecPODS.2019.8884935
  • P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Systems and Software Process Improvement and Innovation, 2019, pp. 111–122 DOI: https://doi.org/10.1007/978-3-030-28005-5_9
  • M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in IEEE International Conference on Digital Ecosystems, 2019, pp. 205–212 DOI: https://doi.org/10.1145/3297662.3365827
  • M. Gasparaite, S. Ragaisis, “Comparison of devops maturity models,” in International Conference on Information Technologies, 2019, pp. 65–69
  • J. M. Radstaak, Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations, Grade Thesis, University of Twente, 2019
  • A. Caprarelli, E. Di Nitto, D. Tamburri, “Fallacies and pitfalls on the road to DevOps: a longitudinal industrial study,” in Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment: Second International Workshop, 2019, pp. 200–210 DOI: https://doi.org/10.1007/978-3-030-39306-9_15
  • M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019 DOI: https://doi.org/10.35940/ijrte.C6888.098319
  • J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in Jornadas Iberoamericanas de Ingeniería de Software e Ingeniería del Conocimiento, 2019.
  • A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020 DOI: https://doi.org/10.1016/j.cosrev.2020.100308
  • C. Marnewick and J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE, 2020. DOI: https://doi.org/10.1109/MS.2020.3023298
  • D. Teixeira, R. Pereira, T. Henriques, M. M. Da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020 DOI: https://doi.org/10.1504/IJASM.2020.112343
  • T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020 DOI: https://doi.org/10.5220/0010177801170129
  • J. Guerrero, K. Zuñiga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/JCI.2020.1.5 DOI: https://doi.org/10.46571/JCI.2020.1.5
  • R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information, 2018, pp. 244–259 DOI: https://doi.org/10.1007/978-3-319-91704-7_16
  • T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the American Society for Information Science and Technology, 2018, pp. 279–285. DOI: https://doi.org/10.1145/3278681.3278714
  • T. Seppä-Lassila, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, Finland, 2017
  • L. König, A. Steffens, “Towards a quality model for devops,” Continuous Software Engineering & Full-scale Software Engineering, vol. 37, 2018
  • O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” Software Process Improvement and Capability Determination, 2016, pp. 135–149 DOI: https://doi.org/10.1007/978-3-319-38980-6_11
  • M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in Software Process Improvement and Capability Determination, 2016, pp. 123–134 DOI: https://doi.org/10.1007/978-3-319-38980-6_10
  • R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Work in Agile. Methods, 2016, pp. 39–49 DOI: https://doi.org/10.1007/978-3-319-55907-0_4
  • R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, Towards the adoption of DevOps in software product organizations: A Maturity Model Approach, 2017
  • S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, Eindhoven University of Technology, 2014
  • J. Smeds, K. Nybom, I. Porres, “DevOps: A Definition and Perceived Adoption Impediments,” Lecture Notes in Business Information Processing, vol. 212, pp. 166–177, 2015. https://doi.org/10.1007/978-3-319-18612-2 DOI: https://doi.org/10.1007/978-3-319-18612-2_14
  • IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Standar 2675-2021, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476 DOI: https://doi.org/10.1109/IEEESTD.2021.9415476
  • G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceedings International Conference on Software Engineering, 2016, pp. 76–85 DOI: https://doi.org/10.1145/2904354.2904372
  • L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in European Symposium on Systems Analysis and Design, 2019, pp. 77–90 DOI: https://doi.org/10.1007/978-3-030-29608-7_7
  • P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760 DOI: https://doi.org/10.1109/ComPE49325.2020.9200149
  • M. A. McCarthy, L. M. Herger, S. M. Khan, B. M. Belgodere, “Composable DevOps: automated ontology based DevOps maturity analysis,” in International Conference on Service-Oriented Computing, 2015, pp. 600–607 DOI: https://doi.org/10.1109/SCC.2015.87
  • B. Kitchenham, S. Linkman, D. Law, “DESMET: A methodology for evaluating software engineering methods and tools,” Computing and Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997 DOI: https://doi.org/10.1049/cce:19970304
  • ATOS, DevOps Maturity Assessment, 2021. https://bit.ly/3uTbPve
  • Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
  • Infostretch, Infostretch DevOps Self-Assessment, 2021. https://bit.ly/3fh4krm
  • InCycle, InCycle Evaluacion de devops, 2021. https://bit.ly/2RqYQCl
  • IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
  • Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
  • Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
  • IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
  • Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
  • Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
  • Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
  • Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
  • Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
  • Unión Europea, Reglamento 651/2014, 2014
  • Unión Europea, Recomendación de la comisión del 6 de mayo de 2003 sobre la definición de microempresas, pequeñas y medianas empresas, 2003

Downloads

Download data is not yet available.

Most read articles by the same author(s)