What is There About DevOps Assessment? A Systematic Mapping
Keywords:Development and Operations, DevOps, Assessment, Appraisal, SLM
DevOps has been established as a framework used by software development companies seeking to set mechanisms to automate their development processes. Consequently, over the last decade, many companies have adopted DevOps to support their project’s development process and perform continuous improvement tasks to ensure that it is applied correctly. To achieve this, companies are looking for solutions that allow them to evaluate the degree of implementation of DevOps in their internal processes. In this sense, the objective of this study focuses on identifying, through a systematic mapping of the literature, the mechanisms used to assess DevOps in software development companies. According to the above, the current state of knowledge related to the proposal of processes, models, techniques, tools, and methodological guides is presented to conduct the DevOps assessment. As a result, it is noted that there are multiple methodological solutions that seek to assess DevOps; however, a high degree of heterogeneity was evidenced in the identified solutions, resulting in the need to establish a clear framework that serves as the basis for proposing a generic, structured, and unambiguous DevOps assessment model applicable to software companies.
H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295
CMMI Institute, Capability maturity model integration for development, 2018.
Rational Software, “Rational Unified Process,” in Best Practices in Software Development Teams, 2020
W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings 9th International Conference in Software Engineering, 1987, pp. 328–338
K. Schwaber, J. Sutherland, The scrum guide the deﬁnitive guide to scrum: The rules of the game, 2017
M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003
K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002
K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000
H. Kniberg, Scrum and XP from the Trenches, 2015
C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009
J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Hawaii International Conference on System Sciences, 2008, pp. 466–466
A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” in Conference on Electrical and Electronic Engineering, 2005, pp. 704-710
J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” in Hawaii International Conference on System Sciences, 2014
ISO/IEC, Calidad de los servicios TI,” 2019
M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in The International Conference on Information and Computer Technologies, 2015, pp. 78–82
S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, 2017, pp. 1–7
P. Debois, Devopsdays - Organizing Guide, 2009
S. Nagpal, A. Shadab, Literature Review: Promises and Challenges of DevOps, 2017
M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017
C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI, vol. 2020, no. E32, 2020
J. Michelsen, Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps, 2014
E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28
M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computing in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29
J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65
F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
J. D. Patón-Romero, M. Piattini, “Green IT maturity models: a systematic mapping study,” in 12th Iberian Conference on Information Systems and Technologies, 2017, pp. 1–6
K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, “Systematic mapping studies in software engineering,” in International Congress on Engineering and Sustainability, 2008, pp. 1–10
J. Biolchini, P. G. Mian, A. C. Natali, G. H. Travassos, “Systematic review in software engineering,” in System Engineering and Computer Science Department, 2005
D. Budgen, M. Turner, P. Brereton, B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Psychology of Programming Interest Group, 2008, pp. 195–204
M. Genero, L. Cruz, M. Piattini, Métodos de investigación en ingeniería de software. Bogota, DC: Ra-ma, 2014
N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference On Cyber Security And Protection Of Digital Services, 2019, pp. 1–8
P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Systems and Software Process Improvement and Innovation, 2019, pp. 111–122
M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in IEEE International Conference on Digital Ecosystems, 2019, pp. 205–212
M. Gasparaite, S. Ragaisis, “Comparison of devops maturity models,” in International Conference on Information Technologies, 2019, pp. 65–69
J. M. Radstaak, Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations, Grade Thesis, University of Twente, 2019
A. Caprarelli, E. Di Nitto, D. Tamburri, “Fallacies and pitfalls on the road to DevOps: a longitudinal industrial study,” in Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment: Second International Workshop, 2019, pp. 200–210
M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019
J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in Jornadas Iberoamericanas de Ingeniería de Software e Ingeniería del Conocimiento, 2019.
A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020
C. Marnewick and J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE, 2020.
D. Teixeira, R. Pereira, T. Henriques, M. M. Da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020
T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020
J. Guerrero, K. Zuñiga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/JCI.2020.1.5
R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information, 2018, pp. 244–259
T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the American Society for Information Science and Technology, 2018, pp. 279–285.
T. Seppä-Lassila, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, Finland, 2017
L. König, A. Steffens, “Towards a quality model for devops,” Continuous Software Engineering & Full-scale Software Engineering, vol. 37, 2018
O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” Software Process Improvement and Capability Determination, 2016, pp. 135–149
M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in Software Process Improvement and Capability Determination, 2016, pp. 123–134
R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Work in Agile. Methods, 2016, pp. 39–49
R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, Towards the adoption of DevOps in software product organizations: A Maturity Model Approach, 2017
S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, Eindhoven University of Technology, 2014
J. Smeds, K. Nybom, I. Porres, “DevOps: A Definition and Perceived Adoption Impediments,” Lecture Notes in Business Information Processing, vol. 212, pp. 166–177, 2015. https://doi.org/10.1007/978-3-319-18612-2
IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Standar 2675-2021, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476
G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceedings International Conference on Software Engineering, 2016, pp. 76–85
L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in European Symposium on Systems Analysis and Design, 2019, pp. 77–90
P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760
M. A. McCarthy, L. M. Herger, S. M. Khan, B. M. Belgodere, “Composable DevOps: automated ontology based DevOps maturity analysis,” in International Conference on Service-Oriented Computing, 2015, pp. 600–607
B. Kitchenham, S. Linkman, D. Law, “DESMET: A methodology for evaluating software engineering methods and tools,” Computing and Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997
ATOS, DevOps Maturity Assessment, 2021. https://bit.ly/3uTbPve
Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
Infostretch, Infostretch DevOps Self-Assessment, 2021. https://bit.ly/3fh4krm
InCycle, InCycle Evaluacion de devops, 2021. https://bit.ly/2RqYQCl
IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
Unión Europea, Reglamento 651/2014, 2014
Unión Europea, Recomendación de la comisión del 6 de mayo de 2003 sobre la definición de microempresas, pequeñas y medianas empresas, 2003
How to Cite
Copyright (c) 2022 Carlos-Eduardo Orozco-Garcés, César-Jesús Pardo-Calvache, Yilber-Hernán Salazar-Mondragón
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles included in the Revista Facultad de Ingeniería are published under the Creative Commons (BY) license.
Authors must complete, sign, and submit the Review and Publication Authorization Form of the manuscript provided by the Journal; this form should contain all the originality and copyright information of the manuscript.
The authors who publish in this Journal accept the following conditions:
a. The authors retain the copyright and transfer the right of the first publication to the journal, with the work registered under the Creative Commons attribution license, which allows third parties to use what is published as long as they mention the authorship of the work and the first publication in this Journal.
b. Authors can make other independent and additional contractual agreements for the non-exclusive distribution of the version of the article published in this journal (eg, include it in an institutional repository or publish it in a book) provided they clearly indicate that the work It was first published in this Journal.
c. Authors are allowed and recommended to publish their work on the Internet (for example on institutional or personal pages) before and during the process.
review and publication, as it can lead to productive exchanges and a greater and faster dissemination of published work.
d. The Journal authorizes the total or partial reproduction of the content of the publication, as long as the source is cited, that is, the name of the Journal, name of the author (s), year, volume, publication number and pages of the article.
e. The ideas and statements issued by the authors are their responsibility and in no case bind the Journal.