What is There About DevOps Assessment? A Systematic Mapping

¿Qué hay acerca de la evaluación de DevOps? Un mapeo sistemático

Main Article Content


DevOps has been established as a framework used by software development companies seeking to set mechanisms to automate their development processes. Consequently, over the last decade, many companies have adopted DevOps to support their project’s development process and perform continuous improvement tasks to ensure that it is applied correctly. To achieve this, companies are looking for solutions that allow them to evaluate the degree of implementation of DevOps in their internal processes. In this sense, the objective of this study focuses on identifying, through a systematic mapping of the literature, the mechanisms used to assess DevOps in software development companies. According to the above, the current state of knowledge related to the proposal of processes, models, techniques, tools, and methodological guides is presented to conduct the DevOps assessment. As a result, it is noted that there are multiple methodological solutions that seek to assess DevOps; however, a high degree of heterogeneity was evidenced in the identified solutions, resulting in the need to establish a clear framework that serves as the basis for proposing a generic, structured, and unambiguous DevOps assessment model applicable to software companies.



Download data is not yet available.

Article Details

Author Biographies (SEE)

Carlos-Eduardo Orozco-Garcés, Universidad del Cauca

Roles: Investigation, Formal Analysis, Methodology, Writing – original draft.

César-Jesús Pardo-Calvache, Universidad del Cauca

Roles: Supervision, Methodology, Validation, Writing-Revision and Edition.

Yilber-Hernán Salazar-Mondragón, Universidad del Cauca

Roles: Supervision, Methodology, Validation.

References (SEE)

H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295

CMMI Institute, Capability maturity model integration for development, 2018.

Rational Software, “Rational Unified Process,” in Best Practices in Software Development Teams, 2020

W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings 9th International Conference in Software Engineering, 1987, pp. 328–338

K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017

M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003

K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002

K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000

H. Kniberg, Scrum and XP from the Trenches, 2015

C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009

J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Hawaii International Conference on System Sciences, 2008, pp. 466–466

A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” in Conference on Electrical and Electronic Engineering, 2005, pp. 704-710

J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” in Hawaii International Conference on System Sciences, 2014

ISO/IEC, Calidad de los servicios TI,” 2019

M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in The International Conference on Information and Computer Technologies, 2015, pp. 78–82

S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, 2017, pp. 1–7

P. Debois, Devopsdays - Organizing Guide, 2009

S. Nagpal, A. Shadab, Literature Review: Promises and Challenges of DevOps, 2017

M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017

C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI, vol. 2020, no. E32, 2020

J. Michelsen, Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps, 2014

E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28

M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computing in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29

J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65

F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201

J. D. Patón-Romero, M. Piattini, “Green IT maturity models: a systematic mapping study,” in 12th Iberian Conference on Information Systems and Technologies, 2017, pp. 1–6

K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, “Systematic mapping studies in software engineering,” in International Congress on Engineering and Sustainability, 2008, pp. 1–10

J. Biolchini, P. G. Mian, A. C. Natali, G. H. Travassos, “Systematic review in software engineering,” in System Engineering and Computer Science Department, 2005

D. Budgen, M. Turner, P. Brereton, B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Psychology of Programming Interest Group, 2008, pp. 195–204

M. Genero, L. Cruz, M. Piattini, Métodos de investigación en ingeniería de software. Bogota, DC: Ra-ma, 2014

N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference On Cyber Security And Protection Of Digital Services, 2019, pp. 1–8

P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Systems and Software Process Improvement and Innovation, 2019, pp. 111–122

M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in IEEE International Conference on Digital Ecosystems, 2019, pp. 205–212

M. Gasparaite, S. Ragaisis, “Comparison of devops maturity models,” in International Conference on Information Technologies, 2019, pp. 65–69

J. M. Radstaak, Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations, Grade Thesis, University of Twente, 2019

A. Caprarelli, E. Di Nitto, D. Tamburri, “Fallacies and pitfalls on the road to DevOps: a longitudinal industrial study,” in Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment: Second International Workshop, 2019, pp. 200–210

M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019

J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in Jornadas Iberoamericanas de Ingeniería de Software e Ingeniería del Conocimiento, 2019.

A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020

C. Marnewick and J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE, 2020.

D. Teixeira, R. Pereira, T. Henriques, M. M. Da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020

T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020

J. Guerrero, K. Zuñiga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/JCI.2020.1.5

R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information, 2018, pp. 244–259

T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the American Society for Information Science and Technology, 2018, pp. 279–285.

T. Seppä-Lassila, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, Finland, 2017

L. König, A. Steffens, “Towards a quality model for devops,” Continuous Software Engineering & Full-scale Software Engineering, vol. 37, 2018

O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” Software Process Improvement and Capability Determination, 2016, pp. 135–149

M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in Software Process Improvement and Capability Determination, 2016, pp. 123–134

R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Work in Agile. Methods, 2016, pp. 39–49

R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, Towards the adoption of DevOps in software product organizations: A Maturity Model Approach, 2017

S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, Eindhoven University of Technology, 2014

J. Smeds, K. Nybom, I. Porres, “DevOps: A Definition and Perceived Adoption Impediments,” Lecture Notes in Business Information Processing, vol. 212, pp. 166–177, 2015. https://doi.org/10.1007/978-3-319-18612-2

IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Standar 2675-2021, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476

G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceedings International Conference on Software Engineering, 2016, pp. 76–85

L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in European Symposium on Systems Analysis and Design, 2019, pp. 77–90

P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760

M. A. McCarthy, L. M. Herger, S. M. Khan, B. M. Belgodere, “Composable DevOps: automated ontology based DevOps maturity analysis,” in International Conference on Service-Oriented Computing, 2015, pp. 600–607

B. Kitchenham, S. Linkman, D. Law, “DESMET: A methodology for evaluating software engineering methods and tools,” Computing and Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997

ATOS, DevOps Maturity Assessment, 2021. https://bit.ly/3uTbPve

Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz

Infostretch, Infostretch DevOps Self-Assessment, 2021. https://bit.ly/3fh4krm

InCycle, InCycle Evaluacion de devops, 2021. https://bit.ly/2RqYQCl

IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW

Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD

Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N

IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd

Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0

Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm

Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment

Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB

Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD

Unión Europea, Reglamento 651/2014, 2014

Unión Europea, Recomendación de la comisión del 6 de mayo de 2003 sobre la definición de microempresas, pequeñas y medianas empresas, 2003

Citado por: