Skip to main navigation menu Skip to main content Skip to site footer

Vol. 31 No. 59 (2022)
January-March 2022 (Continuous Publication)

Papers

What is There About DevOps Assessment? A Systematic Mapping

https://doi.org/10.19053/01211129.v31.n59.2022.13896

Published 2022-03-14

Carlos-Eduardo Orozco-Garcés
Universidad del Cauca

César-Jesús Pardo-Calvache
Universidad del Cauca

Yilber-Hernán Salazar-Mondragón
Universidad del Cauca

Abstract

DevOps has been established as a framework used by software development companies seeking to set mechanisms to automate their development processes. Consequently, over the last decade, many companies have adopted DevOps to support their project’s development process and perform continuous improvement tasks to ensure that it is applied correctly. To achieve this, companies are looking for solutions that allow them to evaluate the degree of implementation of DevOps in their internal processes. In this sense, the objective of this study focuses on identifying, through a systematic mapping of the literature, the mechanisms used to assess DevOps in software development companies. According to the above, the current state of knowledge related to the proposal of processes, models, techniques, tools, and methodological guides is presented to conduct the DevOps assessment. As a result, it is noted that there are multiple methodological solutions that seek to assess DevOps; however, a high degree of heterogeneity was evidenced in the identified solutions, resulting in the need to establish a clear framework that serves as the basis for proposing a generic, structured, and unambiguous DevOps assessment model applicable to software companies.

Keywords

Development and Operations, DevOps, Assessment, Appraisal, SLM

PDF XML

Author Biography

Carlos-Eduardo Orozco-Garcés

Roles: Investigation, Formal Analysis, Methodology, Writing – original draft.

César-Jesús Pardo-Calvache

Roles: Supervision, Methodology, Validation, Writing-Revision and Edition.

Yilber-Hernán Salazar-Mondragón

Roles: Supervision, Methodology, Validation.

References

  1. H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92–99, 2002. https://doi.org/10.1109/MS.2002.1020295 DOI: https://doi.org/10.1109/MS.2002.1020295
  2. CMMI Institute, Capability maturity model integration for development, 2018.
  3. Rational Software, “Rational Unified Process,” in Best Practices in Software Development Teams, 2020
  4. W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings 9th International Conference in Software Engineering, 1987, pp. 328–338
  5. K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017
  6. M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003
  7. K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002
  8. K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000
  9. H. Kniberg, Scrum and XP from the Trenches, 2015
  10. C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009
  11. J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Hawaii International Conference on System Sciences, 2008, pp. 466–466 DOI: https://doi.org/10.1109/HICSS.2008.384
  12. A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” in Conference on Electrical and Electronic Engineering, 2005, pp. 704-710
  13. J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” in Hawaii International Conference on System Sciences, 2014
  14. ISO/IEC, Calidad de los servicios TI,” 2019
  15. M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in The International Conference on Information and Computer Technologies, 2015, pp. 78–82 DOI: https://doi.org/10.1109/INTECH.2015.7173368
  16. S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, 2017, pp. 1–7 DOI: https://doi.org/10.1109/ICTER.2017.8257808
  17. P. Debois, Devopsdays - Organizing Guide, 2009
  18. S. Nagpal, A. Shadab, Literature Review: Promises and Challenges of DevOps, 2017
  19. M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909–3943, 2017 DOI: https://doi.org/10.1109/ACCESS.2017.2685629
  20. C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI, vol. 2020, no. E32, 2020
  21. J. Michelsen, Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps, 2014
  22. E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in International Conference on Global Software Engineering, 2016, pp. 24–28. https://doi.org/10.1109/ICGSE.2016.28 DOI: https://doi.org/10.1109/ICGSE.2016.28
  23. M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computing in Emerging Markets, 2015, pp. 85–89. https://doi.org/10.1109/CCEM.2015.29 DOI: https://doi.org/10.1109/CCEM.2015.29
  24. J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60–65 DOI: https://doi.org/10.1109/IC2E.2015.23
  25. F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. https://doi.org/10.13140/2.1.5125.1201
  26. J. D. Patón-Romero, M. Piattini, “Green IT maturity models: a systematic mapping study,” in 12th Iberian Conference on Information Systems and Technologies, 2017, pp. 1–6 DOI: https://doi.org/10.23919/CISTI.2017.7975693
  27. K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, “Systematic mapping studies in software engineering,” in International Congress on Engineering and Sustainability, 2008, pp. 1–10 DOI: https://doi.org/10.14236/ewic/EASE2008.8
  28. J. Biolchini, P. G. Mian, A. C. Natali, G. H. Travassos, “Systematic review in software engineering,” in System Engineering and Computer Science Department, 2005
  29. D. Budgen, M. Turner, P. Brereton, B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Psychology of Programming Interest Group, 2008, pp. 195–204
  30. M. Genero, L. Cruz, M. Piattini, Métodos de investigación en ingeniería de software. Bogota, DC: Ra-ma, 2014
  31. N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference On Cyber Security And Protection Of Digital Services, 2019, pp. 1–8 DOI: https://doi.org/10.1109/CyberSecPODS.2019.8884935
  32. P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Systems and Software Process Improvement and Innovation, 2019, pp. 111–122 DOI: https://doi.org/10.1007/978-3-030-28005-5_9
  33. M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in IEEE International Conference on Digital Ecosystems, 2019, pp. 205–212 DOI: https://doi.org/10.1145/3297662.3365827
  34. M. Gasparaite, S. Ragaisis, “Comparison of devops maturity models,” in International Conference on Information Technologies, 2019, pp. 65–69
  35. J. M. Radstaak, Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations, Grade Thesis, University of Twente, 2019
  36. A. Caprarelli, E. Di Nitto, D. Tamburri, “Fallacies and pitfalls on the road to DevOps: a longitudinal industrial study,” in Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment: Second International Workshop, 2019, pp. 200–210 DOI: https://doi.org/10.1007/978-3-030-39306-9_15
  37. M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854–4862, 2019 DOI: https://doi.org/10.35940/ijrte.C6888.098319
  38. J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in Jornadas Iberoamericanas de Ingeniería de Software e Ingeniería del Conocimiento, 2019.
  39. A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020 DOI: https://doi.org/10.1016/j.cosrev.2020.100308
  40. C. Marnewick and J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE, 2020. DOI: https://doi.org/10.1109/MS.2020.3023298
  41. D. Teixeira, R. Pereira, T. Henriques, M. M. Da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464–511, 2020 DOI: https://doi.org/10.1504/IJASM.2020.112343
  42. T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020 DOI: https://doi.org/10.5220/0010177801170129
  43. J. Guerrero, K. Zuñiga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Ciencia e Ingeniería, vol. 12, no. 1, pp. 48–62, 2020. https://doi.org/10.46571/JCI.2020.1.5 DOI: https://doi.org/10.46571/JCI.2020.1.5
  44. R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information, 2018, pp. 244–259 DOI: https://doi.org/10.1007/978-3-319-91704-7_16
  45. T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the American Society for Information Science and Technology, 2018, pp. 279–285. DOI: https://doi.org/10.1145/3278681.3278714
  46. T. Seppä-Lassila, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, Finland, 2017
  47. L. König, A. Steffens, “Towards a quality model for devops,” Continuous Software Engineering & Full-scale Software Engineering, vol. 37, 2018
  48. O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” Software Process Improvement and Capability Determination, 2016, pp. 135–149 DOI: https://doi.org/10.1007/978-3-319-38980-6_11
  49. M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in Software Process Improvement and Capability Determination, 2016, pp. 123–134 DOI: https://doi.org/10.1007/978-3-319-38980-6_10
  50. R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Work in Agile. Methods, 2016, pp. 39–49 DOI: https://doi.org/10.1007/978-3-319-55907-0_4
  51. R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, Towards the adoption of DevOps in software product organizations: A Maturity Model Approach, 2017
  52. S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, Eindhoven University of Technology, 2014
  53. J. Smeds, K. Nybom, I. Porres, “DevOps: A Definition and Perceived Adoption Impediments,” Lecture Notes in Business Information Processing, vol. 212, pp. 166–177, 2015. https://doi.org/10.1007/978-3-319-18612-2 DOI: https://doi.org/10.1007/978-3-319-18612-2_14
  54. IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Standar 2675-2021, 2021. https://doi.org/10.1109/IEEESTD.2021.9415476 DOI: https://doi.org/10.1109/IEEESTD.2021.9415476
  55. G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceedings International Conference on Software Engineering, 2016, pp. 76–85 DOI: https://doi.org/10.1145/2904354.2904372
  56. L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in European Symposium on Systems Analysis and Design, 2019, pp. 77–90 DOI: https://doi.org/10.1007/978-3-030-29608-7_7
  57. P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Evaluation, 2020, pp. 757–760 DOI: https://doi.org/10.1109/ComPE49325.2020.9200149
  58. M. A. McCarthy, L. M. Herger, S. M. Khan, B. M. Belgodere, “Composable DevOps: automated ontology based DevOps maturity analysis,” in International Conference on Service-Oriented Computing, 2015, pp. 600–607 DOI: https://doi.org/10.1109/SCC.2015.87
  59. B. Kitchenham, S. Linkman, D. Law, “DESMET: A methodology for evaluating software engineering methods and tools,” Computing and Control Engineering Journal, vol. 8, no. 3, pp. 120–126, 1997 DOI: https://doi.org/10.1049/cce:19970304
  60. ATOS, DevOps Maturity Assessment, 2021. https://bit.ly/3uTbPve
  61. Microsoft, Microsoft DevOps Self-Assessment, 2021. https://bit.ly/2RZCHLz
  62. Infostretch, Infostretch DevOps Self-Assessment, 2021. https://bit.ly/3fh4krm
  63. InCycle, InCycle Evaluacion de devops, 2021. https://bit.ly/2RqYQCl
  64. IBM, IBM DevOps Practice Self Assesment, 2021. https://ibm.co/3w2bWEW
  65. Xmatters, DevOps Maturity Survey Report, 2021. https://bit.ly/33N8iCD
  66. Atlassian, DevOps Maturity model, 2021. https://bit.ly/2Rq1o3N
  67. IVI, IVI’s DevOps Assessment, 2021. https://bit.ly/3w9LGZd
  68. Veritis, Veritis, 2021. https://bit.ly/3yhZhQ0
  69. Boxboat, Boxboat, 2021. https://bit.ly/3yqsDMm
  70. Humanitec, DevOps Assessment, 2021. https://humanitec.com/devops-assessment
  71. Atlassian, DevOps Assessment, 2021. https://bit.ly/3fChUpB
  72. Eficode, Eficode DevOps Assesment, 2021. https://bit.ly/3omPkfD
  73. Unión Europea, Reglamento 651/2014, 2014
  74. Unión Europea, Recomendación de la comisión del 6 de mayo de 2003 sobre la definición de microempresas, pequeñas y medianas empresas, 2003

Downloads

Download data is not yet available.

Most read articles by the same author(s)

Similar Articles

You may also start an advanced similarity search for this article.