EvaScrum: An assessment instrument to support the diagnosis of Scrum – results of two case studies

César Pardo, Wilson Ortega, Hmailton Jojoa, Ricardo A. Zambrano


The purpose of the process evaluation is to obtain relevant information in a qualitative and quantitative way about the current state in which a process is to support processes of evolution and continuous improvement. Scrum is one of the most used agile approaches, however, there some aspects that can hinder its implementation, e.g., lack of detail of artefacts and meetings that raises, including the timetables, application of the approach, among others. In this sense, and in order to facilitate the success in the implementation of Scrum, this paper presents EvaScrum, an assessment instrument that provides the opportunity to professionals and consultants to assess and diagnose the degree of implementation of Scrum through questions, metrics, a spreadsheet and a Web application. EvaScrum is based on Mr. Scrum; a reference model which provides a clear and complete set of process elements based on Scrum and EvaScrumTOOL, a web tool to manage the assessments. This paper presents the detailed analysis of two case studies in software development enterprises where EvaScrum was applied. The results obtained have allowed case study enterprises to identify improvement opportunities to carry out in the processes and projects where Scrum is being applied, all this, in a practical, useful and suitable manner which allow these types of enterprises, to assess and diagnose their implementations of Scrum with feasible resources both time and economically.


Scrum, Assessment, Software industry, Case studies, Software process improvement (SPI)

Full Text:



VersionOne Inc. & Collabnet, “CollabNet & VersionOne,” 15th Annual State Of Agile Report, 2021. https://bit.ly/3x9cTLO (accessed Jul. 30, 2021).

M. Amir, K. Khan, A. Khan, and M. Khan, “An appraisal of agile software development process,” Int. J. Adv. Sci. Technol., vol. 58, no. 1, pp. 75–86, 2013, doi: https://doi.org/10.14257/IJAST.2013.58.07.

T. J. Gandomani and M. Z. Nafchi, “Agility assessment model to measure Agility degree of Agile software companies,” Indian J. Sci. Technol., vol. 7, no. 7, pp. 955–959, 2014, doi: https://dx.doi.org/10.17485/ijst/2014/v7i7.8.

J. Rymer, D. West, M. Gilpin, J. Hammond, R. C. Wildeman, and D. D’Silva, “Lean software is agile, fit-to-purpose, and efficient,” 2008. [Online]. Available: https://bit.ly/2Rp5MOc.

O. Ozcan-Top, “Agilitymod: a software agility reference model for agility assessment,” Middle East Technical University, 2014.

K. Schwaber and J. Sutherland, “The scrum guide the definitive guide to scrum: The rules of the game,” USA, 2020. [Online]. Available: https://bit.ly/3iRzw2d.

K. Beck and C. Andres, Extreme Programming Explained: Embrace Change, 2nd ed. Boston: Addison-Wesley Professional, 2004.

A. Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams, 1st ed. Boston: Pearson Education, 2004.

M. Poppendieck and T. Poppendieck, Lean software development: an agile toolkit, 1st ed. Boston: Addison-Wesley Professional, 2003.

J. A. Highsmith, Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, vol. 12. New York, NY: Dorset House Publishing editors, 2000.

J. Stapleton, DSDM: Dynamic Systems Development Method: The Method in Practice, 1st ed. Addison-Wesley Professional, 1997.

S. R. Palmer and J. Mac Felsing, A Practical Guide to Feature Driven Development, 1st ed. Prentice Hall, 2002.

A. K. Shuja and J. Krebs, IBM Rational Unified Process Reference and Certification Guide: Solution Designer, 1st ed. IBM Press, 2007.

D. J. Anderson, Kanban: Successful Evolutionary Change for Your Technology Business, 3.8.2010. Blue Hole Press, 2010.

C. Mann and F. Maurer, “A case study on the impact of scrum on overtime and customer satisfaction,” in Proceedings - AGILE Conference 2005, 2005, pp. 70–79, doi: 10.1109/ADC.2005.1.

R. N. Jaffar, A. A. A. M. Hussain, and W. Chiad, “A new model for study of quality attributes to components based development approach,” Period. Eng. Nat. Sci., vol. 7, no. 3, pp. 1177–1185, 2019, doi: http://dx.doi.org/10.21533/pen.v7i3.686.

A. Marchenko and P. Abrahamsson, “Scrum in a multiproject environment: An ethnographically-inspired case study on the adoption challenges,” in Proceedings - Agile 2008 Conference, 2008, pp. 15–26, doi: 10.1109/Agile.2008.77.

M. Senapathi and A. Srinivasan, “Sustained agile usage: A systematic literature review,” in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE), 2013, pp. 119–124, doi: 10.1145/2460999.2461016.

C. Pardo, O. S. Gómez, H. Jojoa, R. Zambrano, and W. Ortega, “Mr. Scrum: A Reference Model to Foster and Facilitate the Adoption of Scrum in the Agile Software Development Companies,” Int. J. Adv. Sci. Eng. Inf. Technol., In Press, 2021.

J. F. Abrantes and G. H. Travassos, “Towards pertinent characteristics of agility and agile practices for software processes,” Clei Electron. J., vol. 16, no. 1, pp. 1–16, 2013.

S. Ambler, “IT Project Success Rates Survey Results,” 2018. https://bit.ly/3fwfXcz (accessed Jun. 15, 2020).

V. Babenko, L. Lomovskykh, A. Oriekhova, L. Korchynska, M. Krutko, and Y. Koniaieva, “Features of methods and models in risk management of IT projects,” Period. Eng. Nat. Sci., vol. 7, no. 2, pp. 629–636, 2019, doi: http://dx.doi.org/10.21533/pen.v7i2.558.g330.

M. Pikkarainen and U. Passoja, “An approach for assessing suitability of agile solutions: a case study,” Extrem. Program. Agil. Process. Softw. Eng., pp. 171–179, 2005, doi: 10.1007/11499053_20.

T. J. Gandomani, H. Zulzalil, A. A. A. Ghani, and M. A. B. Sultan, “Towards comprehensive and disciplined change management strategy in agile transformation process,” Res. J. Appl. Sci. Eng. Technol., vol. 6, no. 13, pp. 2345–2351, 2013, doi: https://doi.org/10.19026/RJASET.6.3706.

T. J. Gandomani, H. Zulzalil, A. A. Ghani, A. B. M. Sultan, and M. Z. Nafchi, “Obstacles in moving to agile software development methods,” J. Comput. Sci., vol. 9, no. 5, pp. 620–625, 2013, doi: 10.3844/jcssp.2013.620.625.

Y. Andalib and C. E. Mokhlis, “The contribution of the quality certification process to the improvement of human resources management practices,” Period. Eng. Nat. Sci., vol. 8, no. 3, pp. 1880–1887, 2020, doi: http://dx.doi.org/10.21533/pen.v8i3.1664.g672.

C. E. Mokhlis, A. Elmortada, M. Sbihi, and K. Mokhlis, “The impact of ISO 9001 quality management on organizational learning and innovation: Proposal for a conceptual framework,” Period. Eng. Nat. Sci., vol. 7, no. 2, pp. 944–951, 2019, doi: http://dx.doi.org/10.21533/pen.v6i2.590.

C. Pardo, W. Ortega, and O. S. Gómez, “A Reference model based on agile values, principles and aspects of Scrum, XP and Kanban to foster the adoption of agility in industry,” Int. J. Adv. Sci. Eng. Inf. Technol., In Press, 2021.

O. Ozcan-Top and O. Demirörs, “A reference model for software agility assessment: AgilityMod,” in Software Process Improvement and Capability Determination, 2015, vol. 526, pp. 145–158, doi: 10.1007/978-3-319-19860-6_12.

A. Qumer and B. Henderson-Sellers, “An evaluation of the degree of agility in six agile methods and its applicability for method engineering,” J. Inf. Softw. Technol., vol. 50, no. 4, pp. 280–295, 2008, doi: 10.1016/j.infsof.2007.02.002.

A. Sidky, J. Arthur, and S. Bohner, “A disciplined approach to adopting agile practices: The agile adoption framework,” J. Innov. Syst. Softw. Eng., vol. 3, no. 3, pp. 203–216, 2007, doi: 10.1007/s11334-007-0026-z.

K. Sureshchandra and J. Shrinivasavadhani, “Adopting Agile in Distributed Development,” in 2008 IEEE International Conference on Global Software Engineering, 2008, pp. 217–221, doi: 10.1109/ICGSE.2008.25.

M. James, “An Example Checklist for ScrumMasters,” 2016. https://bit.ly/2OBfw8P (accessed Jul. 30, 2021).

Henrik Kniberg, “crisp.,” The Unofficial Scrum Checklist, 2010. https://bit.ly/3dQCq4T (accessed Jul. 30, 2021).

D. Tousignant, “Cape Project Management Inc.,” Scrum Self Assessment, 2015. https://bit.ly/2QgnEMl (accessed Jul. 30, 2021).

M. P. Kumar, “Scrum Alliance,” Maturity Assessment Model for Scrum Teams, 2014. https://www.scrumalliance.org/community/articles/2014/july/maturity-assessment-model-for-the-scrum-teams (accessed Apr. 10, 2021).

R. Van Roosmalen, “Agile Strides – Coaching & Consultancy,” Excel Scrum Checklist, 2020. https://bit.ly/2D4MHIH (accessed Jul. 30, 2021).

O. R. Yürüm, O. Demirörs, and F. Rabhi, “A Comprehensive Evaluation of Agile Maturity Self-assessment Surveys,” in Software Process Improvement and Capability Determination, 2018, pp. 300–315, doi: 10.1007/978-3-030-00623-5_21.

V. R. Basili, G. Caldiera, and H. D. Rombach, “The Goal, metric, and question Approach,” Kaiserslautern, Germany, 1994. doi: 10.1002/0471028959.sof142.

F. J. Pino, C. Pardo, F. García, and M. Piattini, “Assessment methodology for software process improvement in small organizations,” Inf. Softw. Technol., vol. 52, no. 10, pp. 1044–1061, 2010, doi: http://dx.doi.org/10.1016/j.infsof.2010.04.004.

ISO/IEC, “Online Browsing Platform (OBP),” Information technology -- Process assessment, 2004. https://bit.ly/2qANDlk (accessed Jul. 30, 2021).

D. Avison, F. Lan, M. Myers, and A. Nielsen, “Action Research,” Commun. ACM, vol. 42, no. 1, pp. 94–97, 1999, doi: 10.1145/291469.291479.

P. Brereton, B. Kitchenham, D. Budgen, and Z. Li, “Using a Protocol Template for Case Study Planning,” in 12th International Conference on Evaluation and Assessment in Software Engineering (EASE 2008), 2008, pp. 1–8, doi: https://doi.org/10.14236/EWIC%2FEASE2008.5.

R. K. Yin, Case Study Research: Design and Methods, 5th ed. Boston: SAGE Publications, 2013.

P. Inc., “Learn how PayScale and Payfactors are coming together to make pay more powerful,” Average Certified Scrummaster (CSM) Salary, 2021. https://www.payscale.com (accessed Jul. 30, 2021).

L. H. García Paucar, C. Y. Laporte, Y. Arteaga, and M. Bruggmann, “Implementation and Certification of ISO/IEC 29110 in an IT Startup in Peru,” Softw. Qual. Prof., vol. 17, no. 2, pp. 16–29, 2015, doi: http://dx.doi.org/10.1007/978-3-319-30243-0_9.

M. Höst and P. Runeson, “Checklists for Software Engineering Case Study Research,” in Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 479–481, doi: http://dx.doi.org/10.1109/ESEM.2007.29.

J. H. Price and J. Murnan, “Research Limitations and the Necessity of Reporting Them,” Am. J. Heal. Educ., vol. 35, no. 2, pp. 66–67, Apr. 2004, doi: 10.1080/19325037.2004.10603611.

DOI: http://dx.doi.org/10.21533/pen.v9i3.2188


  • There are currently no refbacks.

Copyright (c) 2021 César Pardo, Wilson Ortega, Hmailton Jojoa, Ricardo A. Zambrano

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

ISSN: 2303-4521

Digital Object Identifier DOI: 10.21533/pen

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License