In the past few days, I attended two different workshops co-located with the International Conference on Software Engineering. The workshops were: the International Workshop on Software Engineering for Computational Science and Engineering (SECSI) and the International Workshop on Conducting Empirical Studies in Industry (CESI). I’ve learned one major thing from both of these workshops:
Software engineers have a lot of fancy terms for concepts that Software Carpentry teaches.
This post outlines some of the buzzwords that I heard at the conference (in the order they appeared), and how I see them relating (or not relating) to principles taught by Software Carpentry (SWC). Note: I am *not* a software engineer; these descriptions of software engineering concepts are my best understanding of the ideas/techniques in lay terms.
Agile software development is a development method that is based on iterative and incremental development. Requirements for the software develop through self-organizing, cross-functional teams. Adaptive planning is encouraged. Rapid and flexible response to change are inherent components of this development process.
SWC: We definitely teach incremental development, and we advocate iterative development (although I’m not sure that we truly teach it). Requirements for software in computational science are often vague, because the researchers may not yet know what the outcomes should be. Adaptive planning, therefore, is a must.
Lean software development is emerging from within the agile development movement. Lean development has seven major principles: (1) eliminate waste, (2) amplify learning, (3) decide as late as possible, (4) deliver as fast as possible, (5) empower the team, (6) build integrity in, (7) see the whole. Specifically, eliminating waste refers to taking out any part of software development that does not add value to the customer (like unnecessary functionality, unclear requirements, insufficient testing). Building integrity in refers to ensuring that all of the system’s components work well together as a whole; one way to achieve this quality is refactoring to keep simplicity, clarity, minimum amount of features in the code. In addition, the complete and automated build process should include developer and customer tests.
SWC: I think most computational scientists practice variations of lean software development. Since the products created in this environment generally have very specific use-cases, they only have a very specific (constrained) set of functionalities. SWC advocates building integrity as well, but I’m not sure how much students actually take away the importance of software integrity from workshops.
Continuous integration in software engineering requires all developer workspaces to merge into a shared mainline several times a day. It is intended to be used in combination with automated unit tests. In addition to the automated tests, organizations that use continuous integration usually use a build server to implement continuous processes of applying quality control. Principles of continuous integration are: (1) maintain a code repository, (2) automate the build, (3) make the build self-testing, (4) everyone commits to the baseline every day, (5) every commit (to baseline) should be built, (6) keep the build fast, (7) test in a clone of the production environment, (8) make it easy to get the latest deliverables, (9) everyone can see the latest of the build, (10) automate deployment.
SWC: With the use of version control and git, I believe we are advocating ideas in line with continuous integration, although we do not outright teach or require it. Often, during workshops participants have to pull additions to a git repo of course materials on day 2, which reinforces the concepts of getting latest deliverables, everyone can see the latest build, and deployment is automated.
Test-driven development is a process that requires software developers to write an automated test case that defines an improvement or new function. Initially, the test will fail. Then, the developer writes the minimum amount of code to pass that test. Finally, the developer refactors the new code to suit project standards. This process results in a short development cycle.
SWC: TDD is taught at workshops now. Usually, the second half of day 2 focuses on testing.
Extreme programming (XP) is a software development methodology that claims to organize people to produce higher-quality software more productively. Five major activities are defined as a part of this process: (1) coding, (2) testing, (3) listening, and (4) designing. XP also recognizes four major values: (1) communication, (2) simplicity, (3) feedback, (4) courage, and (5) respect.
SWC: I do not think we directly advocate XP with SWC. We do encourage working with a partner on programming and facilitating communication between programmers.
Pair programming is an agile software development technique where two programmers work together at a single workstation. The “driver” writes the code while the “navigator” reviews each line of code as it is being written. The programmers switch roles frequently.
SWC: The workshops do not have enough time to do actual pair programming, but participants work in pairs on small exercises assigned by instructors.
Unit testing is a method to isolate and test individual units of source code. Each test case should be independent from other tests. A suite of unit tests provides a strict, written contract that a piece of code must satisfy. Benefits of unit testing are: finding problems early, facilitating change, simplifying integration, providing documentation, replacing formal design.
SWC: Just like TDD, unit testing is taught in SWC boot camps in the second part of day 2. In particular, the benefits of testing are clearly stated for participants.
A scrum in an iterative and incremental technique used in agile software development. In a scrum, there are three core roles and a variety of ancillary roles. The core roles are referred to as “pigs” and the ancillary roles are “chickens.” The core roles are: product owner (stakeholders and voice of the customers), development team (responsible for delivering potentially shippable product increments at the end of each sprint), and scrum master (the facilitator of the scrum, accountable for removing impediments to the ability of the team). A “sprint” is a basic “time boxed” unit of development in scrum. The process of a sprint follows:
- Preliminary planning meeting where tasks for sprint are identified and previous progress is reviewed
- The product owner identifies items for the sprint from the “backlog,” an ordered list of requirements
- The backlog can only be modified by the development team during the sprint
- Development occurs over a fixed time period (one week to one month, generally)
- After a spring is completed, the development team demonstrates how to use the software
A sprint often has daily meetings called the “daily scrum.” Such meetings have specific guidelines: all members come prepared with updates; the meeting starts precisely on time, even if members are missing; the meeting should happen at the same location and time every day; the meeting length is timeboxed for 15 minutes; and all are welcome but only the core roles speak. Generally, the meetings are conducted with all members standing, to discourage a lengthier meeting. During the daily scrum, team members answer the following questions: (1) what have you done since yesterday? (2) what are you planning to do today? (3) any impediments/stumbling blocks?
SWC: Scrumming is not used in Software Carpentry workshops. At first glance, I don’t think it really has a place in SWC. But, when we are teaching participants “best practices” for computational science, maybe we should teach them the ideas of a daily “check-in” with small goals and incremental steps in software development.