Assessing Software Carpentry

Greg Wilson just started a new GitHub repository for SWC assessments. It has me thinking about our plan of attack for assessment development in 2013. After talking with Julie Libarkin at MSU who specializes in research-quality assessment, I want to make sure that we don’t get ahead of ourselves. The process we should be following for this project is: (1) identify the goals (and learning objectives), (2) create the assessment instruments, (3) host SWC workshops which are “instructional interventions” and assess them with the instruments from part (2).


(1) Identify the Goals

We need a list of the top 5-10 goals/learning objectives of SWC overall. These can then be broken into subgoals, because we do have a lot to cover. On my first pass of the SWC curriculum, here is what I am seeing –

(A) The Bash Shell: files and directories; creating and deleting things; pipes and filters; wildcards; shortcuts; loops; shell scripts; finding things;

(B) Version Control: definition; how to checkout a working copy of a repo; view history; add files; submit changes; update working copy; check status of working copy; resolve and merge conflicts; setting up a repo; pr ovenance

(C) Basic Programming with Python: basic arithmetic; assign values to variables; create and run programs; data types; reading files; input and output; repetition; selection (making choices); flags; reading data files; provenance; lists; loops; nesting; aliasing; dictionaries and sets

(D) Functions and Libraries in Python: how functions work; why to use functions; define a new function; pass values; call stack; variable scope; global variables; multiple arguments; returning values; aliasing; importing libraries; filters; functions as objects; numpy

(E) Databases: selecting; removing duplicates; calculating new values; filtering; sorting; aggregation; database design; combining data; self join; missing data; nested queries; creating and modifying tables; transactions; programming with databases

(F) Quality: defensive programming; handling errors; unit testing; numbers; coverage; debugging; designing testable code

(G) Web Programming in Python: how we got here; formatting rules; attributes; metadata; lists and tables; images; links; processing HTML and XML; finding nodes; how the web works; getting data; providing data

(H) Development: the grid; aliasing; randomness; neighbors; handling ties; putting it all together; bugs; refactoring; testing; performance; profiling; a new beginning

I pulled this information from a summary of SWC curriculum, created by getting the website repository from GitHub, then doing:

$ make site
$ make book-summary > summary.html
$ open summary.html

Most importantly, these topics need to be translated into explicit learning objectives. Suggestions?


(2) Create the Assessment Instruments

This is the step that a lot of us are jumping to right now. I think we need to back up and make sure that we all understand and agree on (1). Then, we have to be careful to design quality assessments while not “re-inventing the wheel.” Julie has a lot of references for us with assessments that we can adapt to SWC situations. That’s preferred to writing our own assessments from scratch; it saves us work and reduces the time we have to spend checking reliability and validity of the instrument.

Before we get too far ahead into developing specific questions, let’s decide what we want to know.

(3) Host SWC Workshops as Interventions

When we finally have steps (1) and (2) finished, then we can easily apply a variation of our standard assessment instruments at any SWC workshop.


One thought on “Assessing Software Carpentry

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s