Julie Libarkin led our College Student Cognition in Science class in a lab activity last week centered on developing and writing a research protocol. We worked in cross-disciplinary pairs for about 40 minutes to design a protocol.
Our research protocol focused on the work I am doing with Software Carpentry (SWC). Below, I am outlining each major section of the design process and how it relates to SWC.
Research Questions. What do we want to know?
Does the Software Carpentry Boot Camp teach students good, reliable computational science practices? Are students actually using the skills taught in the camp long term? In other words, is the Boot Camp a viable model for teaching graduate students (and faculty?) to program?
Data. Identify the data required to answer these questions. What might it look like?
- Knowledge students have when entering a boot camp (multiple choice or Likert scale questions, either confidence based or content-specific questions)
- Knowledge students have when exiting a boot camp (multiple choice or Likert scale questions, either confidence based or content-specific questions)
- Confidence students have with solving the problems they face by using programming (Likert scale)
- Workflow practices; for example: how often to they use version control and how do they use it? (Multiple choice for # of hours a week spent doing a task, qualitative essay/short response about their experiences with computational science)
- A diagram of an individuals workflow, or how they approach a specific type of computational science problem
- Long-term impact of SWC, with a focus on using online resources/lessons and tutoring
- Background information like programming experience, year in school, research topic
- Demographic information like institution, gender, department, zip code, age
Some questions that may be important to look into:
- What do we need to do about the fact that SWC is voluntary? How does this change our audience (as opposed to a truly random audience)?
- What about faculty and staff who take the boot camps? What information do we need to survey for them?
- What sort of information is necessary to collect with each workshop (like location and instructor) and associate with students?
Hypotheses. What do we expect the outcomes to be?
We expect to see workshop attendees to have higher confidence levels when it comes to computational science. We also expect to see increased use of version control, working at the terminal, and writing scripts to solve problems.
Materials. What do we need in order to do the research / collect the data?
Aside from the workshop intervention itself, materials necessary are (1) online survey forms, (2) internet access for students, (3) a method for (reliably) reaching students six months after the boot camp.
Screening. Who will the research target?
All SWC participants, ideally. Perhaps we should make some stipulations like only attendees of a 2-day workshop.
The procedure is experimental (not solely observational) because SWC is in itself an educational intervention that we are testing.
Analysis. How will we process and interpret the data we collect?
We want to analyze the change in inferred student knowledge/understanding over the boot camp itself. We also want to see how background and discipline can affect a student’s long-term use of SWC skills. Coding the workflow diagrams so that we can do quantitative analysis on them to find patterns is ideal.