Valid, Reliable Assessment in Computer Science

I’m currently reading Allison Tew’s dissertation, Assessing Fundamental Introductory Computing Concept Knowledge in a Language Independent Manner

Her research questions are:

  1. How can existing test development methods be applied and adapted to create a valid assessment instrument for CS1 conceptual knowledge?
  2. To what extent can pseudocode be used as the mechanism for achieving programming language independence in an assessment instrument?
  3. To what extent does the language independent instrument provide a valid measure of CS1 conceptual knowledge?

Her work clearly documents good practices for assessment instrument development across any discipline. She also notes how these practices apply in computer science. I find it very interesting that Tew has gone to the trouble to develop this how-to guide for valid, reliable instruments, and yet the field (some 3 years later) has still not largely improved its standards for publishing education-related research. I wonder how to have a bigger impact on the research community, how to force journals and other publication venues to reform their standards and require researchers to establish reliability and validity for their instruments.

Tew’s process, documented chronologically, was:

  1. Define conceptual content
  2. Expert review of test specification
  3. Build test bank of questions
  4. Verify language independence
  5. Pilot questions
  6. Establish validity
  7. Establish reliability

Future research topics identified by Tew that intrigues me are:

  • Do students who learn in drag-and-drop programming environments like Alice transfer their conceptual understanding to pseudocode?
  • Compare the use of different programming languages for introductory courses. Michigan State uses Python for its intro courses because it is believe to be novice-friendly. Is this assumption actually true?

Other research topics that I see branching out of her work:

  • Develop a concept inventory applicable for nonmajors
  • Develop a concept inventory for computational science, possibly graduate-level
  • Develop a concept inventory for software design, object-oriented programming, or another Junior-Senior level course
  • Find ways to increase awareness of the DBER community in computer science about the importance of using valid, reliable instruments
  • Test Tew’s concept inventory with different cultural groups for bias



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s