7A: Evaluation Guide
- Page ID
- 55438
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)What are concept inventories?
(Excerpted from presentation by J. Knight, UC Boulder).
- Multiple choice (usually) instruments that address fundamental concepts and contain known student misunderstandings
- Developed through an iterative process that includes gathering evidence of validity and reliability through student and faculty interviews
- Diagnostic: can identify specific misunderstandings and measure student learning over time
- Objective: not tied directly to a course, but rather to a set of concepts
Guidelines for using concept inventories
(Dirks, Wenderoth, Withers Assessment in the College Science Classroom, 2013).
- Protect the test!
- Must be given in a proctored environment to keep questions from getting out to students.
- Use for evaluation only
- Not a learning tool.
- When used for pre-/post-testing
- Use the same testing context
- Can use same or isomorphic questions (Resource: Research Methods Knowledge Base – W. Trochim, 2013)
- Normalized learning gain
- <g> = (%post - %pre)/(100-%pre)
Content-independent metrics
- Typically assess skills or affective domain
- Critical thinking, views of science, enthusiasm for the discipline…
- Can be used as pre/post, but typically post only
- Resource: FLAG – Field-tested Learning Assessment Guide - http://www.flaguide.org/index.php
- Mental Measures Yearbook http://buros.org/mental-measurements-yearbook
- http://www.salgsite.org/
- https://www.tntech.edu/cat/
- http://www.criticalthinking.org/pages/critical-thinking-testing-and-assessment/594
- https://www2.viu.ca/studentsuccessse...sInventory.pdf
Other Resources
- Summer Institutes website: http://www.summerinstitutes.org/
- University of Colorado – SEI: http://www.colorado.edu/sei/
- SERC: https://serc.carleton.edu/index.html
- https://serc.carleton.edu/NAGTWorkshops/departments/degree_programs/metrics.html
- UW BERG: http://uwberg.com/teaching-resources/
Table below from Dirk et al., (2014) Assessment in the College Science Classroom, Ch7 Appendix A; Freeman, NYC.
Concept Inventories in Astronomy |
|
|
Astronomy Diagnostic Test (ADT) Lunar Phases Light and Spectroscopy |
Hufnagel 2002 Lindell and Olsen 2002 Bardar et al., 2007 |
Concept Inventories in Biology |
|
|
Genetics Concept Inventory (GCA) Genetics Literacy Assessment Instrument 2 (GLAI-2) Conceptual Inventory of Natural Selection (CINS) Biology Literacty (http://bioliteracy.net/) Diagnostic Question Clusters: Biology Host Pathogen Interactions (HPI) Introductory Molecular and Cell Biology Assessment (IMCA) |
Smith et al., 2008 Bowling et al., 2008 Anderson et al., 2002 Klymkowsky et al., 2010 Wilson et al., 2006; D’Avanzo 2008 Marbach-Ad et al., 2009 Shi et al., 2010 |
Concept Inventories in Chemistry |
|
|
Chemistry Concept Inventory |
Mulford and Rbonison 2002 Krause et al., 2003 |
Concept Inventories in Engineering |
|
|
Engineering Thermodynamics Concept Inventory Heat Transfer Materials Concept Inventory Signals and Systems Concept Inventory Static Concept Inventory Thermal and Transport Science Concept Inventory (TTCI) |
Midkiff et al., 2001 Jacobie et al., 2003 Krause et al., 2003 Wage et al., 2005 Steif et al., 2005 Streveler et al., 2011 |
Concept Inventories in Geoscience |
|
|
Geoscience Concept Inventory (GCI) |
Libarkin and Anderson, 2005 |
Concept Inventories in Math and Statistics |
|
|
Statistics Concept Inventory (SCI) Calculus Concept Inventory (CCI) |
Allen 2006 Epstein 2005 |
Concept Inventories in Physics |
|
|
Force Concept Inventory (FCI) The Force and Motion Conceptual Evaluation (FMCE) Thermal Concept Evaluation Brief Electricity and Magnetism Assessment (BEMA) Conceptual Survey in Electricity and Magnetism (CSEM) |
Hestenes et al., 1992 Thornton and Sokiloff 1998 Yeo and Zadnick 2001 Ding et al., 2006 Maloney et al., 2001 |
Measuring Students Science Process and Reasoning Skills |
|
|
Rubric for Science Writing Student-Achievement and Process Skills Instrument |
Timmerman et al., 2010 Bunce et al., 2010 |
Measuring Student Attitudes about Science, Research or Study Methods |
|
|
Colorado Learning Attitudes about Science Survey (CLASS) Revised Two-Factor Study Process Questionnaire Student Assessment of Their Learning Gains (SALG) Instrument Survey of Undergraduate Research Experiences Views About Science Survey (VASS) |
http://www.colorado.edu/sei/class Biggs et al, 2001 Lopatto 2004 Halloun and Hestenes 1998 |