9: Assessment and Grading Issues

Grading in Online Courses

Grading and assessment in learning is a complex subject that can (and often does) fill an entire book alone. It is also an area that all instructors with any experience in teaching know well. Grading can run the gamut from informal off the cuff questions thrown out spontaneously in class to gauge understanding, all the way to extensively controlled, proctored high stake tests.

To add to this complexity, every LMS and CMS system has slightly or majorly different ways to create and administer assessments. Your options may be limited to a few tools, or overwhelmingly complex in options. It is outside the scope of this manual to cover every single option of every tool, as there are too many to realistically cover, and they will change immediately after any list is completed. Instead, we will cover some issues concerning assessment and grading in general, giving a few ideas for better ways of approaching the concept of assessment. You may be teaching as part of program or company that has required tests or certifications that are beyond your scope to control. Therefore, this will be a brief examination of those scenarios that you can control.

Keep in mind that grading and assessment online is very different from grading and assessing in face to face classrooms. In face to face classrooms, there is much that you can do to tell if your learners are learning without giving a formal assignment or test. The immediacy of being in the same room with your learners allows you to gauge non-verbal clues, as well as allowing learners to ask questions in the moment. But when teaching online, you will need to find some way of determining what was actually learned by your learners. These are often referred to as “traces” or “artifacts.” Limitations and affordances of the platform you utilize will affect what these traces or artifacts look like. For instance, learners that are online will have access to the entire Internet while learning, making high stakes standardized testing problematic. Learners can just look up answers on the web (and proctoring tools that promise security are often easy to get around). Therefore, you might want to consider moving away from multiple choice tests and consider other more authentic ways of assessing what was learned.

Another aspect to consider when grading is the context of your course itself. For example, MOOCs will be assessed differently than smaller closed online courses. There would really be no way for an instructor to grade hundreds or thousands of papers in a MOOC, so if you are teaching a MOOC you would need to think of other ways to assess learning. Maybe multiple choice quizzes can be more of a “self-check” than a grade? Can those students that want to do projects utilize peer review if you give them a rubric to guide them? See the later section on “Testing Issues Specific to MOOCs” for more on this topic.

Also keep in mind that there could be other differences in how you create graded assignments – such as the difference between undergraduate and graduate courses, or the difference between training new employees versus re-training existing employees that need skills upgrades. In general, the more experience your learners will bring to your course (like in graduate courses, continuing education seminars, labor force re-skilling programs, etc), the more you will want to build on that existing knowledge and skills with more authentic projects than to transmit new information that needs to be memorized.

Earlier in this book, we discussed the decision to grade or not grade. Much of this chapter will be focused on those courses that have decided to grade. However, even in courses where the grade is more of a conversation, or students are grading themselves, the concepts covered here can still guide the creation of assessments in those scenarios. Additionally, even when you have your learners creating their own assignments, this section can help give you tips for how to guide your learners in that process.

Framing Learners

The first step in effective assessment is to take a good look at how you are framing your students. Does the way you design your tests frame them positively as co-learners or apprentices with you, or negatively as possible cheaters? Are you designing assessments that genuinely help with the learning process, or serve as a “gotcha!” for weeding out “weaker” students or those “not doing the work”? We would suggest taking a more positive look at your learners and their learning process, and designing your assessment strategy from that viewpoint.

Of course, there are students that are going to cheat. This happens in face-to-face courses, so of course it will happen in online courses. You will need to assume that learners can Google a lot of things while taking assessments. There are many companies selling monitoring devices or systems aimed at cutting down on cheating. Keep in mind these are not perfect – most students know how to fool them. Just do a search online if you don’t believe me: as soon as any company releases a new proctoring solution, there are soon videos online how to fool that new system. Some further thoughts on proctoring tools and student framing:

• “Shaming and Framing: Imagining Students at an Education Conference” by Chris Gilliard

Something to keep in mind is that learners who cheat do so because they have been given an assessment that is cheatable. In many ways, they are treating the test as a game to beat. This is the problem with standardized testing based on rote memorization: the answers are the same for everyone – and typically already on WikiPedia or some other website somewhere – so why do I need to memorize them in the first place? That is a good question to filter all of our course activities thorough: are students learning a skill and then just being tested on factoids about that skill? Is that really a good way to help them learn?

Types of Assessment

As previously mentioned, there is a long and ever-changing list of assessment tools and options. You are probably already familiar with many of them. While not an exhaustive list, here are some of the more common types of assessments you will see utilized in courses:

• Informal Polling: Usually used to gauge understanding in recent topics, or interest in future topics. These might be specific tools used to aggregate responses, or multiple choice questions utilized to collect more nuanced differences.
• Standardized Testing: These are the most common assessments, utilizing problem types such as multiple choice, true/false, matching, and other forms of standardized question formats. The answers are usually the same or similar for all learners, and tend to focus on rote memorization of facts or application of ideas to scenarios.
• Written Responses: This can take many forms, from open ended questions in a standardized tests, to comprehensive exams, to term papers. Learners may be required to remember a large amount of specific information, or to take what they know and apply it to problem-based scenarios.
• Oral Presentation: Learners either need to prepare a speech or other presentation for the instructor and/or other learners, or answer a series of questions correctly.
• Skills Test/Demonstration: When courses focus more on skills than information, often instructors will create some type of hands-on demonstration of skills gained.
• Activities: Any course activity can potentially be assessed in different ways, from discussion forums to blog entries to creative artifact creation (video, graphics, sound recordings, etc.) or any other work learners produce.
• Group Projects: Group projects are sometimes class activities, and at other times are assessments. These projects usually involve teams working together over a long amount of time to create a complex artifact or presentation.
• Portfolios/Cumulative/Capstone Projects: These are usually a collection of many assignments assembled over the entirety of the course (or multiple courses). These are often evaluated on not only the overall content, but also how they are presented or assembled.

Suggestions for Assessment Practices

Due to the large number of assessment options, it would take several books to cover them all. What we suggest is to determine what assessment plan would work best for your course, and then search for effective practices and tips on those specific options and tools. There are a few ideas we would suggest you take in to consideration before making decisions:

• Involve the learners as much as possible in the process of assessing their own knowledge or skills. The more they can take ownership of their learning process, the more they will learn. Don’t just look at how to involve them in the actual assessment, but in the design of what is tested or how learning is assessed overall.
• Don’t look at testing as a way to catch bad students or punish lazy learners. Many standardized tests fall into this category. Standardized tests can work as a way to help learners know what they should take away from a section of content. But that will work best if you give them unlimited attempts to take the test as a self-check-up.
• Build in ways for learners to give you feedback on any form of assessment, even required testing that you have no control over.
• Consider giving learners the option to determine how they can best communicate to you what they have learned. You might be surprised by their creativity. This requires some flexibility, but it also becomes more difficult for learners to cheat if every person is creating a custom artifact.
• Look at assessment as a conversation over time, not a one-time event. Can they turn in a rough draft of their paper or portfolio or project to you for feedback? If they get behind, are there pathways for them to get caught up?
• Make your job more focused. Some courses have so many discussion boards and tests and papers and so on to grade, that the instructors spend all of their time grading. Some grading rubrics get so complex that it takes forever to read, understand, fill out, and explain them. Focus your attention on what parts really needs to be assessed, and then focus on what you really need to know to fully assess. So much assessment can fall into the “busy work” category. This just burdens you and the learner.
• Where possible, focus on skills demonstrations, portfolios, and other more comprehensive forms of assessment. Have learners apply what they have learned, not just regurgitate factoids.

Standardized and Open-Ended Rubrics

Rubrics are a popular tool to use in grading online assignments. Some think they are a great way to standardize the grading process, while others think they are a way to stifle individual thinking in learners. This difference in opinion probably stems from the fact that some courses may need a more standardized approach, while others may need a more open-ended approach. Standardized rubrics can be seen as a more fair approach to grading by some learners that gives them a specific road-map for every part of an assignment. On the other hand, more open rubrics can save time and headaches, while allowing your learners to have some freedom and flexibility in their learning.

The most common way to design standardized rubrics is to have extensive details on each line of the rubric explaining what each point counts for, with several rows of criteria covering every aspect of each assignment:

Introduction / Topic

10 points: Student properly generates questions around a topic and introduces them fully.

5 points: Student generates questions and has a partial introduction.

2 points: Student is missing either introduction or topical questions.

0 points: Student has no questions or topical questions.

This method creates a strict roadmap that helps many learners. However, it also rarely allows learners to wrestle with what they want to do for an assignment. Additionally, there are many gaps that learners could fall into between point values or descriptions. These issues may not apply to all assignments, so you may find that a standardized rubric works best for your course design. Just keep in mind that these can be more time consuming to create and utilize from the instructor viewpoint.

Another way to approach grading issues is to take a more flexible, open-ended approach to creating rubrics. For example, try to have fewer rows that focus more on learners showing their knowledge than focusing on various assignment details like word count or assignment formats. Simply give the general area that will be graded, the parameters that you will look for, and a total number of points possible. Then leave an open slot for you to enter the points they earn in that area and a place for you to explain why you gave those points. For example:

Subject Knowledge

Subject knowledge is evident throughout the project. All information is clear, appropriate, and correct.

20

(comment)

(points earned)

This format can help you look at grading as more of a conversation than a final decree, especially if you allow learners to turn in early versions of assignments for formative feedback. However, keep in mind that learners may not be accustomed to this more open-ended approach, so you may cause anxiety if you do not scaffold them into this less structured approach.

Testing Issues Specific to MOOCs

Testing in a MOOC environment offers many unique challenges. The large number of students means that grading individual assignments is nearly impossible for most instructors. Some MOOC platforms will provide a number of self-grading options that can help in these situations. Additionally, you can create self- and peer-reviewed rubrics if that method will fit your design. There are also developments in the area of auto-graded papers that will analyze how you grade 100 papers, and then apply that logic to the rest of the papers. However, this feature is still in the testing phase.

The main concept to keep in mind is that MOOCs are open, and all participants will have access to the entire Internet when taking tests. This means that you do not want to design your tests as a big “gotcha” to catch learners that didn’t memorize the content. You will want to design the tests as a means to see if students can find the correct answer using whatever resources they can, and then indicate the correct answer on a test. Or, you may wish to design peer-reviewed assignments that aren’t based as much on correct answers as much as evaluating how well content has been applied. Finally, you can also create self-review rubrics that allow participants to reflect on their own learning in a deeper manner.

The main idea to keep in mind is that MOOC culture does not require an instructor to grade thousands of tests, papers, and forum posts every week. That would be unthinkable. So design your courses accordingly.

Certification and Badges

Many MOOC providers will offer certificates and / or badges to participants that complete their courses. Certificates and badges can be part of regular online courses as well. Typically these forms of reward are optional, meaning learners can still participate in a course even if they do not want to seek a badge or certificate. However, in many cases, badge or certificates are part of the official requirement to complete the course.

When badges or certificates are optional, keep in mind that some participants will want the certificates, and some will just want to learn the content for personal reasons. So you might want to have a certain set of activities for all participants, and then a special set of activities and / or assessments for those participants that are seeking certification.

In theory, anyone can use any tool to make a badge or certificate and send it to those that earn them. However, this could quickly lead to learners sharing these images with others that have not completed the work. Therefore, it is recommended to use a system that credentials secure badges and / or certificates in some ways. Usually, these are centralized spaces that allows you to create the credential and then serve as the gatekeeper to handing them out. Learners often have to create an account on the service to display their badges or certificates earned. If you are interested, here are some credentialing resources:

• Credly: free service for up to 500 badges.
• OpenBadges: widely accepted open credentialing system.
• Issuing OpenBadges.
• Badgr: free, open-source credentialing system.