Matching a specification in SmartRubric - Weighting Assessment Objectives

As exam season approaches, ensuring that you are providing accurate grades for mock exams and practice questions is really important. In this post, I'm going to explain the tools available to you in SmartRubric for setting up correctly weighted assessment objectives. 

SmartRubric is a GIGO system -- if you set up a good rubric, your marking experience will be smooth, accurate and painless. If you set up an awkward or incorrect rubric, you will not get the most out of SmartRubric. But don't worry -- I'm here to help! You can always email caroline@smartrubric.com to get personal help. 

Typically, exam specifications will give you guidance about how much each assessment objective is worth, but it isn't always clear how best to represent that in SmartRubric. I'm going to go through an example where some assessment objectives are weighted more heavily (worth more) than others. I'll break down the mark scheme, and then build it in SmartRubric.

EXAMPLE:  OCR A-Level English Literature, unit 1

Looking over this paper, we can establish the following structure to the exam: 

• Section 1: Shakespeare (two questions)
• Part a - 15 marks in total
• Part b - 15 marks in total
• Section 2: Drama and Poetry pre-1900 (one question - 30 marks)

In total, the paper is worth 60 marks. 

Using the structure above, it makes sense to create a rubric that has 3 tabs - one for Section 1(a), another for Section 1(b) and the last for Section 2. It will look something like this:

If you aren't sure how to build a rubric, you can check out the basics here

Let's just focus on Section 1(a). Once we have that right, it'll be clear how to do the rest. 

Now let's take a look at the mark scheme for that question:

The first thing we need to do is establish how many strands we need to accurately represent this mark scheme in SmartRubric. The rule of thumb is one bullet point, one strand, but I would advise you to use your judgement. Some mark schemes don't helpfully separate objectives out into individual strands for you. In this case, one bullet point, one strand works fine BUT there is a problem. OCR doesn't helpfully match the number of bullet points to the weighting. Instead, each strand in AO2 is worth 25% but each strand in AO1 is only worth 12.5%. 

Curses!!

There are two ways to handle this. 

1. (The WRONG WAY) You can combine both bullet points in AO1 into a single strand in SmartRubric. That means you'll end up with four strands in total: 

• Discussion of effects (AO2)
• Use of analytical methods (AO2)
• Reference to the text (AO2)
• Understanding & expression (AO1)

If you do it this way, each strand will be worth 25% of the whole, and AO2 will be worth 75% and AO1 will be worth 25%. The reason this is not a great solution is that it isn't really granular enough. What do you do if you have a student who really understands the question and knows her vocab inside and out, but her structure is all over the place? You'll need to make a judgement call and an estimate every time you mark a student using this scheme, and your accuracy will suffer. These 'estimates' are exactly the kind of thing SmartRubric is designed to kill off.

2. (The RIGHT WAY) You can adjust the weighting of each level descriptor so that you can still have five, but they're weighted correctly. You may have noticed that strands (rows) in the SmartRubric rubric builder have a field for 'weight': 

This number is a multiplier. What we are going to do is create a strand for each bullet point, like so: 

• Discussion of effects (AO2)
• Use of analytical methods (AO2)
• Reference to the text (AO2)
• Understanding of text and question (AO1)
• Expression (AO1)

Now we are going to correctly weight these strands by using the multiplier. To make things simple, we're just going to multiply each strand by its percentage weight: 

• Discussion of effects (AO2) WEIGHT: 25
• Use of analytical methods (AO2) WEIGHT: 25
• Reference to the text (AO2) WEIGHT: 25
• Understanding of text and question (AO1) WEIGHT: 12.5
• Expression (AO1) WEIGHT: 12.5

Adjusting the weight of a strand like this will NOT affect the total number of marks this question is worth. It just means that before SmartRubric averages out an overall score for a question, it multiplies the marked value of each strand by the weighting, and then divides the whole thing by the sum of the weights.

That's all there is to it!

So, before you start building your rubric in SmartRubric, take a close look at the mark scheme and ensure that you are weighting your assessment objectives correctly. That means that SmartRubric will be able to accurately calculate point scores for your students, and you can stop doing estimates!

If this is all too much, you can always email caroline@smartrubric.com to have your rubric created professionally for you (all users get a minimum of an hour of setup and support). 

Alternatively, if you want to go it alone, here are some other useful resources to help you build accurate rubrics in SmartRubric: 

1. How to create a rubric in SmartRubric
2. How to import and export rubric files
3. How to add rubric templates to your library
4. How to customise rubric templates to meet the needs of your specific classes
5. How to clear out old rubrics