How Marker's Black Box Works

Marker's Black Box is a standalone marking application that runs from a PC with Windows XP and Winodws Vista. It can be used to mark and grade student coursework or exam paper that has been submitted as a hard copy (hand written or word processed) or in electonic form (e.g., online submission of a word processed document). The output of Marker's Black Box is a word processed marker's feedback sheet that contains the written comments and assessment of the marker. It also contains the college, faculty, course, and assignment details and an additional option is to include the marks awarded for each criterion.

Marker's Black Box works on the principle of marking student coursework with reference to a set of criteria. Student work is assessed on a number of criteria and this assessment draws on a rich, highly contextualised remark bank to form the basis of the report. In addition, other remark banks are used for common conceptual, grammatical, and communication errors. A further remark bank is used for remarks that are specific to the current assignment.

The application has a number of additional features that reduce the burden of marking and automate much of the marking process. An important design impetus in developing the application is the need for good intermarker reliability. Just as in-flight data recorders record the behaviour of an aircraft and in-vehicle data recorders record driving behaviour, Marker's Black Box also records the activity of its user and stores this data into an Excel file. This can be used to analyse one's own marking behaviour and style and can even be used to compare markers in a number of different ways.

Marking Templates
Each assignment has its own 'marking template', which consists of a set of marking criteria, criterion definitions, and a bank of remarks that are specific to each mark for each criterion. Each criterion is weighted so that it represents a certain percentage towards the overall mark. Some criteira can be weighted more than others. The marking team can set their own criteria, their weightings, and can agree on the remarks to use. This greatly improves inter-marker reliability. As marking proceeds, remarks are selected based on the mark awarded for each criterion. Thus, remarks are highly specific to the grades being given. It is easy to add to the remark bank or to edit existing remarks. The overall mark is calculated, taking into account how each criterion is weighted.

Criterion-specific Remarks and Comments that are Frequently Used
In addition, there are more remark banks that refer to conceptual issues, grammatical issues, and assignment-specific issues. These are remarks that tutors commonly find themselves using. They are listed by their label, such as 'paragraphs too long', 'too many quotations', and so on. When selected, a full explanation is added to the report. There is no limit to how many such remarks can be selected. Thus, feedback to students can be very rich and instructional. This is done by simple mouse clicks, rather than having to hand write or type the remarks completely.

Not only do these remark selection processes save time from having to write or type them, they also provide the marker with ideas and suggestions for comments. At the same time the remark banks and marking criteria keep the marker mindful of the marking rubric. This is something that really comes to the fore further on in a marking session.

Saving Student Marks in EXCEL
Another useful feature is that student lists can be imported from Excel and marks are saved automatically back into the same Excel file. There is also the option to save each mark for each criterion for every student. This might be especially useful when one wishes to compare the marking from different members of a marking team.

Use Marker's Black Box for All Types of Assignments
The key point about Marker's Black Box is that the marking template can be customised for each assignment. Thus, it can be used for the essay, research report, poster presentation, seminar presentation, portfolio, viva, oral exam, thesis, dissertation, and even references for students applying for jobs or further study. Criteria can be defined by the user and written comments can be edited or written from scratch. They can also be imported from other marking templates. So, if you create a marking template for a new assignment and want to use a criterion from another template, you can use the import function.

Export the Feedback Report to Your Word Processor
Once the item has been graded, the marker can then edit the written feedback, save it and then print the report from the marker's own word processor, such as Microsoft Word. The report automaticlly prints the course name and code, the student's name and ID, the marking criteria, the grades awarded for each criterion, the final mark, and the marker's name. All of these are optional (e.g., if the assignment is marked anonymously, obviously only the student's ID will be shown and not the name; also, the marker may not wish to show the marks awarded for each criterion).

Create References for Students
It's easy to view all of the remarks and edit them. It is easy to create your own set of remarks. In fact, it's so adaptable it can be used for many other kinds of assessment or evaluative purposes, such as providing a personal reference or a staff appraisal. Indeed, the application comes with a template for providing references, such as those required when students apply for jobs or further study.

In-Marking Data Recording
A great feature of this application is that it records the marker's behaviour and saves this data into an Excel file. For each item marked, it records the date, the coursework item being marked, the marker's name, the mark, the word length of the marker's comments, the number of additional remarks selected, the time taken to mark the item, and the marks awarded for each criterion. This data would be valuable for research purposes, such as testing for correlations between the time taken to mark a report and the actual mark awarded, the word length of the feedback and the mark awarded, and so on. If used for all marking, a tutor has a vast amount of data that could give useful insights into their marking behaviour. If used by a team of markers, it may be possible to characterise different types of markers, test for inter-marker reliability, and so on. Each time the application starts, some of this data is presented to the marker, such as the all-time average mark awarded. You can also view this data any time from within the application.

 

If you wish to reference this application, please use the following:

Fulcher, E. P. & Fulcher, G. T. (2009). Marker's Black Box. GEFT Consultancy Services, UK.