Art der Publikation: Beitrag in Sammelwerk

Automatic Assessment of Source Code Highlighting Tasks - Investigation of different means of measurement

Autor(en):

Kramer, M.; Barkmin, M.; Brinda, T.; Tobinski, D.

Herausgeber:

Joy, M.; Ihantola, P.

Titel des Sammelbands:

Proceedings of the 18th Koli Calling Conference on Computing Education Research

Seiten:

1-10

Verlag:

ACM Press

Ort(e):

New York

Veröffentlichung:

2018

ISBN:

978-1-4503-6535-2

Sprache:

englisch

Schlagworte:

Object-Oriented Programming, Highlighting, Assessment, Evaluation

Digital Object Identifier (DOI):

doi:10.1145/3279720.3279729

Link zum Volltext:

https://www.ddi.wiwi.uni-due.de/forschung/publikationen/acm/#kramer-barkmin-brinda-2-2018

Zitation:

Download BibTeX

Kurzfassung

In order to define and create certain elements of object-oriented programming source code, a necessary prerequisite for prospective programmers is being able to identify these elements in a given piece of source code. A reasonable task is therefore, to hand out existing source code to students and to ask them to highlight all occurrences of a certain concept, e.g. class identifiers or method signatures. This usually results in wide range of highlights. To quantify the received results and therefore to make plausible inferences about possible abilities, it is vital to have a reliable and valid method of measurement. First, we investigate various means of measurement towards this concern, including measures from inter-rater reliability and agreement, Cohen's κ and Krippendorff's α, as well as measures from binary classification, such as F1. Those values are then applied on constructed examples. We found that Cohen's κ already represents a given response in an adequate manner.