Shared publicly  - 
Proper way to assess programming knowledge.

I've been thinking about this recently and was wondering if anyone else out there has any opinions about this. I currently teach 2 different programming classes and both have similar approaches to their final exams. There are two parts; one part written and one part practical.

For the practical the students are given a sheet with an objective as well as a list of requirements they must include. They are also provided a grading rubric in order to know what their grade is going to be based on. For example a practical may require students to create an application that loads an XML file that they create, which contains a set of nodes with paths to a sound file and a name for that file. They then create a button for each sound and then when you click on a button it will playback that sound. There also needs to be controls for things like volume and other UI elements for things like displaying the name of the sound currently being played.

With that example in mind here is the issue at hand. Should a student know how to do everything needed for this practical except how to adjust the volume level, they will do pretty well. They will be able to complete most of the assignment except for one little bit. However, if a student knows how to do everything needed except for how to load the XML file, there is a pretty good chance they will do very poorly. If they can't load the file, then they can't test their parsing code. Without the data, they may make mistakes in the code that would show the buttons, but not be able to see the mistakes they made since that code may not run or be compiled. This leads to a cascading problem where a student who may not know one particular part of the practical winds up losing far more points then they should. In both situations the students are missing one bit of information, but the result winds up costing the second student many more points.

A weak foundation leads to a weak house

One way that this can be looked at is like the practical is a house. Loading the XML file could be viewed as a foundational concept. One where the missing of that knowledge should lead to a much poorer grade as a house without a foundation is not one that could be constructed. Not controlling the volume could be viewed more as a poorly constructed door, something that would be an irritation but could be easily remedied and doesn't prevent the rest of the house from being constructed.

If this is the view that should be taken for some of the concepts on the practical then they could all be wrapped into a single line on the rubric worth 70 points as missing any one bit of these foundational concepts should lead to the student not passing. In that case I think it could be argued that the Final Practical could be a pass/fail grade.

I suppose it could also be argued that you could make that one line worth 31 points as that would lead to a failing grade, but then most of the points on their practical are coming from more superfluous aspects. I think then that the higher you shift that grade, the more it becomes like a pass/fail situation.

Inedible Bread

When I was younger I attempted to make bread using a bread machine. I measured out all the ingredients and sent them on their way. It cooked away and was smelling great, one of the big benefits of home made bread. Dinner time rolled around and we took the bread out and as soon as we cut into it we realized it was a gooey mess. I forget exactly which ingredient it was that I messed up, but I think it was the baking soda. Whatever it was I had managed to use Tablespoons of it instead of Teaspoons. The bread was inedible, but you could say that I was very close to having made it correctly and it was really one mistake I made that lead to the problem.

If their practical is then viewed more as bread then the building of a house, then the making of one mistake seems more forgivable. Most of the ingredients were correctly measured and put into the mix. Messing up on one ingredient didn't lead to a situation where I was no longer able to measure the other ingredients. While the end result is bread you can't eat, you can still demonstrate your understanding of the rest of the project.

Final Thoughts

I guess both these examples have the same conclusion; a promised result that is lacking. But I suppose that's not how I truly view the practicals. The real question isn't "Can you create a sound board?" it's "Can you demonstrate the concepts from this class by making a sound board?"

I feel like it's unfair to harshly grade a student whose simple mistake in the beginning leads to errors later. Especially errors in code that they may otherwise be very good at. I could break the practical up into a series of smaller assignments, making the testing of each concept isolated from the rest but a part of what is being taught is the interacting between these different concepts, so completely isolating them seems poor.

One thing I have considered is an option for students to "spend" their rubric points. That student who is having trouble loading their XML could then "spend" those points and I would then provide the code that is needed for that. They would then not receive any points for line on the rubric, but would then be able to progress with the rest of the project.

It may have been obvious already given my apparent desire to get to this conclusion, but I have also considered making a final that is pass/fail. If you accomplish all of the core concepts in the final you pass it. If you don't, then you fail it. In some ways this can make a lot of sense. The two courses I teach are both a part of a series of courses and a student leaving my course should know all the core concepts as not to cause problems in their following courses. This still seems like it could be a rather harsh approach but it could lead to students passing the final being those who truly deserve it.

The other thing that comes to mind is that perhaps I should spend more time teaching students how to work around mistakes. If you know your XML isn't loading correctly, you should be able to fake that information correctly within your code. You could hardcode in some values and be able to see that the rest of your application is working. I think this is a great idea in general, and one worth pursuing, but I think can still lead to the same exact situation. Sometimes, especially when under pressure, it can difficult to even see what it is that is going wrong much less figure out a work around.

I guess this leads to why I'm posting this here. It's partially because I am starting the process of taking all this thought and turning it into some kind of action, but it's also because I would like some other views on this. Please share if you have any thoughts regarding this monstrosity of a post.
Paul Cummins's profile photo
Very interesting post but you bring up some great points. I agree that 2 people both missing 1 bit of info shouldn't have different outcomes in that scenario. It makes sense if one is minor and one isn't but how you put it they are about the same.

I think it would be great to teach to work around mistakes because that is how the real world is. Sometimes you do get data last min so temp data will do. They just need to know how to work around those things correctly so it isn't a pain when you actually get the data.
Add a comment...