@dvanhorn Nice. What's the distribution of scores among the students? Is there some kind of intense competition on the top of the leaderboard?

@dwallach I released it last night and woke up this morning to discover students are very good at finding under-specified corners of the reference implementation, but I'll let you know how it goes toward the end.

@dwallach More accurately, the students found places where the reference interpreter was *over-specifying* behavior, e.g. the interpreter has bignums and never overflows, but the compilers only have fixnums. So I had to change the interpreter to check for integers out of fixnum range and crash, thus making the compiler behavior unspecified. The interpreter also produced "the error result" on unbound variables, thereby specifying a run-time error, but it really should've been unspecified, so now it crashes.

@dvanhorn where did you get the 133 compilers? last year's students?

@wilcoxjay but they're writing next year's test harness 😏

@wilcoxjay next time I will probably play this game for every assignment and release the test assignment *before* the implementation assignment.

@dvanhorn awesome. dare I ask how you know that 100 out of 133 compilers can be eliminated? :)

@wilcoxjay Just a guess, really. I was able to eliminate ~80 pretty easily, so I figured 100 was possible. I will adjust the score if it's not possible. Current leader has knocked out 92 though. I did grab a snapshot of the submissions before the deadline in hopes of boosting the numbers.

@dvanhorn makes sense. I was worried the answer was something like "it's impossible to write a correct compiler", which I would also believe

@wilcoxjay yeah, I believe it too. I have lots of thoughts on extending this. One is there should be a bonus for eliminating the compiler written by me. Two I'd like to maintain a collection of compilers that have not previously been eliminated in years past and you get a bonus for any you take out.

@dvanhorn that would be cool. how do you do the latter with a gradescope autograder? I guess re-run the autograder locally so you can collect not just the score but the list of compilers (not) knocked out?

@wilcoxjay yeah, I'd have to do the curation outside of gradescope.

@dvanhorn aha! I missed the part where it was the previous assignment's compiler they were testing. cool!

Sign in to participate in the conversation
types.pl

A Mastodon instance for programming language theorists and mathematicians. Or just anyone who wants to hang out.