Do You Really Code?

Reviewed by Greg Wilson / 2022-04-13
Keywords: Interviewing, Research Methods

How much do you need to know about Python to be considered a Python programmer? It may seem like asking "how long is a piece of string?" but the question is immensely important in recruiting, both for jobs and for research studies. While the Dunning-Kruger Effect might not be a real thing, the problem of people mis-estimating their own level of understanding certainly is.

Danilova2021 tackles this problem head-on. They recruited participants in four groups:

  • programmers (including both professionals and experienced CS students);
  • non-programmers (economics students and Clickworkers without programmer skills);
  • a test group (Clickworkers who claimed to have programming skills); and
  • an "attack group" (Clickworkers without programming skills).

They then presented each group with sixteen questions:

  1. Which of these lesser-known programming languages have you worked with before?
  2. Which of these websites do you most frequently use as aid when programming?
  3. Choose the answer that best fits the description of a compiler’s function.
  4. Choose the answer that best fits the definition of a recursive function.
  5. Choose the answer that best fits the description of an algorithm.
  6. Which of these values would be the most fitting for a Boolean?
  7. Please pick all powers of 2.
  8. Please translate the following binary number into a decimal number 101.
  9. Please select all even binary numbers.
  10. Please select all valid hexadecimal numbers.
  11. When multiplying two large numbers, your program unexpectedly returns a negative number. What might have caused this?
  12. What is the run time of the following code?
  13. When running the code, you get an error message for line 6: Array index out of range. What would you change to fix the problem?
  14. What is the purpose of the algorithm?
  15. What is the parameter of the function?
  16. Please select the returned value of the pseudo code.

Of these, six seemed to distinguish reliably between programmers and non-programmers:

Danilova2021 results

Before you read the paper, try to guess which six questions are most effective, then compare the actual answer to the questions you use in screening interviews. The result may surprise you: it certainly surprised me, but that's what research is for.

Danilova2021 Anastasia Danilova, Alena Naiakshina, Stefan Horstmann, and Matthew Smith. Do you really code? designing and evaluating screening questions for online surveys with programmers. In Proc. ICSE 2021, doi:10.1109/icse43902.2021.00057.

Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.

The six questions this paper recommended in the end are:

  1. Which of these websites do you most frequently use as aid when programming?
    • Wikipedia
    • LinkedIn
    • Stack Overflow
    • Memory Alpha
    • I have not used any of the websites above for programming
    • I don't program
  2. Choose the answer that best fits the description of a compiler’s function.
    • Refactoring code
    • Connecting to the network
    • Aggregating user data
    • I don't know
    • Translating code into executable instructions
    • Collecting user data
  3. Choose the answer that best fits the definition of a recursive function.
    • I don't know
    • A function that runs for an infinite time
    • A function that does not have a return value
    • A function that can be called from other functions
    • A function that calls itself
    • A function that does not require any inputs
  4. Which of these values would be the most fitting for a Boolean?
    • Small
    • I don't know
    • Solid
    • Quadratic
    • Red
    • True
  5. What is the parameter of the function [in the code shown below]?
    • String out
    • String in
    • I don't know
    • int i=x-1; i>=0; i--
    • Outputting a String
    • int x = len(in)
  6. Please select the returned value of the pseudo code [shown below].
    • hello world
    • hello world 10
    • dlrow olleh
    • world hello
    • HELLO WORLD
    • I don't know
    • hello world hello world hello world hello world
  main {
    print(func("hello world"))
  }

  String func(String in) {
    int x = len(in)
    String out = ""
    for (int i=x-1; i>=0; i--) {
      out.append(in[i])
    }
    return out
  }