I’ve got my first first-author-paper published in Personality and Individual Differences. The paper is titled “Reliability and completion speed in online questionnaires under consideration of personality” (doi:10.1016/j.paid.2017.02.015) and was written together with Lina and Christian. 

When online surveys are performed, data is in practice often filtered based on the completion time: very fast and very slow respondents are removed from the study. In this context we had two research questions:

1. Does completion time of an online questionnaire affect psychometric measures of reliability?
2. Does personality predicts the time a participant needs to complete a self-report questionnaire?

While the first question is mainly a replication of Montag & Reuter (2008), the second has rarely been studied before, at least with a focus on Impulsivity. Prior research has mixed findings mostly for paper-pencil questionnaires (which is prone to measurement errors as experimenters have to note the time manually). In particular, we were interested in the effect of Impulsiveness. Moltó, Segarra & Avila (1993) found that participants high in Impulsiveness needed less time to complete several questionnaires.

Abstract: While self-administered web questionnaires are increasingly used in psycho- logical and market research, the quality of collected data has to be continuously inspected. Two key questions were investigated in this paper: Does the overall completion speed in an online self-report questionnaire influence the reliability of personality scales? We assessed the Five-Factor Model of Personality (using NEO-FFI) and Impulsiveness (using BIS-11). Second, we asked if Impulsiveness predicts the overall completion speed.

In total, 532 participants (436 females, 96 males; mean age = 25.57 years) engaged in an online study to answer these questions. Replicating previous findings, no difference in the reliabilities was found for fast or slow respondents. While underlining the effect of Age on the completion speed, our data indicated evidence against our hypothesis of an influence of Impulsiveness on completion time using a Bayesian approach. Similar results could be observed using classical inferential methods. Of note, no effect could be observed for the Five-Factor Model of Personality and completion time either. Therefore, personality traits are not associated with individual differences in completion time in our investigated sample. We discuss our findings in a broader context of survey research and give a perspective for future research opportunities.

In short, we did not find evidence neither for differences in reliability nor for an effect of Impulsiveness on completion time. For the latter, we report Bayes factors in the result section of the paper but also a traditional hierarchical regression in the Appendix.

If I would do the same study again, I would perhaps do some things differently. First of all, I would pre-register the study including some kind of exclusion rule for outliers on completion time. Second, I would include some further measures of data quality to have a broader view on the topic. Lastly, I might try a full Bayesian analysis. Since this can also be done with the present data-set I might write another blogpost soon with this analysis.

For subscribers, the paper is available at the PAID journal website.¹ Everyone else can find the post-review manuscript here or on ResearchGate.

References

• Harms, C., Jackel, L., & Montag, C. (2017). Reliability and completion speed in online questionnaires under consideration of personality. Personality and Individual Differences, 111, 281–290. http://doi.org/10.1016/j.paid.2017.02.015
• Moltó, J., Segarra, P., & Avila, C. (1993). Impulsivity and total response speed to a personality questionnaire. Personality and Individual Differences, 15(1), 97–98. http://doi.org/10.1016/0191-8869(93)90047-7
• Montag, C., & Reuter, M. (2008). Does Speed in Completing an Online Questionnaire Have an Influence on Its Reliability? Cyberpsychology & Behavior, 11(6), 719–721.