Most discussed and published findings from psychological research claim universality in some way. Especially for cognitive psychology it is the underlying assumption that all human brains work similarly — an assumption not unfounded at all. But also findings from other fields of psychology such as social psychology claim generality across time and place. It is only when replications fail to show an effect again, the limits of generality are discussed, i.e. in which way American participants differ from German participants. 

A recent paper by Patricia Greenfield (2017) argues that those differences and those limits to universality of psychological findings is a key reason why so many replications failed in the Reproducibility Project. She argues, based on her Theory of Social Change and Human Development, that between 2006 und 2014 culture and norms have changed in ways that should even lead one to expect replications not to show the same effects as in the original studies.

While I believe that the general notion of cultural and societal influences on attitudes and behaviour should play a stronger role in the evaluation and discussion of psychological effects, I think Greenfield’s final verdict (“Replicability Should Not Be the Gold Standard in Psychological Science”) is fundamentally wrong.

In her argumentation she follows a pattern that is partly to blame for the replication crisis and the looming theory crisis in Psychology. Her explanations are primarily post-hoc conditional on the fact that the replication failed. If the replication were successful her theory might have also explained that the cultural developments did not influence the effect in question. Of course, discussing studies that have already happened is always post-hoc and can still be valid — I believe, however, that this strongly weakens her argument.

If the general idea of social and cultural influences on our dependent variables is worth discussion, how should we proceed? At best, we would have a theory of how exactly those influences play a role so we can incorporate these into our studies, making exact predictions (which we pre-register, of course) and testing our hypotheses.
I don’t think this will happen in the short term, as psychological theories will remain weak without an ability to make exact predictions.

What we should aim for then is, first, to minimise these impacts on our conclusions. It is still too common to recruit WEIRD student samples for studies in social and cognitive psychology. Instead we should aim for more diverse, that is ultimately more representative, samples. How should it be possible – from both epistemological and statistical perspective – to make an inferential claim about universality if our sample is this limited and biased? Recruiting in different cultures, integrating and comparing results should be mandatory for bold claims of universality. For smaller studies, recruiting in different cities and through different channels is recommended.

Second, we need to be explicit about the limits of our theories and claims. Instead of arguing post-hoc how cultural change might have hindered a replication from succeeding, we need to include it in our theories. This is especially relevant for social psychology. If urban and rural participants differ in their individualistic-vs-collectivistic orientation we should measure these attitudes and include it in our theory and analysis.

Third, diversity in samples should go along with diversity in time. Cultural change happens over time, so we should closely investigate how our phenomena change over time, too. While this is a strong argument in favour of independent replications, it is also an argument for more longitudinal studies.

Greenfield’s (2017) perspective should not simply be dismissed as reactionary, but we should show through our methods and data if and how culture effect our results. Dismissing replications only on the ground of some cultural change without showing this through data is equally bad for the increase in the veracity of psychological research.

As a last note and only to mention it briefly here: One could make an even stronger argument in favour of Greenfield, which is to say that ultimately all studies of psychological phenomena are bound to their time and place. Thus, we are only able to make statements about singular events without being able to generalise. This would drive psychological closer to being part of the humanities than to the natural sciences. Cronbach (1975) already made an argument in this direction.

References

• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116–127. http://doi.org/10.1037/h0076829
• Greenfield, P. M. (2017). Cultural Change Over Time: Why Replicability Should Not Be the Gold Standard in Psychological Science. Perspectives on Psychological Science, 12(5), 762–771. http://doi.org/10.1177/1745691617707314
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716-aac4716. http://doi.org/10.1126/science.aac4716