Pinker, evolution and a lot of confusion

Steven Pinker recently created a bit of a debate when he attacked group selection models in an essay entitled: THE FALSE ALLURE OF GROUP SELECTION
His argument is probably best summarized in his final words:

The idea of Group Selection has a superficial appeal because humans are indisputably adapted to group living and because some groups are indisputably larger, longer-lived, and more influential than others. This makes it easy to conclude that properties of human groups, or properties of the human mind, have been shaped by a process that is akin to natural selection acting on genes. Despite this allure, I have argued that the concept of Group Selection has no useful role to play in psychology or social science. It refers to too many things, most of which are not alternatives to the theory of gene-level selection but loose allusions to the importance of groups in human evolution. And when the concept is made more precise, it is torn by a dilemma. If it is meant to explain the cultural traits of successful groups, it adds nothing to conventional history and makes no precise use of the actual mechanism of natural selection. But if it is meant to explain the psychology of individuals, particularly an inclination for unconditional self-sacrifice to benefit a group of nonrelatives, it is dubious both in theory (since it is hard to see how it could evolve given the built-in advantage of protecting the self and one's kin) and in practice (since there is no evidence that humans have such a trait).

It is a thought provoking but superficial critique of an important issue, namely, at what level does selection operate. I just want to highlight three issues that I particularly grappled with, but if interested, you can find lots more here.

My first problem is the very narrow definition of variables. His definition of natural selection only allows genes as carriers of information, therefore, by definition any other source that may influence selection is already ruled out. Once he defined selection like this, there is no further point arguing. Case closed and many interesting social phenomena and their evolution remain unexplained.

The second problem is the variables that he is considering. The main focus is on altruism at the individual level. Of course, if you want to explain altruistic behaviour of individuals, natural selection is without doubt important. However, group selection works on different variables and at a different level. Group selection concerns the survival of groups over a long period. What is important is how information within a group is passed on that allows the survival of group (and the survival of the actual traits). I am traveling in Africa right now and I would be hard pressed to survive by my genetic information alone if I have no cultural tools that help me survive. Of course, I have modern technology that helps me navigate the hostile environment, but once my car runs out of fuel I am done and my credit card is not going to help me find water or food in the desert. Kalahari bushmen or Masai have very different technologies that has helped them survive for thousands of years. Jared Diamond in Collapse provides excellent examples of how cultural technology is important for group survival and how changing environmental conditions can push groups over the edge. Peter Richerson and Robert Boyd in Not by Genes Alone provide a more scientific account of these cultural processes. What is important here is not that it is a historical analysis, but that these accounts use formal models to understand what groups with what technologies are more likely to survive.

Finally, the description of natural selection used by Pinker is simplistic and does not reflect the emerging complexities of biological processes. The area that interests me most is that of epigenetics: cultural and social variables interact with biological mechanisms, such epigenetic processes switch genes on and off in ways that conflict with classic genetic ideas and the whole idea of a simple translation of genetic information to phenotypes is not tenable anymore. To understand natural selection of complex social traits, looking at genes alone is not cutting the cake.

I really enjoyed reading the essay and the many comments around it. Yet, the argument actually strengthened my conviction that we need to study group processes to better understand human evolution. The learning continues...

How do computers change cultures?

In a study to be published soon in Social Psychology, Nina Hansen and colleagues examined whether and how giving laptops to children in developing countries changes their cultural beliefs and values. It is a fascinating and important study because it looks at how Western technology affects cultural systems in very basic but profound ways. They gave a group of Ethiopian school children laptops. One year later, they did a follow up and compared those children whose laptop was still functioning with a control group as well as a group of students whose laptop had broken down. They note that after this year the children with functioning laptops had become more independent and endorsed more individualistic values. This change was apparently mediated by a change in self-construal, that is the children had changed their values because of changes in how they saw themselves as individuals. The authors also stated that traditional cultural beliefs and norms were not changed, that is collectivistic values and interdependent self-construals were not affected by laptop use. A fascinating finding, but I believe only a first step towards understanding how technology use affects traditional cultural systems.

Here are some of my questions that appeared while reading their article.
They proposed three different mechanisms of how technology affects culture change:
1. Operation of a modern laptop requires a set of complicated actions which children would have to master completely independently of their elders (a self-efficacy explanation)
2. Social usage of technology can change social relations and self-perceptions, eg., through changes in interaction patterns (email) and well-being. This was one mechanisms that I did not quite understood and a bit more explanation of how this may play out in an African context would have been great.
3. Even a cheap and basic laptop of the kind given to children is an immensely valuable object by local standards, increasing their local status vis-à-vis their elders and peers (a social status explanation).


For me, these mechanisms are actually quite important and could have been tested more directly and effectively. The implications of these mechanisms are certainly quite different from a pragmatic and practical perspective (think of interventions and education programmes).

One important aspect for future studies is a more comprehensive test of these mechanisms. For example, the first mechanism requires a change in cognitive structures and abilities. Therefore, a test of cognitive abilities and knowledge pre-post would have been very useful. This mechanism fits in with a number of sociological theories that argue that cultural change happens via changes in values that then lead to different cognitive strategies. Alternatively, you could argue that greater knowledge and self-efficacy in operating technology provides access to different perspectives which then allows children to develop new ideas and values. It would have been a fascinating opportunity to test some of these bigger theories and ideas in a realistic experiment like this one.

The third mechanisms could have been tested directly as their measure of independent values included both self-direction values - which capture self-directed values focused on exploring and experimenting with new ideas as well as power and achievement values - values focused on dominating others, concerns with status and one's position in the social hierarchy according to social norms and expectations. Hence, it would have been very easy to test whether the status explanation is a plausible explanation or not. I think it is important to differentiate whether these changes are driven by knowledge and self-efficacy or by status processes. If the former is true, than providing schools with laptops and sharing resources is a viable mechanism for empowerment, whereas this would not be effective if it was driven by status mechanisms.

Some of their reported data also did not quite gel with the overall message. The usage pattern suggested increasing sharing of the laptop – this suggests more collectivistic behaviour rather than more individualistic behaviour in the classical sense. Hence, the more individualistic changes were accompanied by more collectivistic behaviours!

The design is also problematic in some respects. For example, the criteria for selection of students into the two groups is not discussed. Was allocation to groups random? Were the two groups matched in any other important characteristic?
The authors used single items to measure self-construals. On one hand it is difficult to use standard scales in traditional non-Western communities. On the other hand, given the unknown properties of such Western imposed items, it is hard to understand what the item actually measured and validity and reliability can not be assessed.
It was also curious to see that this item of self-construals only mediated effects of condition on independent values in the laptop condition. In the no laptop condition, there was no correlation between self-construal and values. This may indicate validity problems with the item. Alternatively, it could also mean that our Western findings of relatively close links between the self and values are in fact culturally mediated - that is these links are formed by the use of our cultural products.

Overall, a great field study, but it is not quite answering some of the important questions that are posed in the article.


Hansen, N., Postmes, T., van der Vinne, N., & van Thiel, W. (in press). Information and communication technology and cultural change. How ICT usage changes self-construal and values. Social Psychology.

Applied Cross-Cultural Psychology: Some ideas for a meaningful science

I just spent the last 72 hours in 3 different countries. Lots of random thoughts raced through my mind while spending time in small eateries, big airports and on roads wide and narrow. How can cross-cultural research contribute to the development and well-being of societies? What are the tools that psychologists interested in culture can use to inform politicians and political decision-making? How can we make cross-cultural relevant to everyday actions and events, considering the massive challenges that humanity faces through globalization, climate change and increasing interdependencies at a global level?


I think there are three different paths that may address these broad questions of policy relevance and societal development. For lack of better words, I will call them culturally sensitive understanding, culturally sensitive change and culturally sensitive evaluation of change. In other words: a) an examination of processes that are of societal importance and relevance, b) development and application of culturally sensitive change programs and c) a culture-sensitive evaluation of existing intervention programs so that the needs of communities are better met. Engaging with bigger questions and practical problems entailed in these three approaches can help sharpening our basic research questions and theories as well as contributing to understanding and managing global issues.

Culturally sensitive understanding of societal level problems


The first option is a focus on a better understanding of psychological processes related to important societal outcomes. There are many debates about how society can be made more humane, healthy and prosperous. What are the psychological processes that are associated with these outcomes? Here, the strength of cross-cultural psychology is the quasi-experimental nature of culture. Societies differ along a number of important outcomes and potential antecedents, cross-cultural psychologists can take these variabilities and study what variables are most likely implicated in the different outcomes across societies. An open, but critical mind about potential antecedents about potential contributing factors is important. Once certain variables have been identified as potentially important, more controlled experiments to test the causality may be conducted. Not all variables can be manipulated in experimental settings (just think of the difficulty of manipulating national histories or seasonal patterns). This option is probably closest to standard psychological research. The main difference is a closer alignment between scientific research topics and questions of practical and societal relevance. 


My own focus has been more along the multi-country, sociological level of inquiry. One example is the work by Seini O'Connor. Corruption and political transparency has been on the minds of politicians, philosophers and political scientists for millennia. One of the major unaddressed questions though is what variables might be implicated in changes of corruption levels over time. There are many theories and ideas of what makes societies more or less transparent. Seini's honours project addressed these ideas through an innovative longitudinal method and found some pretty surprising findings (see http://www.victoria.ac.nz/home/about/newspubs/news/ViewNews.aspx?id=4815&newslabel=, the actual study can be found here: http://jcc.sagepub.com/content/early/2011/06/08/0022022111402344.abstract).


Implementing culturally sensitive change programs


Second, cross-cultural psychologists can engage in developing and running culturally sensitive interventions that address practical problems. Psychologists interested in culture have been relatively successful in developing and running intercultural training programs. At the same time, programs that focus on developing and changing behaviours of individuals and groups have largely been left to general psychologists or other disciplines (e.g., developmental workers, economists, sociologists, political scientists). Only few programs have taken a culturally sensitive approach when trying to change behaviours (for a cool example, have a look at this project: https://blog.itu.dk/MOSP-F2010/files/2010/03/rkhaled_siggraph09.pdf). There is much scope for innovative and important work to be done.


Evaluating interventions in culturally sensitive ways


Third, cross-cultural psychologists could get involved more in assessing existing change programs as they are applied and implemented in diverse cultures around the world. For example, micro-crediting – that is the provision of small loans to individuals or groups - has been used in many disadvantaged communities to fight poverty and contribute to economic growth. Yet, we know relatively little about the effectiveness of these initiatives, especially about how they fit in with the larger cultural norms, beliefs and practices. One of the interesting studies in this regard was reported in a study in Science last year (http://www.sciencemag.org/content/332/6035/1278.abstract) . Karlan and colleagues demonstrated that micro-crediting in the Philippines led to down-sizing of enterprises and higher stress among recipients, which is contrary to common expectations about the effectiveness of micro-crediting. This study was conducted by economists who have little interest in examining the cultural (or even psychological) processes. Cross-cultural psychologists could significantly contribute to such research and help in evaluating programs so that they better meet the needs of the communities.

Tales from the field: the dawn of day 3

It is a refreshing morning, the birds are chirping in the trees, a gentle breeze is playing with the banana leaves and the village roosters are advertising the blood red sun over the sea. 

Today is the day that turned yesterday into a strange and nearly frustrating experience. The local world does not play by the rules of the minds of the Western educated, science-oriented aliens that descended upon this little island to study their strange customs. A pre-test a few days ago revealed that the main measure is likely to be contaminated – a beautiful word for saying that somebody had worked out what the main dependent measure of the field study was and is likely to have instructed people how to answer it. A major debacle for the motley crew of international researchers hoping to study a fascinating religious ritual, with the high hopes to help humanity understand why engaging in seemingly insane and dangerous things (think of getting pierced, walking 4 to 6 hours in the tropical heat to finish off the day with a nice stroll over some gentle burning fire – who in their right Western mind would want to do something like this?). 

However, the one thing that should have sealed the study, the brilliantly devised and simple variable to measure how truly connected people feel to their religion and their religious fellows may not work anymore. The frustration turned into a heated debate about behavioural economics, a field of science that most villagers probably will never encounter in their whole life. Hours passed debating the pros and cons of games with the appealing names like dictator or prisoner dilemma game. 

It is fascinating to see the research work and weeks of preparation descend into an abyss of confusion, personal convictions, Western bias and scientific despair. One thing that I am wondering is, we don’t understand what these economic games are measuring with well-educated Western participants, despite nearly a century of research. What will it show us in a group that has problems understanding our humble attempts to ask them ‘how do you feel right now’? It makes me wonder how some famous studies published (like the famous series of studies by Joseph Henrich and others, see http://www.sciencemag.org/content/327/5972/1480.abstract) managed to explain complex games that take a page to describe in their widely cited publications to nomadic hunter and gatherer groups in the African bush. The appeal of our measure was its elegant simplicity and meaningfulness in a local community context. Yet, it might have been too easy and too transparent for the smart minds of some local people.

Now it is the dawn of day 3. A new day and a gentle breeze that calms the jetlag and insomnia. The debate was settled in the end late last night over some dinner and beer, we are going to use a similarly simple design, focusing on an unknown local entity, a potential Mead’esque faux pax, but the best that can be done within the time constraints of the study and better than other measures. It will be an exciting study nonetheless. 

The meeting last night hammering out the details, nine curious minds bent on making it work, 70 heart rate monitors to be connected to people participating in the ritual, a pre-post design with control groups and a multi-method design to study a fascinating ritual. And best of all – despite over 12 hours of tormenting debates and tiring preparations – the sun is shining, it is nice and warm and the sea is just meters away. 

And most importantly, it will be a fascinating day following new won local friends in their religious quests. The true beauty of field work. 

How to do Procrustean Factor Rotation with more than 2 groups

Today, I am continuing the torture with a bit more detail on options for comparing factor loadings across three or more groups within SPSS. This is a crucial issue for cross-cultural research and is becoming increasingly important, because researchers start studying more than two groups. More complex designs are more powerful in uncovering processes that can explain emerging behavioural differences, so this research should be strongly encouraged!

Aim: Compare the factor structure when you have more than two cultural groups, get an estimate of factor similarity

Why are we concerned with Procrustean Rotation? Factor rotation is arbitrary, therefore apparently dissimilar factor structures might be more similar than we think; procrustean rotation is necessary to judge structural and metric equivalence

Statistical Procedure:

The same syntax as for the two group case (see previous post: http://culturemindspace.blogspot.co.nz/2012/03/how-to-do-procrustean-factor-rotation.html) can be run with SPSS, but the greater number of countries adds additional problems. You have various options:

1. Run all pairwise comparisons. However, this will lead to a substantive number of comparisons (especially if you have many samples). This also leads to a number of statistical problems (remember family-wise error rate and increased Type I errors)
2. Select one country as your target group. For example, if an instrument was developed in the US, you may want to compare each group to the US.
3. Compute the average correlation matrix and use it for your factor analysis. The average is sometimes called pooled-within matrix. Therefore, you would compare each sample with the average structure across all samples (this can be done via discriminant function analysis in SPSS, you can then read the resulting correlation matrix into spss and use as an input for your factor analysis - see my discussion of how to do this here). This is highly appealing if you have many samples. This procedure of computing the average correlation matrix as input to the factor analysis can be simplified if (a) you have samples with similar sample size (no sample is dominating others; eg., if you have one sample of 10,000 and three samples of 50 participants each, the large sample is driving the factor structure) and (b) you mean centre each item within each sample prior to the overall factor analysis. This is necessary to account for any group mean differences that might obscure relationships if the samples are pooled. See below for a graphical explanation of why this might be a problem. As you can see, the relationship within each sample is negative, more sleep problems within each sample are associated with less laughter by participants. However, one group is consistently higher, for both the reported sleep problems as well as laughing. There may be reasons of why this is the case (I will come back to this example when talking about multilevel analysis), but for our analysis, combining the two samples would mean that we have a positive relationship across both samples combined (compared to negative relationships within both samples separately). This effect is due to the mean differences across both groups (I will post something soon on the beautiful complexity of these multi-level problems in psychology - very fascinating stuff). As a consequence of this confounding of group differences with individual differences, we need to take any such mean differences into account before we can combine the samples. This can easily be done using the z-transformation option in SPSS (‘Save standardized values as variables’ under the ‘Analysis’ -> ‘Descriptives’ option). 

I believe the last option is the most appealing with large data sets.

 However, cross-cultural psych never stops to be complicated. What happens if you find that some samples show good factor congruence with the average factor structure and others not? Ideally, you would exclude those samples from the average factor structure and re-run the analysis. Proceed iteratively till no sample shows any problems with factor similarity anymore.

If you have lots of cultural samples, you are really curious (and stats savvy) and want to find out what is happening in the strange worlds of culture, you may want to run cluster analysis on the congruence coefficients to identify clusters of samples that show greater similarity with each other. This might provide some interesting insights from a cross-cultural perspective. However, it is computationally demanding and relies on purely statistical criteria. There is a neat paper discussing various options and strategies, written by Welkenhuysen-Gybels and van de Vijver (2001, published in the Proceedings of the Annual Meeting of the American Statistical Association – I think this gives you an idea about what level of analysis we are talking about[1]). You can also download a SAS macro (the link is in the paper) that does much of the computational work for you. I have never worked with SAS, it seems a parallel universe to me and I am fascinated, but scared of it. But there are people who think it is easy. Conceptually, it is a nice tool.  

---

[1] You can download the paper at: http://www.amstat.org/sections/srms/Proceedings/y2001/Proceed/00106.pdf

How to do Procrustean Factor Rotation

Procrustean Factor Rotation

 Today, it is a little bit less light-hearted, but hopefully a bit more practical. 

Aim: To make factor structures maximally comparable & provide a statistical estimate of factor similarity

Why are we concerned with Procrustean Rotation? Factor rotation is arbitrary, therefore apparently dissimilar factor structures might be more similar than we think; procrustean rotation is necessary to judge structural and metric equivalence

Statistical Procedure:

 A SPSS routine to carry out target rotation needs to be run (adapted from van de Vijver & Leung, 1997)

The following routine can be used to carry out a target rotation and evaluate the similarity between the original and the target-rotated factor loadings. One cultural group is being assigned as the source and the second group is the target group. The varimax rotated (or unrotated) factor loadings for at least two factors obtained in two groups need to be inserted. The loadings need to be inserted, separated by commas and each line is ended with a semicolon. The last line is not to end with a semicolon, but with a ‘}’. Failure to pay attention to this will result in an error message and no rotation will be carried out. To use an example, Fischer and Smith (2006) measured self-reported extra-role behaviour in British and East German samples. Extra-role behaviour is related to citizenship behaviour, voluntary and discretationary behaviour that goes beyond what is expected of employees, but helps the larger organization to survive and prosper. These items were supposed to measure a more passive component (factor 1) and a more proactive component (factor 2). The selection of the target solution is arbitrary, in this case we rotated the East German data towards the UK matrix. 

Table 1. Items and varimax-rotated loadings in each sample separately

           

	UK		Germany
	Factor 1	Factor 2	Factor 1	Factor 2
I am always punctual.	.783	-.163	.778	-.066
I do not take extra breaks.	.811	.202	.875	.081
I follow work rules and instructions with extreme care.	.724	.209	.751	.079
I never take long lunches or breaks.	.850	.064	.739	.092
I search for causes for something that did not function properly.	-.031	.592	.195	.574
I often motivate others to express their ideas and opinions.	-.028	.723	-.030	.807
During the last year I changed something. in my work....	.388	.434	-.135	.717
I encourage others to speak up at meetings.	.141	.808	.125	.738
I continuously try to submit suggestions to improve my work.	.215	.709	.060	.691

Syntax:

This can not be done using the windows interface within SPSS. You should run a factor analysis in each sample separately first. Use Varimax (orthogonal) rotation.  Then insert the loadings in the loadings and norm matrices in the SPSS syntax described in Fischer and Fontaine (2011, in Matsumoto and Van de Vijver’s Cross-Cultural Research Methods in Psychology). I can also email this syntax to you (contact me at Ronald.Fischer@vuw.ac.nz).

The start of the syntax is printed below. Be careful to separate the loadings by a ‘,’ and the last loading for each item needs to be followed by ‘;’. The last loading should be indicated by }.

matrix.

compute LOADINGS={

.778,    -.066;  

.875,    .081;   

.751,    .079;   

.739,    .092;   

.195,    .574;   

-.030,   .807;   

-.135,   .717;   

.125,    .738;   

.060,    .691     }.

compute       NORMs = {

.783,    -.163;  

.811,    .202;   

.724,    .209;   

.850,    .064;   

-.031,   .592;   

-.028,   .723;   

.388,    .434;   

.141,    .808;   

.215,    .709}.

…

Output and Interpretation:

The edited output for this example is shown below. It shows the rotated matrix of the group (East Germany in our case) that was rotated to maximal similarity:

*********************************************************************

Run MATRIX procedure:

FACTOR LOADINGS AFTER TARGET ROTATION

   .77  -.10

   .88   .04

   .75   .05

   .74   .06

   .22   .57

   .00   .81

  -.10   .72

   .16   .73

   .09   .69

DIFFERENCE IN LOADINGS AFTER TARGET ROTATION

  -.01   .06

   .07  -.16

   .03  -.16

  -.11   .00

   .25  -.03

   .03   .08

  -.49   .29

   .02  -.08

  -.13  -.02

Square Root of the Mean Squared Difference per Variable (Item)

   .05

   .12

   .12

   .08

   .18

   .06

   .40

   .05

   .09

Square Root of the Mean Squared Difference per Factor

   .19   .13

IDENTITY COEFFICIENT per Factor

   .94   .97

ADDITIVITY COEFFICIENT per Factor

   .86   .92

PROPORTIONALITY COEFFICIENT per Factor

   .94   .97

CORRELATION COEFFICIENT per Factor

   .86   .93

------ END MATRIX -----

The output shows the factor loadings following rotation, the difference in loadings between the original structure and the rotated structure as well as the differences of each loading squared and then averaged across all factors (square root of the mean squared difference per variable column).

The first matrix could be pasted in a new table, showing the rotated loadings (instead of using the loadings from the original analysis as reported above in the table). The second matrix shows the differences after rotation. You should look for large values, because they indicate that some items are problematic. A low value would indicate good correspondence.

The column of values entitled: Square Root of the Mean Squared Difference per Variable (Item) gives you information about each item. The larger the value, the more problematic is an individual item. The next row (Square Root of the Mean Squared Difference per Factor) shows the same information per factor. Again, smaller values are better, larger values indicate trouble for a particular factor. There are no hard and fast criteria for any of these indices above, you should look at the relative values and particular discrepant values.

The most important information is reported in the last four lines, namely the various agreement coefficients. As can be seen there, the values are all above .85 and generally are beyond the commonly accepted value of .90. The most common indicator is Tucker’s Phi which is called Proportionality coefficient here.

It is also worth noting the first factor shows lower congruence and that the estimate vary across indicators. An examination of the differences between the loadings shows that one item (During the last year I changed something. in my work....) in particular shows somewhat different loadings. In the British sample, it loads moderately on both factors, whereas it loads highly on the proactivity factor in the German sample. Therefore, among the British participants making some changes in their workplace is a relatively routine and passive task, whereas for German participants this is a behaviour that is associated more with proactivity and initiative (e.g., Frese et al., 1996). We might want to exclude this item and re-run the analyses. Overall, we could cautiously conclude that our scales meet structural equivalence and most items might even meet metric equivalence (although this syntax routine does not provide a statistical test for this higher level of equivalence). 

Good on ya... if you made it to this point ; ) Hope your eyes are looking slightly better than that of a Tarsier...

Are cultural differences (in psychological processes) reducible to individual differences?

A colleague alerted me to an interesting paper published in the prestigious journal Proceedings of National Academy of Sciences. In this article (www.pnas.org/cgi/doi/10.1073/pnas.1001911107), Jinkyng Na and colleagues around senior author Richard Nisbett argue that differences in social orientation and cognition that exist between social classes and national cultures can not be reduced to individual differences. They present data from a moderately sized sample (N=235) of US citizens. The researchers administered a total of 20 tests, which were subdivided into 10 tests supposedly measuring social orientation and 10 tests measuring some form of cognitive style. They found significant differences between two groups of participants described as low and middle class for 5 of the cognitive tests (4 in the predicted direction) and 4 significant differences in the social orientation (3 in the predicted direction). Combining all measures, they also found significant effects. At the individual level, correlations were close to zero and mainly not significant. I have no problems so far.

The findings capitalize on a well-known statistical phenomena that within-group correlations and between-group correlations are independent and that the total correlation is the combination of both types of correlations. Put differently, correlations between variables (within groups) are statistically independent from the mean differences between groups. The correlation between all variables across all groups can be predicted if the variances and covariances are known. To their credit they discuss this phenomenon and present some simulations in this direction (without acknowledging that these issues have been discussed for at least 30 years in methodological circles, but also fail to realize the importance of separating the within-group from the between-group effects).

What really concerns me is their interpretation of this pattern. They argue that cultural differences are valid and can not be reduced to individual level differences. Where is the problem here?

In their starting example, they refer to both IQ and personality. For both variables, reliable and stable instruments had been constructed first and subsequently, differences were also found between ethnic and cultural groups. Therefore, we have a clear idea of what is happening at the individual level and then need to figure how between-group differences can be understood (which is an on-going debate). Following on from this, they now proceed to two vaguely defined and heterogeneously measured domains, namely social orientation (defined as independence and interdependence) and cognitive style (broadly defined as analytic versus holistic). The cross-cultural literature is full of studies that show small to moderate differences between different societies (most typically US student samples versus East Asian student samples). Yet, there is no consensus on definitions, measurement or a clear understanding of what these variables really are. Indeed, variables are lumped together that are studied as different phenomena in different areas of psychology. For example, happiness is certainly not the same as self-construals or intensity of emotions. Attributions are not the same construct as thematic versus taxonomic categorization tasks. The tests also involve various different methods (which can be seen as strength or weakness depending on the viewpoint). A stroop task is often capturing different psychological processes compared to self-reports, as the heated debate on implicit versus explicit attitudes demonstrates. Looking at this array of tests, I would simply not expect a strong correlation. We are not dealing with a coherent psychological phenomenon.

The mean differences between the two groups could be due to a large number of variables. Some of the causal variables may be strongly related: education, opportunities in life, and various other variables related to wealth immediately spring to mind. In a famous quote, one of the pioneers on cognitive styles Herman Witkin explicitly argued that field independence (as measured by the FLT) is associated with formal education. Wealth is probably also the most important variable influencing social orientation, a fact that is widely known since the famous study by Hofstede published in 1980. Hence, nothing new here. But the problem is  how wealth is influencing these variables and this most likely happening through different psychological processes. Hence, there is no reason to assume that these group differences are actually related rather than being probably influenced by a host of similar but distal variables.

In the latter parts of the paper, the authors then sink in a mess of tautological arguments. Because previous studies have shown these differences, then these differences need to be real and valid and hence, the individual and group level are distinct. Remember, this was what they wanted to test!

Yet, the authors also fail to notice the major shortcomings in many of the previous studies. For example, many of them did not test whether the instruments were equivalent or whether there were instrument, method, sampling or administration biases that could have influenced these results. Furthermore, these previous studies did not link potential explanatory variables to the observed differences, which is a standard practice in comparative psychological research (see for example http://pps.sagepub.com/content/1/3/234.abstract).

The current paper also fails to provide evidence on the equivalence of the measures. More importantly, it is not discussed how the two different groups are actually defined. What is social class and how were people grouped and on what criteria? The current study essentially is non-replicable.

What is a better approach? This has been discussed by a number of eminent cross-cultural psychologists, including Michael Bond, Kwok Leung, Ype Poortinga, Fons van de Vijver, David Matsumoto and others; and I have also occasionally added my two cents to this debate in a number of publications. The way forward is to come up with meaningful, reliable and valid instruments in each group that are equivalent in all cultures of interest. Next, we need to identify the situational, biological or psychological variables that are likely to explain the (predicted) differences between groups. The important element for understanding differences comes in empirically measuring these variables and demonstrating the link between our expected explanatory variable and the observed difference. As an added bonus, we could examine the influence at individual and group (cultural) level. Essentially, we are talking about a multi-level model here. There are now good examples and an increasing number of people use this important approach.

This is a scientifically more productive endeavor than setting up straw arguments based on statistical artefacts. One only wonders how something like this could get into a prestigious journal like PNAS....