So here we are, forming an orderly queue at the slaughterhouse gate. The punishment of the poor for the errors of the rich, the abandonment of universalism, the dismantling of the shelter the state provides: apart from a few small protests, none of this has yet brought us out fighting.

The acceptance of policies that counteract our interests remains the pervasive mystery of the 21st century. In the United States, blue-collar workers angrily demand that they be left without health care, and insist that millionaires should pay less tax. In the UK we appear ready to abandon the social progress for which our ancestors risked their lives, with barely a mutter of protest.

What has happened to us?

The answer, I think, is provided by probably the most interesting report I read last year. Common Cause: The Case for Working with our Cultural Values, written by Tom Crompton of the environment group WWF, examines a series of fascinating recent advances in the field of psychology, and offers a remedy to the blight that now afflicts every good cause from welfare to climate change.

Progressives have been suckers for a myth of human cognition, which Tom Crompton labels the Enlightenment Model. This holds that people make rational decisions by assessing facts. All that has to be done to persuade people is to lay out the data: they will then use it to decide which options best support their interests and desires.

But a host of psychological experiments have shown that it simply doesn’t work like this. Instead of performing a rational cost-benefit analysis, we accept only that information which confirms our identity and values, rejecting any of the information that conflicts with them. In other words, we mould our thinking around our social identity, protecting it from serious challenge, and so confronting people with inconvenient facts is likely only to harden their resistance to change.

Our social identity is shaped by values that psychologists classify as either extrinsic or intrinsic. Extrinsic values concern status and self-advancement. People with a strong set of extrinsic values fixate on how others see them. They cherish financial success, image and fame. Intrinsic values concern relationships with friends, family and community, and self-acceptance. Those who have a strong set of intrinsic values are not dependent on praise or rewards from other people. They have beliefs that transcend their self-interest.

Few people are all extrinsic or all intrinsic; our social identity is more usually formed by a mixture of values. But what psychological tests in nearly 70 countries do show is that values cluster together in remarkably consistent patterns. Those who strongly value financial success, for example, have less empathy, stronger manipulative tendencies, a stronger attraction to hierarchy and inequality, stronger prejudices towards strangers and less concern about human rights and the environment. Those who have a strong sense of self-acceptance have more empathy and a greater concern about human rights, social justice and the environment. These values suppress each other: the stronger someone’s extrinsic aspirations, the weaker his or her intrinsic goals.

Of course we are not born with our values; they are shaped by our social environment. By changing our perception of what is normal and acceptable, politics alters our minds as much as our circumstances. Free, universal health provision, for example, tends to reinforce intrinsic values. Shutting the poor out of health care normalises inequality, reinforcing extrinsic values.

The sharp shift to the right, which in the UK began with Margaret Thatcher and persisted under Blair and Brown – all of whose governments emphasised the virtues of competition, the market and financial success – has changed our values. The British Social Attitudes survey, for example, shows a sharp fall over this period in public support for policies that redistribute wealth and opportunity. And this shift has been reinforced by advertising and the media.

The media’s fascination with power politics, its ‘Rich Lists’, its catalogues of the 100 most powerful, influential, intelligent or beautiful people, its obsessive promotion of celebrity, fashion, fast cars, expensive holidays: all these inculcate extrinsic values. By generating feelings of insecurity and inadequacy – which means reducing self-acceptance – they also suppress intrinsic goals.

Advertisers, who employ large numbers of psychologists, are well aware of this. Crompton quotes Guy Murphy, global planning director for the marketing company JWT. Marketers, Murphy says, “should see themselves as trying to manipulate culture; being social engineers, not brand managers; manipulating cultural forces, not brand impressions”. The more they foster extrinsic values, the easier it is to sell their products.

Right-wing politicians have also, instinctively, understood the importance of values in changing the political map. Thatcher famously remarked that “economics are the method; the object is to change the heart and soul”. Conservatives in the United States generally avoid debating facts and figures. Instead they frame issues in ways that both appeal to and reinforce extrinsic values. Every year, through mechanisms that are rarely visible and seldom discussed, the space in which progressive ideas can flourish shrinks a little more.

And the progressive response to this trend has been disastrous.

Instead of confronting this shift in values, we have sought to adapt to it. Once-progressive political parties have tried to appease altered public attitudes: think of all those New Labour appeals to Middle England, which was often just a code for self-interest. In doing so they endorse and legitimise extrinsic values. Many greens and social justice campaigners have also tried to reach people by appealing to self-interest: explaining how, for example, relieving poverty in the developing world will build a market for British products, or suggesting that, by buying a hybrid car, you can impress your friends and enhance your social status. This tactic also strengthens extrinsic values, making future campaigns even less likely to succeed. Green consumerism has been a catastrophic mistake.

Common Cause proposes a simple remedy: that we stop seeking to bury our values, and instead explain and champion them. Progressive campaigners, it suggests, should help to foster an understanding of the psychology that informs political change and show how it has been manipulated. They should also come together to challenge forces – particularly the advertising industry – that make us insecure and selfish.

The leader of Britain’s Labour Party, Ed Miliband, appears to understand this need. He told last year’s Labour conference that he wants “to change our society so that it values community and family, not just work” and “to change our foreign policy so that it’s always based on values, not just alliances…We must shed old thinking and stand up for those who believe there is more to life than the bottom line.”

But there’s a paradox here, which means that we cannot rely on politicians to drive these changes. Those who succeed in politics are, by definition, people who prioritise extrinsic values. Their ambition must supplant peace of mind, family life, friendship – and even brotherly love!

So we must lead this shift ourselves. People with strong intrinsic values must cease to be embarrassed by them. We should argue for the policies we want, not on the grounds of expediency but on the grounds that they are empathetic and kind; and against others, on the grounds that they are selfish and cruel.

In asserting our values we can then become the change we want to see.

This is an edited version of an article published in George Monbiot’s weekly column in The Guardian newspaper. See also:

George Monbiot is an author and regular contributor for The Guardian.