Ventotene is a small island 25 miles off the Italian coast near Naples. It briefly made international news in August 2016 when the leaders of Italy, Germany and France met there to discuss the future of the EU in the wake of Brexit. Though little known, it is prized by discerning tourists: TripAdvisor is full of reviews mentioning its beauty and its quiet, one referring to it as “an island paradise”.

Two thousand years ago it did not seem so delectable. Then it was called Pandateria and it was there that the Emperor Augustus, in 2 BCE, chose to exile his wayward daughter Julia, after the revelation of her multiple adulteries. He chose it specifically as a place where, although she would not be physically harmed, she would endure daily misery from the horridness of her location.

Patrick Barkham does not mention Ventotene/Pandateria in his new study of islands and the human response to them, as he is concerned solely with the British archipelago; but he is very much aware of the paradox enshrined in that lump of Italian rock. “As often as islands are hailed as utopias, they have been dystopias,” he writes, as he sets out to unpick just why small pieces of land surrounded by water seem to exercise such a powerful hold over the emotions of many of us.

In Islander he runs his rule over eleven of them from around the edge of the United Kingdom, ranging downwards in size from the Isle of Man and its 221 square miles in the Irish Sea to Ray Island and its 110 acres in the estuary of the river Blackwater in Essex; the others in descending size order are Orkney (Mainland), Barra, Eigg, Rathlin, Alderney, Scilly (St Martin’s), St Kilda, Bardsey and Osea.

On all of them he is looking for the magic ingredient, the peculiar factor that pulls people to small islands, not just to visit, but to live – with, on occasion, unfortunate consequences. For many it is escape: sometimes simply a flight from the stresses and tensions of modern urban life and careers, sometimes a hunt for a deeper and more spiritual meaning. The reasons it can go wrong range from loneliness, and the inability to rub along with people in a tiny community, to, more prosaically, just running out of money.

Yet Barkham seeks to go beyond the conventional travelogue. He binds his odyssey together with a literary device: the revisiting of the life of a largely forgotten writer. (His model is clearly Helen Macdonald’s best-selling H is for Hawk, where the account of the training of a goshawk is blended with the biography of the failed falconer T.H. White, who wrote The Sword in the Stone.) Barkham’s overlooked author is Compton Mackenzie, the Englishman turned fervent Scottish nationalist who churned out nearly a hundred books but is now only remembered, if at all, for Whisky Galore, his 1947 novel about the Hebridean shipwreck of a cargo of Scotch whisky, which as a film became one of the best-loved of the Ealing comedies.

Forgotten as he may be, Mackenzie is a suitable subject here, as he was himself a confirmed nesomane, or lover of small islands (the coining, Barkham tells us, is from the Greek, and is John Fowles’), and lived on no fewer than eight of them at various periods of his life. What makes him more suitable still is that D.H. Lawrence took him and his island fixation as the model for a rather grim short story, The Man Who Loved Islands, in which a Mr Cathcart seeks his own paradise in decamping offshore, and ends up in hell.

Barkham uses Lawrence’s dystopian vision to examine closely the disadvantages of island living, and draws a striking conclusion: “They are dangerously seductive places for people seeking to escape the mainstream who swiftly discover they cannot escape themselves.”

Yet Mackenzie himself, in his lifelong island-progression through Man, and Capri, and Syra, and Herm, and Jethou, did not come to an infernal end: he lived most happily on his last one, Barra in the Western Isles, and following him through it all, in parallel with Barkham’s own island progress, certainly adds much to our understanding of what makes a nesomane tick (though Barkham as good as admits that Mackenzie’s is not a literary reputation that is going to be revived).

But it is not the literary device that makes the book work: it is the vivid portraits of the individual islands in their wild beauty, and their islanders, often turning up the unexpected. It is still not widely appreciated, for example, that charming Alderney in the Channel Islands, occupied by the Germans during the war, housed an SS concentration camp – yes, on British soil – in which hundreds and possibly thousands of slave labourers died.

None of the other islands has such a forbidding history, although on many the past is still vividly present. Bardsey, or Ynys Enlli, off the coast of North Wales, is known as “the island of 20,000 saints”, and it has a deeply spiritual air still, yet it is fascinating to learn that some of the religious solitaries drawn to it can be a pain to the other members of its small community. On Rathlin, off the Ulster coast, where once living was truly burdensome, “No one is here now who doesn’t want to be here.” Most of all we have the example of St Kilda, far out from the Hebrides, these days exercising the most enormous fascination, even longing, for many modern island-lovers – but the last of its native inhabitants abandoned it in 1930 because life was just too tough.

There is the paradox: small islands enshrine the dream life, but also the real life. By turns lyrical, humorous and moving, and always sharply perceptive, Barkham’s book captures rivetingly the alluring charm of these miniature worlds, while keeping a keen weather eye out for their pitfalls.

Michael McCarthy is a former environment editor of The Independent, and is the author of The Moth Snowstorm: Nature and Joy.