I’VE JUST RETURNED from a working week under the humpbacked thatch at Totleigh Barton in Devon, UK, where I was teaching an Arvon Foundation course on writing fiction. At the end of the week sixteen more people went away strengthened in their desire to make their names as writers, and the continuing appeal of such courses, along with the proliferation of creative writing programmes across the land, suggests that thousands more share their ambition. Clearly only a few stand much chance of success, yet the enthusiasm generated is so impressive that the question arises as to why the need to write should be so widespread now. Perhaps we should also be asking what values we imagine we are serving when we take up the difficult challenge of writing a novel.

Part of the answer must lie in the unprecedented quest for individuation that has gathered force as increasing numbers of people have felt the need for a greater degree of self-knowledge and self-determination. It’s through language that we strive to open a passage from feeling into meaning, and imaginative writing can offer a lively way to explore fresh perspectives on how we see the world and our place within it. Whether the results get published or not, such an effort of discovery constitutes an important value in itself, yielding many satisfactions and subtle pleasures, and for some people that is already enough. But most hope for the rewards that may come from public recognition of their work, and to count themselves among that curious species of human being – the writer.

These were certainly my own, less than half-understood aspirations when I first felt the fire to write burning in my belly many years ago; but it wasn’t long before I discovered that, far from the romantic image I had cherished, the writing life was that of a drudge sitting alone in a room for hour after hour, day in, day out, beating his brows while doubts did their best to silence him. Being a writer meant being a re-writer, reworking the material over and over again till a pattern of words emerged with something close to the quick of life moving through its cadences. Not, I imagine, the way most people would choose to spend their days, and if an aura of romance still hangs around the novelist’s role it’s perhaps a hankering after some illusory bohemia or, more profitably, an heirloom of a time more than a century ago when such champions of the imagination as Dickens, George Eliot, Balzac and Tolstoy were widely revered as Western culture’s principal agents of consciousness.

But those authors thrived in a more coherent age than ours, an era before Freud, relativity theory and quantum uncertainties, when it was still feasible for their generous imaginations to convey with magisterial intelligence both the inward lives and outward behaviour of diverse characters along with a shrewd understanding of the social forces that shaped their destinies. By contrast, our own fissive and turbulent age so speedily out-trumps the novelist’s imagination at every outrageous turn that J. G. Ballard insists that, “in a sense, the writer knows nothing any longer. He has no moral stance … All he can do is devise various hypotheses and test them against the facts.”

Yet, earlier in the same essay, Ballard declared that “we live in a world ruled by fictions of every kind – mass-merchandising, advertising, politics conducted as a branch of advertising, the pre-empting of any original response to experience by the television screen.” In such circumstances what are the facts against which ‘hypotheses’ can be tested; and what kind of values can novelists hope to celebrate or investigate each time they choose to add a story to the enormous novel in which we live these days? Or to put it another way, is the art of the novel now reduced to an attempt to articulate the relative truth of private experience over the spin and jangle of the times, or does some larger, more unifying vision remain possible?

IT’S CERTAINLY THE case that, in the absence of any myth that carries general authority, all serious writers have to answer such questions for themselves, and my own working life has largely been shaped by them. Like many educated young men of my generation I grew up in thrall to the gravely sceptical, all but nihilistic vision of the existentialist writers. What other stance held any dignity, one wondered, over against the atrocious facts of the Holocaust, nuclear weaponry, and the age-old history of religious conflict? A clear-eyed, tough-minded atheism calmly confronting the absurdity of existence with its own intellectual poise seemed the mark of a truly adult consciousness. Some version of that story still prevails in most serious contemporary fiction, but as I see things now, I was lucky not to become that kind of novelist before life took me by the scruff of the neck and shook me into what I take to be a richer, less reductive view of things.

Briefly expressed, a temporary melt-down of the ego under intense personal pressure opened me up to intimations of the unitary nature of all being and to subtly enlarging frequencies of intelligence which had been jammed out of consideration by my previous, tightly managed model of how things are. Thereafter, though the claims of the ego would often close me in again, the terms on which I lived my life were irrevocably altered and the course of my work as a writer was determined by that change. All the novels I’ve written since then are attempts to dramatise as engagingly as I can the various ordeals of transformation by which we make the journey into larger consciousness, from ego to soul.

In particular I’ve been concerned with the trouble we men have in responding to the claims made by the feminine principle. For the deeper we enter into life the more we are confounded by its contradictory nature until we come awake to the possibility that only through the difficult reconciliation of contrary forces can something new begin to happen. Such reconciliations are accomplished by an athletic exercise of imagination in both its inventive and ethical aspects, so it’s in service of the imagination as a creative and compassionate value in itself that I try to write. The hope is that my fictions will contribute in an entertaining way to the emergence of a new mythology for our time.

FOR ME, THE imagination is the means by which we bring our experience of the outer world into negotiation with our inner world in order to create our personal myth of who we deeply are. The art is to keep the faculty as open as possible both ways, attending alertly to the intelligence reaching us through our senses and to the voices speaking through our dreams, our intuitions and the claims of conscience. A novel written in that reciprocating spirit may persuade a wider audience that, as Ted Hughes once said, “the laws of these two worlds are not contradictory at all: they are one inclusive system.”

I know that my own best work gets done when I’m at my most receptive – more pipeline than engineer – listening for the words that enter the space briefly evacuated by the ego. Many writers have acknowledged the mysterious power of such inspiration as something that, in Emily Brontë’s words, “strangely wills and works for itself”. Increasingly I believe that the source is the living Earth itself, speaking to us through the imagination, urging us to live by better stories than are told by our desires for power and dominion.

True or not, I find it a fruitful tale to tell myself. My hope for the future of literature is that more writers find it works for them too, so that our words perform an ever richer and more conscious song of the Earth. And it may be that the stories we discover will encourage a more open and generous relationship to the delicate web that supports our life and invests it with a deeper sense of meaning.

Lindsay Clarke’s latest novels are The War at Troy and The Return from Troy.