(taken from The Guardian's extract of Jon Ronson's new book, The Psychopath Test)

It was visiting hour at Broadmoor psychiatric hospital and patients began drifting in to sit with their loved ones at tables and chairs that had been fixed to the ground. They were mostly overweight, wearing loose, comfortable T-shirts and elasticated sweatpants. There probably wasn't much to do in Broadmoor but eat. I wondered if any of them were famous. Broadmoor was where they sent Ian Brady, the Moors murderer, and Peter Sutcliffe, the Yorkshire Ripper.

A man in his late 20s walked towards me. His arm was outstretched. He wasn't wearing sweatpants. He was wearing a pinstripe jacket and trousers. He looked like a young businessman trying to make his way in the world, someone who wanted to show everyone that he was very, very sane. We shook hands.

"I'm Tony," he said. He sat down.

"So I hear you faked your way in here," I said.

"That's exactly right," Tony said.

He had the voice of a normal, nice, eager-to-help young man.

"I'd committed GBH," he said. "After they arrested me, I sat in my cell and I thought, 'I'm looking at five to seven years.' So I asked the other prisoners what to do. They said, 'Easy! Tell them you're mad! They'll put you in a county hospital. You'll have Sky TV and a PlayStation. Nurses will bring you pizzas.'"

"How long ago was this?" I asked.

"Twelve years ago," Tony said.

Tony said faking madness was the easy part, especially when you're 17 and you take drugs and watch a lot of scary movies. You don't need to know how authentically crazy people behave. You just plagiarise the character Dennis Hopper played in the movie Blue Velvet. That's what Tony did. He told a visiting psychiatrist he liked sending people love letters straight from his heart, and a love letter was a bullet from a gun, and if you received a love letter from him, you'd go straight to hell.

Plagiarising a well-known movie was a gamble, he said, but it paid off. Lots more psychiatrists began visiting his cell. He broadened his repertoire to include bits from Hellraiser, A Clockwork Orange and David Cronenberg's Crash. Tony told the psychiatrists he liked to crash cars into walls for sexual pleasure and also that he wanted to kill women because he thought looking into their eyes as they died would make him feel normal.

"Where did you get that one from?" I asked.

"A biography of Ted Bundy," Tony replied. "I found it in the prison library."

I nodded and thought it probably wasn't a great idea for prison libraries to stock books about Ted Bundy.

"But they didn't send me to some cushy hospital," Tony continued. "They sent me to bloody Broadmoor!"

Tony said the day he arrived at the dangerous and severe personality disorder (DSPD) unit, he took one look at the place and realised he'd made a spectacularly bad decision. He asked to speak urgently to psychiatrists. "I'm not mentally ill," he told them. It is an awful lot harder, Tony told me, to convince people you're sane than it is to convince them you're crazy.

"When you decided to wear pinstripe to meet me," I said, "did you realise the look could go either way?"

"Yes," said Tony, "but I thought I'd take my chances. Plus most of the patients here are disgusting slobs who don't wash or change their clothes for weeks on end and I like to dress well."

I looked around the Wellness Centre at the patients, scoffing chocolate bars with their parents who, in contrast to their children, had made a great effort to dress well.

"I know people are looking out for 'nonverbal clues' to my mental state," Tony continued. "Psychiatrists love 'nonverbal clues'. They love to analyse body movements. But that's really hard for the person who is trying to act sane. How do you sit in a sane way? How do you cross your legs in a sane way?"

I suddenly felt self-conscious. Was I crossing my legs like a journalist?

"So for a while you thought that being normal and polite would be your ticket out of here?" I said.

"Right," he replied. "I volunteered to weed the hospital garden. But they saw how well behaved I was and decided it meant I could behave well only in the environment of a psychiatric hospital and it proved I was mad."

I glanced suspiciously at Tony. I instinctively didn't believe him about this. It seemed too catch-22, too darkly-absurd-by-numbers. But later Tony sent me his files and, sure enough, it was right there. "Tony is cheerful and friendly," one report stated. "His detention in hospital is preventing deterioration of his condition."

After Tony read that, he said, he started a kind of war of non co-operation. This involved staying in his room a lot. On the outside, Tony said, not wanting to spend time with your criminally insane neighbours would be a perfectly understandable position. But on the inside it demonstrates you're withdrawn and have a grandiose sense of your own importance. In Broadmoor, not wanting to hang out with insane killers is a sign of madness.

"The patient's behaviour is getting worse in Broadmoor," a report written during Tony's non co-operation period stated. "He does not engage [with other patients]."

Tony was funny and quite charming for most of my two hours with him, but towards the end he got sadder. "I arrived here when I was 17," he said. "I'm 29 now. I've grown up wandering the wards of Broadmoor. I've got the Stockwell strangler on one side of me and the Tiptoe Through The Tulips rapist on the other. These are supposed to be the best years of your life. I've seen suicides. I saw a man take another man's eye out."

Tony said just being there can be enough to turn someone crazy. Then one of the guards called out a word – "Time" – and with barely a goodbye, Tony shot from the table and across the room. All the patients did the same. It was a display of tremendous, extreme, acute good behaviour.

I didn't know what to think. Unlike the sad-eyed, medicated patients all around us, Tony had seemed perfectly ordinary and sane. But what did I know?

The next day I wrote to Professor Anthony Maden, the head clinician at Tony's unit at Broadmoor – "I'm contacting you in the hope that you may be able to shed some light on how true Tony's story might be."

A few days later a letter arrived from Tony. "This place is awful at night-time, Jon," he wrote. "Words cannot express the atmosphere."

Tony had included in the package copies of his files. So I got to read the exact words he used to convince psychiatrists back in 1998 that he was mentally ill. He'd really gone to town. He said the CIA was following him, that he enjoyed taking things that belonged to other people because he liked the idea of making them suffer, and that hurting people was better than sex.

I felt the ground shift under my feet. Suddenly I was a little on the side of the psychiatrists. Tony must have come over as extremely creepy.

There was also a description of the crime he committed in 1997. The victim was a homeless alcoholic called Graham who apparently made "an inappropriate comment" about the 10-year-old daughter of one of Tony's friends; something to do with the length of her dress. Tony told him to shut up. Graham threw a punch. Tony retaliated by kicking him. Graham fell over. And that would have been it – Tony later said – had Graham stayed silent. But Graham said, "Is that all you've got?"

Tony "flipped". He kicked Graham seven or eight times in the stomach and groin, returning later to kick him again. I remembered that list of movies Tony said he plagiarised to demonstrate he was mentally ill. A Clockwork Orange begins with a gang of thugs kicking a homeless man while he is on the ground.

My phone rang. I recognised the number. It was Tony. I didn't answer.

A week passed and then the email I had been waiting for arrived from Professor Maden.

"Tony," it read, "did get here by faking mental illness because he thought it would be preferable to prison."

"Oh!" I thought, pleasantly surprised. "Good! That's great!"

But then I read Maden's next line: "Most psychiatrists who have assessed him, and there have been a lot, have considered he is not mentally ill, but suffers from psychopathy."

I looked at the email. "Tony's a psychopath?" I thought.

I didn't know very much about psychopaths back then, but I did know this: it sounded worse.

Faking mental illness to get out of a prison sentence, Maden explained, is exactly the kind of deceitful and manipulative act you'd expect of a psychopath.

A psychologist friend, Essi Viding, agreed. "Classic psychopath!" she said when I described Tony's pinstripe suit.

Tony rang again. I took a breath and picked up the phone.

"Jon?" he said. He sounded small and far away and echoey.

"Yes, hello, Tony," I said, in a no-nonsense way.

"I haven't heard from you in a while," he said.

"Professor Maden says you're a psychopath," I said.

Tony exhaled, impatiently. "I'm not a psychopath," he said.

"How do you know?" I asked.

"They say psychopaths can't feel remorse," said Tony. "I feel lots of remorse. But when I tell them I feel remorse, they say psychopaths pretend to be remorseful when they're not. Trying to prove you're not a psychopath is even harder than trying to prove you're not mentally ill."

"How did they diagnose you?" I asked.

"They give you a psychopath test," said Tony. "The Robert Hare Checklist. They assess you for 20 personality traits. Superficial charm. Proneness to boredom. Lack of empathy. Lack of remorse. Grandiose sense of self-worth. That sort of thing. For each one they score you a 0, 1 or 2. If your total score is 30 or more out of 40, you're a psychopath. That's it. You're doomed. You're labelled a psychopath for life. They say you can't change. You can't be treated. You're a danger to society. And then you're stuck somewhere like this."

It was the French psychiatrist Philippe Pinel who first suggested, early in the 19th century, that there was a madness that didn't involve mania or depression or psychosis. He called it "manie sans délire" – insanity without delusions. He said sufferers appeared normal on the surface, but they lacked impulse controls and were prone to outbursts of violence. It wasn't until 1891, when the German doctor JLA Koch published his book Die Psychopathischen Minderwertigkeiten, that it got its name: psychopathy.

The consensus from the beginning was that only 1% of humans had it, but the chaos they caused was so far-reaching, it could actually remould society. And so the urgent question became, how could psychopaths be cured?

In the late 1960s, a young Canadian psychiatrist believed he had the answer. His name was Elliott Barker and he had visited radical therapeutic communities around the world, including nude psychotherapy sessions occurring under the tutelage of an American psychotherapist named Paul Bindrim. Clients, mostly California free-thinkers and movie stars, would sit naked in a circle and dive headlong into a 24-hour emotional and mystical rollercoaster during which participants would scream and yell and sob and confess their innermost fears. Barker worked at a unit for psychopaths inside the Oak Ridge hospital for the criminally insane in Ontario. Although the inmates were undoubtedly insane, they seemed perfectly ordinary. This, Barker deduced, was because they were burying their insanity deep beneath a facade of normality. If the madness could only, somehow, be brought to the surface, maybe it would work itself through and they could be reborn as empathetic human beings.

And so he successfully sought permission from the Canadian government to obtain a large batch of LSD, hand-picked a group of psychopaths, led them into what he named the "total encounter capsule", a small room painted bright green, and asked them to remove their clothes. This was truly to be a radical milestone: the world's first ever marathon nude LSD-fuelled psychotherapy session for criminal psychopaths.

Barker's sessions lasted for epic 11-day stretches. There were no distractions – no television, no clothes, no clocks, no calendars, only a perpetual discussion (at least 100 hours every week) of their feelings. Much like Bindrim's sessions, the psychopaths were encouraged to go to their rawest emotional places by screaming and clawing at the walls and confessing fantasies of forbidden sexual longing for each other, even if they were, in the words of an internal Oak Ridge report of the time, "in a state of arousal while doing so".

My guess is that this would have been a more enjoyable experience within the context of a Palm Springs resort hotel than in a secure facility for psychopathic murderers.

Barker watched it all from behind a one-way mirror and his early reports were gloomy. The atmosphere inside the capsule was tense. Psychopaths would stare angrily at each other. Days would go by when nobody would exchange a word. But then, as the weeks turned into months, something unexpected began to happen.

The transformation was captured in an incredibly moving film. These tough young prisoners are, before our eyes, changing. They are learning to care for one another inside the capsule.

We see Barker in his office, and the look of delight on his face is quite heartbreaking. His psychopaths have become gentle. Some are even telling their parole boards not to consider them for release until after they've completed their therapy. The authorities are astonished.

Back home in London, I felt terribly sorry for Tony. So many psychopathic murderers – fortunate to have been under Barker's radical tutelage – had been declared cured and freed. Why couldn't Broadmoor adopt some of his ideas? Of course, they seemed dated and naive and perhaps overly reliant on hallucinogenics, but they were surely preferable to locking someone up for ever because he happened to score badly on some personality checklist.

Then I learned that two researchers had in the early 90s undertaken a detailed study of the long-term recidivism rates of psychopaths who'd been through Barker's programme and let out into society. In regular circumstances, 60% of criminal psychopaths released into the outside world go on to reoffend. What percentage of their psychopaths had? As it turned out: 80%.

The capsule had made the psychopaths worse.

"They had psychopaths naked and talking about their feelings!" Bob Hare laughed, shaking his head at the idealism of it all. It was an August evening and we were drinking in a hotel bar in rural Pembrokeshire, west Wales, at one of Hare's three-day residential courses for psychiatrists, care workers and criminal profilers. It was exciting finally to meet him. While names such as Elliott Barker have all but faded away, Hare is influential. Justice departments and parole boards all over the world have accepted his contention that psychopaths are quite simply incurable and everyone should concentrate their energies instead on learning how to root them out using his PCL-R (Psychopathy Checklist-Revised), which he has spent a lifetime refining.

In the mid-60s, Hare was working as a prison psychologist in Vancouver. He put word around the prison that he was looking for psychopathic and non-psychopathic volunteers for tests. He strapped them up to various EEG and sweat- and blood pressure-measuring machines, and also to an electricity generator, and explained to them that he was going to count backwards from 10 and when he reached one they'd receive a very painful electric shock.

The difference in the responses stunned Hare. The non-psychopathic volunteers (theirs were crimes of passion, usually, or crimes born from terrible poverty or abuse) steeled themselves ruefully, as if a painful electric shock were just the penance they deserved. They were, Hare noted, scared.

"And the psychopaths?" I asked.

"They didn't break a sweat," said Hare. "Nothing." The tests seemed to indicate that the amygdala, the part of the brain that should have anticipated the unpleasantness and sent the requisite signals of fear to the central nervous system, wasn't functioning as it should. It was an enormous breakthrough for Hare, his first clue that the brains of psychopaths were different from regular brains.

He was even more astonished when he repeated the test. This time, the psychopaths knew exactly how much pain they'd be in, and still: nothing. Hare learned something that others wouldn't for years: psychopaths were likely to reoffend. "They had no memory of the pain of the electric shock, even when the pain had occurred just moments before," Hare said. "So what's the point in threatening them with imprisonment if they break the terms of their parole? The threat has no meaning for them."

He did another experiment, the startle reflex test, in which psychopaths and non-psychopaths were invited to look at grotesque images, such as crime-scene photographs of blown-apart faces, and when they least expected it Hare would let off an incredibly loud noise in their ear. The non-psychopaths would leap with astonishment. The psychopaths would remain comparatively serene.

Hare knew that we tend to jump a lot higher when startled if we're on the edge of our seats anyway. But if we're engrossed by something, a crossword puzzle, say, and someone startles us, our leap is less pronounced. From this Hare deduced that when psychopaths see grotesque images of blown-apart faces they aren't horrified. They're absorbed.

Thrilled by his findings, Hare sent them to Science magazine.

"The editor returned them unpublished," he said. "He wrote, 'Frankly we found some of the brain wave patterns depicted in your paper very odd. Those EEGs couldn't have come from real people.'"

Then, disastrously for Hare, electric shocks were outlawed in the early 70s. He was forced to change tack. How could psychopaths be rooted out in a more hands-off way? In 1975 he organised a conference on the subject, so experts could pool their observations on the minutiae of psychopaths' behaviour, the verbal and non-verbal tics. Were there patterns? Did they involuntarily use giveaway turns of phrase? Their conclusions became the basis for his now-famous 20-point Hare PCL-R . Which was this:

Item 1 Glibness/superficial charm

Item 2 Grandiose sense of self-worth

Item 3 Need for stimulation/proneness to boredom

Item 4 Pathological lying

Item 5 Cunning/manipulative

Item 6 Lack of remorse or guilt

Item 7 Shallow affect

Item 8 Callous/lack of empathy

Item 9 Parasitic lifestyle

Item 10 Poor behavioural controls

Item 11 Promiscuous sexual behaviour

Item 12 Early behaviour problems

Item 13 Lack of realistic long-term goals

Item 14 Impulsivity

Item 15 Irresponsibility

Item 16 Failure to accept responsibility for own actions

Item 17 Many short-term marital relationships

Item 18 Juvenile delinquency

Item 19 Revocation of conditional release

Item 20 Criminal versatility

Hare said if he were to score himself either 0, 1 or 2 on each item of his checklist, he'd probably get a four or a five out of the possible 40. Tony in Broadmoor had told me that on the three occasions they scored him, he got around a 29 or a 30.

Over the three-day course in Wales, my scepticism drained away entirely and I became a Hare devotee. I think the other sceptics felt the same. He was very convincing. I was attaining a new power, like a secret weapon. I felt like a different person, a hardliner, not confused or out of my depth as I had been when I'd been hanging around with Tony in Broadmoor. Instead, I was contemptuous of those naive people who allowed themselves to be taken in by slick-tongued psychopaths.

My mind drifted to what I could do with my new powers. If I'm being honest, it didn't cross my mind to become some kind of great crime fighter, philanthropically dedicated to making society a safer place. Instead, I made a mental list of all the people who over the years had crossed me and wondered which of them I might be able to expose as having psychopathic character traits. Top of the list was AA Gill, who had always been very rude about my television documentaries and had written a restaurant column in which he admitted to killing a baboon on safari.

"Item 8 Callous/lack of empathy," I thought, and smiled to myself.

After the conference, though, Hare seemed introspective. He said, almost to himself, "I shouldn't have done my research just in prisons. I should have spent some time inside the Stock Exchange as well."

"But surely stock-market psychopaths can't be as bad as serial-killer psychopaths," I said.

"Serial killers ruin families," shrugged Hare. "Corporate and political and religious psychopaths ruin economies. They ruin societies."

It wasn't only Hare who believed that a disproportionate number of psychopaths can be found in high places. Over the following months, I spoke to scores of psychologists who all said the same. Everyone in the field seemed to regard psychopaths in this same way: inhuman, relentlessly wicked forces, whirlwinds of malevolence, forever harming society but impossible to identify unless you're trained in the subtle art of spotting them, as I now was.

I met an American CEO, Al Dunlap, formerly of the Sunbeam Corporation, who redefined a great many of the psychopath traits to me as "business positives": Grandiose sense of self-worth? "You've got to believe in yourself." (As he told me this, he was standing underneath a giant oil painting of himself.) Cunning/manipulative? "That's leadership."

But I became incredibly disappointed whenever Dunlap said things to me that were reasonable. There had been – he swore – no early behavioural problems or juvenile delinquency: "I was a focused, serious kid. In school I was always trying to achieve." And he had a loyal wife of 41 years. There were no rumours of affairs. This would score him a zero on items 17 and 11: many short-term marital relationships, and promiscuous sexual behaviour.

Becoming a psychopath-spotter had turned me power-crazed and a bit psychopathic. I was starting to see the checklist as an intoxicating weapon that was capable of inflicting terrible damage if placed in the wrong hands. And I was beginning to suspect that my hands might be the wrong hands.

I met up with Hare again. "It's quite a power you bestow upon people," I said. "What if you've created armies of people who spot psychopaths where there are none, witchfinder generals of the psychopath-spotting world?"

There was a silence.

"I do worry about the checklist being misused," Hare said.

"Who misuses it?" I asked.

"Over here, you have your DSPD programme," he said.

"That's where my friend Tony is," I said. "The DSPD unit at Broadmoor."

Two years had passed since I'd first met Tony in Broadmoor. I hadn't heard from him in months, and then out of the blue he called.

"Jon!" he said. He sounded excited. "There's going to be a tribunal. I want you to come. As my guest."

"Ah," I said, trying to sound pleased for him. Tony was forever pushing for tribunals, year after year, for the 14 years he had been inside Broadmoor's DSPD unit. His optimism was tireless. But the outcome was always the same. They'd come to nothing.

Journalists hardly ever made it to a DSPD unit and I was curious to see inside. According to Maden, the chief clinician at Tony's unit, it wouldn't exist without Hare's psychopath check-list. Tony was there because he had scored high on it, as had all 300 DSPD patients. The official line was that these were places to treat psychopaths with a view to one day sending them back out into the world. But the widespread theory was the whole thing was in fact a scheme to keep psychopaths locked up for life.

The unit was a clean, bland, modern, calmingly pine-coloured fortress. Nurses and security guards came over to ask me who I was. I said I was a friend of Tony's.

"Oh, Tony," one nurse said. "I know Tony."

"What do you think of Tony?" I asked him.

"I do have strong thoughts about Tony," he said, "but it would not be appropriate for me to tell you what they are."

"Are your thoughts about Tony strongly positive or strongly negative?" I asked.

He looked at me as if to say, "I am not telling you."

And then it was time. We entered the tribunal room.

The hearing lasted all of five minutes, one of which involved the magistrates telling me that if I reported the details of what happened inside the room, I would be imprisoned. So I won't. But the upshot – Tony was to be free.

He looked as if he'd been hit by a bus. In the corridor his barrister congratulated him. The process would take three months – either to find him a bed for a transitional period in a medium-secure unit, or to get him straight out on to the street – but there was no doubt. He smiled, hobbled over to me, and handed me a sheaf of papers.

They were independent reports, written for the tribunal by various psychiatrists who'd been invited to assess him. They told me things I didn't know about Tony: how his mother had been an alcoholic and used regularly to beat him up and kick him out of the house; how most of her boyfriends were drug addicts and criminals; how he was expelled from school for threatening his dinner lady with a knife; how he was sent to special schools but ran away because he missed his mother.

I wondered if sometimes the difference between a psychopath in Broadmoor and a psychopath on Wall Street was the luck of being born into a stable, rich family.

I spotted Professor Maden. I thought he might seem disappointed, but in fact he looked delighted. I wandered over.

"Ever since I went on a Bob Hare course, I've believed that psychopaths are monsters," I said. "They're just psychopaths – it's what defines them, it's what they are." I paused. "But isn't Tony kind of a semi-psychopath? A grey area? Doesn't his story prove that people in the middle shouldn't necessarily be defined by their maddest edges?"

"I think that's right," he replied. "Personally, I don't like the way Bob Hare talks about psychopaths almost as if they are a different species."

Tony was standing alone now, staring at the wall.

"He does have a very high level of some psychopathic traits," Maden said. "He never takes responsibility, everything is somebody else's fault. But he's not a serious, predatory offender. He can be a bully in the right circumstances, but doesn't set out to do serious harm for its own sake. I would also say you can never reduce any person to a diagnostic label. Tony has many endearing qualities when you look beyond the label."

"The thing is, Jon," Tony said as I looked up from the papers, "what you've got to realise is, everyone is a bit psychopathic. You are. I am." He paused. "Well, obviously I am," he said.

"What will you do now?" I asked.

"Maybe move to Belgium," he said. "There's this woman I fancy. But she's married. I'll have to get her divorced."

"Well, you know what they say about psychopaths," I said.

"We're manipulative!" said Tony .

• This is an edited extract from The Psychopath Test by Jon Ronson, published by Picador on 3 June.

[youtube=http://www.youtube.com/watch?v=A6aCir5bu-c]