"I'm expecting a backlash, there's no way they won't try and come back at me, there's a buzz around Belfast about all of this," he says nervously.
To an outsider it may seem like paranoia, but for Brown, 55, who has just written a book about his 30 years as an RUC detective in Belfast, it is a possibility.
His book, Into The Dark, goes on sale this Friday, and in it he accuses the RUC Special Branch of covering-up a series of loyalist murders, of acting with impunity to thwart investigations into those murders and ultimately of urging loyalists to kill him in the 1990s.
The allegations he makes are similar to those made against the RUC down the years – that they colluded with loyalist paramilitaries in carrying out sectarian murders, – but Brown's career as a police detective lends the claims an extra credibility.
Among the allegations he makes is that a series of murders were carried out by a particular UVF informer in Belfast in the 1990s who was protected by Special Branch.
His book also tells how a cadre of Special Branch men effectively subverted the RUC's operations against loyalists in order not only to protect their 'agents' but to guard their own personal power bases.
Brown also tells how as a rookie policeman he was beaten up by a colleague for arresting a gang of UVF men in north Belfast in the 1970s.
Although Brown stops short of accusing Special Branch of whole-scale and institutionalised collusion, his book is nevertheless a damning indictment of the RUC's conduct throughout the Troubles.
He insists that the instances of collusion he refers to in his book should be the subject of a full, public inquiry.
"I am with Pat Rabbite on that one," he says, making reference to the Labour leader's call in the Dáil last week for an independent inquiry into RUC collusion in a series of loyalist murders in north Belfast in the 1990s.
The obvious first question though is why Brown continued to serve in a police force which he ultimately believed to be corrupt.
"I wanted to play my part in combating the trouble that was going on around me and I think I did that to the best of my ability," he says.
"I didn't see that me leaving the RUC would help anyone, and I wasn't going to be intimidated out of the force by the actions of a few.
"Most of the RUC men I served with, including 90 per cent of Special Branch, were good honest law-abiding men.
"It was a band in Special Branch that was responsible for the madness, for allowing murders to go ahead without even trying to prevent them and then actually making sure we couldn't investigate them afterwards."
Among the murders Brown believes went ahead was that of Sharon McKenna, a 23 year-old Catholic woman shot dead in 1993 by a UVF man and Special Branch agent referred to in the book as X.
X is infact Mark Haddock, a former UVF commander in Belfast who is currently in jail on charges related to loyalist paramilitary activity.
It is the activities of his UVF unit, believed to have killed more than a dozen people, which are currently being investigated by the Police Ombudsman and which led to Rabbite's speech in the Dáil last week.
"The Branch men were protecting informers who were infact murderers. They were telling us in CID, 'don't investigate that murder or you're in trouble.
"Their power was frightening, a constable in Special Branch could tell a Detective in CID what to do and how to do it, they run the show."
Brown's role as a CID detective meant he crossed paths with both loyalist paramilitaries and their Special Branch handlers.
In some instances, he says, Special Branch would debrief their agents in front of him.
Jonty Brown's stomping ground in the early 1990s was north Belfast, the most dangerous part of Northern Ireland for a Catholic to live in. Both the UDA and UVF were particularly active in the area, targeting innocent Catholics in random sectarian murder bids.
Often loyalist gunmen would cruise along a Catholic street and burst into the first house with an open front door to shoot dead whoever was in the house. One of the main loyalists directing those attacks in the 1990s was Shankill Road UFF boss Johnny Adair.
It was Brown who was key in putting Adair behind bars.
"I would go up to his house and sit and talk to him, he actually thought I was his friend and he would boast to me about what he had done and what he was going to do.
"I was recording his every word without him knowing it."
It was those transcripts which provided the key evidence which led to Adair's conviction for directing terrorism in 1993.
And that was just one of the successes Brown scored against loyalists.
In 1991 Ken Barrett, a seasoned UFF gunman, confessed to Jonty Brown that he had been the triggerman in one of the most controversial killings in Northern Ireland – the murder of Pat Finucane, a human rights lawyer from Belfast.
Ultimately it was that confession which led to Brown taking early retirement from the police and in writing his book.
What should have been a crowning glory for a detective turned into his Achilles heel.
Within weeks Special Branch were pressurising Brown to suppress the confession, soon pressure turned to threats.
Special Branch, long implicated in Finucane's murder, were desperate to ensure the confession didn't act as a springboard for a fresh investigation into the murder.
Barrett, was after all, one of their agents in the UFF.
"I think that was certainly a turning point for my career, there was non-stop pressure after that.
"But I was determined not to bow down."
However by the mid-1990s Brown became aware of a number of Special Branch officers who were actively encouraging loyalists to shoot him.
The conspiracy was relayed to him by one of his own informers, who reassured him that neither the UDA nor UVF had the stomach to shoot RUC men.
Not surprisingly Brown is fearful of the Special Branch, even today in the "post-Patten reform era".
He is eager to point out that he doesn't want revenge or even to see those he labels as wrongdoers in court.
Although he refuses to comment on the PSNI, Brown says he can still detect the fingerprints of Special Branch in current policing operations.
"I know there are policemen who are still frustrated by Special Branch today.
"You have to understand the power Special Branch had to see why some of them are so reluctant to give up their positions.
"The Troubles kept these people alive, it gave them status, power they couldn't dream of, money, and they didn't want to give it up without a fight."
The allegations outlined by Brown dwarf the findings by the Morris Tribunal of the McBrearty case.
If even a fraction of them are provable Brown's book could have a serious impact on the PSNI and in particular those who wielded power in the Special Branch during the Troubles.p