Britain's role in supplying information to an American military "kill list" in Afghanistan is being subjected to legal challenge amid growing international concern over targeted strikes against suspected insurgents and drug traffickers.

An Afghan man who lost five relatives in a missile strike started proceedings against the Serious Organised Crime Agency (Soca) and the Ministry of Defence demanding to know details of the UK's participation "in the compilation, review and execution of the list and what form it takes".

Legal letters sent to Soca and the MoD state the involvement of UK officials in these decisions "may give rise to criminal offences and thus be unlawful". They say Britain's contribution raises several concerns, particularly in cases where international humanitarian laws protecting civilians and non-combatants may have been broken.

"We need to know whether the rule of law is being followed and that safeguards are in place to prevent what could be clear breaches of international law," said Rosa Curling from the solicitors Leigh Day & Co. "We have a family here that is desperate to know what happened, and to ensure this kind of thing never happens again."

Targeting Taliban commanders in precision attacks has been an important part of Nato's strategy in Afghanistan, and it has involved US, British and Afghan special forces, and the use of drones.

But who is put on the "kill list" and why remains a closely guarded secret – and has become a huge concern for human rights groups. They have questioned the legality of such operations and said civilians are often killed.

Soca refused to discuss its intelligence work, but the agency and the MoD said they worked "strictly within the bounds of international law". Its role in the operation to compile a "kill list" was first explained in a report to the US Senate's committee on foreign relations.

The report described how a new task force targeting drug traffickers, insurgents and corrupt officials was being set up at Kandahar air field in southern Afghanistan. "The unit will link the US and British military with the DEA [Drug Enforcement Agency], Britain's Serious and Organised Crime Agency, and police and intelligence agencies from other countries." The 31-page report from 2009 acknowledged the precise rules of engagement were classified.

But it said two generals in Afghanistan had explained they "have been interpreted to allow them to put drug traffickers with proven links to insurgency on a kill list, called the joint integrated prioritised target list".

"The military places no restrictions on the use of force with these selected targets, which means they can be killed or captured on the battlefield," the Senate report said. "It does not, however, authorise targeted assassinations away from the battlefield. The generals said standards for getting on the list require two verifiable human sources and substantial additional evidence."

The legal challenge has been brought by an Afghan who believes his relatives were unlawfully killed in a case of mistaken identity during one "kill list" operation. A bank worker in Kabul, Habib Rahman lost two brothers, two uncles and his father-in-law in a US missile attack on their cars on 2 September 2010. They had been helping another member of the family who had been campaigning in Takhar province in northern Afghanistan in the runup to the country's parliamentary elections. In total, 10 Afghans were killed and several others injured.

Rahman says most of those who died were election workers. But the attack was praised by Nato's International Security and Assistance Force (Isaf) which said the target had been a man in the convoy called Muhammad Amin. The US accused him of being a Taliban commander and member of the Islamic Movement of Uzbekistan, and said the people who had been travelling with him had been insurgents.

A detailed study of the incident by the research group Afghanistan Analysts Network contradicted the official account, saying Isaf had killed Zabet Amanullah. Amin was tracked down after the incident and is still alive, said the study's author, Kate Clark. "Even now, there does not seem to be any acknowledgment within the military that they may have got the wrong man," she said. "It is really very bizarre. They think Amin and Amanullah are one and the same."

Rahman's lawyers acknowledge they do not know whether information provided by Britain contributed to this attack, but hope the legal challenge will force officials to be more open about the British contribution to the "kill list".

The letters to Soca's director general, Trevor Pearce, and the defence secretary, Philip Hammond, point to the Geneva conventions, which say that persons taking no active part in hostilities are protected from "violence to life and person, in particular murder of all kinds".

They also draw on the International Committee of the Red Cross, which has said anyone accompanying an organised group who is not directly involved in hostilities "remains civilian assuming support functions".

The legal letters, the first step towards seeking judicial review, say "drug traffickers who merely support the insurgency financially could not legitimately be included in the list" under these principles. The lawyers believe that, even if Isaf had targeted the right man, it may have been unlawful for others to have been killed in the missile strike.

"The general practice of international forces in Afghanistan and the experience of our client suggest that proximity to a listed target is, on its own, sufficient for an individual to be considered a legitimate target for attack. Such a policy would be unlawful under the international humanitarian law principles," they say.

Curling said: "Ensuring the UK government and its agencies are operating within their legal obligations could not be more important. Our client's case suggests the establishment and maintenance of the 'killing list' is not in line with the UK's duties under international humanitarian law. Our client lost five of his relatives in an attack by the international military forces as a result of this list. It is important that the Ministry of Defence and Soca provide us with the reassurances sought."

Soca said: "Soca does not discuss intelligence. Soca works strictly within the bounds of international law.

"Our activity overseas is conducted in line with other UK government departments, which comply with the principles of international humanitarian law and human rights."

The MoD said: "As part of Isaf, UK forces operate alongside numerous partner nations in the fight against an insurgency that seeks to maim and kill both innocent Afghan civilians and allied forces alike. We continue to work towards a stable Afghanistan that can effectively manage its own security by the time our combat operations cease at the end of 2014.

"In doing so, UK forces operate strictly within the bounds of international law under rules of engagement which, for reasons of operational security, we do not discuss in detail."

Britain Faces Legal Challenge Over Secret US 'Kill List' In Afghanistan