Aaron Zindani was murdered in the Yemeni capital of Saana on Tuesday, Army Radio reported.
Zindani – a Jew who lived in Israel for the past few years – had returned to his native Yemen.
A friend of the Zindani in Yemen said Al-Qaeda may have been involved.
“He had an argument with a man from Al-Qaeda, and for them killing someone is like killing a fly,” the friend told Army Radio.
The man said that the few dozen Jews living in Sanaa “do not feel they are in danger” despite multiple murders in recent years, and discrimination by security officials and the judiciary.
In March 2009, a Yemeni court ruled a Muslim who murdered Yemeni Jew Moshe Yaish-Nahari – a father of nine – would only pay a fine because he was "an unstable extremist."
Witnesses say Abed el-Aziz el-Abadi, a former MiG-29 pilot in Yemen's air force, walked up to Nahari on 10 December 2008 and said "Jew, accept Islam's message." He then shot Nahari five times with a Kalashnakov rifle.
Yemeni authorities moved 70 Jews from northern Yemen to a compound in Sanaa – openly admitting they could not protect them elsewhere. The transfer left them with no means to earn a living.
The continued violence against Jews in Yemen, coupled with the ongoing unrest in the country, has also led to an exodus of Jews from Yemen.
Yemen’s Jews – who numbered fewer than 400 in 2009 – has since seen its numbers drop by some 20 percent due to emigration, mostly to the United States.