The independent candidate, a 51-year-old physician and prominent human rights and democracy campaigner, was forced by three Israeli security officials into a police car outside the Lion’s Gate entrance to the walled Old City.
He flashed a “V” sign with his fingers before he entered the car.
It was the second time Mr Barghouti had been detained by Jerusalem police during the campaign. He was detained in the Old City last month, but released hours later.
Israel has promised to allow free and fair Palestinian elections on Sunday and agreed to limited campaigning in the disputed city. But it insists that only Palestinians with proper permits be allowed into the disputed city of Jerusalem.
Mr Barghouti lives in the West Bank.
More than 5,000 Palestinians will be permitted to vote in east Jerusalem, but tens of thousands more will have to travel to outlying suburbs to vote.
The leading contender in Sunday’s vote, Mahmoud Abbas, had cancelled tentative plans to visit the Old City in a campaign stop, fearing a large Israeli security presence could embarrass him among voters.
But addressing a crowd of around 2,000 supporters in the West Bank Jerusalem suburb of Bir Babala, he said: “Al-Quds (east Jerusalem) is ours. We are committed to the creation of a Palestinian state with Jerusalem as its capital.”
Mr Abbas later wrapped up his campaign with a press conference in which he said he intended to retain the current prime minister Ahmed Qorei as the head of his government, if elected.
As Palestinians prepared to vote, violence continued in the occupied territories, with two Palestinians shot dead by the Israeli army in the Gaza Strip, while an Israeli was killed and three others wounded in a Palestinian shooting attack in the West Bank.
Polls have put Abbas at least 30 points ahead of his nearest challenger Barghouti.
On Saturday, the Israeli army is due to start easing its clampdown on the Palestinian territories to give voters more freedom to travel on election day.