The International Cricket Council rejected England's request to switch their World Cup match against Zimbabwe and awarded the points to the home team on Saturday.
The tournament's technical committee rejected England's call to reschedule the Group A game from Harare to South Africa, ruling that the venue was safe despite the team's security concerns.
ICC chief executive Malcolm Speed said: "It's a decision that gives the committee no pleasure."
Saturday's decision should end months of wrangling involving politicians, administrators and players.
England, however, still have yet another right of appeal which could prolong a saga which has blighted the World Cup.
Nasser Hussain's team have yet to play a game in the tournament, spending more time discussing whether to play in Zimbabwe than training.
Many of their opponents have already played twice.
The loss of the four points, from a game England expected to win, means the team face a huge task to reach the second round of the tournament.
This was England's second attempt to have the game moved.
The same committee threw out England's first request, a decision ratified by an ICC-appointed judge, last week in a supposedly binding decision.
But the England and Wales Cricket Board (ECB) continued to argue and eventually pulled out of the match, originally scheduled for February 13, saying they had received new information regarding death threats sent to the team and their families.
That prompted the ICC to grant them a second hearing.
Calls for a boycott had begun among British and Australian politicians, who said it would be inappropriate to play in Zimbabwe.
They accuse President Robert Mugabe of rigging his re-election in 2002 and compounding a food crisis by seizing white-owned farms to give to blacks.
Half of the country's 14 million people face food shortages while the leader of the main opposition party is facing a possible death sentence after being accused of plotting to kill Mugabe.
The World Cup could yet see a match boycott.
New Zealand are refusing to play a Group B against Kenya in Nairobi scheduled for February 21, also because of security worries after a November bombing killed 16 people in Mombasa.
