The rules ofcourse, at least in stroke play I'm not sure about match play, don't insist that the player furthest away goes first at any time, on the green or not, merely that you can't seek to gain an advantage by going out of order.

10.1 Match Play
a. When Starting Play of Hole
The side that has the honour at the first teeing ground is determined by the order of the draw. In the absence of a draw, the honour should be decided by lot.
The side that wins a hole takes the honour at the next teeing ground. If a hole has been halved, the side that had the honour at the previous teeing ground retains it.
b. During Play of Hole
After both players have started play of the hole, the ball farther from the hole is played first. If the balls are equidistant from the hole or their positions relative to the hole are not determinable, the ball to be played first should be decided by lot.
Exception: Rule 30-3c (best-ball and four-ball match play).
Note: When the original ball is not to be played as it lies and the player is required to play a ball as nearly as possible at the spot from which the original ball was last played (see Rule 20-5), the order of play is determined by the spot from which the previous stroke was made. When a ball may be played from a spot other than where the previous stroke was made, the order of play is determined by the position where the original ball came to rest.
c. Playing Out of Turn
If a player plays when his opponent should have played, there is no penalty, but the opponent may immediately require the player to cancel the stroke so made and, in correct order, play a ball as nearly as possible at the spot from which the original ball was last played (see Rule 20-5).

10.2 Stroke Play
a. When Starting Play of Hole
The competitor who has the honour at the first teeing ground is determined by the order of the draw. In the absence of a draw, the honour should be decided by lot.
The competitor with the lowest score at a hole takes the honour at the next teeing ground. The competitor with the second lowest score plays next and so on. If two or more competitors have the same score at a hole, they play from the next teeing ground in the same order as at the previous teeing ground.
b. During Play of Hole
After the competitors have started play of the hole, the ball farthest from the hole is played first. If two or more balls are equidistant from the hole or their positions relative to the hole are not determinable, the ball to be played first should be decided by lot.
Exceptions: Rules 22 (ball assisting or interfering with play) and 31-5 (four-ball stroke play).
Note: When the original ball is not to be played as it lies and the player is required to play a ball as nearly as possible at the spot from which the original ball was last played (see Rule 20-5), the order of play is determined by the spot from which the previous stroke was made. When a ball may be played from a spot other than where the previous stroke was made, the order of play is determined by the position where the original ball came to rest.
c. Playing Out of Turn
If a competitor plays out of turn, there is no penalty and the ball is played as it lies. If, however, the Committee determines that competitors have agreed to play out of turn to give one of them an advantage, they are disqualified.
(Making stroke while another ball in motion after stroke from putting green - see Rule 16-1f)
(Incorrect order of play in threesomes and foursomes stroke play - see Rule 29-3)

You don't HAVE to play 'away' (there is no insistence). But it's obviously the way the game was meant to be played.

Sorry - I did miss your point.

I don't understand trying to rush around the golf course. I don't understand the complaints about a 5 hour round.

Ah maybe my gripe isn't actually with slow play at all but at not being let through on an empty course.