The victim, a 20-year-old student, had cuts and bruises but declined medical treatment, police said.
The violent attack happened about 6:05 p.m. Tuesday in the 2600 block of Foothill Boulevard, not far from the Fruitvale District.
Police said the man and suspects were all riding a 40-line AC Transit bus going east on Foothill.
The suspects, all friends, got off the bus when the man did, surrounded him and began to hit him in the face, causing him to fall to the pavement, where the assault continued police said.
One of the boys took the man's phone from his pocket before they all fled on foot northbound on 26th Avenue.
The attack was witnessed by another bus passenger, who called police with a description of the suspects.
Officers soon arrested the five boys in the 2400 block of 26th Avenue and recovered the victim's phone from one of them. The victim and the witness also identified them as the attackers, police said.
Police said four of the boys admitted their involvement, including the one who had the phone. Some said the goal of the robbery was money. They were arrested on suspicion of strong-arm robbery.