Aylward had let Rickie Lee Fowler's family stay at his home, and Fowler stayed there from time to time, he said.
But in September 2003, about a month before the Old Fire, Aylward said he did not want Fowler in his life anymore.
"At that point in time, Rickie was using a lot of drugs ... he was a very angry person at the time," Aylward testified during Fowler's trial in San Bernardino Superior Court.
Aylward gave the defendant a ride to his aunt's Upland home - to get him out of the mountain area.
The witness testified that he told Fowler he was not welcome at his Forest Lane home anymore.
"He didn't take it well, but he left," Aylward said.
Prosecutors say Fowler started the fire with the intention of burning Aylward's home.
Supervising Deputy District Attorney Robert Bulloch said Fowler was angry, selfish and seeking revenge.
The Old Fire, which ignited on Oct. 25, 2003, scorched more than 91,000 acres, destroyed more than 1,000 structures - including Aylward's home - and cost nearly $38 million, officials said.
Fowler, now 30, of San Bernardino, has been charged with five counts of murder and special circumstances, which allow prosecutors to pursue the death penalty if he is convicted.
Fowler liked notoriety, Aylward testified.
He "liked to be the center of attention. The more attention, the better," the witness said.
"If he had started the fire, do you think he would have told everyone?" defense attorney Donald Jordan said.
"Yes," Aylward responded.
"If he didn't start the fire, would he have still told people?" Jordan asked.
"It's possible," Aylward said.
Witness testimony will continue on Monday.