He was even willing to take a polygraph test, he told officials.
Fowler, 30, of San Bernardino, is suspected of starting the 2003 Old Fire that burned more than 91,000 acres and destroyed more than 1,000 structures after starting on Oct. 25, 2003. The fire cost nearly $38million.
The defendant has been charged with five counts of murder and special circumstances, which allow prosecutors to pursue the death penalty if he is convicted. Prosecutors say five residents in burn areas died from heart attacks caused by the stress of evacuation and threats to homes and belongings.
Fowler was strapped to the polygraph in February 2004 and asked a series of questions. When it was over, Fowler was told he failed the exam - specifically the part where the defendant said he was not present at the time the fire was started.
"On the questions - were you present when the fire was started? - When you said `no,' that's not true. That's the part you failed, " said Robert Heard, the polygrapher, in a recording.
"I wasn't there," Fowler responded.
The defendant and the polygrapher went back and forth about what was true.
Fowler maintained for a while he wasn't there, but later changed his statement in a conversation with Heard, according to recorded audio played Wednesday at the San Bernardino Superior Court trial.
"There is no doubt in my mind that you were there when that fire started. Now, your involvement, I don't know," Heard said.
Fowler continued to deny being there.
"I know you were there. You know you were there," Heard said, adding that law enforcement had an eyewitness placing Fowler at the Old Fire.
"I did not start the fire," Fowler said on the recording. "I refuse to say I did because that would be lying on myself."
But in his next statement, Fowler gave in a little bit.
"I had no knowledge of it being started," Fowler said. "Was I there when it started? I don't know. Someone started it when I was on the hill."
After more back and forth, Fowler rephrased his statement.
"OK, yeah, maybe I was there maybe I was in the van. Maybe someone threw a cigarette," Fowler said. "I know I didn't start the fire."
He eventually told Heard he was sitting up in the van, talking on a cell phone, when the fire started. Fowler also told the polygrapher two other people were in in the van.
But he refused to say who started the fire or how it was started because that would incriminate him.
"You guys have to figure that out - you're the detectives," Fowler said to a sheriff's detective in the room.
Testimony in the trial is expected to continue today.