This week, he did something about it.
The McCaskey grad asked for and was granted a release from his athletic scholarship and is leaving the Nittany Lions to transfer to Division I-AA Delaware.
I tried to fit in at Penn State, but it didnt work out, Holler said. All I can say it that Im excited about going to the University of Delaware to try and help the Blue Hens.
Rumors of Hollers move started popping up on Penn State fan Web sites as early as a week ago.
Penn State granted his release on Tuesday, and hes set to report to Delaware on July 6.
They literally just finished the paperwork, Holler said. I havent even talked to the coaches at Delaware yet.
But Holler said he knew the coaches there were expecting him.
The former Pennsylvania Lineman of the Year and Big 33 participant recently met with Paterno to discuss his future with the team.
At that meeting, Holler, who steadfastly denied rumors of a possible transfer earlier this spring, informed Penn States coaching staff that he was packing his bags and heading south.
He never played a down in a regular-season game for the Nittany Lions.
I have no regrets at all as far as Penn State goes, Holler said. Its a wonderful place and I really did have a wonderful time.
It just didnt work out. Sometimes that happens.
Holler had a topsy-turvy three years in Happy Valley. He constantly battled a weight issue an issue that perturbed Paterno.
At a press conference prior to the annual Blue-White Game on April 23, the veteran coach openly criticized the 6-5, 324-pound offensive lineman for not getting in shape.
Joel is OK until he gets home and has his moms home cooking, Paterno said. Hes too heavy. Hes 325 (324 according to Penn States spring practice media guide) and we want him to be 305.
Weve been on Holler for two years Get your weight down! Hes lost 10 pounds.
The former McCaskey star has lost more weight than that, actually.
Before the Big 33 Classic in 2002, he tipped the scales at 375, so hes lost at least 50 pounds since high school. Still, hes nowhere to be found on Penn States depth chart.
He saw limited time in the Blue-White Game which was shortened by rain and he truly was a player without a position.
Holler didnt even make the three-deep list for the annual spring scrimmage.
Hes bounced from guard to tackle since arriving at Penn State, where Holler landed after a standout, well-decorated career at McCaskey.
He was a two-time Lancaster-Lebanon League Lineman of the Year and he was one of the most highly recruited offensive lineman in the country when he started making recruiting trips in 2001-02.
It has been a struggle, Holler admitted after the Blue-White Game. I got down to 330 last year, and Im at 320 right now. I think anybody in my position would be frustrated not with anybody else, but with themselves.
I need to get where I need to be, so Im trying to buckle down and get where I need to be.
Where Holler will be this fall is at Delaware, a I-AA powerhouse that is one year removed from a national championship season.
Another player with L-L League roots, former Manheim Central star Mike Byrne, is expected to be a key contributor on Delawares offensive line in the fall.
Things started going south for Holler in State College when he reportedly flunked a wind-sprint drill during summer workouts in 2004.
He fell from second to fourth on the depth chart last fall when Penn State stumbled through a 4-7 season and at one point Paterno toyed with leaving Holler off the 105-man roster.
Being lighter will make me a better football player I cant argue with that, Holler said after the Blue-White Game. Id love to be 10 or 15 pounds lighter. Im trying. Its just taking a while.
Its taking too long for Penn States liking, apparently. Thats why Holler is shoving off to Delaware, where, according to NCAA rules, hell be able to play right away as a sophomore eligible.
Joel Holler is a good, tough kid, Paterno said before the Blue-White Game. But if hes over 305, he cant play, because hes not that good of an athlete. I think he knows that, but he needs the discipline.
If he doesnt get his weight down, people are going to ask me what happened to Holler.
Hes headed out of State College and is on the road to Delaware, thats what.