Well you aren't the only one. Debate rages as to where the line originated, and whether or not popularizing it merits any credit too.
The American Film Institute seems to think it does, and they have a bit of credibility around the movie world.
They listed "Go ahead, make my day," as the sixth-greatest film quote of all time in their "100 Years... 100 Movie Quotes" list, and credited the line to Clint Eastwood's 1983 tough-guy classic "Sudden Impact," the fourth Dirty Harry film.
But interestingly enough, a year earlier a slightly different version was uttered in the hard-boiled 1982 B-movie "Vice Squad."
In that classic-of-its-kind exploitation film, the necessarily tough vice-squad cop played by Gary Swanson says to a vengeful pimp (it's that kind of movie), "Go ahead scumbag, make my day."
The plot is thickened even further though (already we have more plot than "Vice Squad" did) by the fact that the late screenwriter Dean Riesner, who wrote two of the other Dirty Harry films (there were five in total) claims he wrote the line in his draft of the third installment, 1976's "The Enforcer," but it was cut from the final version. (It's worth noting that Reisner also claimed to have penned the second-most-famous Dirty Harry line, "Do you feel lucky," for the first film, but this is also being disputed by a script doctor who worked on the screenplay after him.)
With all this disagreement swirling over who did it first, it should be noted that "... make my day" is best remembered coming from the mouth of a time-hardened, 1980s Clint Eastwood.
Not only did the AFI immortalize it under his name, but the line also made the equally venerable "Oxford Book of Quotations" in 1999.
"Sudden Impact" was the fourth film featuring Eastwood's famed "Dirty" Harry Callahan (a character nearly as famous as the Man With No Name who made him a spaghetti-western star). The franchise got rolling in 1971 with "Dirty Harry," followed in '73 by "Magnum Force," '76 by "The Enforcer," 1983 by "Sudden Impact" and finally in 1988 by "The Dead Pool."