It's standard practice in science fiction to go beyond the limits of currently known science, often to the extent of violating presently known scientific principles, and that doesn't bother me. Faster than light travel, hand weapons with gigajoule effects, and sensible human governments don't bother me, and even excessively humanlike aliens and intelligent computers don't bother me that much. I can overlook really egregiously bad science for long enough to read the story if it's otherwise a good story. But time travel -- in the standard sense of visiting your own past -- is like fingernails on a chalkboard. And that probably seems strange to a lot of people who enjoy SF.
Ultimately, the reason I hate time travel stories is that the notion of time travel conflicts with my core beliefs. Now, I don't have a whole lot of core beliefs; for the most part, I'm still working out what I really believe about the world. I'm still on the fence on that is-there-a-god thing, for instance. But I have two fundamental assumptions that I've based the rest of my thinking on.
The first assumption is the fundamental assumption of science: it is possible for us to understand the universe. Ultimately, there is no reason why this has to be true; it might turn out that what what makes the universe tick really is fundamentally beyond the human mind. But if that's the case, it's quixotic at best to try anyway, so I choose to assume that it is possible to understand the universe.
The second assumption is that we truly have free will. I make this assumption, again, not because there is any clear reason why it must be true, but because to assume otherwise is to make the whole exercise a farce. If I believe that no amount of hard work, effort, or strife on my part weill actually affect what happens to me, I will not have the motivation to put out the hard work and endure the strife. So I make this assumption out of my own personal philosophical necessity.
Any time travel story that centers on visiting the past and seeing the results has to conflict with one of these assumptions. Actually changing the past violates the principle of cause and effect and creates a paradox. Some people in modern physics, probably smarter than me, will say that cause and effect may not be a valid principle, but I know that I can't understand a universe where it doesn't apply. And a true mathematical paradox always means that one of the assumptions by which you arrive at the paradox is not valid. I go back into the past and kill my father, so I was never born to go back into the past, so my father sires me after all, so I kill him after all. Paradox. Logic's way of telling you you made an incorrect assumption, and in this case, the incorrect assumption was that I could go into the past to kill my father.
There's another (much less common) variety of time travel story, where you go into the past intending to change something, only to end up creating the conditions that actually happened. Robert A. Heinlein's "All You Zombies" is the classic example. There's no paradox here, no violation of causality, so it passes the first assumption, but only by making time immutable. The hero of "All You Zombies" (whose name I completely forget in the 20+ years since I've read it) is predestined at birth to become his own mother and father, and can't change it because it's already happened. But this only holds together logically if Einstein is right, time is a dimension just like the spatial ones, and the four dimensional object that is reality through all time already exists and can't actually change. Which I can't accept because it makes life pointless.
Ultimately, fiction of any kind must create a situation I can believe at some level, or it fails. Science fiction in particular is about creating situations outside of our real world experience, putting people in them, and seeing what happens. When a story is based on something that is flat out not possible according to the very foundation of my belief about possible things, it is extremely difficult to get past that problem and consider the actual story happening to the characters. I have read time travel stories that were good enough that I enjoyed them in spite of the time travel, but I've never failed to be bothered by time travel in a story since I thought about what I've written here. And I find that the more often I encounter time travel in a story, the more it bothers me.