Don't try to apply the grammar you have learned at school to speaking all the time.

Sometimes an infinitive (to do something) and a gerund (doing something) are interchangeable. If you try to apply this to all sentences, though, you'll make a mistake. Here is some examples from Mr. Hiroshi Ogura's book I mentioned before:

Hiroshi Ogura, a teacher at a cram school, has written some books from his experience of teaching writing with a native English teacher.

Wrong: To collect stamps is my hobby.
Right: Collecting stamps is my hobby.

Wrong: I prefer dying rather than stealing.
Right: I'd prefer to die rather than (to) steal.
Right: I prefer walking to riding.