He dislikes the fact that you give your past too much leeway in the here and now. From his point of view, past hurts and fears prevent a harmonious relationship between the two of you.
For more detailed information on topics like love, career and more, check out The Death.