First Damon does NOT hate Stefan. A little (okay a lot) jealous that everyone likes him better, but he loves his brother. Second, Damon doesn't really love Elena, he only thinks he loves her. I'm sure I'll get flack for that last statement, but it's true. he only flirts with her to tick his brother, but you know whatever. Third, Stefan will ALWAYS be Elena's first choice so if he were to come back and she was with Damon Elena would leave Damon so fast all of our heads would spin from the sudden blast of wind.
Don't get me wrong, I think Elena has some love for Damon and is attracted to him (I mean hello have you SEEN him) but there is no mutual respect between them and they are too destructive (with each other and the people around) to ever really work. But like you said only time will tell and who knows what twists and turns the show will take.
Team Damon (being happy with himself)and Team Ian(for saving the world and making a pair of jeans and scruff look good)!!!