Well at least that problem is out of the way because Elena and Klaus made a deal, Jeremy in exchange for Rebekah. And Originals do that whole
"you have my word thing" (although they have gone against it before) so I would hope that no one is going to attack Jeremy in Denver.
It's a good thing though because if Elena invited that guy into the house, it wouldn't have lead to good things in the future.