I agree with you it wasn't love at first sight. I think you found the perfect definition when you said Brian had a "soft spot". There is indeed something that makes Brian let Justin come around and that draws him to Justin, but it's not love, not in the beginning at least. I don't think Brian thinks about it as love untill season 2. When Debbie asks him "you love him, don't you", I think that's around the time Brian starts thinking he's in love. I believe he was in love before, but I don't think Brian saw it as that for a long while.
The first night it was definitely not love, but I think it was something more than lust. I think there was something that made Brian be more patient with Justin that he usually is with tricks. Had Justin been any other trick, he would have kicked him out with no second thoughts when he was called by Melanie. Hell, probably with another trick he wouldn't have had the patience to go through all the rambling Justin did before they got things started and even after
But Justin was young, sweet, naive and a virgin and Brian immediately developed that soft spot. Also when Michael complains about Justin being at Woody's, Brian says "he's kinda sweet".
I also agree about the relationship being a two way street when it comes to teach things to each other. I think Brian had a big role in helping Justin become "the best homosexual he could possibly be", but there is no doubt that Justin taught Brian about love and trust and also about expecting and demanding something better from himself (think S3, the battle against Stockwell). They both gave and took in the relationship and that's what makes it so beautiful.