I don't think you'd get expelled for going to another House unless you purposely cheated your way to get the password. Or unless you thrashed the Common Room and you're from another house. Although the other students may question what's your business there so best to ask permission.