Your name describes my thoughts on that.

Though to be honest, it wouldn't surprise me. If you think about it, each season so far has had a different location for a "home base" for Rick's group.

Season 1: Camp on Atlanta's outskirts.
Season 2: Hershel's farm.
Season 3: Prison.

Following that, it seems pretty logical to assume that at some point in season 4, they'll have to change location again.