Yup! It's been debated for awhile between me and some others. We all had agreed that he was between 50 and 55. After watching the episode again though it was like "DUH!"
Oh also, Gibbs being 53/54 years old means that he has another 6-7 years of field work before he would be forced to go to desk duty. NCIS (the real one) takes their agents off of field work at the age of 60.