The show was dropping a bit ratings wise (more than at any other point in the history of the show) in season 2 and even season 3, so that's why it was in trouble in getting renewed for season 4 on the CW since most of the other WB shows had higher ratings. In fact the start of season 4, the ratings were higher than at any point in season 3 or the back end of season 2 as well. The first three seasons on the CW, the show was pretty constant ratings wise since all three seasons average similar averages and even season 7 was still doing decently ratings wise despite of the loss of Leyton. Only seasons 8 and 9 dropped lower the 2 million viewers threshold.