You're clever to wait until everything is maxed before upgrading. You get half as much loot at higher town hall levels (although you can go up in trophies easier and find good payouts still). A lot of people forget to do their walls.. I wouldn't upgrade town halls below 7 unless your walls are maxed out... once you hit TH8, purple walls are probably good enough for most situations.
It has been a long slow slog to do the upgrades, but will be worth it.