I have gone through this as well. I try to focus most of my habitat improvements between Xmas and May and then use a low impact approach between Summer and hunting season. This approach has been turning things around on our property. Deer sightings are going back up with this approach and only hunting the fringes. I have seen several times how heavy pressure reduces deer activity during daylight hours.
Yep. Best time.