In our area it is the oak tree...boom acorn crop means larger racks the next year and heavier deer...when we have poor to no acorn crop we have racks that didn't get to where they could have been and poorer deer...this is all in a big woods type setting...we are looking to have a great acorn crop this fall...
One forest crop that has been sorely missing from much of the Eastern US is the American chestnut tree. If and when the USDA and FDA approve the GMO version for release to the public, I can't imagine a better permaculture choice than to plant a goodly number on any suitable property. Even if you sacrifice food plot acreage or have to clear other forest trees to accomplish it.
Mainly white oak with a few reds here and there. Lots of post oaks throughout loaded with acorns as well. Hickory is very prolific and I am declaring war on it...does nothing for deer and shades everything out...Do you have a lot of diversity in your forest, or is it mainly one or the other; red or white oaks?
It sounds like a nutrition or feeding program would work well in a setting like yours. If you can see a difference from yr to yr based on acorn crops imagine what those deer would do if they never had a bad yr. We have great plant diversity and soil fertility, I think our bad yrs can be traced to dry conditions. Bucks in general seem to be smaller after a yr or two of very low rainfall.