Actually the seed, bark and leaves are poisonous as well.
When I was a kid, old timers told me that one side of a buckeye was poisonous and the other side wasn't. And that squirrels knew which side to eat and which one to leave alone. They actually believed that and I did too for a long time.
It is true that squirrels will cut on them some. I've read that they are the only animal that has immunity. If cows eat them it's bye bye.