Actually the seed, bark and leaves are poisonous as well.
When I was a kid, old timers told me that one side of a buckeye was poisonous and the other side wasn't.  And that squirrels knew which side to eat and which one to leave alone.  They actually believed that and I did too for a long time.
It is true that squirrels will cut on them some.  I've read that they are the only animal that has immunity.  If cows eat them it's bye bye.