Why is Andrew Jackson so unfairly hated?

Andrew Jackson has been unfairly reviled by modern day people, because his accomplishments has always been more than whatever shortcomings were in his character.
He was a war hero, the only President to ever pay off the US debt, fought against the banks, founder of the Democratic party and was overall a well respected President well into the 20th century. Why do people hate him so much?

Because of one thing.

Trail of Tears – Wikipedia

Now if people actually bother reading this, look at the details: “By 1838, about 2,000 Cherokee had voluntarily relocated from Georgia to Indian Territory (present day Oklahoma). Forcible removals began in May 1838 when General Winfield Scottreceived a final order from President Martin Van Buren to relocate the remaining Cherokees”

Note: VOLUNTARILY until 1838

Jackson had made a deal with the Cherokee chiefs. Martin Van Buren is the one who forcibly relocated them.

So… why isn’t Van Buren’s grave getting shit on then? Why is Jackson getting all the hate?