Slavery made it impossible for small white farmers to compete with cotton plantation owners on cost. They were, for the most part, left to engage in subsistence farming. Since the Civil War (notwithstanding what you hear from some diehard Southerners) was about maintaining slavery, it would appear that there was no economic reason for these people to support the war. And yet, they did, and they died by the hundreds of thousands. Why?
There were a variety of reasons, but one of them undoubtedly was status anxiety; slavery might have cost them money, but it boosted their self-esteem, because it meant they would never be the lowest man on the totem pole, no matter how unsuccessful they might be. They were willing to go to extremes to protect that interest.
You should keep that in mind the next time you hear someone ask why white workers vote for Republicans who cut their benefits.