Well, as P.J. O'Rourke so aptly phrased it "There's no such thing as 'we'; there's only you and me. And sometimes I'm not too sure about me." It's important to not confuse the first person plural pronoun with an actual physical entity.
"We" are not a wealthy "society". You and I live in a place that has some rich people and some poor people and some in-between people. "We" don't "just have to" do anything. There may be things you need to do or things I want to do, but we aren't part of some borg-like collective with collective responsibilities, wants and needs.
If you want to be compassionate, go be compassionate. I know that's usually what I do when I'm feeling compassionate, not expect some entity called "the government" to go be compassionate for me. Mailing a check to the government to help the poor because you're feeling compassionate is like handing the local crackhead a twenty to fetch you a pizza because you're feeling hungry.
If you think something needs to be done, you should do it. You should not assume everybody else thinks the same way or that somebody else will take care of it for you. You and I are not sticks in a fasces or cells in a jellyfish; we are individuals with the right to live our own individual lives without someone else telling us what we have to do.