It should be, "It's hard not to feel bad for him."
"Bad" is not an adverb describing the verb. It is an adjective.
The verb "feel" is a linking verb. It links the adjective that follows to the beginning of the sentence.
To say, "I feel badly," is like saying, I am badly."
Sense verbs like "feel" and the verb "to be" are linking verbs. The word that comes after them must be an adjective or a noun, not an adverb.