I have no doubt that harm to one's (for lack of a better term) karma (please don't commence with the "you don't know what karma really is" lecture, because I do know) lies within taking pleasure in someone else's pain, and being bitter that someone else is joyful when you are not; in other words, being mean-spirited. It's amazing how easily we humans slip into such behaviour - I have done so myself in my misspent past, and it is not something to be prideful of, though many are.