I think it depends on your relationship with the person.

Generally I think it's more polite to address the person, but if you're just firing off an observation to a close friend or coworker, I don't think it's necessary.