Well, theoretically and legally its not racist, when used in its general meaning. Though people like that are the ones that don't understand its real meaning, because they're too incompetent to allow their vocabulary to go that far with the word
People like that annoy me, I mean to each his own, but he's just a moron