Ultimately, if I am OK with just leaving my answer at "No," and I don't experience a defensive feeling, then that's fine too. If I am feeling defensive, then there is something there to be further investigated--some fear/anger based emotion that could use healing by being brought to light and I owe myself that--I like being able to contribute to my own healing/consciousness.
If I feel absolutely at peace with just saying, "No," and the other party appears to have a problem with my response, then that's OK too. The other person is entitled to feel how s/he chooses to feel ;)