If people have put forth energy in your direction - at work, at home, even in a casual business transaction - and all they hear back is a deafening silence, you're busting their trust. Yes, sometimes it takes time and effort to give tactful, candid feedback - but when you don't, you're not building trust.
It takes less time to say, "I hear you. I'll get back to you as soon as I can," than to repair the damage that ignoring can do to a relationship.