Fellow speaker,
When a person makes a good point or wants to have a complex discussion during your speech, let the person know you are interested in what they have to say but don't have time during the speech to answer them. Tell them you will follow up with them on their point after your speech.
Afterwards,
Tim