At the left of the timeline, there are Padlock icons on each of the video/audio timelines. Select the clip you wish to delete(which also selects the audio), and lock the audio tracks you wish you keep, after it is locked, just delete the clip.
Just another option.