Open ~/Library/Preferences/com.apple.Bluetooth.plist (using Property List Editor) and under Root go to RecentDevices. Mark the ones you want to delete and press the delete button.
Mark that the plist only have the BD_ADDR of the devices, on the form 00-0a-d9-2f-ef-17, if you haven't added them to favorites then you have to guess the friendly name (or check the date which either shows last used or added, not sure).