My external Samsung BD/DVD drive works fine with my rMBP when converting a DVD. (Although, it did prompt me to load a version of "libdvdcss" since it can no longer just get it from an installed version of VLC.) With a Blu-Ray disc, I get the same error you did: "no valid source found".
My solution is to first rip the disc with MakeMKV. You need a lot of temporary disc space to dump the raw BluRay movie, but I prefer this method. It's great when you have multiple discs to convert. You can queue up the multiple conversions in HandBrake as soon as the MakeMKV rips complete. It also keeps the drive streaming at full speed while ripping, rather than doing a lot of stopping and starting in order to feed HandBrake at the pace it can handle.