That thing you saw in Bosnia was piano, guitar, and bass, right? I can't remember... I haven't done many gigs with that lineup, I've done far more with piano, bass, and a horn player (trumpet, sax, etc.). I've also done just a few with just piano and bass, though I hope to do more of those, they're always fun! And of course I do MANY, MANY, duo acts with a vocalist or horn player, and quartet acts with a bass player, drummer, and horn player.
I sort of wish there was a modern equivalent to the theorbo (is there??), that'd make for some interesting lineups in jazz groups... Best you can get currently is the guitar, which still only has six strings that don't go very low.
The double bass (rather than a cello) that night in central Bosnia, with three (excellent) musicians, really blew me away; it was brilliant, and I hadn't been expecting a double bass in such an ensemble; I was dining with some colleagues in a small (but packed) restaurant the night before the election, and the musicians (I imagine it was some manner of residency) showed up and played for around an hour, hour and a half, with a break of twenty minutes in the middle of their set, which was that nexus of seemingly effortless jazz, modern Balkan trad, and the 'chanson' (but instrumental) music you used to find in French, (and Italian) and old style Serb and Croatian (50s, 60s, 70s), café music.
Piano and bass is a classic pairing, for a very good reason; you get the sound, texture, tone, and (above all) tune, not forgetting depth, rhythm and bass all wrapped up in two instruments.
The Baroque equivalent pairing would have been Baroque lute and theorbo, (as keyboards seem to have been less portable).
No reason you can't play a theorbo in jazz, unless you want to insist on historical - or traditional - verisimilitude at all times when playing, and jamming, and enjoying and composing music, which would be something of a shame; it is an amazing instrument, - the bass is extraordinary - and I'd love to see (well, hear) how it would (could) work in - released from the (wonderful) confines of the Baroque era.
Now, obviously, it works brilliantly with Baroque music; this was its era, and the music of the time was written with a deep knowledge of the instrument and an awareness of how best to play to its strengths; however, I'd imagine - at the very least - that it could work well with some modern 'trad' inspired music (some Irish trad, or stuff such as Steeleye Span, for example), as such music salutes (and recognises) Renaissance and Baroque ancestry.
Having said that, if modern instruments can be used - along with some with a more venerable ancestry - to inform the music of much modern trad, why not see (hear, listen to) what a theorbo (or archlute, or lute) sounds like when reimagined in a different musical context, sound and setting?
On this (or a closely related) topic, the wonderful (I love his stuff) American artist (and specialist in early music plucked instruments) Brandon Acker made a terrific (short) video with the title "Doth My Lute Hath The Courage To Shred?" and decided to demonstrate that it is indeed possible to 'shred' on the Baroque lute; I have always loved the rosette soundhole of the Baroque lute, it is a gorgeous instrument.