That depends on what other accessories you want to have on you. Anything goes with black, so you have to look beyond that and consider the colors of the shoes and wallets and other accessories you use or wear most. When I'm on the go in jeans and t-shirt and running shoes, midnight blue would suit best. When I'm wearing my hiking shoes and khakis and my brown leather jacket or green or khaki hoodie, I'd go with brown. Black does go with anything. So that's a good pick for casual, formal...just general versatility.
Edit to add that I do have an iPad mini retina with black bezel.