Never heard of the app (looks interesting though, but I already have Toast). Are you going to the proper section of the app? I states it can do DVD videos, so you should be heading to the VIDEO section as opposed to the DATA section. It says it also does Divx and VideoCD's as well, so there should be a specific sub-option if you aren't doing it as such.
If it were burned as a DVD video, then by default when it is inserted into your machine, Apple's DVD Player should auto launch and begin playing it.
When you look in the contents of the disk, is there a "Video_TS" and maybe an "Audio_TS" folder as well (the Video is the important one)? That's what it should be, so if it isn't then it might not be converted/burned properly. If you see the original movie file (.mpg or whatever) in the disk by itself then you aren't choosing the correct option to burn.
The app page says it will do the converting for you, so if it formatted the .mkv file then it should be able to convert just about any popular format to the proper DVD format.