For Windows you'll likely need to use NTFS for your backup drive. For Mac you'll definitely need HFS+ or AFPS for Time Machine.
While there is software drivers you can install that allows Windows to read and write HFS+ and Mac to read and write NTFS, you cannot use the default backup applications to backup to these drives, only third party software.
That said I have a NAS setup that my MacBook Pro and my Surface Pro both can backup to the same drive, and the drive is formatted Ext3…