Take a look at the free community edition of Pydio! If you're already paying for a hosting package with enough space, traffic-limit, PHP, MySQL, SSH and preferably SSL, just install it there. Make sure that the provider doesn't forbid self-installed file-sharing services.
If configuring a NAS at home with dynamic DNS, Pydio, a bundled file-sharing or a simple ftp solution is going to make sense, will heavily depend on the speed of your internet connection and the users habits and expectations.
ownCloud for a server you own ( small number of clients < 1000 )?
I've no experience running Pydio on a NAS. I just made installs in shared hosting and development environments. As shared hosting usually has quite limited computing resources for each user, I'd expect that even the cheapest recent Synology would somehow work for average needs. For the enterprise edition of Pydio a 4 core processor is recommended.What is the bare minimum Synology Nas to run Pydio at a decent speed.
Look up ownCloud.. easy to setup in a *nix environment and might be just what you are looking for.Under 1000
For such an online order/download system why not use a dedicated online shop software? Magento for example should be able to sell downloads. Set the price to zero for the first 100 images. If there is an easy way to restrict a certain file/folder to a certain client/group you'll need to find out.I do photo scanning and would like to upload the first 100 photos of every photo scanning order online for my customers to share and download.