They can't do anything with your IP, except do a WHOIS lookup which would provide details of which ISP you use and a location (we're talking at a regional level of accuracy, not your address or anything personal).
This is assuming you're not serving files/vnc on open ports...