You'll be fine. Just use some common sense and wipe it down once in a while. I noticed for my iphone 4 bumper, it held debris in there but didn't cause major ugly scratches. The marks may from the plastic bumper itself. So for the incase book jacket, the inner is soft lined to prevent scratches and it doesn't have a hard plastic lining that directly rubs on your ipad.
With my iPad 1, I had the apple case and lots of debris go inside but no scratches! Again, I think it's because of the soft case/lining that is in contact with the ipad. I just took it out of the case and with a microfiber cloth, dry wiped it down and if I was really worried, I used the air spray can for computers inside the case. You must also be conscious of how you're handling your stuff and if you do things calmly and think before you throw your ipad on the couch or table, then you'll be at a less risk of causing any type of damage.
I used to have the BSE on my iphone 3GS and that collected lint. The ipad may be different... My ipad 1 was naked except for the apple case and I plan to keep my ipad 2 naked without a skin as well, except for the smart cover and this book jacket.