Subclass NSDocument. Give it a read-only property that returns the custom "only-one-per-NSDocument" class. In the getter for the property, only make an instance of the custom class when the ivar supporting it is null. Manage ownership to match the NSDocument instance lifetime.
http://developer.apple.com/library/...ptual/Documents/Tasks/SubclassNSDocument.html
If you can't subclass NSDocument, then it's harder. You have to correlate each NSDocument instance with a single instance of your custom class. An NSMutableDictionary can do the correlation, but you won't be able to use an NSDocument instance as the key, because a copy will be stored as the key. You can use the object-address of the NSDocument to make an NSNumber, and use that as the key. Then you're basically using the object's pointer as a key.
There's probably an NSSomethingOrOther collection class that does this mapping between any-pointer-as-key and any-other-pointer-as-value, but I'm not recalling what the class name is. It would behoove you to read the Collections Programming Guide. It would behoove me to do the same, but I have less vested interest in doing so right now.