Or, I DO understand it and the patent system is completely out of touch with what we need from it.
It is out of touch, but I do not think you understand the patent system as it is now.
Actually, the right as you describe is limited. If I had the same idea and built it, and used it, and never sold it, you can't do anything to me.
That's incorrect. See law quote below.
So REALLY we ARE talking about the right to sell the product and make money.
Well, no. A patent is most definitely NOT about the right to make or sell a product.
You see, a patent cannot confer the right to make or sell a product, because whether the product is legal or not, is up to other laws. E.g. it might be illegal for you yourself to make a weapon that you've patented.
There's also the fact that many patents depend on other patents to work, which means you cannot produce anything using your patent unless you license those other patents. Hint: that't sometimes why patent holding companies cannot produce anything themselves. They're just one of many patent holders selling a piece of the whole puzzle.
THAT IS THE PROBLEM! FIX THAT! Yes, you created something. Congrats. That does not mean that you get to prevent society from benefitting from it.
Legally, that's exactly what a patent means. Here is the definition from 35 USC § 154:
"Every patent shall contain a short title of the invention and a grant to the patentee, his heirs or assigns, of the right to exclude others from making, using, offering for sale, or selling the invention throughout the United States or importing the invention into the United States, and, if the invention is a process, of the right to exclude others from using, offering for sale or selling throughout the United States, or importing into the United States, products made by that process, referring to the specification for the particulars thereof."
So your idea of a patent, is exactly opposite of what a patent actually is. It is NOT a right to make. It IS a right to prevent anyone else from using or selling your invention, whether you use it yourself or not, unless they license it from you. Courts have upheld this time and again.
--
This was just FYI. Continue with your ideas for improving the system. It's just better that one should start from the current legal basis.
There are many who agree with some of what you say. For example, the noted IP jurist Judge Posner once said in an Apple trial, that it would be mistaken to think that "just because someone has a patent, he has a deep moral right to exclude everyone else (from using it)". Legal right, yes. But with benefits for society? Not necessarily.
Side note: the moment that anyone tries to propose that patents are not a property right that can be sold and licensed without having to use it yourself, is when your proposal is going to fall on deaf legal ears. People need to drop this whole unworkable / unsalable idea that a patent holder must produce something themselves. It's sometimes not even feasible (see examples above).
The idea that society benefits if a patent is used, is good though. Thus my own opinion is that a non-practicing patent holder who is willing to license to practicers at a reasonable price, is far better for society than a patent holder who exercises their right to refuse to license their IP at all. Think about it.