You're stopping short of the logical extension of this line of thinking.
We've made the costs of labor in the US extremely high due to all kinds of regulations. Minimum wages, health care, employee and employer level taxes, etc. We've done this largely because we think everyone should have a "living wage", "access to health care" and pay for things like safety nets (social security, unemployment insurance, etc). More than that, we've imposed regulations on how clean doing business needs to be, from everything to costs of energy to the production of the product. So, what we've done is taken jobs away from people who's labor is not "worth" paying for all those things and moved the job to a place that has decided not to care about those things.
This is one of the great hypocrisies in the modern liberal mindset. This idea that we can both have and should support globalization and free trade, but that we should also have this free trade with countries that do not meet our standards. So, instead of having livable jobs here in the US, we're supporting sending jobs overseas where companies need nets on their buildings and/or the air quality is horrifying (which damages the planet in ways we eventually see too).
This is why free trade is BS and the combination of the factors above is leading to the increasing disparity between the haves and the have-nots. If you're relatively rich, you can, among other things, take advantage of "free trade" to sell cheap stuff on the back of non-US labor, but then if you're not rich and lower down the education scale, you have trouble finding jobs because your labor has been priced out of the country.
To help remedy these problems, we should be imposing tariffs on products produced by means we have decided are not acceptable or scale back the costs of labor domestically. Basically, we either need to bring the cost of foreign labor up (either through tariffs or the political pressure of tariffs that causes other countries to adopt our standards), bring our cost of labor down, or I suppose more effectively, some of both.