I don't like this because of how easy it would be to subtle manipulate a population. The gamut of possibilities are endless. Companies could manipulate the data in ways to gain more business, and political organizations could manipulate elections.
Say you could bump into your friend John on the street. Your AR display has some information about John floating next to him. It pops up with an alert next to him that says "John's birthday is in three days. He really loves Dairy Queen's Ice Cream Cakes! Would you like to order one?" even if John doesn't really like those cakes, or maybe he likes ice cream cakes but not necessarily Dairy Queen ones—perhaps Cold Stone instead. But whoever pays the most to the puppet master gets the order.
Then there are more nefarious things, such as subtly inducing a population to vote a certain way through slight suggestions. For instance, when walking down a street, it could tag a few more strangers than normal in your city as being of Mexican origin (and it knows who are strangers to you), and if a candidate has a strong platform against immigration, it could influence the election. Or it could use geofence based manipulations to get votes in areas that are leaning one way or the other, or target people of a certain race who would respond to a message in a certain way. There are all sorts of weird manipulations you can do when everything in the world has metadata that is constantly logged into a huge system.
Furthermore, aren't some things just meant to be forgotten? Do we need every single thing logged? You know, forgive and forget. Do you want metadata reminding you about horrible things as well? What if it only shows positive things—is that just as bad and could lure us into a false sense of security? How does this manipulate our emotions, even unintentionally? I feel like it's part of the human experience to forget things—sometimes to only be reminded of them many years later when encountering a certain smell or voice.
I obviously see the upside to a system like this. But when we set out to design it, it's smart to realize the downsides so we can try to protect against it. Something like this has the potential to vastly enhance mankind and allow us to break free of some of our limitations and create a more peaceful society with more access to resources. But I'm also a student of history and anything with this much potential for power will likely be corrupted at some point.