Not just location, orientation too, if someone on one phone moves it, modifies the object, the info about what happened must go real fast to the other phone or the effect of being in the same space is broken. If you block off the sun and put a shadow on the virtual object on one side, how it looks 90 degree from there is not the same, so you have to not just know about the object itself but how the rest of the scene interacts with it as seen from all orientations.
Wonder if it were not more efficient in such a case to split the work accross the devices than have each device do the whole work. A kind of grid of computation would be interesting as a device on the back would have a different view of the real world scene than one on the front. Both device could share their depth map and other scene info (for example sound arrives at first phone first means you get a very good idea of its direction, better than you could do from just one phone), (another example, reflections, shadows in the scene from both view can reconstruct sun position even if the sun is not in view) to create something really intriguing.
The idea of all the devices with a view of a scene sharing data about it, and sharing processing, is something that goes beyond just AR. Many people see the Augmented par as merely visual, but it could also be any type of sensor data originating not only from your device, but from every linked device around you. These pervasive sensors (vision, motion, orientation, heat, chemicals, wind, sound, etc) would create immense flows of real time info.
Anyway, all this is something that sounds simpler than it actually is and the consequences are deeper than you would think at first.
That's why i think AR and the sharing/analysis of this flow of real world info and action upon in will lead to things pretty crazy and it won't take that long to happen.
Many people, tend to only see the limitations of new tech, or see tech in isolation. It's when you integrate one tech into all other emerging and old ones that you could the true measure of what can happen.