Indeed.
It's extremely hard to know what we are aiming for.
It's a bit like people from a few 100 years ago trying to copy what a silicone chip does.
They don't understand how the thing they are trying to copy works in the 1st place.
Or us confronted with an alien space ship just floating few feet another ground without any obvious power keeping it there.
I think we need to accept there is no magic in the brain and there is no separate little person inside running the stuff.
You only have to damage areas of the brain and see how certain things no longer work.
I'm of the opinion personally that the brain basically runs itself, and that's due to all the things the brain has experienced over many years, which is of course laid on top of some hard coding that's developed over many years of evolution.
The conscious bit, personally I think is some created layer over the top of it all that somehow gets created perhaps simply due to this complexity.
We don't decide we want water, our brain wants water and then afterwards our consciousness in making it feel like WE chose to get some water.
Now do we need to believe a artificial brain also has this layer on top, and is conscious. I don't think it's needed.
If it appears to be and passes all tests then I honestly can't see it mattering What's actually going on inside the black box.