Assuming you bought the Apple sanctioned/compatible upgrade kit, everything you need will be in the box. Start by pulling off the side panel, and focus your attention on the middle section of the case.
1.Unplug AUX power connector that's in your card( may not have one if your current card uses <75W)
2. Unscrew the PCIe card bracket, pull it off.
3. take out the old card, put in the new, assuring all ports are flush w/ the back of the machine. The 5770 uses one 6-pin power connector, so plug that in.
4. Put your card bracket back on, and you should be ready to boot the machine.
If you didn't buy the Apple upgrade kit, you may have to flash the card which is discussed in other threads.