Different "levels" of cache are basically different speeds. Level 1 cache is very small and very fast, all chips I know of have it. Level 2 cache can be either on chip (fast and small, but slower and bigger than L1) or off chip (larger, but slower), all recent chips I know of have Level 2 cache (almost all have on chip Level 2 cache). Level 3 cache is always off chip (except on Intel's Itanium series), and is just like a larger slower level 2 cache. With a G4+, here's how it works:
If the program uses 32k or less of data, it's stored in the L1 cache, if not, then the most current pieces of data are. Everything in the L1 cache is in the 256k L2 cache, along with anything slightly less recently used. Even less recently used things are in the 1 or 2 MB L3 cache (which is off chip). If it's not in any of the caches, it has to be fetched from ram, which is really slow.
Overall, it boosts performance a bit and helps negate the effects of the slow system bus that G4s have.