They are both.
Culinarily speaking they are vegetables like lettuce, potatoes, carrots, onions peppers, olives...
Botanically speaking they are fruits because they are seeds surrounded by edible flesh (just like peppers, eggplants, cucumbers...). This is opposed to potatoes, carrots, onions and lettuce, as those comprise other parts of the plant.
It gets interesting with olives and melons. Botanically they are fruits. Culinarily olives are vegetables and melons are fruits (they are sweet).
But, olives, like the items that everyone agrees are fruits (apples, oranges, "summer" fruits...) grow on trees. While melons, like the items that are classified as vegetables, grow on plants that must be replanted every year (unlike trees that live from season to season).