You can delete the default locations. Is it your current location? If you have Weather turned on in your Location Services preferences, your current location will show as the first item in the Weather list.
Otherwise, you can delete any other location. Are you sure you're are swiping on the location panel, starting on the right side of the screen and swiping to the left? The panel will shift to the left, and give you the option to delete.
See my screenshot. Atlanta is my current location, shown by the arrow next to the time. This location, your current, you cannot delete. You can only make it go away if you turn off Weather under Location Services.
I swiped from right to left on Cupertino, which reveals the delete option. Just tap delete to delete that location.