Let me pick apart a section of my own design and explain why each element was chosen to be the way it is to explain how skeumorphism can be done well without interfering with functionality, which is of coarse most important. (This isn't an advertisement, the app isn't released)
This section is a fairly basic programmable MIDI keypad. Users can assign whatever MIDI signals they want to each button, choose the colors of the buttons, and add a label.
The greenish section at the top is a scrollable live display of the MIDI commands sent from the app, this allows the user to see if everything is working correctly, It also displays important messages to the user, if the user taps and holds on the display it goes fullscreen for easier viewing. When a critical message is displayed it turns red. It's modeled after your typical cheap LCD display, although with a clean legible font. This was chosen because it offers a neutral color thats easy to read in all lighting conditions, a pure white would be too bright in low light conditions, and the app is meant to be used for live performances where lighting conditions can be poor, either dark, or overly bright. The screen was positioned in the top left corner because it has the least user interaction, and with the majority of people being right handed it is the least comfortable position to reach. You might notice that I have left the status bar displayed at the top. This is because displaying the current time, and battery life would be very important to someone on stage.
To the right is the program button and program mode indicator, which flashes. the user presses the Program button, then selects the UI element that they wish to change settings for, any of the other UI elements can be selected. The program button changes when pressed. It turns slightly darker and and reduces in size a small amount, this gives visual indication that the button is activated, and appears depressed. This is modeled after a toggle button from an electric organ (not a hammond). This is positioned in the top right corner because it's easy to reach held in one hand by swinging the thumb upwards, but it is also a much further reach than any of the other buttons because pressing it by accident could cause confusion. When it's activated the bottom section of the app slides down on sliders to reveal a menu of settings beneath. This gives the user a sense of place, the settings have a physical location within the app.
Below the display screen are the channel buttons, this allows the user to have four different pages within the app. The screenshot shows one mode, the MIDI keypad mode. The user can choose which mode each channel is assigned to, so they can have any combination of four different devices, one for each channel. They glow amber when selected. This gives good indication as to which channel the user is on. having a separate button for each channel gives much faster indication with a quick glance at the screen the user can tell by the position of the glow, which is aligned with the grid of buttons, without having to read anything.
The background is the dark textured plastic I mentioned earlier. dark materials were chosen because it makes the controls stick out more. For someone using this as an instrument it's important that it has a strong visual contrast because a touch screen device has no tactile feedback, and in many situations an audible feedback can't be used, it is however an option. The metal border sharply contrasts the dark background an isolates the channel and program controls from the buttons below, which the user is expected to be tapping away at possibly with vigor. This helps to reduce accidental presses. The border also serves as a dead zone between the buttons below and the channels buttons.
The drumpad like buttons are the user configurable controls, with many possible color combinations and editable labels. The user is able to configure them to do pretty much anything they want with the MIDI spec, so being able to identify what each button is assign to is very important. It's not implemented yet, but the user will be able to assign overlay images to each button as well as labels. This will help because the human brain is much faster at identifying shapes than it is reading words, this is why each group of buttons in the app have distinct shapes, square, rectangle, and circle. The buttons have a glow, the glow is modeled after rubber drumpad buttons which have an LED mounted below them in real life, besides the change in brightness the glow affect on the outside of the button gives the illusion of the button getting slightly larger and give a much better feedback when the user's finger is covering most of the button. There is a visual separation between the buttons to help with more accurate aiming of the user's finger, but the reality is that each button is edge to edge. There is no dead space between the buttons, so the user will be hitting one of them if they hit anywhere in the bottom 3/4 of the app. If the user were to press between the buttons, and have nothing happen it would cause confusion.
None of the skeumorphism is completely necessary, but non of it detracts from the app, and in many cases it adds to the ease of use. I hope that this has helped to explain how skeumorphism can be helpful, or at the least not harmful, and how much really needs to be considered in UI design.