A combination of all of the following hurt Nintendo.
1) Ugly Hardware - If Apple has taught us anything it's that design sells. Apple's products are sleek, sexy and functional. Nintendo tried copying the feel of Apple products with the white, minimalist style of the Wii and DS-line. The problem is they choose to be inspired by late 90's Apple instead of modern day Apple. The Wii and DS up close look like the cheap, bulky, plastic products they are.
2) Sucky Software - Nintendo controls both the hardware and software of their devices, they should work together in harmony. Instead we have half baked features and services that don't work well together, work well sometimes, or don't work at all. Their eStore is a joke and friend codes is a debacle. Why bother spending precious resources on the gimmicky 3D low res camera on the 3DS that no one will use on a regular basis when you could build out the proper infrastructure for a great eStore and online gaming. Why am I calibrating a Wiimote, what is this, 1996? Why does the 3DS still use a stylus?
3) No infrastructure - Sony learnt their lesson the hard way with the PS2 non-existent online network. Microsoft launched the Xbox 360 with a mature Xbox live leaving the PS3 to play catch up. Up to this day they are still trying to match Microsoft feature for feature. It takes a long time to build a fast, stable network. They should have jumped on that bandwagon 3 generations ago. The big tech buzzword nowadays is "cloud computing", Apple being the most successful, Sony and Microsoft and getting their. Point is, they are generating extra revenue by offering services like movies, music, game packs, DLC and more. It's not enough just selling games, where are the services to keep gamers playing those games. Pokemon would make a killing in DLC sales.
4) Casual gaming debacle - Nintendo is paying the ultimate cost of going after the casual market. I guess their mommy never told them to never leave what you have for what you don't have. The casual gamer might outnumber the hardcore gamer, but they are also very fickle. They will run out and buy the newest gizmos if it's priced right or looks cool enough but they seldom come for seconds or thirds or fourths. Nintendo all but abandoned the people that gave them guaranteed money with lackluster, sporadic hardcore games to focus on Wii Sports and various other crapware. The truth is, a lot of hardware was flying off the shelves, but no one was buying games. The casual market was more than happy with Wii Sports, while hardcore gamers starved.
5) No HD/HDMI - Yep, this gets it's own bullet point. How on earth could you release a system with no HD or HDMI support? Sure tube TV's were still in use back when the Wii first came out, but they were already well on their way out. If the Wii support HD gaming, it would still be relevant. Everyone and their mother has an HD TV now, tube TVs are officially extinct as in, not being made anymore. Have you ever seen an SD game running on an HD TV? One word... terrible.
6) Same old, same old - The Wii U and the 3DS feel a continuation of the gimmickry. The Wii U has a huge controller that is a not quite tablet, not quite handheld potential debacle. While the 3DS is nothing more than a bigger, uglier version of the DS featuring 3D, a far more polarizing feature than a crisp, high pixel HD screen. One might argue that the PS Vita is a souped up PSP, at least Sony covered all the bases and future proofed the thing with a capacitive touch screen, buttons when you need them, and the all important 2nd analog stick. It's packed to the gills with great features and can easily cater to both the hardcore and casual markets.
7) To risk, or not to risk - Granted Nintendo takes on huge risks with underpowered hardware that uses some gimmickry to stand out, but they play it too safe in their software. There is no need for a Nintendo rumor site, because everybody knows their next game will be a Zelda, a Mario, a Metroid that is exactly the same as the last. They have strong franchises, but enough with the milking, gamers are getting disillusioned. Where are the new IP's, where is the new gameplay that takes existing franchises to the next level.
8) App Store with a twist - At this point in time all the major players need to have an app store, and I'm not talking about a general purpose app store like Apple, but a specialized app store with full featured games, old-school catalog games, bite-sized games, episodic content, DLC, demos, and apps that lean toward gamer centric use. This app store has to have a twist, a well thought out, well implemented social component so people can easily meet new friends and like minded gamers, kinda like Facebook but without the monthly privacy issues, and friend codes can go bye-bye too. They can still use physical media, to keep trade-in gamers happy, but give gamers the choice of physical media or download. There is a huge market for this, just look at the top sellers on Apple's store, they are all games.
Nintendo doesn't need to be a third party publisher to thrive. They need nicely designed, sleek, future-proof hardware with fully-functional, easy to use software that is not only good at doing multiple things, but doing them well. They need a library of fantastic, innovative, full-featured games, old-school catalog games, bite-sized games and new IP. They need some kind of app store with solid social features to build a community around, and the infrastructure to support online play and services. They need to put more effort in taking care of their base, the hardcore gamer who will spend the money, while still addressing the casual crowd. They need to spend those R&D dollars wisely, the era of the attachment is over. People no longer have space in their homes for a million crappy plastic peripherals, Nintendo needs to KISS (keep it simple stupid). If Nintendo took the time to address these issues, they'll be fine in no time.