All package managers have many downsides. You just have to pick which you can live with.
Bower is created solely for the front-end and is optimized with that in mind.
Size of repo
country-data for country information or
sorts for sorting functions that is usable on the front end or the back end).
Bower has a much smaller amount of packages.
Handling of styles etc
Bower includes styles etc.
The biggest difference is that npm does nested dependencies (but is flat by default) while Bower requires a flat dependency tree (puts the burden of dependency resolution on the user).
A nested dependency tree means that your dependencies can have their own dependencies which can have their own, and so on. This allows for two modules to require different versions of the same dependency and still work. Note since npm v3, the dependency tree will by flat by default (saving space) and only nest where needed, e.g., if two dependencies need their own version of Underscore.
Some projects use both is that they use Bower for front-end packages and npm for developer tools like Yeoman, Grunt, Gulp, JSHint, CoffeeScript, etc.
- Nested Dependencies - Insight into why node_modules works the way it does