All package managers have many downsides. You just have to pick which you can live with.

History

npm started out managing node.js modules (that's why packages go into node_modules by default), but it works for the front-end too when combined with Browserify or webpack.

Bower is created solely for the front-end and is optimized with that in mind.

Size of repo

npm is much, much larger than bower, including general purpose JavaScript (like country-data for country information or sorts for sorting functions that is usable on the front end or the back end).

Bower has a much smaller amount of packages.

Handling of styles etc

Bower includes styles etc.

npm is focused on JavaScript. Styles are either downloaded separately or required by something like npm-sass or sass-npm.

Dependency handling

The biggest difference is that npm does nested dependencies (but is flat by default) while Bower requires a flat dependency tree (puts the burden of dependency resolution on the user).

A nested dependency tree means that your dependencies can have their own dependencies which can have their own, and so on. This allows for two modules to require different versions of the same dependency and still work. Note since npm v3, the dependency tree will by flat by default (saving space) and only nest where needed, e.g., if two dependencies need their own version of Underscore.

Some projects use both is that they use Bower for front-end packages and npm for developer tools like Yeoman, Grunt, Gulp, JSHint, CoffeeScript, etc.


Resources