An XML sitemap only helps search engines find your pages — it has no effect on their ranking.
If you're concerned about robots wasting their time crawling the less important pages, you can assign them a lower priority in your sitemap. Providing accurate last modification timestamps in your sitemap can also help robots crawl your site more efficiently.
One possible optimization, if you have so many pages that the sitemaps themselves get impractically large, is to split your pages into multiple sitemaps collected under one sitemap index and include lastmod timestamps in the index so that robots can download only the maps that have changed. This works best if you sort the most frequently changing pages into one sitemap, the somewhat less frequently changing pages into another, and so on.