Figured it out. What you need to do is create a separate src folder called debug that holds the different icons. For example, if your project layout is as follows, and your launcher icon is called ic_launcher.png:

[Project Root]
  -[Module]
    -src
      -main
        -res
          -drawable-*
            -ic_launcher.png

Then to add a separate icon for the debug build type, you add:

[Project Root]
  -[Module]
    -src
      -main
        -res
          -drawable-*
            -ic_launcher.png
      -debug
        -res
          -drawable-*
            -ic_launcher.png

Then, when you build under the debug build type, it will use the ic_launcher found in the debug folder.