What I generally do is create an interface for an application service that performs this function. In my examples I'll assume you are using something like the MVVM Toolkit or similar thing (so I can get a base ViewModel and a RelayCommand).

Here's an example of an extremely simple interface for doing basic IO operations like OpenFileDialog and OpenFile. I'm showing them both here so you don't think I'm suggesting you create one interface with one method to get around this problem.

public interface IOService
{
     string OpenFileDialog(string defaultPath);

     //Other similar untestable IO operations
     Stream OpenFile(string path);
}

In your application, you would provide a default implementation of this service. Here is how you would consume it.

public MyViewModel : ViewModel
{
     private string _selectedPath;
     public string SelectedPath
     {
          get { return _selectedPath; }
          set { _selectedPath = value; OnPropertyChanged("SelectedPath"); }
     }

     private RelayCommand _openCommand;
     public RelayCommand OpenCommand
     {
          //You know the drill.
          ...
     }

     private IOService _ioService;
     public MyViewModel(IOService ioService)
     {
          _ioService = ioService;
          OpenCommand = new RelayCommand(OpenFile);
     }

     private void OpenFile()
     {
          SelectedPath = _ioService.OpenFileDialog(@"c:\Where\My\File\Usually\Is.txt");
          if(SelectedPath == null)
          {
               SelectedPath = string.Empty;
          }
     }
}

So that's pretty simple. Now for the last part: testability. This one should be obvious, but I'll show you how to make a simple test for this. I use Moq for stubbing, but you can use whatever you'd like of course.

[Test]
public void OpenFileCommand_UserSelectsInvalidPath_SelectedPathSetToEmpty()
{
     Mock<IOService> ioServiceStub = new Mock<IOService>();

     //We use null to indicate invalid path in our implementation
     ioServiceStub.Setup(ioServ => ioServ.OpenFileDialog(It.IsAny<string>()))
                  .Returns(null);

     //Setup target and test
     MyViewModel target = new MyViewModel(ioServiceStub.Object);
     target.OpenCommand.Execute();

     Assert.IsEqual(string.Empty, target.SelectedPath);
}

This will probably work for you.

There is a library out on CodePlex called "SystemWrapper" (http://systemwrapper.codeplex.com) that might save you from having to do a lot of this kind of thing. It looks like FileDialog is not supported yet, so you'll definitely have to write an interface for that one.

Hope this helps.

Edit :

I seem to remember you favoring TypeMock Isolator for your faking framework. Here's the same test using Isolator:

[Test]
[Isolated]
public void OpenFileCommand_UserSelectsInvalidPath_SelectedPathSetToEmpty()
{
    IOService ioServiceStub = Isolate.Fake.Instance<IOService>();

    //Setup stub arrangements
    Isolate.WhenCalled(() => ioServiceStub.OpenFileDialog("blah"))
           .WasCalledWithAnyArguments()
           .WillReturn(null);

     //Setup target and test
     MyViewModel target = new MyViewModel(ioServiceStub);
     target.OpenCommand.Execute();

     Assert.IsEqual(string.Empty, target.SelectedPath);
}

Hope this is helpful as well.