The short answer is: You can't do that using Google's form builder.
The longer answer is that you can copy the form's HTML code and host it on your own website. That HTML code may then be modified to have the button display any text you'd like.
Copying and hosting the HTML code does have its drawbacks. If you edit the form from Google's form builder, the copied HTML code will of course not be updated, so you'd have to copy and mofiy it again.
See this article from ohdoylerules.com for how to copy and host the HTML code. Note that in this article, they are using Google Drive to host their customised HTML. That feature is not available in new Google Drive, so you will have to host your HTML somewhere else.
I have created a demo form here: http://vvv.vidarramdal.com/stackoverflow/webapps-66392/
I have searched the online lots for the answer to my question and several times the setText('text button') function has been mentioned. From what I can see this only applies to the these more complex customised forms. If that's not correct ...
No, you are correct.