This border is used to show that the element is focused (i.e. you can type in the input or press the button with Enter). You can remove it, though:

textarea:focus, input:focus{
    outline: none;
}

You may want to add some other way for users to know what element has keyboard focus though for usability.

Chrome will also apply highlighting to other elements such as DIV's used as modals. To prevent the highlight on those and all other elements as well, you can do:

*:focus {
    outline: none;
}