I have Blue Yeti USB microphone which also has an audio output for monitoring. I do not use the output but once pluged-in - Ubuntu recognize it as new sound card and switch output to the microphone immediatelly. This drives me nuts because I always need to switch back audio output manually to previous device.
Pulseaudio has a feature automatically switch to hot-plugged devices which is enabled by default. All is needed - comment out this setting on the configuration.
Find and remove or comment-out the following lines on
### Use hot-plugged devices like Bluetooth or USB automatically (LP: #1702794) .ifexists module-switch-on-connect.so load-module module-switch-on-connect .endif ``