Every now and then I'd get a message about supervisor not being able to find its socket. So I'd blindly start supervisor again and everything would appear alright.
However, I finally did some investigating yesterday and it turns out that I was just starting a new instance on top of the one that was already up. Running
ps aux | grep supervisor showed that there were now two processes for supervisor running. Oops. Good thing this wasn't for anything serious. Not that anything bad happens - the original supervisor instance was still managing the app in some cases. In others, though, the new process was doing it. This could lead to some confusion where either 1) the application appears to be running independently of supervisor or 2) you can't start the application on the new supervisor process because the port is being used on the old one, and you can't kill that old application because it's constantly being restarted.
Anyways, it turns out that the supervisor default is to save the socket in
/tmp. That's bad because it will get overwritten (different times depending on the system). So, the solution is to move that shit into
/var/log for the log file).
I'm not 100% sure this is the right answer - I'll have to keep checking up on those applications - but it seems very plausible.