Well as a status report, we are still scratching our heads a bit but we have taken some measures that we know could be to blame, and hopefully corrected them.
The problem that happens about 11 PM to 1 am EST should be corrected. What was happening was all the VPSs were getting their cPanel updates at the exact same moment. This was due to the fact that the VPS's come from a template that we have built, but overlooked the fact that the crontab entries were all set to fire off at the same time.
We have adjusted the crons on each VPS to fire off at different times to that the disk I/O is not maxed out trying to install files for 20 something cPanel servers at the same time.
Additionally, we have removed one VPS from the server which should free up some resources for the others.
We are also in the process of putting one another one of the load consuming VPS in quarantine on a separate server.
So that should help us get past that nightly issue that we were dealing with, and the random fluctuations in the day should be better after this second VPS is transferred out to its own dedicated machine.