You can check your site through HostTracker [Website Uptime Monitoring Service] to make sure it's not your particular connection that's slow.

In my experience, it was the overloaded VPS that was the bottleneck. It's my understanding that disk access is what slows the VPS down. Your site may be well within its resource allotment but disk access is like rush hour on an L.A. freeway.