The Pingdom reports for the last year do not suggest a general slowdown. There is a variation month-to-month of about 10%. For the Home page response times, you can see the history at
http://stats.pingdom.com/w17m1yxzjyin/49229
If interested in other services, start at
http://stats.pingdom.com/w17m1yxzjyin
There might be a problem with the specific server that hosts your account, though FastMail seem to be pretty good at finding such issues. If the problem is glaring, I would suggest submitting a support ticket.