Earlier today, ClickTime began to experience performance issues at 8:20am. We immediately started investigating the issue, and decided to prevent additional logins from exacerbating the issue by disabling logins at 9:05 a.m. At 9:45 a.m. we identified the likely source and began to address it. At 10:15, we re-enabled logins for all users.
This issue was related to a check we perform to see if a user has left stopwatches running. There was a database issue that prevented this check from running quickly, and as those requests accumulated those queries began to time out. While this function isn’t essential to using ClickTime, it does occur on every page in the site, so it impaired site performance for many users. When we improved performance on this check, it resolved the issue for all site usage.