Update on system issues – December 19th

Update 12:59 pm: We are back online! Thank you for your patience as we worked to restore access to ClickTime.

Update 12:15 pm: We are in the final stages of restoring access to the system. Thank you for bearing with us, as it took our team longer than expected to address this issue. ClickTime access should be restored within the hour! 

Update 9:40 am: Our team is still working to restore full ClickTime service. Our current estimate is noon Pacific Time (20:00 GMT). Thank you for your patience.  

Update 7:40 am: Our migration to backup systems is moving smoothly, but slower than expected. We are now targeting 10:00 am Pacific Time (18:00 GMT) for full service restoration.

Update 5:00 am: We are currently moving data to backup systems and we are targeting 16:00 GMT for service restoration. 

Update: 3:45 am: We have a current estimate of service restoration at 8am Pacific time, 16:00 GMT. We apologize for the delay and inconvenience.

Update: 2:30 am: Our team has identified a hardware failure of one of our databases. We are preparing to transfer operations to backup servers if we cannot restore the primary server. We will have another update within 1 hour.

Update 1:36 am: Starting at approximately 1:00 am Pacific Time (9:00am GMT) on December 19th, ClickTime has been experiencing login failures. We have disabled logins temporarily while our operations team investigates the cause and works to restore service. We will post an update as soon as we have new information.

ClickTime System Update

We've addressed the time entry issue that had affected some customers in the U.K, Europe, Australia, New Zealand and Asia. 

All customers should be able to enter and report on time as normal. 

We apologize for any inconvenience this may have caused.

Please reach out to support@clicktime.com if you have questions, or if we can help in any way.

ClickTime Performance Recap

Earlier today, ClickTime began to experience performance issues at 8:20am. We immediately started investigating the issue, and decided to prevent additional logins from exacerbating the issue by disabling logins at 9:05 a.m. At 9:45 a.m. we identified the likely source and began to address it. At 10:15, we re-enabled logins for all users.

This issue was related to a check we perform to see if a user has left stopwatches running. There was a database issue that prevented this check from running quickly, and as those requests accumulated those queries began to time out. While this function isn’t essential to using ClickTime, it does occur on every page in the site, so it impaired site performance for many users. When we improved performance on this check, it resolved the issue for all site usage.

ClickTime Update

Earlier today, some customers experienced an issue logging into ClickTime. We apologize for any inconvenience this may have caused.

You and your team should now be able to log into ClickTime as normal.

Please reach out to support@clicktime.com if you have questions, or if we can help in any way.

We’ll be posting more details shortly.