ClickTime will be unavailable Saturday, March 28th, between 11:00am and 4:00pm Pacific Daylight Time(GMT -07:00).
The ClickTime system is undergoing important maintenance to improve system security and performance.
Scheduled maintenance and performance updates
ClickTime will be unavailable Saturday, March 28th, between 11:00am and 4:00pm Pacific Daylight Time(GMT -07:00).
The ClickTime system is undergoing important maintenance to improve system security and performance.
We've been closely monitoring our systems throughout the day today (Feb 3rd) and site speed has been better than average for a Tuesday. It's not quite time to declare victory, but we do think that ongoing site performance should remain up to our standards moving forward.
In brief, we experienced excessive slowness while user sessions were being passed across our farm of web servers. (A bit of background: all actions taken by our users are spread out across numerous web servers for scalability and maximum speed. We pass information between those servers behind-the-scenes to maintain a seamless user experience as users bounce from one machine to another.) This slowness led our users to re-attempt their actions and logins more and more often, which created more slowness, and this compounded itself. We believe this happened yesterday because a threshold was crossed where our session-management found its limit and became congested.
So we've done several things to fix this: yesterday afternoon we made improvements to streamline our session-management. This is a step towards fixing the underlying cause of the original speed issue. More changes along those lines will come today. In coming releases we're going to limit the ability to re-try certain actions to avoid multiplying the load of any one user. Lastly we are accelerating our plans to totally replace our session-handling system with something many times faster.
Some customers have asked if we're unprepared to handle the load of a growing customer base. Ironically, yesterday's problems were the result of mistakes made pursuing that goal. We recently expanded our hardware capacity to handle future transactional load, but the plumbing between those new systems couldn't keep up. We've now got a clear idea of how to remedy those flaws.
I hope anyone with questions will contact us: support@clicktime.com
-Alex Mann
CEO
alex@clicktime.com
As of this evening (Feb 2) we have implemented certain site changes that we believe will mitigate the root cause of today's performance problems. More complex changes will be made over the coming days and weeks.
We're also maintaining enhanced monitoring procedures overnight and into the typical Tuesday morning busy period. If any problems do arise, we'll know immediately and we'll keep you posted on this blog. Traffic has been very smooth this afternoon but load has been fairly low so we're still being cautious.
(And while we don't anticipate the need to bring the site offline again for further changes, we'll reserve that option if the situation requires it.)
All of us are truly sorry for the frustration and inconvenience this has caused many of our customers. We'll do a full post-game analysis in a future posting; for now we're all working very hard to maintain a good user experience on our application.
Contact us at support@clicktime.com if there's anything we can do to help.
Thanks,
-ClickTime Web Operations
Performance continues to be slower than usual but we have kept the site up for user logins. System response is ranging from moderately slow to normal. (If your session is too slow, we suggest you log out and try again a few minutes later.) We continue to work on this issue and will post updates as they become available.
We are still experiencing performance issues but as of 11:45am Pacific Time ClickTime is available for login. You may see periods of slowness throughout the day while we continue to work on this.
(We hope to keep the site up continuously but we may need to disable logins again in order to perform additional maintenance.)
We will continue to post information as it becomes available.
We're continuing to diagnose this morning's performance issues. Because users are seeing very slow login and load times, we've taken the site down temporarily. We will bring it back up as soon as the issue is resolved. Our web operations team is hard at work on this and we'll be posting updates as we know more. You can also check our twitter feed (@clicktime) for news or contact support@clicktime.com and we'll add your email to an update list.
We are experiencing technical difficulties and are temporarily unavailable. We are actively working on the issue and will post updates as soon as we have more information. Thank you for your understanding.
The scheduled maintenance has been completed. The ClickTime system is now fully operational.
We appreciate your patience.
Our maintenance is taking a longer than anticipated. We are currently estimating that we will be back up around 6:00pm PST but that is an estimate at thios point. We will update you with more information as it becomes available.
ClickTime will be unavailable Saturday, November 8th, between 11:00am and 4:00pm Pacific Standard Time(GMT -08:00).
The ClickTime system is undergoing important maintenance to improve system security and performance.