Category: Status Updates

Performance back to normal

We've been closely monitoring our systems throughout the day today (Feb 3rd) and site speed has been better than average for a Tuesday. It's not quite time to declare victory, but we do think that ongoing site performance should remain up to our standards moving forward.

In brief, we experienced excessive slowness while user sessions were being passed across our farm of web servers. (A bit of background: all actions taken by our users are spread out across numerous web servers for scalability and maximum speed. We pass information between those servers behind-the-scenes to maintain a seamless user experience as users bounce from one machine to another.) This slowness led our users to re-attempt their actions and logins more and more often, which created more slowness, and this compounded itself. We believe this happened yesterday because a threshold was crossed where our session-management found its limit and became congested.

So we've done several things to fix this: yesterday afternoon we made improvements to streamline our session-management. This is a step towards fixing the underlying cause of the original speed issue. More changes along those lines will come today. In coming releases we're going to limit the ability to re-try certain actions to avoid multiplying the load of any one user. Lastly we are accelerating our plans to totally replace our session-handling system with something many times faster.

Some customers have asked if we're unprepared to handle the load of a growing customer base. Ironically, yesterday's problems were the result of mistakes made pursuing that goal. We recently expanded our hardware capacity to handle future transactional load, but the plumbing between those new systems couldn't keep up. We've now got a clear idea of how to remedy those flaws.

I hope anyone with questions will contact us:  support@clicktime.com

-Alex Mann
CEO
alex@clicktime.com

Feb 2nd End-of-Day Update

As of this evening (Feb 2) we have implemented certain site changes that we believe will mitigate the root cause of today's performance problems. More complex changes will be made over the coming days and weeks.

We're also maintaining enhanced monitoring procedures overnight and into the typical Tuesday morning busy period. If any problems do arise, we'll know immediately and we'll keep you posted on this blog. Traffic has been very smooth this afternoon but load has been fairly low so we're still being cautious.

(And while we don't anticipate the need to bring the site offline again for further changes, we'll reserve that option if the situation requires it.)

All of us are truly sorry for the frustration and inconvenience this has caused many of our customers. We'll do a full post-game analysis in a future posting; for now we're all working very hard to maintain a good user experience on our application. 

Contact us at support@clicktime.com if there's anything we can do to help.

Thanks,

-ClickTime Web Operations

Update on Feb. 2 Performance Issues

We're continuing to diagnose this morning's performance issues. Because users are seeing very slow login and load times, we've taken the site down temporarily. We will bring it back up as soon as the issue is resolved. Our web operations team is hard at work on this and we'll be posting updates as we know more. You can also check our twitter feed (@clicktime) for news or contact support@clicktime.com and we'll add your email to an update list.