The Daycast service was offline from 2:33 to 2:48PM Pacific time while a user-reported issue was being resolved.

Remediation: Though we got the affected user up and running again, the issue needs a deeper fix. To completely squash the bug, we intend to update the way Daycast moves a user from one organization to another. The process is currently outdated and needs to be brought into alignment with the rest of the application.

The Daycast service was offline from 9:07AM to 10:51AM Pacific time while planned maintenance was being performed.

The Daycast service was offline from 9:52AM to 11:11AM Pacific time while service and improvements were being deployed. Users who were running the app when work began may have noticed an in-app offline message. All time and tasks were tracked as usual during the downtime. Users who attempted to start Daycast during the offline window will have encountered an error screen. See the release notes for details on the improvements we made during this downtime.

FreshBooks users who attempted to finalize time logged to FreshBooks projects in Daycast between the hours of 8:18AM and 10:58AM Pacific time will have encountered an error message. Once upstream service interruption was resolved, Daycast submitted all previously unsubmitted hours to FreshBooks successfully.

Between 1AM and 5:05AM Eastern time, password resets and invitations to join a team on Daycast were not available due to an upstream email provider's planned service outage. The Daycast service was otherwise uninterrupted.

The Daycast service experienced an unexpected outage between 3:30 and 6:25PM Pacific time. A critical IP address was inadvertently changed during routine maintenance of a server, which caused connectivity failure with our authentication provider. Once we'd identified the source of the problem, remediating was a simple matter of correcting the IP address.

The Daycast service experienced intermittent interruptions between 10:30AM and 1:30PM Pacific time. Users attempting to run the app during that window may have encountered slow loading times or failure to load altogether. We sourced the problem to issues with external Domain Name Service resolution and addressed/resolved those issues. It took some time, hence the three hour window of potential service interruption, but we did get it solved, returning Daycast service to normal.

Users who attempted to start Daycast between 10:47AM and 11:54AM Pacific time may have encountered sluggish loading or even failure to load. We sourced the problem to our datacenter, who in turn found a bug in a router that was causing the latency issues. Once identified, the problem was immediately corrected, and Daycast's loading trouble subsided.

The Daycast service was offline from 12:46 to 13:16 PST while service and improvements were being deployed. Users who were running the app when work began may have noticed an in-app offline message. All time and tasks were tracked as usual during the downtime. Users who attempted to start Daycast during the offline window will have encountered an error screen. (Offline startup is on our long-range roadmap, so that users can open and run Daycast even when the service is undergoing maintenance.) See the release notes for details on the improvements we made during this downtime.

The Daycast Service experienced unexpected downtime from 09:00 to 16:32 PST. After investigating, we sourced the issue to unanticipated changes in an upstream provider’s service. Responses to authorization requests were suddenly and inexplicably much, much slower. (We still don’t know why; we reached out to the provider but have not received an explanation.) The outage affected Daycast users as follows:

• Anyone already running Daycast when the outage began will have seen the Reconnecting banner throughout the outage, though use of the app was otherwise unaffected.
• Anyone already running Daycast when the outage began who also attempted either a refresh or a restart of the app during the outage will have encountered a failed loading screen.
• Anyone not already running Daycast when the outage began who attempted to start the app will have encountered a failed loading screen.

We resolved the problem by making changes on our end to how authorization requests are handled. Even if we never receive an explanation from the upstream service provider, the issue is solved by the adjustments we made. No time or data was lost in the outage.

Users who attempted to push hours logged in Daycast to a FreshBooks account between 08:00 and 10:00 PST today may have noticed a failure to submit their records to FreshBooks. This was due to a temporary outage of the FreshBooks API. Once the outage was resolved, all the previously failed submissions were successfully pushed. No data or time was lost.

The Daycast service was offline from 16:00 to 17:29 PST while service and improvements were being deployed. Users who were running the app when work began may have noticed an in-app offline message. All time and tasks were tracked as usual during the downtime. Users who attempted to start Daycast during the offline window will have encountered an error screen. (Offline startup is on our long-range roadmap, so that users can open and run Daycast even when the service is undergoing maintenance.) See the release notes for details on the improvements we made during this downtime.

Daycast users who were running the app between 17:00 and 18:00 Pacific time may have noticed an in-app message indicating that Daycast was offline briefly (~30 seconds). Users who attempted to open and run the app during that brief offline window may have encountered an error screen prompting them to try again. This short service interruption was expected and due to planned system maintenance.

The Daycast service was interrupted from 16:21 to 17:00 US/Pacific time due to an outage experienced by a third-party vendor that provides our data storage. No data was at risk, but the outage disrupted Daycast’s ability to connect to the third-party vendor, which compromised its ability to load properly. Once the outage was resolved, Daycast was again able to connect and load fully.

Planned maintenance work resulted in expected downtime from 01:00 UTC to 01:27 UTC. Users may have found the Daycast service unavailable or slow to respond during that brief window.

Daycast experienced an unexpected outage today from 19:24 UTC to 19:53 UTC due to an issue with Google Cloud Platform: https://status.cloud.google.com/incident/cloud-networking/18012. Daycast users may have been unable to run the app or, if it was already in use, may have encountered a Reconnecting message or other problems. Once the issue with Google Cloud Platform was resolved, the Daycast service recovered. If you were tracking time or otherwise made changes in your Daycast during the outage, all your changes and updates have been saved accurately. No data was affected by this outage.

While deploying a new version, Daycast experienced twenty-two minutes of unexpected downtime today from 18:38:00 to 19:00:00 UTC. Users who were already running Daycast would have seen a Reconnecting message in the app, and any changes they made or time they tracked during the downtime was pushed once Daycast reconnected. However, any users who tried to start the app during this twenty-two minute window will have encountered a screen indicating that Daycast could not connect. To those users in particular we say, thank you for your patience. We know how frustrating it can be to not be able to load the software you need when you need it, and we apologize for the inconvenience. We’re actively working on ways to ensure that you experience as little disruption as possible during upgrades like these.

Network maintenance resulted in expected downtime from 16:04:00 UTC to 16:12:00 UTC. Users may have found the Daycast service unavailable or slow to respond during that brief window.

Daycast experienced several brief, intermittent disconnections and reconnections between 16:30:00 and 16:52:00 UTC. Upon investigation, we discovered the issue occurred during server replication. To remediate, we’ve made changes to our replication schedule and will monitor to ensure the problem doesn’t recur.

At 17:00:00 UTC one of our team reported an error when attempting to log in. (Error message: “The requested feature is not enabled in this environment.”) We easily reproduced the error and found that both login and new account creation attempts were failing. We sourced the problem to an issue with our authentication management service and reached out to them for resolution. They resolved the issue and normal service was restored at 18:27:00 UTC.

An infrastructure upgrade at our authentication management service limited Daycast use for around fifteen minutes starting at 04:00:00 UTC:

• New account creation was temporarily disabled
• Existing account deletion was temporarily disabled
• The invite feature was temporarily disabled

Daycast use was otherwise unaffected. The upgrade is complete and all of the above capabilities have been restored.

There was an unexpected power outage today from 22:16:00 to 23:20:00 UTC at Daycast's datacenter, during which Daycast was down. We believe high winds in the area caused the power outage. No data was lost and service has been restored.

Network improvements at our datacenter created expected downtime from 07:32:27 to 07:33:58 UTC. Users may have experienced longer than usual load times during that brief window.

Planned maintenance beginning at 02:00:00 UTC and lasting for around fifteen minutes limited the Daycast service as follows:

• New account creation was temporarily disabled
• Existing account deletion was temporarily disabled
• The invite feature was temporarily disabled

Daycast use was otherwise unaffected. The maintenance period has concluded and all of the above capabilities have been restored.

Outgoing, app-triggered email (e.g., password resets and user-generated invitations to others to join Daycast) was interrupted again today from 20:46:00 UTC to 21:00:00 UTC due to load testing. We caught the problem almost immediately, stopped the test, and cleared the resulting backlog of emails.

Remediation: We’re in the process of flagging critical emails (time-sensitive communications like password resets and invitations to others) so that they get priority delivery from our email platform regardless of whether load testing generates emails or not.

Outgoing, app-triggered email (e.g., password resets and user-generated invitations to others to join Daycast) was interrupted today from 17:04:00 UTC to 23:00:00 UTC. We noticed the problem at 18:37:00 UTC and identified the cause as a bottlenecked email queue resulting from our load testing—the load test generated a flurry of Welcome to Daycast emails that created a hefty backlog.

Remediation: We’re currently investigating the best way to avoid sending emails generated by load testing.

There was an unexpected Daycast outage today from 18:17:00 to 19:07:00 UTC during a recent hardware upgrade at Daycast's datacenter. The impact on our production servers was unexpected, and we're investigating the cause now.

Update: We planned the hardware upgrade and expected it to be a seamless transition with no downtime. However, due to an existing and unknown configuration error, the old hardware did not failover—that is, pick up the slack when the error occurred—as we expected it to.

Remediation: We've resolved the configuration error that was the source of the outage and are currently reviewing our approach to service-affecting maintenance with an eye towards mitigating opportunities for interruptions.