tag:status.lookback.io,2005:/historyLookback Status - Incident History2024-03-28T05:25:12-07:00Lookbacktag:status.lookback.io,2005:Incident/197684912024-01-19T11:05:26-08:002024-01-19T11:05:26-08:00Slow loading times<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>11:05</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>10:39</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.lookback.io,2005:Incident/185983292023-09-25T00:56:30-07:002023-09-25T00:56:30-07:00Participate links not working<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>00:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>00:46</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.lookback.io,2005:Incident/173781412023-05-26T09:00:54-07:002023-05-26T09:00:54-07:00Emails, notifications, transcriptions, and player feed not working<p><small>May <var data-var='date'>26</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Resolved</strong> - We have restored our queue system and all effected systems are now operational.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>08:36</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the situation.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>06:43</var> PDT</small><br><strong>Identified</strong> - We've identified that our queue system has gone down, and are working on bringing it up again.</p>tag:status.lookback.io,2005:Incident/162079902023-02-20T10:56:15-08:002023-02-20T10:56:16-08:00Not possible to create new organizations, slow loading of dashboard, organization settings, and player<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>10:56</var> PST</small><br><strong>Resolved</strong> - All systems are operational. We're continuing to keep an eye out on any degraded performance when it comes to organization creation from new signups, as well as loading times.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>08:55</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>08:33</var> PST</small><br><strong>Update</strong> - Slow loading times are gone, but we're continuing to investigate the core issue. Team creation might still be slow and not work.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>07:13</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.lookback.io,2005:Incident/160582942023-02-07T23:59:44-08:002023-02-07T23:59:44-08:00Login unavailable<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>23:59</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>23:58</var> PST</small><br><strong>Update</strong> - Our fix worked. Considering the matter resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>16:03</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:44</var> PST</small><br><strong>Investigating</strong> - It's currently not possible to login to the platform. We are investigating the issue.</p>tag:status.lookback.io,2005:Incident/126249182022-10-26T08:41:15-07:002022-10-26T08:41:15-07:00Users cannot login to dashboard and player<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>08:41</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>08:29</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.lookback.io,2005:Incident/91598242022-01-25T05:42:25-08:002022-01-25T05:42:25-08:00Elevated errors<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>05:42</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved and dashboard is stable again.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>05:28</var> PST</small><br><strong>Investigating</strong> - We have detected elevated errors in the Dashboard and are investigating this.</p>tag:status.lookback.io,2005:Incident/88423502022-01-06T02:45:05-08:002022-01-06T02:45:05-08:00Information about the Log4j issue.<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>02:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>00:19</var> PST</small><br><strong>Update</strong> - AWS has confirmed that the services we rely on were not vulnerable to attacks due to Log4j.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>03:16</var> PST</small><br><strong>Monitoring</strong> - Log4j is a Java-based logging utility found in a wide number of software products.<br /><br />The CVE-2021-44228 vulnerability (aka the “Log4Shell” vulnerability) was disclosed by the Apache Log4j project. If exploited, this vulnerability could potentially allow a remote attacker to execute code on the server.<br /><br />We don't use Java in our systems and as such have no dependencies on Log4j. The one exception to this is our Android app, Participate, which does not use Log4j.<br /><br />Our database providers are not affected. We rely on AWS as our datacenter and some of their systems may have been vulnerable in a way that affects us. We are investigating that further.<br /><br />We continue to monitor the situation.</p>tag:status.lookback.io,2005:Incident/79339822021-09-06T21:14:16-07:002021-09-06T21:14:16-07:00Users cannot log in to lookback.io<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>21:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>21:13</var> PDT</small><br><strong>Update</strong> - Rollback of previous deploy has fixed the issue. Still investigating root cause.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>21:04</var> PDT</small><br><strong>Investigating</strong> - Users are reporting an error "Something is wrong" when attempting to access the login page https://auth.lookback.io/login.</p>tag:status.lookback.io,2005:Incident/76063302021-07-26T11:30:10-07:002021-07-26T23:35:34-07:00Chrome 92: Participants not being able to hear Moderators in Participate for Desktop (LiveShare)<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Resolved</strong> - A fixed Chrome extension is approved in the Chrome Web Store, and published as version 2.24.3.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>10:37</var> PDT</small><br><strong>Monitoring</strong> - We've submitted a fix for Participate for Desktop to the Chrome Webstore. It's pending review. If approved, it'll roll out in a few hours.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>06:57</var> PDT</small><br><strong>Identified</strong> - Since the release of Chrome 92, we can reproduce an issue where Participants using Participate for Desktop (our Chrome extension) cannot hear the Moderator in LiveShare sessions.<br /><br />We recommend using Interview mode instead until a fix has been implemented.</p>tag:status.lookback.io,2005:Incident/73043502021-06-22T06:22:16-07:002021-06-22T06:37:01-07:00Sessions not becoming watchable<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>06:22</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>06:10</var> PDT</small><br><strong>Identified</strong> - We've identified the issue and are doing rollbacks. Non-uploaded sessions will be uploaded automatically.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>05:55</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>04:30</var> PDT</small><br><strong>Investigating</strong> - After ending a session, it does not become watchable in the player or dashboard. This is for new recordings. We're investigating the issue.</p>tag:status.lookback.io,2005:Incident/73023102021-06-22T02:08:10-07:002021-06-22T02:08:10-07:00Watching replay sessions doesn't work – stuck in "Loading..."<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>02:08</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>02:06</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>02:01</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>00:35</var> PDT</small><br><strong>Update</strong> - We're continuing to troubleshoot this issue. A _potential_ root cause has been identified.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>00:01</var> PDT</small><br><strong>Investigating</strong> - No sessions can be watched in the player. We are currently investigating.</p>tag:status.lookback.io,2005:Incident/71350952021-05-31T23:07:18-07:002021-05-31T23:23:42-07:00Users are unable to access player, dashboard, or settings<p><small>May <var data-var='date'>31</var>, <var data-var='time'>23:07</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>21:43</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Investigating</strong> - The backend service that powers multiple services in the Lookback platform is not responding. We are currently investigating and attempting to roll back to a working state.</p>tag:status.lookback.io,2005:Incident/68469952021-04-26T14:35:22-07:002021-04-26T14:35:22-07:00Experiencing intermittent issues when loading dashboard and player<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:35</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:31</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>14:29</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating.</p>tag:status.lookback.io,2005:Incident/67655262021-04-15T08:00:00-07:002021-04-16T01:44:00-07:00Slowdown in main backend – affecting dashboard, player, and participant's sessions<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Resolved</strong> - We had a near outage in one of our main backends, which affected queries from<br /><br />- the dashboard<br />- the player<br />- and participant clients (iOS, Android, Chrome)<br /><br />It was a "near" outage, since the service didn't go down, but had a major slowdown in responding to queries. Queries would be queued up and cause a snowball effect.<br /><br />We rolled back to an earlier version, and are now investigating the root cause.</p>tag:status.lookback.io,2005:Incident/62205692021-02-08T08:45:00-08:002021-02-08T09:16:27-08:00Recordings DB outage<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>08:45</var> PST</small><br><strong>Resolved</strong> - We are currently investigating this issue.</p>tag:status.lookback.io,2005:Incident/61272152021-01-27T06:42:08-08:002021-01-27T06:42:08-08:00Users are unable to start new sessions or end existing sessions<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>06:42</var> PST</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>06:30</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>06:29</var> PST</small><br><strong>Identified</strong> - Users are currently unable to start new sessions or end existing sessions. We are investigating.</p>tag:status.lookback.io,2005:Incident/52529542020-10-06T09:37:00-07:002020-10-07T13:57:09-07:00Some moderators running sessions with iOS Participate app users on iOS are unable to hear Participants wearing headphones<p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>09:37</var> PDT</small><br><strong>Resolved</strong> - Fixed in Participate iOS 4.3.2</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>12:31</var> PDT</small><br><strong>Update</strong> - We have tested a fix for the audio problems and have submitted a new iOS Participate release to Apple. We will update this incident once the review process has been completed and the app is available.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>12:15</var> PDT</small><br><strong>Identified</strong> - We have identified the root cause of the issue creating audio problems and have created a fix. A new iOS release is being tested. Once testing has completed we will submit the updated app for review by Apple.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>10:21</var> PDT</small><br><strong>Update</strong> - We are continuing our investigation to identify and resolve the root cause of this issue. In the mean time we recommend that customers pause testing on iOS devices until a fix has been made available. We will continue to update this page as work progresses.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>09:26</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>09:25</var> PDT</small><br><strong>Investigating</strong> - Customers have informed us that using the latest release of the iOS app they are sometimes unable to hear Participants on iOS 14 when the Participant is wearing headphones. We are currently investigating whether this is due to a recent app update as well as exact steps required to reproduce the problem.</p>tag:status.lookback.io,2005:Incident/40864342020-05-11T10:12:05-07:002020-05-11T10:12:05-07:00Some customers are unable to access the dashboard, player, and cannot carry out sessions<p><small>May <var data-var='date'>11</var>, <var data-var='time'>10:12</var> PDT</small><br><strong>Resolved</strong> - We are not seeing any further errors at this time.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>10:05</var> PDT</small><br><strong>Monitoring</strong> - We are now seeing very few errors for customers and are continuing to monitor the situation.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>09:46</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the inability for some customers to access the dashboard and run sessions due to an outage at a dependent service.</p>tag:status.lookback.io,2005:Incident/39217002020-04-20T12:06:49-07:002020-04-20T12:06:49-07:00Some customers are having issues loading the dashboard or videos<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:06</var> PDT</small><br><strong>Resolved</strong> - Errors have subsided and this incident has been marked as resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:45</var> PDT</small><br><strong>Monitoring</strong> - We've introduced a fix and are no longer encountering this behavior, we will continue to monitor the situation.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:37</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where some customers are unable to load their dashboard or videos.</p>tag:status.lookback.io,2005:Incident/34098502020-01-09T12:19:08-08:002020-01-09T12:19:08-08:00Users are unable to export recordings<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>12:19</var> PST</small><br><strong>Resolved</strong> - Issues with exporting for recordings and highlights have been addressed.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>11:58</var> PST</small><br><strong>Identified</strong> - We have identified the issue with exports and are working on deploying a fix.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>11:55</var> PST</small><br><strong>Investigating</strong> - Some users are unable to export their recordings and highlights, we are investigating the cause and will provide updates as available.</p>tag:status.lookback.io,2005:Incident/33958202020-01-06T09:02:05-08:002020-01-06T09:02:05-08:00Some users are unable to play recordings<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>09:02</var> PST</small><br><strong>Resolved</strong> - We have identified the issue and a fix has been deployed to address all issues.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>08:46</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>08:36</var> PST</small><br><strong>Investigating</strong> - Some users are unable to play recordings, are having issues starting sessions, and are seeing errors in the player, we are investigating now.</p>tag:status.lookback.io,2005:Incident/31159092019-10-17T13:03:50-07:002019-10-17T13:03:50-07:00Users are unable to start sessions<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>13:03</var> PDT</small><br><strong>Resolved</strong> - We have resolved this incident.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:39</var> PDT</small><br><strong>Monitoring</strong> - We have identified the root cause of the issue and are continuing to monitor the situation.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:29</var> PDT</small><br><strong>Update</strong> - Functionality should now be restored for customers. We are continuing to investigate the root cause of this issue.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:14</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of users being unable to start live, in-person, and self test sessions.</p>tag:status.lookback.io,2005:Incident/28577312019-08-13T14:35:04-07:002019-08-13T14:35:04-07:00Some users are unable to play recordings<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>14:35</var> PDT</small><br><strong>Resolved</strong> - We have addressed this issue and believe mitigation steps are in place, we will continue to monitor.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Monitoring</strong> - We're continuing to monitor the situation and have not encountered further issues.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>12:23</var> PDT</small><br><strong>Identified</strong> - We have identified the cause of playback issues and have addressed it for a majority of customers, we will continue to monitor the situation.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>12:07</var> PDT</small><br><strong>Update</strong> - This should be resolved for many users however we're continuing to investigate the root cause of this issue.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>11:28</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating some user's inability to play back recordings.</p>tag:status.lookback.io,2005:Incident/27524092019-07-23T13:22:01-07:002019-07-23T13:22:01-07:00Users may be unable to download recordings<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>13:22</var> PDT</small><br><strong>Resolved</strong> - We have confirmed that download functionality is now fully operational for all users again.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>13:01</var> PDT</small><br><strong>Monitoring</strong> - We have addressed issues related to some recordings being unavailable for download and have confirmed that downloads are processing correctly again.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>12:29</var> PDT</small><br><strong>Identified</strong> - We have identified why some recordings cannot be downloaded and are working to address it now.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>12:01</var> PDT</small><br><strong>Investigating</strong> - Some users have notified us that they are unable to download recordings. We have confirmed this and are investigating now.</p>