Running Updates - Reporting Data Inconsistencies

Update - 21 Oct 2021

The replacement of the reporting extraction component has been successful, and we have not experienced any new issues since.

We are now running the scripts to fix the historical data over the last two months.

After this, we will run a fix for the introduced dates and correct those.

We expect this work to be completed by the end of next week (October 29th).

If you are experiencing issues and are not sure if they are related to this, you can raise a support ticket (https://support.snyk.io/hc/en-us/requests/new) and we will confirm.



Update - 14 Oct 2021 

We have replaced the component in our reporting that was causing the extraction process to not always run correctly. This means that from 14:00 UTC today (14th Oct 2021), current reporting information should be correct.

This means the Security and License issues current totals will be correct.

However, the issues over time and exposure window graphs and the new issues and fixed issues totals in the UI (and the issues counts over time function in the API) may continue to be incorrect until we have been able to fix all of the historic data.

We are currently working on a plan to remediate the historic data and will advise on a date we hope to complete this on or before Tuesday 19th October.


Again, your patience is appreciated. As a reminder, if you are experiencing issues and are not sure if they are related to this, you can raise a support ticket (https://support.snyk.io/hc/en-us/requests/new) and we will confirm.




Update - 12 Oct 2021

Snyk is experiencing issues with data consistency in the reporting functionality.

Below is the latest status and information on the situation.

Firstly, please be assured that:

1) This issue is only in our reporting function. Snyk products are working as normal in terms of identifying and notifying customers of their vulnerabilities. The issue is in the roll up of this data into reporting.

2) We understand this is a serious issue and are working hard to resolve it. We appreciate your patience and feedback.


Summary of the issue:

Due to our recent significant increase of customers, projects and issues, a component in our reporting process is, on some days, not processing the data from our main database into our reporting database as fast as we need it to. The result of this is that sometimes when data is rolled up in reporting it is not complete, but not always, and not always by the same amount. The result of this is inconsistencies in reporting.

More details:

The extraction service we use in reporting is at times not running correctly, resulting in instance when not all of the data needed was extracted. In a given snapshot of time this means that the issues appears to disappear and so we report it as fixed. When the extraction works again and previously missing issue 'reappear' we then assume them to be a new issue.

Areas of Snyk affected:

Our reporting API (https://snyk.docs.apiary.io/#reference/reporting-api), any API call starting snyk.io/api/v1/reporting, and the Summary tab of our Reporting Section in the UI.

In the UI specifically, issues counts may be impacted in terms of freshness, dips may been seen in the issues over time, graphs and exposure window graph and the new issues and fixed issues count may be inflated.

Solutions:
  • [Long-term] We have already begun work on replacing our reporting infrastructure. This will enable us to offer more flexibility and capability, building on some different technologies that have proven scaling capability significantly beyond that which we see today and into the future. However, this work is in progress and not due to be complete for a few months. 
  • [Short-term] We are currently actively investigating a number of possible options to stop this from happening. We have currently identified a viable solution, which we are implementing. We hope to have a plan by the end of Friday the 15th of October for resolving the problem.
We have updated our status page (https://status.snyk.io/) to reflect the current status of reporting and will continue to make updates there. You can also visit this webpage for updates.

If you are experiencing issues and are not sure if they are related to this, please raise a support ticket (https://support.snyk.io/hc/en-us/requests/new) and we will confirm.