Incident Summary:

On December 27th, 2021,12:35 AM GMT+01, webhooks subscriptions and webhooks history no longer appeared on the Developer dashboard for PandaDoc users. The configured integrations stopped working for a number of PandaDoc users.

The issue was resolved on January 6th, 2022, 4:00 PM GMT+01 by returning the missing webhook subscriptions and related webhook events history.

Customer Impact:

For some PandaDoc users, the configured integrations which use webhooks (Zapier, Salesforce, Hubspot, and any custom integrations) stopped working.

Missing subscriptions had later reappeared automatically for our clients who had integrations configured before August, 2021; for other users the subscriptions were returned between January 5th and 7th; webhook events history returned on the same dates.

We do not expect further issues related to the case, however, please note the potential scenarios below:

• If there are extra/duplicate subscriptions, please delete or deactivate these manually;
• If some subscriptions are missing or integrations are not working, we recommend re-creating the missing webhook subscription. Please reach out to support if you need assistance with this.

Also please note that webhooks that were not fired during the incident cannot be re-fired, thus document statuses in PandaDoc and in external systems (CRMs, custom applications, any other integrations) may be out of sync.

Preventative Measures:

At this point, we are reviewing our infrastructure and its configuration process to exclude the possibility of this issue occurring again.