Validating PayPal Webhooks Offline (Almost)

Update 4/9/2021: Now includes a Python example!

PayPal offers the abillity for you to receive webhooks for transaction notifications.  This isn’t exactly new — it was introduced with the REST APIs back in 2013(-ish?).  But for those of you still using IPN, you should know that webhooks has some big advantages over IPN.

First, webhooks provides a more structured way to find out exactly what happened.  Each webhook event includes an event_type — so you can figure out just by looking at that what happened.  Second, PayPal provides APIs to let you create webhooks, retrieve events, replay events, and even see samples of the different event types.  Third, you can have more than one webhook per account — this is a big advantage over IPN, which would only let you have one IPN listener per account.  There are more advantages, but that’s not what I want to focus on for this post.

As with IPN, there’s the question of “how do I know that this webhook event is genuine?”  PayPal has the Verify Webhook Signature API to do this — but what if I want to do it without making another API call?  There is actually a way to do this.

PayPal crytographically signs the webhook event when it’s sent to you — and (almost) all the information that you need to verify the signature (as well as the signature itself) are included in the HTTP post.  Let’s look at the different elements:

First, there are a number of HTTP headers that PayPal includes when it makes the post to your site:

  • PAYPAL-TRANSMISSION-ID is a unique ID (more specifically, a UUID) for the transmission.
  • PAYPAL-TRANSMISSION-TIME is the time when PayPal initiated the transmission of the webhook, in ISO 8601 format.
  • PAYPAL-TRANSMISSION-SIG is the Base64-encoded signature.
  • PAYPAL-CERT-URL is the URL to the certificate which corresponds to the private key that was used to generate the signature.
  • PAYPAL-AUTH-ALGO is the algorithm that was used to generate the signature.  (I’ve only ever seen PayPal use SHA256withRSA, but it’s possible that PayPal might switch in the future if/when SHA256 is broken.)

And lastly, there’s the body of the HTTP post itself — the webhook JSON.

How do you validate the signature?  Well, the signature isn’t based off the body of the webhook itself; rather, it’s based off the following string:

<transmissionid>|<timestamp>|<webhookid>|<crc>

  • <transmissionid> and <timestamp> are the verbatim values given in the PAYPAL-TRANSMISSION-ID and PAYPAL-TRANSMISSION-TIME HTTP headers, respectively.
  • <webhookid> is the ID that PayPal assigned to your webhook when you created it.  You can find this a few different places:
    • If you used the Webhooks API to create the webhook, this would have been the value of /id in the response.
    • You can use the List All Webhooks API to see the webhooks you have registered.  You can grab the webhook ID from there.
    • You can also see your webhooks from developer.paypal.com.  (Go to the Dashboard, then the My Apps & Credentials page.  Scroll down to the REST API Apps section and find your REST app.  Click on it, then scroll down to the “Sandbox Webhooks” or “Live Webhooks” section.  The webhook ID will be displayed in the “Webhook ID” column.)

  • <crc> is the CRC32 of the body of the HTTP post (e.g., the raw, unaltered webhook JSON), and expressed as a base 10, unsigned integer.

Let’s look at a quick example.  Suppose this is what PayPal posted to you.  (This is an actual webhook I received, albeit slightly modified:)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
POST /paypal-webhook-handler HTTP/1.1
Accept: */*
PAYPAL-TRANSMISSION-ID: 6e3b26a0-9287-11e7-ac1e-6b62a8a99ac4
PAYPAL-TRANSMISSION-TIME: 2017-09-05T22:13:22Z
PAYPAL-TRANSMISSION-SIG: Hdwao5lBJ9R6IX1JgOuyKdA1oyw2edUGhJ4ovHDqA7XXJS9BvVMQJL/51nXzVu5mI0iDTfkXk8XophZnkXB+srwtdxkjjIeW+fNMsp9qsI64gywFK40AqD6YvyIbbBhGm8SPecfVGOWYeAy16jHx/6F6e/wxeSClM8XcQMrp6jwy5NZRyD/0BsijjI6KQedonrg6jiq3BqrzbvIyuMW32DtiqXPg/2Inog0ZItpTmHDu71Xci6zgiTmb4BsKHX/vyBwRZE6wo4NwtiP1NoNr+l32H3JCAvOvjvPRBAFbaG+SKjUGn3NL8nV3EQGXV20rJI4l5wWRYh5C4DBzppXgkA==
PAYPAL-AUTH-VERSION: v2
PAYPAL-CERT-URL: https://api.sandbox.paypal.com/v1/notifications/certs/CERT-360caa42-fca2a594-aecacc47
PAYPAL-AUTH-ALGO: SHA256withRSA
Content-Type: application/json
User-Agent: PayPal/AUHD-211.0-33754056
Host: www.bahjeez.com
correlation-id: 42e699ec204cc
CAL_POOLSTACK: amqunphttpdeliveryd:UNPHTTPDELIVERY*CalThreadId=0*TopLevelTxnStartTime=15e541b2066*Host=slcsbamqunphttpdeliveryd3002
CLIENT_PID: 21282
Content-Length: 965

{"id":"WH-36687761JL817053T-6SY78077XN391202M","event_version":"1.0","create_time":"2017-09-05T22:13:22.000Z","resource_type":"payouts","event_type":"PAYMENT.PAYOUTSBATCH.SUCCESS","summary":"Payouts batch completed successfully.","resource":{"batch_header":{"payout_batch_id":"2AZEQUD4YPAEJ","batch_status":"SUCCESS","time_created":"2017-09-05T22:12:56Z","time_completed":"2017-09-05T22:13:22Z","sender_batch_header":{"sender_batch_id":"2017021897"},"amount":{"currency":"USD","value":"1.0"},"fees":{"currency":"USD","value":"0.0"},"payments":1},"links":[{"href":"https://api.sandbox.paypal.com/v1/payments/payouts/2AZEQUD4YPAEJ","rel":"self","method":"GET"}]},"links":[{"href":"https://api.sandbox.paypal.com/v1/notifications/webhooks-events/WH-36687761JL817053T-6SY78077XN391202M","rel":"self","method":"GET"},{"href":"https://api.sandbox.paypal.com/v1/notifications/webhooks-events/WH-36687761JL817053T-6SY78077XN391202M/resend","rel":"resend","method":"POST"}]}

In this example:

  • <transmissionid> would be 6e3b26a0-9287-11e7-ac1e-6b62a8a99ac4.
  • <timestamp> would be 2017-09-05T22:13:22Z.  (Remember — use the exact value that PayPal passed to you.  Don’t try to change this into your local timezone or change its format.)
  • <id> would be my webhook ID, which in this case is 2R269424P6803053B.
  • <crc> would be 1330495958.

Which means that the string PayPal signed would be:

6e3b26a0-9287-11e7-ac1e-6b62a8a99ac4|2017-09-05T22:13:22Z|2R269424P6803053B|1330495958

The last thing to do is to verify the signature.

So far, we’ve been able to do everything without pulling in any external resources, but unfortunately that ends here.  To verify the signature, we need a copy of the certificate that corresponds to the private key that was used to generate the signature.  PayPal provided us a URL where we can fetch that certificate (in the PAYPAL-CERT-URL header) — we’ll need to fetch a copy of that.  Bad news is that means pulling in an outside resource (which will slow down the verification process); good news is that the certificates don’t change that often (in fact, I’ve only ever seen PayPal use one certificate), so you can cache the certificate for future use.

The only thing that’s left is to verify the signature against the string we formed above.  I won’t get into specifics on this — each language has their own way of pulling this off.  Java has built-in classes and methods that will help you out with this; for PHP, you can use the built-in OpenSSL functions to help you out.

If the signature verification is successful, and you trust the certificate that was used to sign the message, then you can be sure that the message you’re receiving is genuine.

Side note: there’s a weakness here in that CRC32 is used to hash the actual message body.  CRC32 isn’t a secure hashing algorithm (not sure it was ever meant to be), so I’m not sure why PayPal decided to use that instead of something like SHA256.  (Edit: I’m told something new is in the works.)

Anywho…I wrote a couple of example implementations.  Note that these examples don’t cache the certificates — you’ll need to figure out how to do that on your own.  But, feel free to use what I have.

First, a PHP example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
<!--?php

$headers = apache_request_headers();

$cert_url = $headers[ 'PAYPAL-CERT-URL' ];
$transmission_id = $headers[ 'PAYPAL-TRANSMISSION-ID' ];
$timestamp = $headers[ 'PAYPAL-TRANSMISSION-TIME' ];
$algo = $headers[ 'PAYPAL-AUTH-ALGO' ];
$signature = $headers[ 'PAYPAL-TRANSMISSION-SIG' ];
$webhook_id = "09A5628866464184S"; // Replace with your webhook ID

$webhook_body = file_get_contents( 'php://input' );

try {
  if( verify_webhook( $cert_url, $transmission_id, $timestamp, $webhook_id, $algo, $signature, $webhook_body ) ) {
    // Verification succeeded!
  } else {
    // Verification failed!
  }
} catch(Exception $ex) {
  // Something went wrong during verification!
}

/**
 * Verifies a webhook from PayPal.
 *
 * @param string $cert_url The URL of the certificate that corresponds to the
 *                         private key that was used to sign the certificate.
 *                         When the webhook is posted to you, PayPal provides
 *                         this in the PAYPAL-CERT-URL HTTP header.
 * @param string $transmission_id The transmission ID for the webhook event.
 *                                When the webhook is posted to you, PayPal
 *                                provides this in the PAYPAL-TRANSMISSION-ID
 *                                HTTP header.
 * @param string $timestamp The timestamp of when the webhook was sent. When
 *                          the webhook is posted to you, PayPal provides
 *                          this in the PAYPAL-TRANSMISSION-TIME HTTP header.
 * @param string $webhook_id The webhook ID assigned to your webhook, as
 *                           defined in your developer.paypal.com dashboard.
 *                           If you used the Create Webhook API to create your
 *                           webhook, this ID was returned in the response to
 *                           that call.
 * @param string $signature_algorithm The signature algorithm that was used to
 *                                    generate the signature for the webhook.
 *                                    When the webhook is posted to you, PayPal
 *                                    provides this in the PAYPAL-AUTH-ALGO
 *                                    HTTP header.
 * @param string $webhook_body The byte-for-byte body of the request that
 *                             PayPal posted to you.
 *
 * @return bool Returns true if the webhook could be successfully verified, or
 *              false if it was not.
 *
 * @throws Exception if an error occurred while attempting to verify the
 *     webhook.
 */

function verify_webhook( $cert_url, $transmission_id, $timestamp, $webhook_id, $signature_algorithm, $signature, $webhook_body ) {
  // This is used to translate the hash methods provided by PayPal into ones that
  // are known by OpenSSL...right now the only one we've seen PayPal use is 'SHA256withRSA'
  $known_hash_methods = [
    'SHA256withRSA' => 'sha256WithRSAEncryption'
  ];

  if( array_key_exists( $signature_algorithm, $known_hash_methods ) ) {
    $algo = $known_hash_methods[ $signature_algorithm ];
  } else {
    $algo = $signature_algorithm;
  }

  // Make sure OpenSSL knows how to handle this hash method
  $openssl_algos = openssl_get_md_methods( true );
  if( !in_array( $algo, $openssl_algos ) ) {
    throw new Exception( "OpenSSL doesn't know how to handle message digest algorithm "$algo"" );
  }

  // Fetch the cert -- we have to use cURL for this because PHP's built-in
  // capability for opening http/https URLs uses HTTP 1.0, which PayPal doesn't
  // support
  $curl = curl_init( $cert_url );
  curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
  $cert = curl_exec( $curl );

  if( false === $cert ) {
    $error = curl_error( $curl );
    curl_close( $curl );
    throw new Exception( "Failed to fetch certificate from server: $error" );
  }

  curl_close( $curl );

  // Parse the certificate
  $x509 = openssl_x509_read( $cert );
  if( false === $x509 ) {
    throw new Exception( "OpenSSL was unable to parse the certificate from PayPal\n" );
  }

  // Calculate the CRC32 of the webhook body
  $crc = crc32( $webhook_body );

  // Assemble the string that PayPal actually signed
  $sig_string = sprintf( '%s|%s|%s|%u', $transmission_id, $timestamp, $webhook_id, $crc );

  // Base64-decode PayPal's signature
  $decoded_signature = base64_decode( $signature );

  // Fetch the public key from the certificate
  $pkey = openssl_pkey_get_public( $cert );
  if( false === $pkey ) {
    throw new Exception( "Failed to get public key from PayPal certificate\n" );
  }

  // Verify the signature
  $verify_status = openssl_verify( $sig_string, $decoded_signature, $pkey, $algo );

  openssl_x509_free( $x509 );

  // Check the status of the verification
  if( $verify_status == 1 ) {
    return true;
  } else if( $verify_status == -1 ) {
    throw new Exception( "Error occurred while trying to verify webhook signature" );
  } else {
    return false;
  }
}

And second, a Java servlet (written for Apache Tomcat 8):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
package com.bahjeez;

import java.io.IOException;
import java.net.URL;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.Signature;
import java.security.SignatureException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.Base64;
import java.util.stream.Collectors;
import java.util.zip.CRC32;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * Servlet implementation class ValidateWebhook
 */

@WebServlet(name = "ValidateWebhook", urlPatterns = { "/ValidateWebhook" })
public class ValidateWebhook extends HttpServlet {
    private static final long serialVersionUID = 1L;

    /**
     * @see HttpServlet#HttpServlet()
     */

    public ValidateWebhook() {
        super();
    }

    public static boolean verifySignature(String webhookBody, String certUrl, String transmissionId, String transmissionTimestamp, String authAlgo, String signature, String webhookId) throws Exception {

        CertificateFactory fact;
        try {
            fact = CertificateFactory.getInstance("X.509");
        } catch (CertificateException e) {
            throw new Exception("Failed to construct CertificateFactory object");
        }

        URL url = new URL(certUrl);
        X509Certificate cer;
        try {
            cer = (X509Certificate) fact.generateCertificate(url.openStream());
        } catch (CertificateException e) {
            throw new Exception("Failed to create X509Certificate object");
        }

        Signature sigAlgo;
        try {
            sigAlgo = Signature.getInstance(authAlgo);
        } catch (NoSuchAlgorithmException e) {
            throw new Exception("Failed to initialize Signature object (maybe unrecognized signature algorithm?)");
        }

        CRC32 crc = new CRC32();
        crc.update(webhookBody.getBytes());

        String verifyString = transmissionId + "|" + transmissionTimestamp + "|" + webhookId + "|" + crc.getValue();

        try {
            sigAlgo.initVerify(cer);
        } catch (InvalidKeyException e) {
            throw new Exception("Failed to initialize signature verification");
        }

        try {
            sigAlgo.update(verifyString.getBytes());
        } catch (SignatureException e) {
            throw new Exception("Failed to update signature verification object");
        }

        byte[] actualSignature = Base64.getDecoder().decode(signature);
        try {
            return sigAlgo.verify(actualSignature);
        } catch (SignatureException e) {
            throw new Exception("Failed to verify signature");
        }
    }

    /**
     * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
     */

    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        String webhookBody = request.getReader().lines().collect(Collectors.joining());
        String certUrl = request.getHeader("PAYPAL-CERT-URL");
        String transmissionId = request.getHeader("PAYPAL-TRANSMISSION-ID");
        String transmissionTimestamp = request.getHeader("PAYPAL-TRANSMISSION-TIME");
        String authAlgo = request.getHeader("PAYPAL-AUTH-ALGO");
        String signature = request.getHeader("PAYPAL-TRANSMISSION-SIG");
        String webhookId = "24N36863A45710219";

        try {
            if(this.verifySignature(webhookBody, certUrl, transmissionId, transmissionTimestamp, authAlgo, signature, webhookId)) {
                response.setStatus(200);
            } else {
                response.setStatus(400);
                response.getWriter().write("Failed to verify signature on incoming webhook");
            }
        } catch(Exception ex) {
            response.setStatus(500);
            response.getWriter().write("Failed to verify signature due to internal error: " + ex.getMessage());
        }
    }

}

And finally, a third example written for Python 3. This example will need the cryptography library (pip install cryptography) and the requests library (pip install requests).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
#!/usr/bin/env python3

import http.server as SimpleHTTPServer
import socketserver as SocketServer
import logging
import pprint
import zlib
import json
import os.path
import requests
from cryptography import x509
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat import backends
import base64

PORT = 8000

# This example caches the PayPal signing certs, as they don't change very often
CERT_FILE_CACHE = './cert_cache.json'

# Set to your webhook ID
#
# If you created the webhook through the developer.paypal.com site, the webhook
# ID is shown on the details page for your REST API application.
#
# If you created the webhook through the POST /v1/notification/webhooks API,
# the ID is returned in the response.  You can also use the
# GET /v1/notifications/webhooks API if you forgot the webhook ID.
WEBHOOK_ID = ''

class GetHandler(
        SimpleHTTPServer.SimpleHTTPRequestHandler
        ):

    def fetchPayPalCert(self, url):
        # Assumes that the cert doesn't already exist in the cache
        # Also assumes that the cert it's fetching is genuine
        r = requests.get(url)
        if r.status_code >= 400:
            raise Exception("Unable to fetch certificate")

        certdata = r.text

        if not os.path.exists(CERT_FILE_CACHE):
            cache = {'certs': [{'url': url, 'cert': certdata}]}
            cache_file = False
            try:
                cache_file = open(CERT_FILE_CACHE, 'w')
                json.dump(cache, cache_file)
            except:
                # Just ignore it, we made a best effort
                pass

            cache_file.close()

            return certdata
        else:
            try:
                cache_file = open(CERT_FILE_CACHE)
                cache_json = json.load(cache_file)
                cache_file.close()

                new_cache = {'certs':[]}

                if 'certs' in cache_json:
                    for cert in cache_json['certs']:
                        if 'url' in cert and 'cert' in cert:
                            new_cache['certs'].append(cert)

                new_cache['certs'].append({'url':url, 'cert':certdata})

                cache_file = open(CERT_FILE_CACHE, 'w')
                json.dump(new_cache, cache_file)
                cache_file.close()
            except:
                # Just ignore it, we made a best effort
                pass

        return certdata

    def getPayPalCert(self, url):
        if not os.path.exists(CERT_FILE_CACHE):
            return self.fetchPayPalCert(url)

        cache_json = False
        cache_file = False
        try:
            cache_file = open(CERT_FILE_CACHE)
            cache_json = json.load(cache_file)
        except:
            cache_file.close()
            return self.fetchPayPalCert(url)

        cache_file.close()

        if "certs" in cache_json:
            for cert in cache_json["certs"]:
                if "url" in cert and cert["url"] == url and "cert" in cert:
                    return cert["cert"]

        return self.fetchPayPalCert(url)

    def do_POST(self):
        self.close_connection = True

        # Check for required headers
        required_headers = (
            'Content-Length',
            'Content-Type',
            'PAYPAL-TRANSMISSION-ID',
            'PAYPAL-TRANSMISSION-TIME',
            'PAYPAL-TRANSMISSION-SIG',
            'PAYPAL-CERT-URL',
            'PAYPAL-AUTH-ALGO'
            )

        for header in required_headers:
            if header not in self.headers:
                self.send_response(400)
                self.end_headers()
                self.wfile.write(("Required header missing from request: " + header).encode())
                return

        content_type = self.headers['Content-Type']
        if content_type != "application/json":
            self.send_response(400)
            self.end_headers()
            self.wfile.write("Invalid Content-Type".encode())
            return

        content_length = int(self.headers['Content-Length'])
        transmission_id = self.headers['PAYPAL-TRANSMISSION-ID']
        transmission_time = self.headers['PAYPAL-TRANSMISSION-TIME']
        transmission_sig = base64.b64decode(self.headers['PAYPAL-TRANSMISSION-SIG'])
        cert_url = self.headers['PAYPAL-CERT-URL']
        auth_algo = self.headers['PAYPAL-AUTH-ALGO']
        body = self.rfile.read(content_length)

        if auth_algo != 'SHA256withRSA':
            self.send_response(400)
            self.end_headers()
            self.wfile.write(("Don't know how to handle signing algorithm " + auth_algo).encode())
            return

        checksum = zlib.crc32(body)
        verify_str = transmission_id + "|" + transmission_time + "|" + WEBHOOK_ID + "|" + format(checksum)
        cert_data = self.getPayPalCert(cert_url)

        cert = x509.load_pem_x509_certificate(cert_data.encode('ascii'), backend=backends.default_backend())
        public_key = cert.public_key()

        try:
            public_key.verify(
                transmission_sig,
                verify_str.encode("ascii"),
                padding.PKCS1v15(),
                hashes.SHA256()
            )
        except:
            self.send_response(400)
            self.end_headers()
            self.wfile.write("Signature verification failed".encode())
            return

        # If you've made it to this point, then verification succeeded -- you can proceed to
        # parse out the webhook
        self.send_response(204)
        self.end_headers()

Handler = GetHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)

httpd.serve_forever()

Leave a Reply

Your email address will not be published. Required fields are marked *