At the end of this guide, you will be able to set up a Gmail API webhook. Once this is set up, you will be able to get the details of any email that is hitting the mailbox you are monitoring. You can use this for various purposes.
For example: Maybe you want to turn any incoming emails into support tickets. Or maybe you want to read the contents of the email, and pass it through a machine-learning model that sends someone else a notification if it’s an “angry customer email”.
There are many different possibilities. But, in order to get things set up, you will have to navigate various steps on the Google Cloud Platform. This is an in-depth guide on how to set it all up.
Set Up Project On the Google Cloud Platform (Authentication)
The first step is that you need to set up a new project on GCP so that the user who owns the email account gives your webhook permission to read the incoming emails.
Creating A New Project
Go to Cloud Resource Manager
Here you need to click on the “Create Project” button and create a new project.
Once your project is created, by default GCP usually makes it the default and selected project. But, just in case you need to, you can always select it from the top menu.
Getting Credentials & Adding Test Users
The next step is that we need a “credentials JSON file”. So, use the side menu of GCP and head over to the credentials page as shown in the screenshot below.
Next, we are going to try and create OAuth credentials. This will be used for the “Login With Google” authentication screen.
At this point, you might get an error like this:
The above error is basically saying: “Hey when we show the login screen to the user, we are going to provide your app name, your logo, your privacy policy link, etc. So, you need to give me that info first.” So, click on the “Configure Consent Screen” button to go to a form that asks you for the relevant info.
The first step is to choose if the application is: “Internal” or “External”. If you are going to set this up only for your company’s GSuite Email account you can choose “Internal”. The process is simpler. Google does not want to verify your app. But, if you want this to work for any GMail email ID, choose “external”. In this guide, we will go down the “external” flow. Because that one is a little bit more elaborate.
Once you hit the “Create” button, you will reach a new screen. Here you need to fill in some details. Don’t worry too much about these right now. Maybe if you need to “verify” your app in the future, you can come back to this and update this info. For now, I have just filled in the info as shown in the screenshot below.
The privacy policy and terms URL are of something else. But it does not matter right now. For the logo, I just found and put a NodeJS logo. I will put it below. You can download it and put it if you need to use it.
Hit “Save And Continue” when you are done. There are other steps in the flow. But, at this stage, they are not important. There is only one more thing that is important. You need to add test users. Only these users will be able to give Gmail Access. So, find that section and fill it up with your test users email ID.
Make sure you add any users whose Gmail account you want webhook access to. Without this, things will not work. If you release this application to the general public, you will need to go through the verification process. Once that is done, any user will be able to use your app. But currently, it is limited only to test users.
Downloading Your Credentials JSON File
So, with the OAuth Consent screen work done, you can go back to Credentials, and once again try to create OAuth Credentials.
Now, you should be sent to a new screen like the one below. Choose “Desktop App”.
Once the credentials are created, download the JSON file by clicking on the “Download JSON” button:
Turning On The Gmail API For The Project
Next, from the side menu, we need to turn on access to the Gmail API. You can click on the “Enable APIs” link on the side menu like so…
And then, search for the Gmail API and click on the enable button.
Testing Everything Out By Getting The Gmail Inbox Labels
We are ready to test if we actually have access to Gmail. We are going to run the test, by following the instructions from the official Google docs.
Below are the rough steps:
# create a new app folder $> mkdir my-new-app # run npm init to create a new app $> npm init # Install all the required packages $> npm install googleapis@105 @google-cloud/local-auth@2.1.0 --save
In the above application folder, we need to put in the credentials JSON file we got from the above step. We need to save it as “credentials.json”. This is because the script below expects that.
Finally, create a script called “index.js” with the following code.
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const {authenticate} = require('@google-cloud/local-auth');
const {google} = require('googleapis');
// If modifying these scopes, delete token.json.
const SCOPES = ['https://www.googleapis.com/auth/gmail.readonly'];
// The file token.json stores the user's access and refresh tokens, and is
// created automatically when the authorization flow completes for the first
// time.
const TOKEN_PATH = path.join(process.cwd(), 'token.json');
const CREDENTIALS_PATH = path.join(process.cwd(), 'credentials.json');
/**
* Reads previously authorized credentials from the save file.
*
* @return {Promise<OAuth2Client|null>}
*/
async function loadSavedCredentialsIfExist() {
try {
const content = await fs.readFile(TOKEN_PATH);
const credentials = JSON.parse(content);
return google.auth.fromJSON(credentials);
} catch (err) {
return null;
}
}
/**
* Serializes credentials to a file comptible with GoogleAUth.fromJSON.
*
* @param {OAuth2Client} client
* @return {Promise<void>}
*/
async function saveCredentials(client) {
const content = await fs.readFile(CREDENTIALS_PATH);
const keys = JSON.parse(content);
const key = keys.installed || keys.web;
const payload = JSON.stringify({
type: 'authorized_user',
client_id: key.client_id,
client_secret: key.client_secret,
refresh_token: client.credentials.refresh_token,
});
await fs.writeFile(TOKEN_PATH, payload);
}
/**
* Load or request or authorization to call APIs.
*
*/
async function authorize() {
let client = await loadSavedCredentialsIfExist();
if (client) {
return client;
}
client = await authenticate({
scopes: SCOPES,
keyfilePath: CREDENTIALS_PATH,
});
if (client.credentials) {
await saveCredentials(client);
}
return client;
}
/**
* Lists the labels in the user's account.
*
* @param {google.auth.OAuth2} auth An authorized OAuth2 client.
*/
async function listLabels(auth) {
const gmail = google.gmail({version: 'v1', auth});
const res = await gmail.users.labels.list({
userId: 'me',
});
const labels = res.data.labels;
if (!labels || labels.length === 0) {
console.log('No labels found.');
return;
}
console.log('Labels:');
labels.forEach((label) => {
console.log(`- ${label.name}`);
});
}
authorize().then(listLabels).catch(console.error);
This code is directly from the official google docs. When you run the script from the console, the following will happen:
- The script will open up the browser with a “Login With Google” type screen.
- The app that you set up above will be used. All the details you set up in the OAuth Consent screen will be shown to the user.
- Make sure you log in with the “test user” that you specified above.
- The user will be asked to confirm that he/she really does want to give access to Gmail.
- Finally, once the user gives permission, a “token.json” file will be created in the folder. (This will be used in the future requests. Login and consent will not be required for all actions.)
- Then a Gmail API to list the labels in the inbox will be called. (See line 73 above)
- If all goes properly, the labels will all be printed on the console.
If you see the labels, congrats! Everything is set up right! You are ready for the next step!
The “token.json” file that got created here is important. So, don’t delete it.
Set Up “Pub / Sub” To Receive All Changes To The Mail Box
First, let me start by sharing the official docs on this topic.
So, in order to set up a webhook and subscribe to all the things that happen in the mailbox, you have to use another GCP service called “Pub / Sub”. Which stands for “Publish-Subscribe” as you are probably aware.
What Is “Pub / Sub”?
The idea is simple. When something happens with the email account (for example a new email), a notification is sent to the “Pub / Sub” service. In order words, a notification is “Published”. Now, you can set up many different “Subscribers” who want to be notified when these events happen. In our case, we are going to set things up such that the event is “pushed” to a “webhook” that is subscribed.
Below is a video that explains the Pub-Sub design pattern in brief.
Adding “Pub / Sub” To The Project
We are going to add “Pub / Sub” to our project next and configure a webhook. So, the first step is to look for “Pub / Sub” in the search bar with the project selected.
Next, we are going to create a “Topic”. A “Topic” as the name suggests is just a way to organize things. All the Gmail notifications will come under this this “Topic” we create.
Next, fill out the details in the topic creation form. You just need to choose a name and leave the rest of the things as is.
As soon as you create the Topic, a “Subscription” will also be created automatically. You will be sent to the “Subscription” page. If you are not sent automatically, you can click on the link and go there.
Converting “Pub / Sub” From Pull To Push & Using Webhook.site To Set Up A Test Webhook
Now, we need to tell GCP that whenever something new happens, we need to hit a webhook URL. So, we will start to do that by editing the Subscription.
Now, we are going to change the “Delivery Type” of the subscription to “Push” and also put in the webhook URL. In your case, you will most likely put in an actual URL where the changes to the Gmail Mailbox will be received. But, for the sake of this article, we are going to use https://webhook.site/
Using Webhook.site, we instantly get a webhook URL that we can use to test things out. We will just stick in the URL we get for now and monitor all the things that GCP sends us.
With that done, the Webhook URL and Push settings are now done.
Note About Keeping Your Subscription Alive: The official google docs say:
You must re-call watch() at least every 7 days or else you will stop receiving updates for the user. We recommend calling watch() once per day. The watch() response also has an expiration field with the timestamp for the watch expiration.
– The Docs
This needs to be kept in mind when you take your application to production.
Allowing Gmail To Post To The Pub/Sub Topic That Was Just Set Up
Now, we need to allow Gmail to publish things to our Topic. Without giving this permission, Gmail will not have the authority required to publish messages on the Topic we just set up.
In order to do this, go one step back to the “Topic” screen. In the tab on the right, you should see a “Permissions” section. There we are going to add permissions as follows via the “Add Principle” button.
In the “New Principles” box enter in: gmail-api-push@system.gserviceaccount.com
And in the “Assign Roles” search and select: Pub / Sub Publisher
Once you do this, you are done. GCP & Gmail have all the permissions set up correctly.
Connecting Gmail To The Pub / Sub Topic That Was Just Set Up
Now that everything is set up, we need to run a little script to tell Gmail to start sending the notifications. Below is the script you can put in a file and run.
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const { google } = require('googleapis');
const TOKEN_PATH = path.join(process.cwd(), 'token.json');
const TOPIC_NAME = 'projects/project_name/topics/TopicName';
// Load the credentials from the token.json file
async function loadSavedCredentialsIfExist() {
try {
const content = await fs.readFile(TOKEN_PATH);
const credentials = JSON.parse(content);
return google.auth.fromJSON(credentials);
} catch (err) {
return null;
}
}
// Connect to Pub Sub
async function connectPubSub(auth) {
const gmail = google.gmail({ version: 'v1', auth });
const res = await gmail.users.watch({
userId: 'me',
requestBody: {
'labelIds': ['INBOX'],
topicName: TOPIC_NAME
},
});
console.log(res);
}
// Run the script
(async () => {
let cred = await loadSavedCredentialsIfExist();
await connectPubSub(cred);
})();
On line 7, you will need to replace the “TOPIC_NAME” with your Topic name. You can get this from the topic page of GCP. It looks something like…
projects/project_name/topics/TopicName
Besides this, if you look at line 6, you will see that this script expects that the authentication process is done. And there is a file created called “token.json” in the current folder. The contents of the file should look something like this:
{
"type":"authorized_user",
"client_id":"271119457041-SOME-THING.apps.googleusercontent.com",
"client_secret":"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",
"refresh_token":"1//0gEqBKVE4crbWC-SOME-THING-ToSFBBQgS4PWmg1VEELQEvGFdFR2BY3Vosy7rge76w"
}
On line 30 you can see that we have outputted the response we get from the API. The response will be a 200 OK response if everything got done properly. And, in the response you should see a “data” field that looks something like this:
{
historyId: '8634261',
expiration: '1671717033024'
}
Note: Pub / Sub Is A Paid Service
You can look up the Pub-Sub pricing here
It’s complex to understand what’s going on with the pricing. But, it’s important to note that they do charge you money as a function of messages sent. So, remember to be prepared for that, or delete the “Topic” as soon as you are done.
For testing purposes, you should be able to use it with the GCP free credits.
A Look At The Type Of Events Your Webhook Gets
What You Get In The Webhook
If you go back to your Webhook URL and see what has been received there, you will see that a POST request was made to the URL. And the data sent was something like…
{
"message": {
"data": "eyJlbWFpbEFkZHJlc3MiOiJzb21lLmVtYWlsQGdtYWlsLmNvbSIsImhpc3RvcnlJZCI6ODYzNDI2MX0=",
"messageId": "6458754830483477",
"message_id": "6458754830483477",
"publishTime": "2022-12-15T13:50:32.949Z",
"publish_time": "2022-12-15T13:50:32.949Z"
},
"subscription": "projects/some-project/subscriptions/some-topic"
}
Here is what it looks like at Webhook.site
As you can see, there is a lot of data. The important info you need is base 64 encoded and stored in the “data” key. If you base 64 decode the same, you get something like:
{
"emailAddress": "some.email@gmail.com",
"historyId": 8634261
}
The “email address” is just letting you know which mailbox is this message regarding. There is also a historyId. We are going to see how to make use of that next in order to get the details of emails sent and received etc.
History IDs & How To Use Them
You can think of histoyIds as if they are “points in time”. When we give the Gmail API a particular historyId, it gives back to us a JSON object with whatever has happened after that point. This may be things like, “a message came in” or “a message was moved to trash”.
In order to pass the historyId and ask for a response, below is a sample script:
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const { google } = require('googleapis');
const TOKEN_PATH = path.join(process.cwd(), 'token.json');
// Load the credentials from the token.json file
async function loadSavedCredentialsIfExist() {
try {
const content = await fs.readFile(TOKEN_PATH);
const credentials = JSON.parse(content);
return google.auth.fromJSON(credentials);
} catch (err) {
return null;
}
}
// Function to log the data object to the console
function logCompleteJsonObject(jsonObject) {
console.log(JSON.stringify(jsonObject, null, 4));
}
// Get history details based on history ID
async function getHistory(auth, historyId) {
const gmail = google.gmail({ version: 'v1', auth });
const res = await gmail.users.history.list({
userId: 'me',
startHistoryId: historyId
})
// The main part of the response comes
// in the "data" attribute.
logCompleteJsonObject(res.data);
}
// Run the script
(async () => {
let cred = await loadSavedCredentialsIfExist();
let historyId = 8631681;
await getHistory(cred, historyId);
})();
And below is the type of response you get:
{
"history": [
{
"id": "8631683",
"messages": [
{
"id": "18510ea757da1719",
"threadId": "18510ea757da1719"
}
]
},
{
"id": "8631697",
"messages": [
{
"id": "18510ea757da1719",
"threadId": "18510ea757da1719"
}
],
"labelsAdded": [
{
"message": {
"id": "18510ea757da1719",
"threadId": "18510ea757da1719",
"labelIds": [
"UNREAD",
"IMPORTANT",
"CATEGORY_PERSONAL",
"TRASH",
"INBOX"
]
},
"labelIds": [
"TRASH"
]
}
]
},
],
"nextPageToken": "08268086677003141834",
"historyId": "8638477"
}
As you can see, “history” is an array. And that array has 2 types of things. One is where a message seems to have come in. The other one is “labelsAdded”. It lets you know that the message was moved to the trash in the above case.
But the actual contents of the message are still not got. So, in order to get that, check the next section.
How To Get The Details Of An Email Message
After looking at the history, you should be able to get message ids that you want to look at. They look something like this: 18510ea757da1719
So, we can run a script like the one below to get the details of the message:
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const { google } = require('googleapis');
const TOKEN_PATH = path.join(process.cwd(), 'token.json');
// Load the credentials from the token.json file
async function loadSavedCredentialsIfExist() {
try {
const content = await fs.readFile(TOKEN_PATH);
const credentials = JSON.parse(content);
return google.auth.fromJSON(credentials);
} catch (err) {
return null;
}
}
// Function to log the data object to the console
function logCompleteJsonObject(jsonObject) {
console.log(JSON.stringify(jsonObject, null, 4));
}
// Call the API to get message
async function getMessage(auth, messageId) {
const gmail = google.gmail({ version: 'v1', auth });
const res = await gmail.users.messages.get({
userId: 'me',
id: messageId
})
logCompleteJsonObject(res.data);
}
// Run the script
(async () => {
let cred = await loadSavedCredentialsIfExist();
let messageId = '18510ea757da1719';
await getMessage(cred, messageId);
})();
And in response, we will get a JSON object that looks something like this:
{
"id": "18510ea757da1719",
"threadId": "18510ea757da1719",
"labelIds": [
"UNREAD",
"IMPORTANT",
"TRASH",
"CATEGORY_PERSONAL"
],
"snippet": "Hi, This is a test email. Khoj",
"payload": {
"partId": "",
"mimeType": "multipart/alternative",
"filename": "",
"headers": [
{
"name": "Received",
"value": "by 2002:a59:9d0f:0:b0:30b:1aa4:bbac with SMTP id p15csp383538vqo; Wed, 14 Dec 2022 05:54:39 -0800 (PST)"
},
{
"name": "X-Received",
"value": "by 2002:a67:fb0c:0:b0:3b5:10e6:e47d with SMTP id d12-20020a67fb0c000000b003b510e6e47dmr6093822vsr.4.1671026079093; Wed, 14 Dec 2022 05:54:39 -0800 (PST)"
},
{
"name": "ARC-Seal",
"value": "i=1; a=rsa-sha256; t=1671026079; cv=none; d=google.com; s=arc-20160816; b=EN3co9arAlA/VmNuVMbxgrOs3scS9LODh/L0LkIWxAs5K8XcKdPjt6w09o85lr0rGD VeghUPI1J5r5iQAG2oxXPpt8wLIZ0z0H3FUlTW3bhQxGqCZPlsD646SwMBzfLBPX1URc B7m3SsQH1j/bXE3pbor+AUxfVDruIJl3x7+NzzJ3WHMuLl/zkbs27z6CRSt2Ei/vAuMu TL9aJaHxUrx1+cRvZBwvyuTswGZfAzjMlyo0K8P0T+UtFIifXpvcBjQ0OE7zyQ+MDZPF +EoDcV6GcbGMrS6VU0JcNqsXHgjrL12rj7SG9oxeOHhXQUjQPnmFY9OdXOdU3qd7KUGu uLfQ=="
},
{
"name": "From",
"value": "Some Guy <some.guy@email.com>"
},
{
"name": "Date",
"value": "Wed, 14 Dec 2022 19:24:27 +0530"
},
{
"name": "Subject",
"value": "Test Mail"
},
{
"name": "To",
"value": "Some Name <some.name@gmail.com>"
},
{
"name": "Content-Type",
"value": "multipart/alternative; boundary=\"000000000000cf6af405efca134b\""
}
],
"body": {
"size": 0
},
"parts": [
{
"partId": "0",
"mimeType": "text/plain",
"filename": "",
"headers": [
{
"name": "Content-Type",
"value": "text/plain; charset=\"UTF-8\""
}
],
"body": {
"size": 36,
"data": "SGksDQpUaGlzIGlzIGEgdGVzdCBlbWFpbC4NCg0KS2hvag0K"
}
},
{
"partId": "1",
"mimeType": "text/html",
"filename": "",
"headers": [
{
"name": "Content-Type",
"value": "text/html; charset=\"UTF-8\""
}
],
"body": {
"size": 89,
"data": "PGRpdiBkaXI9Imx0ciI-PGRpdj48ZGl2PkhpLDxicj48L2Rpdj5UaGlzIGlzIGEgdGVzdCBlbWFpbC48YnI-PGJyPjwvZGl2Pktob2o8YnI-PC9kaXY-DQo="
}
}
]
},
"sizeEstimate": 4880,
"historyId": "8631703",
"internalDate": "1671026067000"
}
As you can see, you can get the from, to, subject message contents (base64 encoded) and all the other details you might want. You can now parse this and do as you need to.
Conclusion
Okay, now you have everything you need in order to connect to the Gmail API and set up a webhook for getting all the email notifications. Hope this helped you out. Now you can build on top of this and conquer the world.