Using Amazon SES to send mails from a Ghost blog
As Amazon is blocking all outgoing traffic on port 25 (SMTP) sending mails via Mailgun won’t work. Therefore, I decided to send all mails to subscribers via Amazon Simple Email Service (SES). In this post you can read how I set everything up to make this working.
Why do I want to use Amazon Simple Email Service?
In the beginning I hosted my blog on DigitalOcean and used Mailgun to send emails to subscribers, as you can see in my previous post. But after some time I moved the blog to AWS and now it is running inside an EC2 instance. Unfortunately Amazon is blocking all outgoing SMTP traffic on EC2 instances, which means sending mails via Mailgun won’t work. When I requested this restriction to be removed, Amazon denied it with the following message:
Hello,
Thank you for submitting your request to have the email sending limit removed from your account and/or for an rDNS update.
This account, or those linked to it, have been identified as having at least one of the following:
* A history of violations of the AWS Acceptable Use Policy
* A history of being not consistently in good standing with billing
* Not provided a valid/clear use case to warrant sending mail from EC2Unfortunately, we are unable to process your request at this time, please consider looking into the Simple Email Service. https://aws.amazon.com/ses/
Regards,
Amazon Web Services
Honestly I didn’t really get the reason because points 1 & 2 are definitely not true as my account is rather new and point 3 seems to be wrong as well, because I used exactly the same description when making the request to get out of the Amazon SES sandbox. But as I wanted to give Amazon SES a try anyways, I didn’t elaborate further on this one.
Another advantage is that I have all my stuff in the AWS ecosystem now, which is very convenient in my opinion. On the other hand, the Ghost integration for Mailgun is a lot better than for Amazon SES. This means there are a lot of things you have to do manually.
Set up Amazon SES
When getting started with Amazon Simple Email Service your account will be in a test environment called sandbox. This means you can only send emails to addresses you verified before. So the first step is to get out of this sandbox. To achieve that you need to contact Amazon by creating a new support case. A detailed description about how to get out of the sandbox is available here.
Once you got out of the sandbox, you need to verify your domain. This is necessary to send emails from a domain. I won’t describe how to do this here, because there is a nice tutorial by Amazon again and I don’t see any use in duplicating their content. You can find the tutorial here.
The last step to complete the setup is to get your SMTP credentials. Amazon again provided a nice tutorial for this here.
Use Amazon SES for administrative mails
Sending administrative mails (e.g. signup mails) with Amazon SES is very easy. The only thing you have to do is to edit your Ghost config, which is located in /var/www/ghost/ and is named config.production.json by default. There you need to add the following settings:
"mail": {
"transport": "SMTP",
"options": {
"host": "<SES-HOST>",
"port": 465,
"service": "SES",
"auth": {
"user": "<SES-ACCESS-KEY-ID>",
"pass": "<SES-ACCESS-KEY>"
}
}
}After completing all of the above described steps sending mails via Amazon SES should work.
NOTE: If you didn’t enable member signup yet, you can read about how to do that here.
Use Amazon SES to inform users about new posts
Ghost has a very nice integration to send mails about new posts via Mailgun. When using this, you can automatically send a mail to all your subscribers when creating a new post. Unfortunately this works only when using Mailgun, as one of their staff members stated in a forum post:
Therefore, it requires a lot of work to use Amazon SES for this type of emails. Furthermore, I didn’t integrate this functionality in the Ghost core as I want to keep updates as simple as possible. For this reason I decided to implement a semi-automated solution, which means I created a Shell script, that fetches the latest post from my blog and sends an email to all subscribers via Amazon SES. This shell script has to be invoked manually when a new post is created. In the following I’ll describe the necessary steps to complete this task.
Install the AWS Command Line Interface
To make the script working the AWS Command Line Interface (CLI) needs to be installed on your local machine. You can read more about how to install it here.
Create a mail template
First I created a template for the mails that should be sent. As all administrative mails are created by Ghost automatically I just had to create the template for the mail that informs subscribers about new posts. A mail template basically looks like this (copied from the tutorial provided by AWS):
{
"Template": {
"TemplateName": "MyTemplate",
"SubjectPart": "Greetings, {{name}}!",
"HtmlPart": "<h1>Hello {{name}},</h1><p>Your favorite animal is {{favoriteanimal}}.</p>",
"TextPart": "Dear {{name}},\r\nYour favorite animal is {{favoriteanimal}}."
}
}With this snippet you can set the following properties:
TemplateName: the name of the templateSubjectPart: the subject of the mailHtmlPart: the HTML body of the mailTextPart: the text body of the mail if a mail client can't or doesn't display HTML
It is possible to use replacement tags in the subject and body parts. In the above example {{name}} and {{favoriteanimal}} are used as replacement tags and can be replaced with any text when sending the email.
So I created a new HTML file to create the HtmlPart. I was aiming for a very simple mail body, that just shows a short preview of the post and links to it. In the picture below you can see what it looks like, you can find the code for it on GitHub.
Next, I used a Line Break Removal Tool and a JSON Escape/Unescape Tool to stuff the whole JSON into one String, as it is required to create a valid template. I kept the TextPart very simple so the final template looks like this now:
{
"Template": {
"TemplateName": "NewPostTemplate",
"SubjectPart": "New Post on ksick.dev: {{postTitle}}",
"HtmlPart": "[html I described above - too long to display here]",
"TextPart": "Hey there\r\n\r\n, A new post is available on ksick.dev:\r\n{{postTitle}}\r\n{{postUrl}}\r\n\r\nAll the best,\r\nKathi from ksick.dev "
}
}Again, you can find the whole template, which is stored in a file named newPostTemplate.json, on GitHub. Now we got a nice template but Amazon SES doesn't know about it. So let's upload it with the following command:
$ aws ses create-template --cli-input-json file://newPostTemplate.jsonCreate a SES configuration set to get informed about errors
For this one I didn’t make anything special and simply followed Part 1 of this tutorial. Therefore, I won’t describe the steps I made here. I named the configuration set NewPostConfig.
Write a script to send mails
As I wrote in beginning of this section, my goal is to semi-automate mail sending. This means every time a new post is created, I have to manually trigger a script that automatically fetches the latest post and sends a mail to all subscribers. The full script can be found on GitHub. In the following I’ll explain how it works.
You can use the command below to send a mail to multiple recipients with the AWS CLI. As you can see, it expects a JSON file, which contains information about the mail itself as an argument. This means in the script this JSON file needs to be created first, before the mail is sent with this command.
aws ses send-bulk-templated-email --cli-input-json file://mybulkemail.jsonSo let’s take a look at the described JSON file before we dive deeper into the logic of the script. I’ll first show you how it should look like, then I’ll explain its properties.
{
"Source":"Mary Major <mary.major@example.com>",
"Template":"MyTemplate",
"ConfigurationSetName": "ConfigSet",
"Destinations":[
{
"Destination": {
"ToAddresses": [
"anaya.iyengar@example.com"
]
},
"ReplacementTemplateData":"{ \"name\":\"Anaya\", \"favoriteanimal\":\"angelfish\" }"
}
],
"DefaultTemplateData":"{ \"name\":\"friend\", \"favoriteanimal\":\"unknown\" }"
}Source: mail address (and if wanted name) of the senderTemplate: the name of the template that should be used to create the mail (the one we created above)ConfigurationSetName: the name of the configuration set to use (created in the previous section)Destinations: an array ofDestinationobjects
-Destination: the recipients
*ToAddresses: the addresses the mail should be sent to
*CcAddresses: the addresses the mail should be sent in cc
*BccAddresses: the addresses the mail should be sent in bcc
-ReplacementTemplateData: In the template it was possible to define replacement tags (e.g.{{name}}). You can define their replacements for this special destination here.DefaultTemplateData: In the template it was possible to define replacement tags (e.g.{{name}}). You can define their replacements for all destinations here.
This means within the script we need to generate a JSON file containing all of the above described information. The first three, Source, Template and ConfigurationSetName, are rather easy, because they are known constants. The Destinations and DefaultTemplateData properties are going to be more interesting though. So let's see how we can get them.
Get the DefaultTemplateData
When taking a look at the template file that was created earlier, it becomes clear that the following variables need to be set: {{postTitle}}, {{postExcerpt}}, {{postUrl}} and {{unsubscribeUrl}}. The first three are the same for all people the mail is sent to, so we can put them into the DefaultTemplateData. The unsubscribeUrl is different for each subscriber, so it needs to be put into the ReplacementTemplateData inside Destination.
To get the necessary data of the latest post we can use the Ghost Content API, to be precise the /content/posts endpoint. To only get title, content and URL of the first post the following parameters should be set:
key: an API key you can generate in Ghost's backend. More information can be found herefields: the fields to query, in this casetitle,urlandhtmlfor the contentlimit: how many posts to query, as we only need the latest post, 1 is the value to set
With all that information it is very easy to query the latest post using curl. You can see the code for that below. The keys for title, URL and the content are stored in variables as we will need them again to parse the received data.
title_key="title"
url_key="url"
content_key="html" post_data=$(curl -X GET "https://ksick.dev/ghost/api/v3/content/posts/?key=${ghost_api_key}&fields=${title_key},${url_key},${content_key}&limit=1")
Ghost returns a JSON containing an array of posts with the fields we specified when making this request. When it comes to parsing this JSON I found a simply awesome tool to do this: . I can highly recommend checking this one out and playing around with it, because it is just great in my opinion. In the following code snippet you can see how parsing the title, excerpt and URL from the response JSON works. Parsing title and URL is pretty straight forward. For the excerpt it is nearly the same, you just have to parse the first paragraph out of the full content, which can be done with awk. The tr command is used to escape all quotes, because the result will be stored in a JSON again, which I will write about in one of the next sections. The syntax is quite self-explanatory for simple tasks like the one below.
With the above code the JSON for the DefaultTemplateData is stored in a global variable and will be included in the final JSON later on.
Get all Destinations
Unfortunately the Ghost Content API doesn’t include an endpoint to get all subscribers. Therefore, we need to find another way to get the subscribers. In the Ghost admin panel it is possible to export a JSON file containing lots of information about a blog. You can read more about that here. As it’s not possible to fetch the subscribers via REST, this will be the way to go. So this file needs to be downloaded and passed as an argument to the script. Inside the script jq is used to extract the subscribers (all members that have the subscribed flag set) and store them in a list of Destination objects.
Again the resulting JSON is stored in a global variable and will be part of the full JSON in the end.
Generate the full JSON template
Now the most tricky parts (getting the information about subscribers and the latest post) are done, and the last step is to generate the template file. Generating the content of this file can of course be achieved with jq again. After that the result just has to be written to a file.
Finally send the mail
Now the full template is created and the very last step to complete the task is to send the mail via Amazon SES. This requires the following command.
aws ses send-bulk-templated-email --cli-input-json file://$template_fileFull workflow to send a mail to all subscribers
To send a mail to all subscribers after publishing a post you now need to complete the following steps:
- Go to the Ghost admin panel and head to
Labs->Export Content, clickExportand download the export file. - Invoke the just created script like this:
./inform_users_about_new_post GHOST_API_KEY PATH_TO_EXPORT_FILEThat’s it, now all subscribers are informed about the new post.
TLDR;
Within this blog post I described how I semi-automated the process of informing all Ghost blog subscribers about a new post via Amazon SES. This required some additional work, as Ghost only supports Mailgun out of the box. I used the AWS CLI to send a mail via Amazon SES and automated the template creation with a Shell script. The full source code can be found on GitHub.
Resources
jq (the awesome tool I used for JSON parsing): https://stedolan.github.io/jq/
Originally published at https://ksick.dev on July 8, 2020.
