Triggering Events from Email Messages

In this cookbook recipe, we're going to show how you can raise events to a pico using an external mail service, like mailin. Transforming a mail message into an event isn't too difficult. The fun part is in parsing the body of the email to take further action. This recipe shows how to do that too. 

Strategy overview

An incoming email message (1) will be transformed into a webhook. The webhook will be received (2) by the pico engine as an event for a mail routing pico. This pico will select some fields from the email message and pass these on to another rule (3). This rule will identify the target pico--the logical final recipient of the original email message--and send it an event which includes the body of the email message (4). The target pico will use functions provided (5) by the mail pico to parse the message body and will then raise (i.e. send to itself) the events (6) represented therein. We will define a simple language which will allow Alice to specify a list of events in the body of her email message.

Making a localhost engine available to the outside world

You'll need to have access to an email address that can be used. This implies also having access to a domain name. This can be challenging if you are running a node pico engine locally.

Here at Pico Labs, we have a pico engine running on a Raspberry Pi 3, and make it available to the outside world by using ngrok, which supplies "secure introspectable tunnels to localhost". This means that our engine can be accessed at https://assigned_id_1.ngrok.io etc. where "assigned_id_1" is a pseudo-random identifier given to us by ngrok. (The actual ID is redacted here, to avoid denial of service attacks, because ngrok only allows us 20 uses per minute.) Another alternative is PageKite, but we'll use ngrok in this cookbook.

Obtaining an email address that forwards to a webhook

Here we will use CloudMailin which is a cloud service that assigns us an email address, which we can configure so that it does an HTTP POST to a webhook that we supply. We redact the actual address as "assigned_id_2" because CloudMailin only allows us 200 incoming email messages per month. The address we are using is assigned_id_2@cloudmailin.net

We have configured it to use https://assigned_id_1.ngrok.io/sky/event/<eci>/eid/mail/received where "<eci>" is an ECI (event channel identifier) which we created for the mail pico which we will be using to respond to mail:received events.  We'll call that "assigned_id_3". 

It should also be possible to install, configure, and run your own mailin server, but here we will use the CloudMailin service.

A ruleset that handles the mail:received event

We created a pico, named Mail, and added a new channel with type "secret" and name "incoming mail", whose id is known here as <eci>, and is "assigned_id_3". It has a ruleset containing this rule

  rule mail_incoming {
    select when mail received
    pre {
      attrs = event:attrs().klog("attrs")
      headers = attrs{"headers"}
      date = headers{"Date"}.klog("date")
      to = headers{"To"}.klog("to")
      from = headers{"From"}.klog("from")
      subject = headers{"Subject"}.klog("subject")
      text = attrs{"plain"}.klog("text")
    }
    fired {
      raise mail event "parsed" attributes
        { "date":date, "to":to, "from":from, "subject":subject, "text":text }
    }
  }

This rule logs many things (and these log messages could be removed in a production application), extracts a few simple items from the incoming message and headers, and raises a new mail:parsed event supplying those items as event attributes. You may wish to add rules in this or another ruleset for use within the Mail pico to respond to the mail:parsed event. In our case, we wanted to pass the event on to another pico, the target pico.

Forwarding to another pico

For this purpose, the same ruleset also contains a rule which is triggered by this mail:parsed event, which parses out the ECI of the target pico from the "To" email address.

  rule mail_router {
    select when mail parsed to re#[+]([^@]*)@cloudmailin.net# setting(eci)
    event:send({"eci":eci, "domain":"mail", "type":"events", "attrs":event:attrs()})
  }

This rule is only selected when the "to" address matches the regular expression in line 2. The construct [+] matches a literal plus sign, and the capture group ([^@]*) matches all characters up to the next at sign. The setting clause binds the captured characters as the value of the eci identifier. This ECI is how we refer to the target pico which is to handle the incoming email message.

This means that we must send our email message to an address looking like

assigned_id_2+assigned_id_4@cloudmailin.net

The redacted "assigned_id_4" is an ECI of the target pico to which the mail_router rule (in line 3) sends the mail:events event with selected fields from the email message as attributes. We created a new channel for this purpose, with name "email tests" and type "secret" whose id is known herein as "assigned_id_4". This is easily done in the "Channels" tab of the developer UI.

Further processing of an incoming email message

In this example, the target pico has a ruleset which handles numerous events with domain "aurora" and controls a set of 12 Nanoleaf Aurora units. Rather than modify the existing ruleset, we made a new ruleset and add it to the same pico. Shown here is a global definition and a rule that reacts to the mail:events event.

  global {
    newline = (13.chr() + "?" + 10.chr()).as("RegExp")
  }
  rule mail_events {
    select when mail events
    foreach event:attr("text").split(newline) setting(request)
    pre {
      parts = request.split(re#/#)
      aurora = parts.length() == 2 && parts[0] == "aurora"
    }
    if aurora then noop()
    fired {
      raise aurora event parts[1]
    }
  }

The regular expression created in line 2 will match a carriage return (if present) followed by a newline. It is used in line 6 to split the text into an array of the lines of the email message body. The remainder of the rule is evaluated once for each line of the message, with the name request bound to the line under consideration.

Each line is split into parts using a forward slash as a separator. We ignore any lines which do not begin "aurora/" followed by just one slash-separated string of characters. They are ignored because the rule won't fire for such lines. For lines which match our expectation, in line 13 we raise an aurora event whose type is parts[1] (the second slash-separated string of characters).

Sample email message

A sample email message might look like this:

From: ...
To: assigned_id_2+assigned_id_4@cloudmailin.net
Subject: anything -- ignored by these rules


This comment/annotation is ignored
The next line turns on the aurora nanoleaves
aurora/on
Then we move on to the second next stored theme
aurora/next
aurora/next

Upon receipt of this email message, our Mail pico will extract the "To" field and the "text" of the message body, and begin a cascade of events. Next, the same pico will parse out the ECI ("assigned_id_4") of the Aurora pico and send it the mail:events event. This will trigger its new rule which will loop over the lines of the message body, ignoring comment lines, and raise a series of three events, which the existing ruleset will handle to turn the NanoLeaf display on and move through to the second-next theme.

The person composing the email message needs some knowledge of the events understood by the Aurora pico, but doesn't need to know where its pico is hosted, just the email address to which the message must be sent.

Caveats

In general, the most difficult part of turning an incoming email message into an event is parsing the message body to determine what further events need to be raised.

Notice that email clients differ. For example, the OWA email client running in a Chrome browser will insert a zero-width space character into the message body if the user tabs to it from the subject area.This is just one potential problem which could cause you grief.

You will want to thoroughly test your rules before putting this cookbook into production.

Extended parsing

This modification to the rule in the Aurora pico which parses the incoming email message shows how we can extend the mini-language to include event attributes. Basically, lines in the message follow the syntax of standard event URL's starting with the domain portion, so that a line like

     aurora/tempEffect?name=Nemo&duration=5

can result in the equivalent of

     raise aurora event "tempEffect" attributes { "name": "Nemo", "duration": 5 }

which allow events requiring attributes to be triggered by an email message.

  global {
    newline = (13.chr() + "?" + 10.chr()).as("RegExp")
    var = function(s) {
      p = s.split(re#=#);
      {}.put(p[0],p[1])
    }
    vars = function(s) {
      p = s=>s.split(re#&#)|[];
      p.reduce(function(a,b){a.put(var(b))},{})
    }
    parse = function(text,domain) {
      dlpo = domain.length() + 1;
      text.split(newline)
          .filter(function(v){p=v.split(re#/#);p[0]==domain && p.length()==2})
          .map(function(v){p=v.split(re#[?]#);[p[0].substr(dlpo),vars(p[1])]})
    }
  }
  rule mail_events {
    select when mail events
    foreach parse(event:attr("text"),"aurora") setting(request)
    pre {
      type = request[0];
      attrs = request[1];
    }
    if request.length() == 2 then noop();
    fired {
      raise aurora event type attributes attrs;
      ent:lastRequest := "aurora/"+type;
      ent:lastAttrs := attrs;
    }
  }

As before, line 2 is a pattern matching line breaks, used to split the entire message body in line 13. The function var (lines 3-6) makes a map for a single attribute ("name=Nemo" becomes {"name": "Nemo"}) and is used in line 9. The function vars (lines 7-10) collects all of the attributes into a single map and is used in line 15. The parse function (lines 11-16) first splits the message body into lines, then (line 14) selects only those lines which start with the domain followed by a slash (the length of that leading substring is maintained in dlpo (domain length plus one)), and finally (line 15) collects for each desired message body line an array whose first entry is the event type, and whose second entry is any collected attributes as a map.

The rule loops over the parsed events (line 20), setting the type (line 22) and attributes (line 23) of each, and raises the corresponding events (line 27).

Update

The complete ruleset for the Mail Pico that first receives email, and reacts to the mail:received event, can be found at https://github.com/Picolab/pico_lessons/blob/master/mailin/io.picolabs.mail.krl and nearby is the ruleset to test this with Aurora, the target pico, at https://github.com/Picolab/pico_lessons/blob/master/mailin/aurora_test.krl

We moved the parsing functions into the io.picolabs.mail ruleset, which provides them to rulesets which might want to react to the mail:events event. Such rulesets would include a clause in their meta block like

  use module io.picolabs.mail alias mail

so that parsing can be done with a line like

    foreach mail:parse(event:attr("text"),"aurora") setting(request)

This is a basic use of User Defined Modules and has the advantage that the parsing functions do not need to be replicated into a ruleset for every pico which needs them.