01 CoAP Blinkr

Welcome to the first tutorial, a kind of API-enhanced "Hello World" for embedded devices. In this tutorial, you'll learn how to

  • use the slyft client to manage project, assets and build jobs,
  • specify an API using the RESTful API Modeling Language,
  • describe a data schema for serialization of data structures using JSON Schema and
  • work with MessagePack as a binary serialization format.

Prerequisites

We assume that you have already downloaded and installed the slyft command line client. If not, please do so as described in the Download and Install section of this documentation. Furthermore, we assume that you are registered and logged in, as shown in the documentation section. Please do so to be able to continue with this tutorial:

$ slyft user login
(...)

You do not have to be an expert regarding CoAP, JSON Schema or MessagePack. The examples we're going use are fairly simple ones and easy to comprehend.

A git client will be needed. We're using PlatformIO as a build system. You may of course use your own build system, some of the platformio or pio commands will look different for your development environment.

The tutorial has three parts. The first part is about what we're planning to do: building an API for an embedded device, including writing the necessary input pieces for slyft to work with. The second part is about using the slyft client, creating a project, uploading the input file and generating code. The last part is about merging the code into an embedded device project for the Arduino platform.

Blinkr Base firmware

For this tutorial, we're using NodeMCU, an ESP8266-based device. You can compose it with a huge amount of sensors and actors, but for this core tutorial, we'll rely on the built-in LED.

Let's assume we want to blink the LED in a specified on/off frequency, say, 200 milliseconds on, 100 milliseconds off and so on. This frequency should not be built into the firmware but rather be configurable using an API.

But let's start with a basic blinking example (that connects to WiFi, too). Create a PlatformIO project for the board:

$ pio init -b nodemcuv2

Everything starts with a main sketch. We do not want to put everything there, but move functionality to other source files, with some interfaces in-between. Put the following content into src/main.cpp:

#include <Arduino.h>

#include <tfwifi.h>
#include <blinkr_app.h>

void setup() {
    Serial.begin(9600);
    setup_wifi();
    blinkr_init();
}

void loop() {
    blinkr_loop();
}

Code around WiFi should go to tfwifi.h and tfwifi.cpp, whereas code about blinking should go to blinkr_app.h and blinkr_app.cpp.

Let's quickly fill these files with code:

src/tfwifi.h declares the setup_wifi() function:

#ifndef __TFWIFI_H
#define __TFWIFI_H

#include <ESP8266WiFi.h>
#include <WiFiUdp.h>

void setup_wifi();

#endif

src/tfwifi.cpp contains the WiFi code, do not forget to insert your own WiFi credentials here:

#include <Arduino.h>
#include "tfwifi.h"

const char* ssid     = "<<YOUR SID>>";
const char* password = "<<YOUR PASSWORD>>";

void setup_wifi() {
    Serial.println();
    Serial.println();
    Serial.print("Connecting to ");
    Serial.println(ssid);

    WiFi.mode(WIFI_STA);
    WiFi.begin(ssid, password);

    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }

    Serial.println("");
    Serial.println("WiFi connected");
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
}

In src/blinkr_app.h, we declare 3 functions: One for initializing the blinkr and specifiying a PIN for an LED (default the Built-In led), a function to set the frequency and a loop which turns the LED on or off, depending on the on/off frequency values:

#ifndef __BLINKR_APP_H
#define __BLINKR_APP_H

void blinkr_init(int ledpin = LED_BUILTIN);

void blinkr_set_frequency(int on_duration_msec, int off_duration_msec);

void blinkr_loop();

#endif

src/blinkr_app.cpp implements these functions:

#include <Arduino.h>
#include "blinkr_app.h"

int __on_duration_msec = 500;
int __off_duration_msec  = 500;
int __state = 0;
int __ledpin = LED_BUILTIN;

void blinkr_init(int ledpin) {
    __ledpin = ledpin;
    pinMode(__ledpin, OUTPUT);
}

void blinkr_set_frequency(int on_duration_msec, int off_duration_msec) {
    __on_duration_msec = on_duration_msec;
    __off_duration_msec = off_duration_msec;
}

void blinkr_loop() {
    __state = !__state;
    delay(__state?__on_duration_msec:__off_duration_msec);
    digitalWrite(__ledpin, __state);
}

Try to compile and flash these files using:

$ pio run -t upload

You can use pio device monitor to look at what it's doing. It should connect to the WiFi, after that start blinking with a default frequency of 500msec.

Now let's CoAP-enable this!

Specify an API

Data

Initially, we need to define our data structure. To do this, we make use of a schema to create a model of what structures we need. A good starting point for this is JSON Schema. If you're new to schemas and/or JSON Schema, please take a look at their website. A good read is the "Understanding JSON Schema" documentation, which is also linked at the JSON Schema page.

We'll start with our own schema. Create a directory named model/ in your PlatformIO project directory and cd there:

$ mkdir model
$ cd model

Next up, we create a data schema, which will be written in JSON. Let's start with a skeleton:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "id": "blinkr",
    "title": "Making a blinky API out of this",
    "description": "Specify blink mode, duration etc. for an embedded device LED",
}

The "id" field is an important naming field, as this string will be used by the code generator to build function and struct' names etc. "title" and "description" are optional, but give a hint of what we're about to build: Send blink data to an embedded devices, about frequency, how long to blink, etc.

We want to express a structure not unlike the following:

{ "on_duration": 200, "off_duration": 100, "times": 12 }

meaning that we'd like to have an LED on for 200 msecs, then off for 100msecs, repeating this for 12 times. To save bytes in transmitting, we're going to use shorter keys where possible, so "on_duration" becomes "on_d", and so on. For now, all item values are integers.

Let's continue with our schema. We need to specify that we want to encode an object:

(...)
  "type": "object"

The object has properties, one for each field above: on_d (for on_duration),off_d (for off_duration), t (for times of repetition)

(...)
  "type": "object",
  "properties" : {
      "on_d" : {
        (...)
      },
      "off_d" : {
        (...)
      },
      "t" : {
        (...)
      }
      "additionalProperties": false,
        "required": ["on_d", "off_d"],
        "minProperties": 2
},

Each property has a type, a description and optional parameters, in this case they're all integers. At the bottom we describe that we do not allow additional properties than the one defined above, that both "on_d" and "off_d" are required (and this "t" is optional), and that we expect at least two properties to be present.

Filling out the property details, we can complete our schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "id": "blinkr",
    "title": "Making a blinky API out of this",
    "description": "Specify blink mode, duration etc. for an embedded device LED",
    "type": "object",
    "properties" : {
    "on_d" : {
          "type": "integer",
          "description": "msec, how long to have LED turned on"
        },
    "off_d" : {
          "type": "integer",
          "description": "msec, how long to have LED turned off"
        },
    "t" : {
          "type": "integer",
          "description": "times of repetitions. Optional. If not given, run forever"
    }
    },
    "additionalProperties": false,
    "required": ["on_d", "off_d"],
    "minProperties": 2
}

Open a text editor of your choice, paste the above JSON schema into it and save it to the file blinkr-schema.json within model/ directory.

The API

What use is the data if we cannot send it to the device? Instead of transmitting it over a serial line or sending it as plain UDP, we're aiming to have an API based on CoAP, the Constrained Application Protocol. CoAP is similar to the well-known HTTP, but handles some things differently, which is better suited for constrained devices such as our ESP8266-based unit.

In order to specify the API we make use of RAML, the RESTful API Modeling Language. If you want to learn more about RAML, have a look at raml.org. For this tutorial, we only need some basic constructs of RAML, which are easy to understand.

In RAML we can define endpoints and associate REST method verbs such as GET and POST with individual actions and request/response bodies.

Create a file named api.raml using your favorite text editor, place it in the same directory next to blink_schema.json. Let's start with a skeleton:

#%RAML 0.8
title: coapled
baseUri: coap://api.sylft.io/
version: 0.1

/led:
  post:
    description: |
      Make LED blink with given frequency, returns status code.

This defines an endpoint called /led and specifies a post method underneath it, together with a description. Then we add a reference to our blinkr data schema, because we want to transmit this data to the endpoint:

#%RAML 0.8
title: coapled
baseUri: coap://api.sylft.io/
version: 0.1
schemas:
  - blinkr: blinkr-schema.json
/led:
  post:
    description: |
      Make LED blink with given frequency, returns status code.
    body:
      application/x-msgpack:
        schema: blinkr

Note the schemas list and the body element underneath /led > post. This specifies that the body of POST requests to the /led endpoint shall contain msgpack data adhering to the blinkr schema.

That's it! Save the above RAML contents to the file api.raml. Next, we use the slyft client to upload both files into a new project.

Using slyft

We assume that you have installed slyft, have already used it to register at the slyft service and to log in. If not, please look at the installation, registration, and command reference sections of this documentation to see how.

Slyft does not automatically know what to do with these model files, it needs some information about the Slyftlets that should be run. For this example, we want to use the c99-arduino-coap-msgpack slyftlet, which is specified on the according documentation page.

Create a text file named c99-arduino-coap-msgpack.json, in the same directory as the files above. This file will serve two purposes:

  • Slyft knows what to do with your input (that is, use this very slyftlet)
  • The contents of this file serve as configuration input for the slyftlet

Put the following contents into c99-arduino-coap-msgpack.json:

{
  "c_define_prefix": "tut1",
  "c_identifier_prefix": "TUT1",
  "source_file_extension": "cpp"
}

Right now your model/ directory should look like this:

$ tree
.
├── api.raml
├── blinkr-schema.json
└── c99-arduino-coap-msgpack.json

0 directories, 3 files

As the next step, we need to create a new project, where we can upload our files, using the interactive project create command. The interactive command asks for a project name first, then for an optional description. The -r stands for remember and puts a .slyftproject file in the current directory. So whenever you run slyft from within model/, it will automatically choose this project.

$ slyft project create -r
Please provide project name: slyft-tut-1
Details to the project (optional): 01 CoAP tutorial
Project Details
===============

| KEY       | VALUE                         |
|:----------|:------------------------------|
| Name      | slyft-tut-1                   |
| Details   | 01 CoAP tutorial              |
| CreatedAt | 2017-03-17 10:58:32 +0000 UTC |
| UpdatedAt | 2017-03-17 10:58:32 +0000 UTC |
| Settings  | {}                            |

Remembering this project in file .slyftproject

Upload all three files:

$ slyft asset add api.raml blinkr-schema.json c99-arduino-coap-msgpack.json

The client will output details of uploads, and a list of currently uploaded files can be queried using the asset list subcommand:

$ slyft asset list

| NUMBER     | NAME                          | UPDATEDAT                     | PROJECT NAME     | ORIGIN     |
|:-----------|:------------------------------|:------------------------------|:-----------------|:-----------|
| 1          | api.raml                      | 2017-03-17 11:06:11 +0000 UTC | slyft-tut-1b     | user       |
| 2          | blinkr-schema.json            | 2017-03-17 11:06:11 +0000 UTC | slyft-tut-1b     | user       |
| 3          | c99-arduino-coap-msgpack.json | 2017-03-17 11:06:11 +0000 UTC | slyft-tut-1b     | user       |

With this basic set, we're able to let slyft validate the project. Validations and builds will run as jobs in the background, and the optional --wait 30 parameter waits up to 30 seconds for completion:

$ slyft project validate --wait 30

Job 246 is started, use `slyft project status` to view status details
Waiting (max. 30 seconds) for job completion....Job Details
===========

| KEY           | VALUE                         |
|:--------------|:------------------------------|
| Id            | 246                           |
| Kind          | validate                      |
| Status        | processed                     |
| ProjectId     | 154                           |
| ProjectName   | slyft-tut-1b                  |
| CreatedAt     | 2017-03-17 11:15:19 +0000 UTC |
| UpdatedAt     | 2017-03-17 11:15:33 +0000 UTC |
| ResultMessage | Successful                    |
| ResultStatus  | 0                             |

ResultMessage is Successful, so our model files are valid in a way that Slyft is able to generate API code. Let's build the project:

$ slyft project build --wait 30  
Job 247 is started, use `slyft project status` to view status details
Waiting (max. 30 seconds) for job completion....Job Details
===========

| KEY           | VALUE                         |
|:--------------|:------------------------------|
| Id            | 247                           |
| Kind          | build                         |
| Status        | processed                     |
| ProjectId     | 154                           |
| ProjectName   | slyft-tut-1b                  |
| CreatedAt     | 2017-03-17 11:28:35 +0000 UTC |
| UpdatedAt     | 2017-03-17 11:28:46 +0000 UTC |
| ResultMessage | Successful                    |
| ResultStatus  | 0                             |

After a successful build, a new asset all.tar appears in the asset list. This is what has been generated by Slyft:

$ slyft asset list
(...)
| 4          | all.tar                       | 2017-03-17 11:28:46 +0000 UTC | slyft-tut-1b     | slyftlet   |

We can download it using the asset get subcommand:

$ slyft asset get --file all.tar  
Downloaded all.tar

$ tar tf all.tar
blinkr.cpp
blinkr.h
coapled_app.cpp.template
coapled_app.h
coapled_coapwrapper_ep.cpp
coapled_coapwrapper_impl.cpp
udp_microcoap_wrapper.cpp
udp_microcoap_wrapper.h

There are a bunch of files in this tarball, namely:

  • blinkr.h and blinkr.cpp: These have been generated from the blinkr_schema.json schema and reflect the data structures defined there.
  • coapled_app.h and coapled_app.cpp.template: These are application-level handlers with request and response objects, as well as a handler function for our /led endpoint. Note that the implementation file has a .template suffix, meaning that you would probably want to copy/rename it to .cpp and edit it.
  • coapled_coapwrapper_ep.cpp defines the CoAP endpoint characteristics
  • coapled_coapwrapper_impl.cpp implements the endpoints
  • udp_microcoap_wrapper.* is wrapper code to bridge requests coming from Arduino's UDP class and forward them to the microcoap handler.

These should be unpacked into the projects src/ folder (when in model/ folder):

$ tar xf all.tar -C ../src
$ cd ..

So the project tree should look like this:

$ tree
.
├── README.md
├── lib
│   └── readme.txt
├── model
│   ├── all.tar
│   ├── api.raml
│   ├── blinkr-schema.json
│   └── c99-arduino-coap-msgpack.json
├── platformio.ini
└── src
    ├── blinkr.cpp
    ├── blinkr.h
    ├── blinkr_app.cpp
    ├── blinkr_app.h
    ├── coapled_app.cpp.template
    ├── coapled_app.h
    ├── coapled_coapwrapper_ep.cpp
    ├── coapled_coapwrapper_impl.cpp
    ├── main.cpp
    ├── tfwifi.cpp
    ├── tfwifi.h
    ├── udp_microcoap_wrapper.cpp
    └── udp_microcoap_wrapper.h

Integrating all parts

This does not yet compile, since we need to fulfil some library dependencies, namely microcoap and msgpack for c. Clone the repositories https://github.com/1248/microcoap and https://github.com/ludocode/mpack from github into a vendor/ directory:

$ git clone https://github.com/1248/microcoap vendor/microcoap
$ git clone https://github.com/ludocode/mpack vendor/mpack

We do not need all files from there, so we copy a selection to lib/private_lib, where PlatformIO can pick them up:

$ mkdir -p lib/private_lib/microcoap lib/private_lib/mpack
$ cp vendor/microcoap/coap.c lib/private_lib/microcoap
$ cp vendor/microcoap/coap.h lib/private_lib/microcoap
$ cp -r vendor/mpack/src/mpack lib/private_lib/
$ cp vendor/mpack/src/mpack-config.h.sample lib/private_lib/mpack/

It should look like this:

$ tree             
.
├── README.md
├── lib
│   ├── private_lib
│   │   ├── microcoap
│   │   │   ├── coap.c
│   │   │   └── coap.h
│   │   └── mpack
│   │       ├── mpack-common.c
│   │       ├── mpack-common.h
│   │       ├── mpack-config.h.sample
│   │       ├── mpack-expect.c
│   │       ├── mpack-expect.h
│   │       ├── mpack-node.c
│   │       ├── mpack-node.h
│   │       ├── mpack-platform.c
│   │       ├── mpack-platform.h
│   │       ├── mpack-reader.c
│   │       ├── mpack-reader.h
│   │       ├── mpack-writer.c
│   │       ├── mpack-writer.h
│   │       └── mpack.h
│   └── readme.txt
├── model
(.... and some more ....)

When everything is in the right place now, it should compile:

$ pio run
Building .pioenvs/nodemcuv2/firmware.bin
text       data     bss     dec     hex filename
223720     2316   31592  257628   3ee5c .pioenvs/nodemcuv2/firmware.elf

Slyft's output contained a file coapled_app.cpp.template which is a code template for you to modify. Naming it .template prevents it from being compiled, but we can take a look at its contents:

$ cat src/coapled_app.cpp.template

// Generated by slyft.io
// This code part has been generated on an "as is" basis, without warranties or conditions of any kind.

// application-part

#include "Arduino.h"

#include "microcoap/coap.h"
#include "coapled_app.h"

// include data structure headers
#include "blinkr.h"

//
// CoAP Application level handlers
//

// -- Application level handler for POST to /led
bool TUT1__coapled__post___led(
    struct TUT1__coapled__post___led_req *req,
    struct TUT1__coapled__post___led_resp *resp) {

    // YOUR CODE GOES HERE
    return true;
}

It implements a single function which is a callback to a CoAP-POST request to the /led endpoint. Its parameters are a request and response struct, which contain input and/or output values, defined in blinkr.h and coapled_app.h. Take some time to look into these files as well.

We already have an "application-level" function, blinkr_app.cpp. It already has a function to set the blink frequency, but it's not network-enabled right now. Lets do this by combining the contents of coapled_app.cpp.template into blinkr_app.cpp.

Open blinkr_app.cpp and add all lines from coapled_app.cpp.template at the end of it. Modify the endpoint function TUT1__coapled__post___led to get the values from the request.

src/blinkr_app.cpp:

#include <Arduino.h>
#include "blinkr_app.h"

int __on_duration_msec = 500;
int __off_duration_msec  = 500;
int __state = 0;
int __ledpin = LED_BUILTIN;

(....unchanged. ADD THE FOLLOWING AT THE END OF FILE...)

// Generated by slyft.io
// This code part has been generated on an "as is" basis, without warranties or conditions of any kind.

// application-part

#include "Arduino.h"

#include "microcoap/coap.h"
#include "coapled_app.h"

// include data structure headers
#include "blinkr.h"

//
// CoAP Application level handlers
//

// -- Application level handler for POST to /led
bool TUT1__coapled__post___led(
    struct TUT1__coapled__post___led_req *req,
    struct TUT1__coapled__post___led_resp *resp) {

    // grab out the data structure
    struct TUT1__blinkr *b = &req->data.blinkr;

    blinkr_set_frequency(b->on_d, b->off_d);
    Serial.print("TUT1__coapled__post___led set on_d=");
    Serial.print(b->on_d);
    Serial.print(",off_d=");
    Serial.print(b->off_d);
    Serial.println();

    // give back the CHANGED response code to coap client
    resp->response_code = COAP_RSPCODE_CHANGED;

    return true;
}

So much for the application part, what's still needed is a connection between Arduino's UDP class and the microcoap-Library.

Open src/main.cpp and modify as follows (new code with comment lines):

#include <Arduino.h>

#include <tfwifi.h>
#include <blinkr_app.h>

// include libs
#include "microcoap/coap.h"
#include "udp_microcoap_wrapper.h"

// Instantiate a WiFiUDP object (from esp8266/arduino)
WiFiUDP Udp;

void setup() {
        Serial.begin(9600);
        setup_wifi();
        blinkr_init();

        // UDP-Listen on port 5683 (default CoAP)
        Udp.begin(5683);

        // setup microcoap lib
        coap_setup();
        // connect UDP object to the endpoints
        udp_microcoap_wrapper_init(&Udp);
}

void loop() {
        blinkr_loop();

        // get coap processing object
        udp_microcoap_wrapper *c = udp_microcoap_wrapper_get();

        // let it handle a message (if one is available via WifiUdp)
        c->ops->handle_udp_coap_message(c);
}

This concludes the first tutorial. In the next tutorial we're going to build a python-based client, to remote-control our blinkr device: Hack on!

For the impatient and command line folks, here's a quick hack using CoAPthon and msgpack-tools:

$ sudo pip install CoAPthon
$ brew install https://ludocode.github.io/msgpack-tools.rb
$ echo '{ "on_d": 100, "off_d": 200, "t": 10 }' | json2msgpack >data.mp
$ coapclient.py -o POST -p coap://YOUR DEVICE IP/led -f data.mp

Enjoy!