Steve Spencer's Blog

Blogging on Azure Stuff

Connecting the ESP 8266 to Azure IoT Hub using MQTT and MicroPython

Recently  was introduced to the ESP 8266 processor which is a low cost IoT device with built in Wi-Fi, costing around £3 - £4 for a development board. The thing that interested me (apart from price) was the device is Arduino compatible and will also run MicroPython. The version I purchased from Amazon was the NodeMcu variant with built in power and serial port via a microUsb port, so it makes an ideal board to start with as there are no additional components required.

clip_image001

This board however did not have MicroPython installed and that required a firmware change. The instructions were fairly straight forward and I followed this tutorial.

After installing MicroPython you can connect to the device using a terminal emulator via the USB serial port. Check in Device Manager to find the COM port number and the default baud rate is 115200. I used the Arduino Serial Monitor tool. In the terminal emulator you can press enter and you should get back the python REPL prompt. If not then you have the COM port or Baud rate wrong.

image

You can write you python directly into here but its easier to write the python in you PC then run it on the device. For this I use ampy

In Command Prompt install ampy using:

pip install adafruit-ampy

This allows you to connect to your device. Close the terminal emulator to free up the COM port then type the following to list the files on your device:

ampy --port COM4 --baud 115200 ls

The MicroPython Quick Ref will summarise how to access the GPIO ports etc but in order to connect to the IoT hub you will need to configure the Wi-Fi on the device. This can be done using the network module.

So create a new text file on your PC and write the code to connect to your Wi-Fi. To test this you can use ampy to run the python on the device:

ampy --port COM4 --baud 115200 run networking.py

Its a good idea to use print statements to help debug as once the run has complete the output will be reflected back in your Command Prompt.

Now you are connected to Wi-Fi we can start to look at connecting to the IoT hub. I am assuming that you already have your IoT hub set up. We now need to configure you new device. Navigate to the IoT hub in your Azure Portal. In Explorers click IoT Devices, then New

image

Enter your device id, the name your device will be known as. All your devices need a name that is unique to your IoT hub. Then click Save. This will auto generate the keys needed to generate the shared access signature needed to access the IoT hub later.

image

Once created you may need to click refresh in the devices list to see you new device. Click the device and copy the primary key, you will ned this for later to generate the Shared Access Signature used in the connection string. In order to generate a new Shared Access Token you can use Visual Studio Code with the Azure IoT Hub Toolkit extension installed. This puts a list of devices and endpoints in the explorer view and allows you to create a new Shared Access Token. find your device in the Devices list, Right click and select Generate SAS Token For Device

image

You will be prompted to enter the number of hours the token is valid for and the new SAS token will appear in the output window:

image

SharedAccessSignature sr=[your iothub name].azure-devices.net%2Fdevices%2Fesp8266&sig=bSpX6UMM5hdUKXHfTagZF7cNKDwKnp7I3Oi9LWTZpXI%3D&se=1574590568

The shared access signature is made up of the full address of your device, a time stamp indicating how long the signature is valid for and the whole thing is signed. You can take this an use it to test your access to IoT hub, so make sure you make the time long enough to allow you to test. The ESP8266 doesn't have a clock that can be used to generate the correct time so you will need to create the SAS off board. I’m going to use an Azure function with the code here to generate it.

Back to Python now. In order to connect to the IoT hub you will need to use the MQTT protocol. MicroPython uses umqtt.simple.

There are a few things required before you can connect.

Firstly the Shared Access Signature that you created above.

Next you will need to get the DigiCert Baltimore Root certificate that Iot Hub uses for SSL. This can be found here. Copy the text from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----, including both the Begin and End lines. Remove the quotes and replace the \r\n with real new line in your text editor then save the file as something like baltimore.cer.

Next you will need a ClientId. For IoT hub the ClientId is the name of your device in IoT Hub. In this example it is esp8266

Next you will new a Username. For IoT hub, this is the full cname of your IoT Hub with your client id and a version. e.g. [your iothub name].azure-devices.net/esp8266//?api-version=2018-06-30

The following code should allow you to connect to the IoT Hub:

def connectMQTT():
     from umqtt.simple import MQTTClient

    CERT_PATH = "baltimore.cer"
     print('getting cert')
     with open(CERT_PATH, 'r') as f:
         cert = f.read()
     print('got cert')
     sslparams = {'cert':cert}

   CLIENT_ID='esp8266'
     Username='yourIotHub.azure-devices.net/esp8266/?api-version=2018-06-30'
     Password='SharedAccessSignature sr=yourIotHub.azure-devices.net%2Fdevices%2Fesp8266&sig=bSpX6UMM5hdUKXHfTagZF7cNKDwKnp7I3Oi9LWTZpXI%3D&se=1574590568'

   

    mqtt=MQTTClient(client_id=CLIENT_ID,server='yourIotHub.azure-devices.net',port=8883,user=Username,password=Password, keepalive=4000, ssl=True, ssl_params=sslparams)


     mqtt.set_callback(lightLed)
     mqtt.connect(False)

    mqtt.subscribe('devices/esp8266/messages/devicebound/#')
     flashled(4,0.1, blueled)


    return mqtt

set_callback requires a function which will be called when there is a device message sent from the IoT Hub. Mine just turns a Led on or off

def lightLed(topic, msg):
     if msg== b'on':
         statusled.on()
     else:
         statusled.off()

connect(False) means that the topic this device subscribes to will persist after the device disconnects.

I’ve also configured the device to connect to its bound topics so that any message sent to the device will call the callback function.

Now we need to have a process loop so that we can receive the messages. The ESP8266 does not seem to run async code so we need to call the wait_msq function to get any message back from the IoT hub

mqtt=connectMQTT()
print('connected...')
while True:
     mqtt.wait_msg()

save your python as networking.py (and make sure that all the code you wrote initially to connect to Wi-Fi is included) then run ampy again:

ampy --port COM4 --baud 115200 run networking.py

Your device should run now. I’ve used the Led flash to show me progress for connecting to Wi-Fi then connecting to IoT Hub and also through to receiving a message. There is a blue LED on the board which I’ve been using as well as a standard LED which is turned on/off based upon the device message received from the IoT Hub. The blue LED is GPIO 2.

In order to send a message from the IoT hub to your device then you can do this from the Azure Portal in the devices view. Click on the device then click Message To Device

image

Enter the Message Body (on or off) and click Send Message

image

Alternatively you can do this in Visual Studio Code by right clicking the device and selecting Send C2D Message To Device and enter the message in the box that pops up

image

In my example the Led lights when I enter on and turns off when I enter off. ampy is likely to timeout during this process, but that’s ok as the board will still be running. As we’ve put the message retrieval inside a loop then the board will continue to run. To stop it running you will need to reset the board by pressing the reset button.

My next step is to sort out automatically generating the Shared Access Signature  and then I’ll look at sending data to the IoT Hub

Unlock The Door Demo Software on GitHub

If you attended my DDD East Anglia talk “A Raspberry Pi2, Azure ML and Project Oxford to unlock that door!” where I integrate a Raspberry Pi running Windows 10 IoT core with the service bus , Project Oxford for face recognition and a Windows Store App to take my picture and hopefully unlock my door. Yes I did bring a door with me. Thanks for attending and for your nice comments.

I have started to put my code up on GitHub. The code for the Raspberry Pi is already there - https://github.com/sdspencer-mvp/RaspberryPi2-UnlockTheDoor. More will appear later as I tidy it up and remove all my config secrets Winking smile

I will be repeating this talk at Smart Devs in Hereford on 12 October 2015 and again at DDD North in Sunderland on 24 October 2015.

Windows 10 IoT Core New Release

I’ve just upgraded my Raspberry Pi 2 with Windows 10 IoT Core Build Number 10531.0 (download , release notes). It fixes an issue I’ve been having with setting the application the runs when the Pi first starts up. Prior to this release my application would start up the first time and then shutdown and be replaced by the default app. It would then not start up at power up again. Now my application starts up every time I power on my Raspberry Pi Smile

It is also possible to set the computer name and set the administrator password from the Raspberry Pi administration website. Previously this was done using PowerShell.

In order to navigate to the administration page you must first know either the machine name or ip address of your Raspberry Pi. This can be found in the Windows 10 IoT Core watcher application that runs after you have installed the IoT core SDK. To access the admin website either enter the address into a browser (http://<ipaddressornameofPi>:8080) or right click on the Pi in the IoT Core Watcher application and select “Web Browse Here”. You will need to enter the username Administrator plus your password to access the site.

image

Here  you can enter a new device name (machine name) as well as change the password. A reboot will be required if you change the name.

In order to set the start up app click the Apps link on the menu panel

image

You will need to ensure that you have first deployed your application to the Raspberry Pi. If you have debugged your application using Visual Studio then a debug version will already have been installed on the Raspberry Pi.

From the Installed Apps drop down select your applications and click the Set Default button. Your application should start and replace the Default App in the running apps list. You can check this by clicking reboot or cycling the power to the Raspberry Pi and your app should start up after the Raspberry Pi has booted.

Windows 10 IoT core project issues when upgrading to VS2015RTM

Just updated my VS2015 to RTM and tried to load in my blinky Iot Project for Raspberry Pi 2. It didn’t load and I was informed that the project required updating

clip_image001

Right clicking on the project offers the option to download updates

Selecting this takes you to:

https://msdn.microsoft.com/en-us/library/windows/apps/xaml/mt188198.aspx

It looks like all I can do is to create a new blank project and copy the existing project files over.

Created a new project and copied the contents of MainPage.xaml.cs and MainPage.Xaml over the contents of the files created in the new project. I found it was quicker to do this than to copy the files over manually. Also, change the namespace (if you created a project with a different name) in both MainPage.xaml.cs and MainPage.xaml. Add in all other files you are using by right clicking on the project and clicking Add Existing Items…

Need to add the following reference:

clip_image002

In the project properties: I selected remote debugger and entered the ip address of my raspberry pi.

When I tried to debug the deployment failed because the version of the remote debugger on the raspberry pi2 was out of date. In order to upgrade it I needed to also upgrade my Windows 10 to the latest version.( https://ms-iot.github.io/content/en-US/win10/SetupPCRPI.htm ) then reflash my raspberry pi 2 sd card ( https://ms-iot.github.io/content/en-US/win10/SetupRPI.htm)

I first updated my Win 10 VM but when I ran the WindowsIoTImageHelper it would not recognise the SD card of the host machine and I couldn’t seem to force it to use the SD card on the host. I then updated my surface Pro to the latest Windows 10 and repeated the process to reflash my Pi.

With all the upgrades completed my project now deploys and runs fine on my updated Raspberry PI2.

Raspberry Pi2 , Iot Core and Azure Service Bus

Using Raspberry Pi2 on Windows 10 IoT core has a number of challenges mainly due to the limitations of both the universal app APIs and also the lack of APIs that currently run on the platform. I specifically wanted to utilise Azure Service Bus Topics to send/receive messages on my Raspberry Pi2. After a bit of searching around I decided that the easiest way to achieve this was to use the Service Bus REST API. There are a number of samples included in the documentation:

Receiving a message: https://msdn.microsoft.com/en-us/library/azure/hh690923.aspx

Sending a message: https://msdn.microsoft.com/en-us/library/azure/hh690922.aspx

The full code for the sample uses WebClient but I needed to use HttpClient so I converted the samples accordingly.

[EDIT] The above links don't work anymore so I've published my code on GitHub https://github.com/sdspencer-mvp/RaspberryPi2-UnlockTheDoor/blob/master/UnlockTheDoor/MainPage.xaml.cs 

Sending a message to the service bus requires a POST and receive and delete requires a DELETE. The following code shows how this was achieved using HttpClient

private async void SendMessage(string baseAddress, string queueTopicName, string token, string body, IDictionary<string, string> properties)

{

    string fullAddress = baseAddress + queueTopicName + "/messages" + "?timeout=60&api-version=2013-08 ";

    await SendViaHttp(token, body, properties, fullAddress, HttpMethod.Post);

}

 

 

 

// Receives and deletes the next message from the given resource (queue, topic, or subscription)

// using the resourceName and an HTTP DELETE request.

private static async System.Threading.Tasks.Task <string> ReceiveAndDeleteMessageFromSubscription(string baseAddress, string topic, string subscription, string token, IDictionary<string, string> properties)

{

    string fullAddress = baseAddress + topic + "/Subscriptions/" + subscription + "/messages/head" + "?timeout=60";

    HttpResponseMessage response = await SendViaHttp(token, "", properties, fullAddress, HttpMethod.Delete);

    string content = "";

    if (response.IsSuccessStatusCode)

    {

        // we should have retrieved a message

        content = await response.Content.ReadAsStringAsync();

    }

    return content;

}

 

 

 

private static async System.Threading.Tasks.Task<HttpResponseMessage> SendViaHttp(string token, string body, IDictionary<string, string> properties, string fullAddress, HttpMethod httpMethod )

{

    HttpClient webClient = new HttpClient();

    HttpRequestMessage request = new HttpRequestMessage()

    {

        RequestUri = new Uri(fullAddress),

        Method = httpMethod ,

 

    };

    webClient.DefaultRequestHeaders.Add("Authorization", token);

 

    if (properties != null)

    {

        foreach (string property in properties.Keys)

        {

            request.Headers.Add(property, properties[property]);

        }

    }

    request.Content = new FormUrlEncodedContent(new[] { new KeyValuePair<string, string>("", body) });

    HttpResponseMessage response = await webClient.SendAsync(request);

    if (!response.IsSuccessStatusCode)

    {

        string error = string.Format("{0} : {1}", response.StatusCode, response.ReasonPhrase);

        throw new Exception(error);

    }

    return response;

}

 

There was an issue with the GetSASToken method as some of the encryption classes weren't supported on the Universal App so I converted it to the following:

private string GetSASToken(string baseAddress, string SASKeyName, string SASKeyValue)

{

    TimeSpan fromEpochStart = DateTime.UtcNow - new DateTime(1970, 1, 1);

    string expiry = Convert.ToString((int)fromEpochStart.TotalSeconds + 3600);

    string stringToSign = WebUtility.UrlEncode(baseAddress) + "\n" + expiry;

    string hmac = GetSHA256Key(Encoding.UTF8.GetBytes(SASKeyValue), stringToSign);

    string hash = HmacSha256(SASKeyValue, stringToSign);

    string sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}",

        WebUtility.UrlEncode(baseAddress), WebUtility.UrlEncode(hash), expiry, SASKeyName);

    return sasToken;

}

 

 

public string HmacSha256(string secretKey, string value)

{

    // Move strings to buffers.

    var key = CryptographicBuffer.ConvertStringToBinary(secretKey, BinaryStringEncoding.Utf8);

    var msg = CryptographicBuffer.ConvertStringToBinary(value, BinaryStringEncoding.Utf8);

 

    // Create HMAC.

    var objMacProv = MacAlgorithmProvider.OpenAlgorithm(MacAlgorithmNames.HmacSha256);

    var hash = objMacProv.CreateHash(key);

    hash.Append(msg);

    return CryptographicBuffer.EncodeToBase64String(hash.GetValueAndReset());

}

 

This allowed me to send and receive messages on my Raspberry Pi2 using IoT core. I created the subscriptions for the topic using a separate app using the .NET SDK which is cheating I guess, but I’ll get around to converting it at some point.

 

In order to use this the following parameters are used:

 

SendMessage( BaseAddress, Username, Token, MessageBody, MessageProperties)

 

BaseAddress is “https://<yournamespace>.servicebus.windows.net/”

 

Token is the return value from the GetSASToken method. using the same base address as above and the KeyName and Key are obtained from the Azure portal and is of the format

 

Endpoint=sb://<yournamespace>.servicebus.windows.net/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<Key>.

 

MessageBody – This is the string value of the message body

 

MessageProperties are a Dictionary containing name/value pairs that will get added to the Request headers. For example I have set the message properties when I press the door bell button on my Raspberry PI2

 

Dictionary<string, string> properties = new Dictionary<string, string>();

properties.Add("Priority", "High");

properties.Add("MessageType", "Command");

properties.Add("Command", "BingBong");

 

These are added to the service bus message and allow me to have subscriptions that filer on Command message types as well as the specific command of BingBong

 

Receiving messages are a bit trickier as we need to create a separate task that is continually running. Once the message is received we need to get back to the main tread to execute the action for the message

await Task.Run(async () =>

{

.

.

.

string message = await ReceiveAndDeleteMessageFromSubscription(_BaseAddress

,_TopicName

, _SubscriptionName

, token, null);

 if (message.Contains("Unlock"))

{

   await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(

      CoreDispatcherPriority.Normal,

      () =>

      {

          SwitchLED(false);

     });

}

 

.

.

}

 

You may want to put a delay in this if receiving the messages causes the app to slow down due to the message loop hogging all the resources. There’s a default timeout in the call to SendAsync and this will automatically slow the thread down.

 

I now have a working Raspberry PI2 that can send and receive message to the Azure Service bus. I’ve created a test win forms app that allows me to send messages to the Service bus and it allows me to control the Raspberry Pi2 remotely. The next phase is to build a workflow engine that hooks up to the service bus and allows me to automatically control the Raspberry Pi. 

Issues setting up Raspberry Pi, Windows 10 IoT core and Visual Studio on a Windows 10 VM

After setting up my Surface Pro with Windows 10 and IoT core I decided that in order to demo it all I needed a Windows 10 VM with it all on. I had a couple of issues that I didn’t get on my Surface Pro.

The first issue I had was that the Windows IoT core watcher application would not run properly and kept shutting down. This is a known bug and has a work around:

Launch the "Developer Command Prompt for VS2015" as Administrator
change the working directory over to "C:\Program Files (x86)\Microsoft IoT"
sn -Vr WindowsIoTCoreWatcher.exe
corflags WindowsIoTCoreWatcher.exe /32BIT+ /FORCE

 

The second issue was Visual Studio couldn’t connect to TFS online. When I tried to manage connections I got the following error:

SplitterDistance must be between Panel1MinSize and Width - Panel2MinSize.

This seems to happen on both VS 2015 Enterprise RC and Community RC editions. I found a work around as follows:

Open up Team Foundation Server online at <youraccount>.visualstudio.com. Click code, then navigate to the project you want to open, click on the solution file which then opens the solution in the web editor. Click the visual studio icon and VS opens with the team project now in team explorer. Close VS and open it again and your team project should still be  connected to team explorer

 

Now with Visual Studio working I needed to set Windows into developer mode. This can be done as follows:

Start->settings->Update & Security -> For Developers. However, when I tried this the setting page kept closing. You can also use the Group Policy editor (Gpedit.msc) as follows:

https://msdn.microsoft.com/en-us/library/windows/apps/dn706236.aspx

clip_image001

Raspberry Pi and Windows IoT Core – Push Buttons and Relays

In my previous Raspberry Pi Post I talked about using the Raspberry Pi to turn an LED on an off. Now whilst this is pretty, its not really that useful. So I wanted to use the same program but to turn on something that needed a bit more power than an LED. I’d recently acquired a solenoid (a coil with a bolt that gets draw towards the magnetised coil when 12v is applied to the solenoid’s coil). Now my Pi doesn’t have enough power on its own to drive the solenoid so I needed a mechanism to apply 12v to the coil from a 3.3V output that the PI delivers. This meant I had to think back to my school days, which in my case is a difficult task :-). I remembered that I could use a transistor to turn  on something with a bigger current from a smaller one.  I decided that as the Pi can supply both 3.3V and 5V I would use a 5V relay and a transistor to allow me to turn on a separate 12v supply to the solenoid. I tried to calculate the correct resistors for the circuit but I failed miserably so in the end I decided trial and error was my best plan. I used a NPN transistor and a resistor and I also combined the LED and resistor from the previous post. The other change that I wanted to do was to remove the timer, that was being used to turn the LED on and off, and replace it with a push button switch.

The following shows the circuit I used.

 image

I should really use a diode across the resistor to protect the transistor and I’ve even used my soldering iron without burning my fingers.

For information, the following image shows the assignment of pins for the Raspberry Pi 2:

image

Anyway, In order to change the code to use a push button I took the sample https://www.hackster.io/windowsiot/push-button-sample and added the push button code to my blinky sample and removed the timer turning on the LED.

In order to use the push button I needed to configure one of the GPIO pins for input rather than output that was used for turning on the LED. I still needed to use a timer, as I needed to read the push button pin on a regular basis to see when the input changed to low when the button was pressed.I set the time to 250 ms so that I didn’t have to hold the button down too long for it to register,  but not too long that the timer  would hog all the resources on the PI.

Now when I press the button the LED turns on, the relay clicks and the solenoid pulls the bolt across. It made me jump when I first connected it up as the solenoid made quite a loud bang and I though I’d blown something up!!

I think I know enough now of how to use the GPIO on the Raspberry PI so I am looking at how I can now connect the PI up to Azure and make it part of a distributed system.

More on this to come……

Getting Started with Raspberry Pi 2 and Windows 10 IoT Core

I've got my Raspberry Pi 2 this week and promptly downloaded the Windows 10 IoT core for it.

Scott Hanselman's blog post covers most of what you need to get started

http://www.hanselman.com/blog/SettingUpWindows10ForIoTOnYourRaspberryPi2.aspx

I've summarised the bits that I either didn't read properly or had to go searching for :-)

Download the Windows 10 IoT core and follow the instructions here: http://ms-iot.github.io/content/win10/SetupRPI.htm

In the zip file that is downloaded there is also an MSI file. Install this on your dev machine and you will get an IoT Watcher application that shows all devices on your network. It shows you all the details you need to remote debug your IoT Core device. If you right click on the device you can copy the ip address. This was really useful for this because the only display I could connect my Pi to was my TV (Mainly due to having the wrong cables or no hdmi port on my monitors). Although it was quite impressive to see such a small device on a big screen, it wasn't very practical, plus I keep getting kicked off as the family want to use it to actually watch TV! I'm going to get myself a cheap small monitor just for my Pi. The IoT Watcher application allowed me to check that the Pi was running and also to get its IP address

In order to configure your device including changing the password and setting the machine name the following commands are useful

http://ms-iot.github.io/content/win10/samples/PowerShell.htm

To get started developing for your device download the samples from here: https://github.com/ms-iot/samples

I started with the Blinky sample and this can be the basis for your applications, I picked this one as it shows how to use the GPIO to control something. When this is loaded in Visual Studio 2015 the MainPage.Xaml.cs file is where all the work is done. InitGPIO() sets up the pins for connecting the LED to and there is a timer that ticks to turn the LED on and off

Debugging the application can be done directly on the device and this needs configuring. In order for this to deploy you need to ensure that authentication is turned off in VS as won’t deploy otherwise. When setting the device in VS, I could not get the device to appear in the search tool so I manually configured it with its IP address. This can be done (or to change the device) in the debug section of project properties. Once deployed you can set break points in the code which is running on the device and debug it remotely.

Now I've got that working I've dusted off my soldering iron and the rest of my electronics kit and I am off to play. More later.