(Guest writer) Echelon/NES Smart Meters – NES/ECHELON electrical meter connect to MQTT (Home Assistant) via IR

MQTT and IR

This is another bonus entry on our blog in the Echelon/NES Smart Meters series. As we have mentioned before, it is sometimes hard to find time besides our regular jobs to dabble with this project. We have had a lot of interest in using the IR (Infrared) port on the Echelon/NES Smart Meters, and there have been (at least with N1) some confusion about if it was encrypted or not.

We know that someone at Sommerhack 2021 had been told by N1 that they should be able to use the IR port and that the key was the same as the MBK used for the MEP communication. We have not tried that ourselves as we have chosen to focus our limited time on the MEP integration. And to our knowledge – nobody else have tried the IR approach – that is – until now :-).

Our fellow tinkere Ulrik was faced with the challenge that his power supplying company Konstant, does not allow him to use the MEP-port on his Echelon meter. They are simply following another approach compared to N1, and suggests that their customers should use the IR-port.

Overall/political speaking this is actually really good news. Because if N1 can allow us to use the MEP port in a safe way, why should Konstant be against it for security reasons? Why should they not agree to allow it sometime in the future if they are pushed?
Also – if Konstant can allow their customers to use the IR port in a safe way, why should N1 be against it for security reasons? Why should they not agree to allow it sometime in the future (if not already?) if we push them?

Some have been dreaming of being able to interface with the Echelon/NES meters for years – but it was not possible due to a lot of “red tape”. Maybe we now have or will get TWO separate interfaces for interfacing with Echelon/NES Meters?

Enough dabbling with words from dabbler.dk – let’s hand over this scene to Ulrik and let us hear what he was actually able to do with the IR port:


First of all, I would like to say thank (and sorry) to Gert and Graves for letting me ‘spam’ them with my thoughts, questions and discoveries. And of cause for helping me out from time to time :-).

INFO:
My name is Ulrik and I work as an Automation technician. I started out as an electrician and was fascinated by all the technical stuff you could do. I have around 30 Years of technical background. I build my own house in year 2000 where I put in a PLC to control the house. All lightning, heating, ventilation and alarmsystem is run from that.
Well… Back on the track… I started out collecting the pulses from my main meter by using the S0 pulses. Worked great, until I got solarpanels installed and the meter now send out pulses whenever a watt was produced or consumed, so not that usefull :-(. Then I started looking at other ways and stumbled over a lot of options, among this was the IR port. So here’s a story of how I got this to work :-).
I have a NES 83334 meter. I also use Home Assistant.

I have very little knowledge of Python and NODE RED, so ‘Google was my friend’ 🙂
I also know that this setup could be much more ‘smart’. But for now, it just might get people started to get data from the meters 😉

My supply area is KONSTANT and they will send you the decryptkey if you contact them at info@konstant.dk

They also send me a draft version of Optical Port Programmer’s Guide with some info tables in the meter.

KONSTANT also told me that the meters they are using (ECHELON / NES) have the MEP port disabled, so I was unable to use that. So I went the optical IR eye way…

Communication parameter that I use:
9600, 8, 1, none

LINKS:
Install a RAM disk (to prevent destroying the SD card.)
https://www.hellojona.com/2017/06/create-a-ram-disk-tmpfs-in-raspberry-pi-3/

ANSI C1218 protocol description:
https://kupdf.net/download/ansi-c1218_598ceccadc0d606c4b300d1a_pdf

ANSI C1219 Protocol description:
https://kupdf.net/download/ansi-c12-19-2008_5af9fdd5e2b6f501147de395_pdf

Open Smart Grid Protocol (OSGP)
https://www.etsi.org/deliver/etsi_gs/osg/001_099/001/01.01.01_60/gs_osg001v010101p.pdf

Termineter (Framework for reading data from smartmeters)
https://github.com/rsmusllp/termineter

HARDWARE USED:

IR Probe:
I bought mine from Weidmann Elektronik
https://www.weidmann-elektronik.de/Produkt_IR-Kopf.html

1.00 x IR Schreib/Lesekopf USB (Optokopf), ART0027, EUR 44,99

I have not tried other probes, but I figured at the start of this project, I did not want to have trouble with my hardware. I know it’s kind of expensive but it works 🙂

Raspberry Pi 3 model B+ with 8 gb SD card:
(I used Raspberry Pi Imager and installed Raspberry Pi OS)
Activate VNC and SSH (Raspberry PI configuration – Interfaces)

SOFTWARE:

Install SAMBA

sudo apt install samba samba-common-bin

Edit the config file:

sudo nano /etc/samba/smb.conf

[pi]
path=home/pi
read only = no
writeable = yes
browseable = yes
guest ok = yes
create mask = 0755
directory mask = 0755

Press CTRL-o and CTRL-x to save and exit

Install a RAM disk (to prevent destroying the SD card.)
Link: https://www.hellojona.com/2017/06/create-a-ram-disk-tmpfs-in-raspberry-pi-3/

mkdir /var/tmp

Edit the fstab file.

sudo nano /etc/fstab

Add the following line to /etc/fstab to create a 400MB RAM Disk

tmpfs /var/tmp tmpfs nodev,nosuid,size=400M 0 0

Execute the following command to mount the newly created RAM Disk

mount -a

To verify the RAM Disk is created and mounted successfully, execute the following command

df -h

Temporary files can now be written/read to/from /var/tmp partition.

PYTHON 3.9 (I think that it is installed on the Raspberry by default).

Install NODE-RED
I think I installed it from the Rasberry Pi ‘recommended software’ via the GUI.

Autostart Nodered: (run from a terminal / SSH)

sudo systemctl enable nodered.service

Install Temineter (Python program to communicate C1218 and C1219 protocol)
Start a terminal and install it.

git clone https://github.com/securestate/termineter.git

Change permissions on directory

sudo chmod 777 termineter -R

I edited a some of files (I use Visual Studio Code through the SAMBA share) to get this to work.
You can use the files from the attached zip of use them as inspiration to copy/waste.

Changed files:
lib/c1228/connection.py
lib/termineter/core.py
lib/termineter/modules/get_data.py
lib/termineter/modules/read_part.py

TERMINETER program changes
lib\termineter\core.py:
*** Section around ‘Setup end configure options’

self.options.add_string('SERIAL_CONNECTION', 'serial connection string',default='/dev/ttyUSB0')
self.options.add_string('PASSWORD', 'serial c12.18 password', default='<your decryptkey here!>')
self.advanced_options.add_integer('C1218_PACKET_SIZE', 'c12.18 maximum packet size', default=64)

Run the program:

python ./termineter

Termineter
 <[ termineter v1.0.5
 <[ model: T-1000
 <[ loaded modules: 19

termineter >

Run the module to start getting data:

termineter > run get_data

Hopefully the program starte to get data (and save it in the /var/tmp folder)

[+] Successfully connected and the device is responding
[*] Read 8 bytes
0000 74 40 05 00 9f 1d 00 00                         t@......
[*] Read 40 bytes
0000 96 0a 00 00 00 00 00 00 00 00 00 00 39 02 00 00 ............9...
0010 f8 0c 00 00 dc 0c 00 00 90 19 00 00 27 70 03 00 ............'p..
0020 ef 6f 03 00 76 70 03 00                         .o..vp..

Instead of (or additional) to make the program changes, you can use a ‘resource file’ (look at
https://github.com/rsmusllp/termineter/wiki/GettingStarted at the bottom)

If you use a resource file, you can ‘autostart’ the program.

Errorhandling still needs to be taken care of 😉

NODE-RED Program:
The data from termineter is stored in /var/tmp BT23 and BT28 and is updated every 10 sec-ish.
I use NODE-RED to handle the data (and send them to HomeAssistant via MQTT)
Install the ‘Bufferparser’. (Manage palette and search for ‘node-red-contrib-buffer-parser’)
The Bufferparser handles and changes the data from the files in /var/tmp and makes the data ready.
Afterwards the data is made ready in a function and finally sent by MQTT.

I also tried to make some ‘autodiscovery sensors’ for HomeAssistant.

Screendump of the Flow:

HomeAssistant - Screendump of the Flow
HomeAssistant – Screendump of the Flow

1. Timestamp runs every 10 secs.

2. Function /var/tmp/BT23:

HomeAssistant - Function
HomeAssistant – Function

3. BufferParser BT23:

HomeAssistant - Buffer-parser
HomeAssistant – Buffer-parser

4. Change node ‘fwd_total’:

HomeAssistant - Fwd_total
HomeAssistant – Fwd_total

5. MQTT: Setup according to your MQTT broker settings.

6. Discovery settings for HomeAssistant

HomeAssistant - Discovery settings
HomeAssistant – Discovery settings

Screenshot from HomeAssistant:

HomeAssistant - Screenshot
HomeAssistant – Screenshot

Hope this can inspire other to get data from ECHELON/NES meters via IR eye.

🙂 Ulrik, email: u_dabbler[AT]dumac.dk

(Guest writer) Echelon/NES Smart Meters – Home Assistant integration

Home Assistant Logo

This is a bonus entry on our blog in the Echelon/NES Smart Meters series. We have limited time and are devoting the sparse time we have on the hard- and software needed for the ESP32-MEP module. We are aware that other systems should be able to integrate to the solution, but we currently have no experience with stuff like “Home Assistant” etc.

This project was never supposed to be a dabbler.dk private project anyway. As you know if you’ve been following this blog, the end goal is to get the specification released. As soon as that happen we’ll release our hard- and software as well, and others can choose to join us in improving it, start their own project from the specs or even branch our project and move on from there.

We would love to see this grow in all sorts of directions…

Luckily others have more knowledge in integrating with “Home Assistant” and one of the more persistent persuaders must have been Jan Nielsen. He got a private access to a webservice on Gert’s ESP32-MEP module to see if he was able to integrate to it…

…and best of all: when asked he immediately wrote this blog entry about it – sharing his findings. The rest of this blog post is written by Jan Nielsen, so send all praises and credits to him:


Jan Nielsen
Educated as Data Technician (Data Mekaniker) from EUC Syd in Sønderborg in 1997.

I have worked some years as a full-time developer; then switched over to system administration.
But still with an interest in automation when there is a need, and occasionally some hobby projects.
https://www.linkedin.com/in/jan-nielsen-21a5832/


Hi there, my name is Jan. Like I assume many of the people reading this, I have been following this blog for some time. I found it back in August 2021, after first seeing Gerts post at ing.dk.

At the time I had recently discovered the Home Assistant project. My interest was to get a chance to play around with Zigbee – more than Philips Hue allowed for.
Shortly after the August update was released. The big news there was an energy dashboard, and a few more things related to it like long-term statistics etc. In this context energy meant electricity consumption and production (solar).
Without digital access to the overall household consumption this was not really useful.
You can get Zigbee, Wifi, Bluetooth og probably Z-wave plugs that can measure power consumption, but it is only a portion of the consumption.


Some solutions for reading total electricity consumption:

  • A guy named Jonas Pedersen had made a HA integration that pulls in data from an API at eloverblik.dk. This is working, but data is two days delayed, and for that reason not suitable for the energy dashboard.
  • The HA release mentioned two solutions that were already supported. A solution for Dutch meters that apparently has a serial port with an open and documented protocol, and another one for counting the number of blinks from a LED on the meter. Well yes, that second one would probably work with an Echelon meter also, but still not quite delivering what I hoped for. For example it does not get the total consumption value seen in the display, if for some reason counting pulses is impaired consumption during that time is simply missed. Additionally I assume it will not give current power usage in Watt, or it must be derived from time between two pulses, which can be quite lengthy when consumption is low. If I have to fiddle with electronics, it has to be a solution I believe in, and that I find at least somewhat smart.
    Similarly ready-made devices like this are available for purchase. But it seems fairly expensive, when in doubt how well it can actually do it.
  • Lastly, Develco seems to have developed a really interesting module for the MEP port. But as the dabbler.dk guys discovered it cannot be purchased anywhere, at least not only a single or a few pieces.


    So, when I found this blog here it was the most promising solution. Thus, I have been waiting and regularly checked for news and progress…

At the end of January when I saw Gert mentioning some web service, my interest was raised further.
In the autumn I made a HA integration for Min Volkswagen. It simply pulled in data from a connectedcars.io API. I first learned about this API from Esben on tech at Youtube. However, his approach was to use Node-RED. This I found a bit cumbersome, especially with the location tracker that had to be configured in multiple places. So, I took the challenge and tried to write an integration for HA, which would be easy to use and configure.
The API was really easy to use as Esben had shown how to do it. The HA part was more challenging and had quite some learning curve, but by studying other examples, documentation and at times HA source code I eventually got it working, including a UI dialog for the initial configuration.
Back to the MEP webservice… if the dabbler.dk guys had made a webservice I would rather easily be able to pull this data in, similarly to how I had already done with Min Volkswagen. So, I reached out to Gert offering to be a tester 😉, but also to implement a HA integration that could consume the webservice. Due to the NDA Gert was, understandably, not that keen on the idea, but he did offer remote access to a webservice. As for the development process this would be perfectly ok, and overall bring us faster to my goal of getting this data into HA. So, I accepted Gerts offer.

So, I cloned my previous HA integration and started to adjust the API reader code. As the webservice is very simple this was mostly a matter of removing authentication, and code to handle multiple cars/devices from the same web service instance.
Simplified, the basic steps to read it looks somewhat like this:

try:
async with aiohttp.ClientSession() as session:
async with session.get(req_url, headers = headers) as response:
temp = await response.json()
if temp is not None:
_LOGGER.debug(f"Got meter data: {json.dumps(temp)}")
self._data = temp

except aiohttp.ClientError as client_error:
...

And next to adjust each value, named a sensor in HA, that I wanted to make available in HA:

...
sensors = []
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "energy consumption", "", False, True, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "energy returned", "", True, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L1", False, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L2", False, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L3", False, False, _meterclient))
...
async_add_entities(sensors, update_before_add=True)

class MeterEntity(Entity):
"""Representation of a Sensor."""

def __init__(self, config_entry_id, meterName, itemName, phase, returned, entity_registry_enabled_default, meterclient):
"""Initialize the sensor."""
...
if self._itemName == "energy consumption":
self._unit = ENERGY_KILO_WATT_HOUR
self._icon = "mdi:home-import-outline"
self._device_class = DEVICE_CLASS_ENERGY
self._dict["state_class"] = "total_increasing"
elif ...

async def async_update(self):
"""Fetch new state data for the sensor.
This is the only method that should fetch new data for Home Assistant.
"""
try:

# Measurement
if self._itemName == "energy consumption":
energy = await self._meterclient._get_value(["Fwd_Act_Wh"])
if (energy is not None):
self._state = int(energy) / 1000

Within a day or so data started to roll in.

At first the webservice had the accumulated consumption, returned to the grid as well as voltage and current for each phase. As I find power more relevant, I had to calculate this from voltage and current. Since then the webservice has been extended with power readings and more.

When it all seemed to run smoothly, one day it turned out to have registered a humongous consumption.

Unfortunately, I did not make a screenshot of it before correcting the data, but here is what it normally should look like:

Home Assistant - Echelon energy consumption
Home Assistant – Echelon energy consumption

As it is the kWh counter it should be forever increasing. Little consumption will be slowly increasing, higher consumption will increase steeper.
When the problem appeared, it rose all of a sudden thousands of kWhs, and shortly after fell back to normal.

Due to the way I had defined the sensor in HA, this was a bigger issue than just that short glitch.
For it to be usable with the Energy dashboard it needs to have statistics collected each hour, and this requires a ‘state_class’ defined. For this kind of data it can be ‘total’ or ‘total_increasing’. To support meter reset the ‘total’ type requires a date specified where the meter was reset. Instead the ‘total_increasing’ type, which I had chosen, will see a decrease as a reset.
As the Energy dashboard works with the deltas, not only did the increase add thousands to the depicted consumption, when returning to normal it also added 117.000 kWhs because it assumed it was reset to zero and then consumed the reported value…

Gert had also noticed this at times, and had also mentioned it to me earlier. But in my rush to get something to show up I had also totally forgotten about it.
Rather than having to specify a reset date or risking negative consumption, I decided to continue with ‘total_increasing’. However, I did implement a limitation that would sanitize the kWh reading. If it increased more than 1000 Wh or decreased from the latest reading, it would discard the new value for up to an hour until finally accepting the new value if not returned to a more realistic value. It is only done in memory though, so not totally fail proof if HA is restarted during an issue, but it minimizes the risk.
A few days later Gert reported the issue had been found and firmware updated.

At the same time there was more info added to the webservice.
In a previous blog, it was shown that it could return info about itself like manufacturer, model and version. Although secondary, I find it nice and convincingly to have that reported as well.
Not only did the added info have metadata about the meter, it also turned out that similar data about the MEP module had been added.
In HA a device only has a predefined set of properties. So, how to depict both sets of device info?
Although not originally planned, it turned out simple enough, just make it appear as two devices. The meter itself and the MEP module:

Home Assistant - Echelon
Home Assistant – Echelon
Home Assistant - Echelon MEP
Home Assistant – Echelon MEP

To do this, it requires at least one sensor on the MEP module. This is because it is the sensor that returns info about the device it is part of.
In this case I invented two binary_sensors that tell something about the status. One if the last http request succeeded, and two if the data returned doesn’t look as expected.
The “Connected via” relationship was new to me, and also something I wanted to try out… 😁 But it just worked, so no challenge in that.

As Gert and Graves had taken care of the difficult part, the most challenging part of this integration turned out to be the Configuration button, or as it is known internally the OptionsFlow.

Home Assistant - Edit connection settings
Home Assistant – Edit connection settings

This I had not made with Min Volkswagen. The UI itself was not much different than the initial configuration, but I wanted it to be able to update the url to the MEP module and to change the scan interval (how frequently it polls the webservice).
The challenge with the url was that configuration and options are stored under different keys. And I didn’t like the idea of writing to both keys, and then making sure to read the right one if both were present.
Although I thought it would be a normal use case, I had to find a way to do it in a community thread where other developers had the same need.

Even worse when I wanted to make the scan interval configurable.
Gert had expressed concerns about the ESP32 being overloaded with requests, so I decided that polling it every 5 minutes would be sufficient for many use cases. In particular if the primary interest is the total kWh reading.
However, if you are interested in checking the power usage right now and want to see the difference when some device is on or off you need much more frequent updates.
Although you can buy plugs with the ability to report the consumption for a specific appliance, let’s also enable use cases that require readings from the meter almost as frequently as you might like for whatever purpose. When you have it in your own home and only polled by you, it is of much less risk of being overloaded than the module being exposed online right now.
HA has a really simple way for an integration to set a scan interval. It is a hardcoded value that I have not found a way to change dynamically. For integrations not using ConfigFlow and OptionsFlow it can be set in the configuration file, but making this apply requires a restart…
After studying other examples, I ended up disabling the built in triggering mechanism. Instead setting up a timer, that would fire an event to the sensors when it is time to update.
A change in the OptionsFlow dialog will reload the integration anyway, so although the timer is set up at load time it will update the timer interval when changed.

Besides the expected benefits, being able to reload the integration also turned out to have other advantages. Now it can unload the integration without restarting HA core, or enable and disable sensors without restarting. Just a much more convincing experience.

The integration is made publicly available for anyone to use. But to be usable in any way you will also need the hardware and not least the firmware, which currently awaits release from the NDA. So, continue to keep an eye on the blog for updates.
The integration can be found here: https://github.com/jnxxx/homeassistant-dabblerdk_powermeterreader
You will also see more screenshots in the README file.
Be aware that it does not contain any information about the MEP protocol. It just consumes the webservice provided by the ESP32 firmware, that is where the secret source is.

For ease of use and installation, I will try to get it into the HACS community listing of custom integrations.
The challenge will be if we can get a logo accepted, which is required. I already experienced once it is not so simple if you do not own the trademark rights. So, now we try to put it under the dabbler.dk name and logo.
Until included in HACS, you can add the url above as a repository to HACS yourself (as described in the installation instructions) and it will work just like it was. Only thing missing is the logo in the integrations overview.