Echelon/NES Smart Meters – probably our final post on this topic

Sad face

Today I got a bit of a surprise – without any warning my power distribution company changed my power meter from a NES Echelon meter to a Landis+Gyr E360 meter.

This means that I unfortunately have no way of testing and working on new hardware/software for our Dabbler MEP interface.

I hope someone else with access to a NES meter with a MEP interface will pick up the task and improve the hard- and software further.

We’ll continue to keep the Dabbler GitHub running on https://github.com/dabblerdk.

…and we’ll also keep the Documentation and message board GitHub we have setup in corporation with OSGP Alliance running: https://github.com/OSGP-Alliance-MEP-and-Optical.

Echelon/NES Smart Meters – MEP interface modules is now available again from www.ustepper.com

uStepper

Thanks to our friends at www.ustepper.com hardware to IoT enable your NES/Echelon meter is now available again.

You can purchase it directly from their shop here: https://ustepper.com/shop/home/26-esp32-c3-dabbler-mep-interface.html

Note, although this is a completely redesigned hardware based on a different ESP32 module (it uses the ESP32C3 and not the ESP32-WROOM-32 of our original module), we have extended our software to also support the new hardware.

The source code is the same, and the board selection configures the software so i will run on the chosen module. Simply select the “ESP32 Dev Module” board for the “dabbler hardware” and the “ESP32C3 Dev Module” board for the “uStepper hardware”.

You will also need the .wwws files for both versions.

As always you can find our updated software on our GitHub here: https://github.com/DabblerDK/MEP-SW-ESP32

As always other small fixes and improvements were done and going forward we’ll plan to keep supporting both hardware versions.

Echelon/NES Smart Meters – MEP interface modules will (hopefully) soon be for sale again

uStepper

As you might have discovered we recently shut down our sales of MEP interface modules and kits.

Our initial goals were to:

  • Get the MEP documentation and specs released for the public so everyone could tinker with the interface.
    We succeeded this almost a year ago, and as a bonus the IR interface documentation and specs were also released. You’ll find both on the GitHub we have setup for OSGP Alliance here: https://github.com/OSGP-Alliance-MEP-and-Optical
    Note: The OSGP Alliance is the global non-profit association dedicated to promoting the adoption of the Open Smart Grid Protocol (OSGP) and infrastructure for smart grid applications towards a future proof modern smart grid. Networked Energy Services (NES) is a member and chose that the documentation / specs should be released via OSGP Alliance.
  • Do a prototype implementation of a module that put the Echelon/NES meters online.
    Note: It was NEVER our intention to develop a finished consumer product, nor sell/distribute such products.
  • Get people involved in tinkering with the MEP interface – either by getting them to do their own projects or by getting them involved in developing “our” prototype further.

As you might know we had (and still have) a hard time getting more people involved, but we have had some community contributions to the code and inspiration stories. To try to push this further, we chose to do a “limited time offer” where you could buy either parts (kits) or assembled modules of our prototype.

Unfortunately the offer started just as Graves became a father and I got a new job. So we both had “real life” situations that demanded more of our time. Combined with a higher demand on the offer than expected, we had to extend the offer far longer than expected.

Our “limited time offer” had “limited effects” :-). We had hoped that someone would step in and start producing these modules, and although we had a few interested people contacting us, nothing happened that ended up in something we could refer others to. In the end we kind of gave up. Also we caught up with the initial demand for modules and started shutting down the offer. As usual we purchase too many parts, so it took some time before we were out. But we are now completely out of parts and don’t plan to buy any more.

Well, this “whining” is just to explain that it was neither (and never were) our intentions to bring this project further than to the prototyping state, nor our intentions to sell these modules and kits. We have full time jobs in “real life”, so we don’t want to start a business selling these modules (it won’t be viable without other products too – the demand for MEP modules is simply not high enough for this to happen).

Now for the good news! / we are happy to announce:

After shutting down our offer, we were contacted by a party that do run a business selling various electronic products – and they are currently working on a cost reduced version of the module. We have allowed them to build on our project free of charge and the plan is it will end up in a more or less compatible version running the same software (or a variant of it).

They are still in the development phase (there is a lot of work in bringing a tinkering prototype to a state where it can be produced and sold in an effective/viable manner), but they have allowed us to tell you about it and share their name.

If everything goes as planned you will soon be able to purchase a “Dabbler” MEP module variant from the Danish company uStepper. Stay tuned on both their website and our Blog!

[Limited offer has now ended] Echelon/NES Smart Meters – MEP interfaces modules and kits

Limited Time Offer

Update as per 2023-03-30: We are out of PCBs and parts and are not planning to order any new ones.

The Limited offer has now finally ended. It has been running FAR longer than we expected due to high demand – we were simply not able to catch up, and when we did, we had some extra modules and kits we wanted to provide.

Anyway – everything comes to an end, and we simply don’t have the time to keep providing these kits and modules. We hope that someone else will pick up this task – and we will be happy to refer/forward any requests etc…

Remember all Hard- and Software is (and will continue to be) available on our GitHub here: https://github.com/dabblerdk

Below you’ll find the original text of this entry along with the comment section:


This is a follow up on our previous Echelon/NES Smart Meter post.

We have released all our Hard- and software on GitHub here: https://github.com/dabblerdk, so your cheapest way of getting started if you have soldering skills is to order a PCB and components – and start building your own modules.

We hope that someone will pick up the task of producing/selling kits and assembled modules, but we have decided to offer some ourselves for a limited time to get more people involved.
Note: The price is set so others will have a chance of under-bidding us, that is on purpose.

If you buy one of our modules we expect you to participate in the project in some way. It could be as simple as write your story/experience about the project (as a guest writer on www.dabbler.dk or on the OSGP Alliance GitHub site), or if you have the skills you can help develop/improve the hard- and software.

We have two options:

  1. The MEP module as a kit, all components included. The ESP32 is a 16MB version.
    You are required to have the tools and skills, to assemble the PCB, adjust the buck converter and program the ESP32.
    The PCB/components are classic through-hole-technology (THT) so you don’t need to have good SMD skills.
    But note that the ESP32 is a separate PCB with castellated holes (that is kind of like SMD).
    As the module is NOT assembled, we cannot test it or install the software. You’ll have to do that yourself.
    Price: 300 DKK including freight in Denmark (payment due before shipment (bank transfer, PayPal or Mobile Pay).

  2. An assembled MEP module. The ESP32 is a 16MB version.
    The Module is flashed and should boot into Access Point mode when mounted in the Smart Meter. You should then be able to go to the homepage of the module and configure it to use your WIFI. Here you will also be able to enter the MBK key you got from your Power Company.
    PCB and components are mostly classic through-hole-technology (THT), but the ESP32 is a board with castellated holes.
    We might also choose to provide SMD versions at the same price depending on component availability. You will be asked if SMD is OK with you before shipping.
    Before shipment we will make sure the module boots correctly in our Smart Meter, but we won’t perform a long/extensive test of the module.
    Price: 500 DKK including freight in Denmark (payment due before shipment (bank transfer, PayPal or Mobile Pay).

IMPORTANT:
This is a prototype project without ANY form of warranty. Any use of this module is your risk and costs!
We cannot gurantee delivery times as we purchase most components from China to keep the cost low. Delivery times may vary from 0 to 2 months or in special cases even longer.

NOTE:
AS we currently have no experience with the IR solution, we will NOT be providing any hard- or software for that at this point.

ORDERING:
Please send an e-mail to gert@dabbler.dk and graves@dabbler.dk, stating if you want a kit or an assembled module. Also state your shipment address.

Please let us know if you have any questions.

[Breaking News] Echelon/NES Smart Meters – Documentation for MEP and Optical interfaces have been released on GitHub!

Breaking News!

It is with great pleasure we finally can announce the release of the following official OSGP Alliance documentation for OSGP Smart Meters (i.e. Echelon/NES Smart Meters):

  • The Multipurpose Expansion Port (MEP) interface and protocol
  • Optical interface and protocol

You will find the PDF files in the Documentation repository at GitHub using this link: https://github.com/OSGP-Alliance-MEP-and-Optical.

Thank you for your patience, we know this have been under way for almost two years.

Now, go ahead, please start Tinkering now! 🙂

Note: The hard- and software we’ve been tinkering with is not finished (probably will never be finished as it is a tinkering project 🙂 ), but we’ll get it released ASAP so others can get inspired or join as they wish.

Echelon/NES Smart Meters – dabbling the hardware v1.10 and v2.00

PCB v1.10 and 2.00

Our software fix for the (fake?) MAX3232 hardware issue seems to solve the issue completely.

It is still a rough implementation in the software (using delays), so we’ll have to improve on that later on… It is probably not that big of an issue – because when a MAX3232 starts to behave, it seems it continue that way for a long time…

Next step is then off cause a new hardware prototype PCB, but why not add a few things like:

  • R3 & R4: a few resistors so one can chose if our software fix for the (fake?) MAX3232s should be used (0 ohm R4 installed) or not (0 ohm R3 installed). You might even chose to use a higher resistor value if that works for you. You should only install either R3 or R4 – NOT BOTH!
    Note: If you don’t have 0 ohms resistors, just bridge the pins.
  • R5, R6 & R7: a few resistors on the MAX3232 pins so you have the option to see if your MAX3232 works better if you add some higher resistor values to the pins. If not, you can just install 0 ohm resisters or bridge the pins…
  • …and let us move to SMD now we are at it (actually we also have a PCB version 1.10 which is classic THT (Through Hole Technology) as the earlier versions.

The schematics now looks like this:

Hardware version 1.10 and 2.00 schematics

A 3D rendering of the PCB (for some reason without a 3D MAX3232) looks like this:

ESP32 MEP PCB v2.00 (SMD) - front
ESP32 MEP PCB v2.00 (SMD) – front
ESP32 MEP PCB v2.00 (SMD) - back
ESP32 MEP PCB v2.00 (SMD) – back

The idea around putting C1 – C5 on the back is to shorten the traces from the MAX3232 (supposedly that is also better according to it’s datasheet… But what do we know – we are software guys 🙂 )

We’ve just received the PCBs from JLC PCB (no, they are not a sponsor – we pay for this), but have not had time to populate nor test them yet…

You can see the received PCBs in the feature image of this blog entry.

Stay tuned for more news regarding this project – only on www.dabbler.dk…

Note: We are still waiting for NES to release the MEP protocol specifications…

Echelon/NES Smart Meters – the software solution to a hardware issue

Well, we warned you that software guys like us tend to come up with software solutions – even for hardware problems.

As expected, no electronic engineers reached out to us with a solution for our (probably fake) MAX3232 issues, so we had to come up with a solution ourselves…

That is off cause ok – and with the help of a cut PCB trace:

Hardware v1.03 - cut trace to solve MAX3232 issues
Hardware v1.03 – cut trace to solve MAX3232 issues

… and a bodge wire:

Hardware v1.03 - bodge wire to solve MAX3232 issues
Hardware v1.03 – bodge wire to solve MAX3232 issues

… it seems we can persuade the MAX3232’s to play along.

This is off cause a temporary solution to test this fix – we need a new PCB prototype version if this works.

Note: You will find others struggling with similar MAX232/3232 issues on the Internet. Some are able to solve it with larger decoupling capacitors and some with resisters on the power pin – but that did not work for us.

Our implementation is simply adding control of the power to the MAX3232 through software. When it mis-behaves we punish it by turning it off for a while. Then back on until it behaves…

Note: the max current a ESP32 digital pin can supply seems to be around 40mA and the max consumption of a MAX3232 is around 1mA. So we should be safe doing this “hack”…

If you are an electronic engineer you are probably finding this solution fun – but it seems to be working :-).

While we are still waiting for the MEP Protocol specification to be released (and our NDA to be lifted), we’ll try to keep your entertained with more blog entries.

Stay tuned to www.dabbler.dk…

(Guest writer) Echelon/NES Smart Meters – Home Assistant integration

Home Assistant Logo

This is a bonus entry on our blog in the Echelon/NES Smart Meters series. We have limited time and are devoting the sparse time we have on the hard- and software needed for the ESP32-MEP module. We are aware that other systems should be able to integrate to the solution, but we currently have no experience with stuff like “Home Assistant” etc.

This project was never supposed to be a dabbler.dk private project anyway. As you know if you’ve been following this blog, the end goal is to get the specification released. As soon as that happen we’ll release our hard- and software as well, and others can choose to join us in improving it, start their own project from the specs or even branch our project and move on from there.

We would love to see this grow in all sorts of directions…

Luckily others have more knowledge in integrating with “Home Assistant” and one of the more persistent persuaders must have been Jan Nielsen. He got a private access to a webservice on Gert’s ESP32-MEP module to see if he was able to integrate to it…

…and best of all: when asked he immediately wrote this blog entry about it – sharing his findings. The rest of this blog post is written by Jan Nielsen, so send all praises and credits to him:


Jan Nielsen
Educated as Data Technician (Data Mekaniker) from EUC Syd in Sønderborg in 1997.

I have worked some years as a full-time developer; then switched over to system administration.
But still with an interest in automation when there is a need, and occasionally some hobby projects.
https://www.linkedin.com/in/jan-nielsen-21a5832/


Hi there, my name is Jan. Like I assume many of the people reading this, I have been following this blog for some time. I found it back in August 2021, after first seeing Gerts post at ing.dk.

At the time I had recently discovered the Home Assistant project. My interest was to get a chance to play around with Zigbee – more than Philips Hue allowed for.
Shortly after the August update was released. The big news there was an energy dashboard, and a few more things related to it like long-term statistics etc. In this context energy meant electricity consumption and production (solar).
Without digital access to the overall household consumption this was not really useful.
You can get Zigbee, Wifi, Bluetooth og probably Z-wave plugs that can measure power consumption, but it is only a portion of the consumption.


Some solutions for reading total electricity consumption:

  • A guy named Jonas Pedersen had made a HA integration that pulls in data from an API at eloverblik.dk. This is working, but data is two days delayed, and for that reason not suitable for the energy dashboard.
  • The HA release mentioned two solutions that were already supported. A solution for Dutch meters that apparently has a serial port with an open and documented protocol, and another one for counting the number of blinks from a LED on the meter. Well yes, that second one would probably work with an Echelon meter also, but still not quite delivering what I hoped for. For example it does not get the total consumption value seen in the display, if for some reason counting pulses is impaired consumption during that time is simply missed. Additionally I assume it will not give current power usage in Watt, or it must be derived from time between two pulses, which can be quite lengthy when consumption is low. If I have to fiddle with electronics, it has to be a solution I believe in, and that I find at least somewhat smart.
    Similarly ready-made devices like this are available for purchase. But it seems fairly expensive, when in doubt how well it can actually do it.
  • Lastly, Develco seems to have developed a really interesting module for the MEP port. But as the dabbler.dk guys discovered it cannot be purchased anywhere, at least not only a single or a few pieces.


    So, when I found this blog here it was the most promising solution. Thus, I have been waiting and regularly checked for news and progress…

At the end of January when I saw Gert mentioning some web service, my interest was raised further.
In the autumn I made a HA integration for Min Volkswagen. It simply pulled in data from a connectedcars.io API. I first learned about this API from Esben on tech at Youtube. However, his approach was to use Node-RED. This I found a bit cumbersome, especially with the location tracker that had to be configured in multiple places. So, I took the challenge and tried to write an integration for HA, which would be easy to use and configure.
The API was really easy to use as Esben had shown how to do it. The HA part was more challenging and had quite some learning curve, but by studying other examples, documentation and at times HA source code I eventually got it working, including a UI dialog for the initial configuration.
Back to the MEP webservice… if the dabbler.dk guys had made a webservice I would rather easily be able to pull this data in, similarly to how I had already done with Min Volkswagen. So, I reached out to Gert offering to be a tester 😉, but also to implement a HA integration that could consume the webservice. Due to the NDA Gert was, understandably, not that keen on the idea, but he did offer remote access to a webservice. As for the development process this would be perfectly ok, and overall bring us faster to my goal of getting this data into HA. So, I accepted Gerts offer.

So, I cloned my previous HA integration and started to adjust the API reader code. As the webservice is very simple this was mostly a matter of removing authentication, and code to handle multiple cars/devices from the same web service instance.
Simplified, the basic steps to read it looks somewhat like this:

try:
async with aiohttp.ClientSession() as session:
async with session.get(req_url, headers = headers) as response:
temp = await response.json()
if temp is not None:
_LOGGER.debug(f"Got meter data: {json.dumps(temp)}")
self._data = temp

except aiohttp.ClientError as client_error:
...

And next to adjust each value, named a sensor in HA, that I wanted to make available in HA:

...
sensors = []
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "energy consumption", "", False, True, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "energy returned", "", True, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L1", False, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L2", False, False, _meterclient))
sensors.append(MeterEntity(config_entry.entry_id, config["name"], "voltage", "L3", False, False, _meterclient))
...
async_add_entities(sensors, update_before_add=True)

class MeterEntity(Entity):
"""Representation of a Sensor."""

def __init__(self, config_entry_id, meterName, itemName, phase, returned, entity_registry_enabled_default, meterclient):
"""Initialize the sensor."""
...
if self._itemName == "energy consumption":
self._unit = ENERGY_KILO_WATT_HOUR
self._icon = "mdi:home-import-outline"
self._device_class = DEVICE_CLASS_ENERGY
self._dict["state_class"] = "total_increasing"
elif ...

async def async_update(self):
"""Fetch new state data for the sensor.
This is the only method that should fetch new data for Home Assistant.
"""
try:

# Measurement
if self._itemName == "energy consumption":
energy = await self._meterclient._get_value(["Fwd_Act_Wh"])
if (energy is not None):
self._state = int(energy) / 1000

Within a day or so data started to roll in.

At first the webservice had the accumulated consumption, returned to the grid as well as voltage and current for each phase. As I find power more relevant, I had to calculate this from voltage and current. Since then the webservice has been extended with power readings and more.

When it all seemed to run smoothly, one day it turned out to have registered a humongous consumption.

Unfortunately, I did not make a screenshot of it before correcting the data, but here is what it normally should look like:

Home Assistant - Echelon energy consumption
Home Assistant – Echelon energy consumption

As it is the kWh counter it should be forever increasing. Little consumption will be slowly increasing, higher consumption will increase steeper.
When the problem appeared, it rose all of a sudden thousands of kWhs, and shortly after fell back to normal.

Due to the way I had defined the sensor in HA, this was a bigger issue than just that short glitch.
For it to be usable with the Energy dashboard it needs to have statistics collected each hour, and this requires a ‘state_class’ defined. For this kind of data it can be ‘total’ or ‘total_increasing’. To support meter reset the ‘total’ type requires a date specified where the meter was reset. Instead the ‘total_increasing’ type, which I had chosen, will see a decrease as a reset.
As the Energy dashboard works with the deltas, not only did the increase add thousands to the depicted consumption, when returning to normal it also added 117.000 kWhs because it assumed it was reset to zero and then consumed the reported value…

Gert had also noticed this at times, and had also mentioned it to me earlier. But in my rush to get something to show up I had also totally forgotten about it.
Rather than having to specify a reset date or risking negative consumption, I decided to continue with ‘total_increasing’. However, I did implement a limitation that would sanitize the kWh reading. If it increased more than 1000 Wh or decreased from the latest reading, it would discard the new value for up to an hour until finally accepting the new value if not returned to a more realistic value. It is only done in memory though, so not totally fail proof if HA is restarted during an issue, but it minimizes the risk.
A few days later Gert reported the issue had been found and firmware updated.

At the same time there was more info added to the webservice.
In a previous blog, it was shown that it could return info about itself like manufacturer, model and version. Although secondary, I find it nice and convincingly to have that reported as well.
Not only did the added info have metadata about the meter, it also turned out that similar data about the MEP module had been added.
In HA a device only has a predefined set of properties. So, how to depict both sets of device info?
Although not originally planned, it turned out simple enough, just make it appear as two devices. The meter itself and the MEP module:

Home Assistant - Echelon
Home Assistant – Echelon
Home Assistant - Echelon MEP
Home Assistant – Echelon MEP

To do this, it requires at least one sensor on the MEP module. This is because it is the sensor that returns info about the device it is part of.
In this case I invented two binary_sensors that tell something about the status. One if the last http request succeeded, and two if the data returned doesn’t look as expected.
The “Connected via” relationship was new to me, and also something I wanted to try out… 😁 But it just worked, so no challenge in that.

As Gert and Graves had taken care of the difficult part, the most challenging part of this integration turned out to be the Configuration button, or as it is known internally the OptionsFlow.

Home Assistant - Edit connection settings
Home Assistant – Edit connection settings

This I had not made with Min Volkswagen. The UI itself was not much different than the initial configuration, but I wanted it to be able to update the url to the MEP module and to change the scan interval (how frequently it polls the webservice).
The challenge with the url was that configuration and options are stored under different keys. And I didn’t like the idea of writing to both keys, and then making sure to read the right one if both were present.
Although I thought it would be a normal use case, I had to find a way to do it in a community thread where other developers had the same need.

Even worse when I wanted to make the scan interval configurable.
Gert had expressed concerns about the ESP32 being overloaded with requests, so I decided that polling it every 5 minutes would be sufficient for many use cases. In particular if the primary interest is the total kWh reading.
However, if you are interested in checking the power usage right now and want to see the difference when some device is on or off you need much more frequent updates.
Although you can buy plugs with the ability to report the consumption for a specific appliance, let’s also enable use cases that require readings from the meter almost as frequently as you might like for whatever purpose. When you have it in your own home and only polled by you, it is of much less risk of being overloaded than the module being exposed online right now.
HA has a really simple way for an integration to set a scan interval. It is a hardcoded value that I have not found a way to change dynamically. For integrations not using ConfigFlow and OptionsFlow it can be set in the configuration file, but making this apply requires a restart…
After studying other examples, I ended up disabling the built in triggering mechanism. Instead setting up a timer, that would fire an event to the sensors when it is time to update.
A change in the OptionsFlow dialog will reload the integration anyway, so although the timer is set up at load time it will update the timer interval when changed.

Besides the expected benefits, being able to reload the integration also turned out to have other advantages. Now it can unload the integration without restarting HA core, or enable and disable sensors without restarting. Just a much more convincing experience.

The integration is made publicly available for anyone to use. But to be usable in any way you will also need the hardware and not least the firmware, which currently awaits release from the NDA. So, continue to keep an eye on the blog for updates.
The integration can be found here: https://github.com/jnxxx/homeassistant-dabblerdk_powermeterreader
You will also see more screenshots in the README file.
Be aware that it does not contain any information about the MEP protocol. It just consumes the webservice provided by the ESP32 firmware, that is where the secret source is.

For ease of use and installation, I will try to get it into the HACS community listing of custom integrations.
The challenge will be if we can get a logo accepted, which is required. I already experienced once it is not so simple if you do not own the trademark rights. So, now we try to put it under the dabbler.dk name and logo.
Until included in HACS, you can add the url above as a repository to HACS yourself (as described in the installation instructions) and it will work just like it was. Only thing missing is the logo in the integrations overview.

(Long time no see) Echelon/NES Smart Meters – Draft document received!!

MEP Client Developer's Guide, cover page

Sorry it’s been some time since the last update – but please read the comment sections on the older entries – this project is still kicking, and we try to find time to at least answer the comments.

Let us admit – we were worried… It seems NES went into some kind of radio silence mode during end of 2021 and start of 2022 – but finally in mid January we manage to get a hold on them.

They apologized that the MEP-protocol specifications was delayed – the team simply had to work on other more urgent projects… We fully understand that stuff can get delayed when you are “hit by reality”. After all – this release is for the community, not something a company can pay their bills with :-). And Graves and Gert also have to attend their day job to get bills paid. This is – after all – only a spare time project.

Anyway – NES assured us that it would now get prioritized a bit higher again and hopefully soon be finished.

BUT EVEN BETTER – earlier this week, Gert received a draft PDF with the “release version” of the “MEP Client Developer’s Guide”. This shows us that NES are still actively working on this. Due to the NDA we cannot share much, but you will find an image of the cover page as the featured image of this entry.

It is clear that this is still supposed to be released through OSGP Alliance. Just like promised previously by NES.

A quick glance through through the PDF shows that the document has been revised, even improved a lot. And – more important – it seems that most of the protocol is still in there… So this is REALLY good news. We’ve promised to proof read it a bit and give feedback. But we are really looking forward to be able to link to the release when it is ready.

Sorry for this short – but at least positive – update.

We’ll get back to you with progress on software and hardware soon. And hopefully also soon with some news about the release of the documentation.