Encoding/decoding dictionaries to byte arrays for LoRa/serial

[MrTinkerman]

Hi all,

I'm working with LoRa and currently I use dictionaries/JSON to transmit messages. It keeps everything clear and readable but if I understand correctly each character in the JSON strings (including { " " : }) is transmitted as one byte, right? Since I have a fixed set of variables and my values are 0-255 I thought I'd explore the option to encode my dictionaries to byte arrays.

I wrote the class below and it seems to work fine but I'm hoping for some feedback and answers to my questions about the returned byte array. Below is the code and screenshots with comment (because tabbed comments are totally messed up when posting code)

General question are:

• Is this even going make transmitting more efficient?
• Is there a library that does (something like) this?
• Am I missing something obvious?

Question 1, size:
I would assume that a byte array of 8 bytes has a size of, well 8 bytes, so why is the size reported as 48?

Question 2, characters:
When encoding {'var1':60, 'var2':75, 'var3':88}
The result is printed as : bytearray(b'\x00<\x01K\x02X')
I read that this is just the way a byte array is printed, if it 'sees' an ASCII code it prints the character, is this correct?

Thanks for reading!

The class:

class EcDcDict:

    def __init__(self, var_list):                       
        self.enc_dict = {var:ind for ind, var in enumerate(var_list)}
        self.dec_dict = {ind:var for var, ind in self.enc_dict.items()}

    def encode(self, enc):                
        enc_list = list()                                                                                                 
        for key, val in enc.items():										
            enc_list.extend([self.enc_dict[key], val])						
        return bytearray(enc_list)											

    def decode(self, dec):
        decoded = dict()
        for a in range(0, len(dec), 2):
            decoded[ self.dec_dict[ dec[a] ] ] = dec[a+1] 
        return decoded

Example code:

from ecdcdict import EcDcDict
import json, gc

var_list = ['var1', 'var2', 'var3']					
ecdc = EcDcDict(var_list)					

my_dict = {'var1':10, 'var2':20, 'var3':30}	

gc.collect()
before = gc.mem_alloc()
encoded = ecdc.encode(my_dict)
print(f'Encoded dict: {encoded}')					
gc.collect()
print(f'Size: {gc.mem_alloc() - before}')

decoded = ecdc.decode(encoded)
print(decoded)

gc.collect()
before = gc.mem_alloc()
json_dict = json.dumps(my_dict)
print(f'JSON: {json_dict}')							
gc.collect()
print(f'Size: {gc.mem_alloc() - before}')			

[mendenm]

answer to question 2, first: yes, printing a bytearray does print anything which is a printable, 7-bit ASCII character as such. Non-printable characters below 0x7f and all characters above 0x7f are printed as \x codes. (Note: I believe all printables are 0x20-0x3f (space through ?), 0x40-0x5f (@ through _) and 0x60-0x7e(` through ~))

question 1: don't trust gc.mem_alloc() in too much detail. For example, your byte array header, and the symbols 'before' and 'encoded' take up some space, along with the literal contents of the array.

[projectgus]

answer to question 2, first: yes, printing a bytearray does print anything which is a printable,

Further to this, if you want to see the bytearray as hex then you can print encoded.hex() instead.

[projectgus]

• Is there a library that does (something like) this?

You might find the micropython-lib cbor2 library useful: https://github.com/micropython/micropython-lib/tree/master/python-ecosys/cbor2

CBOR is a standard binary encoding format, and the cbor2 library has a compatible API to the json one. Install with mip install cbor2.

I think A CBOR encoded dict will be bigger than your encoding, because IIUC your encoding doesn't include the dict keys in the messages. However you could make a list of values and encode this as cbor2, and it would probably be a similar size.

[peterhinch]

This document describes options for serialisation in MicroPython. There are some interesting tradeoffs between various methods.

[MrTinkerman]

Thanks for the feedback all. The document on serialisation has some good info and the suggested cbor2 library might interesting in case I need to transmit other datatypes.

I did a quick test on cbor2 and it look like it's around 30% reduction in bytes, but that's just a first limited test.

In my current project transmission speed is priority. My homebrew solution transmits only 2 bytes for each key/value pair, with the limitation that the values can only be 0-255 and TX/RX both need to have the list of all variables. It's all about tradeoffs...

[MrTinkerman]

With some help from perplexity.ai I wrote test code and my class has a reduction rate of 0.16 / 84%

The class:

class EcDcDict:
    
    def __init__(self, var_list):
        self.enc_dict = {var:ind for ind, var in enumerate(var_list)}
        self.dec_dict = {ind:var for var, ind in self.enc_dict.items()}
        
    def encode(self, enc):
        enc_list = list()													
        for key, val in enc.items():										
            enc_list.extend([self.enc_dict[key], val])						
        return bytearray(enc_list)											
                                                                            
    def decode(self, dec):
        decoded = dict()
        for a in range(0, len(dec), 2):
            decoded[ self.dec_dict[ dec[a] ] ] = dec[a+1]
            return decoded

The test code:

from ecdcdict import EcDcDict
import json

def gen_var_list(n):
    
    var_list = [f'var{i:02}' for i in range(1, n + 1)]
    return var_list

def gen_test_dicts(n):
    
    test_dicts = []
    for i in range(1, n + 1):
        test_dict = {f'var{j:02}': j for j in range(1, i + 1)}
        test_dicts.append(test_dict)
    
    return test_dicts

num = 50
var_list   = gen_var_list(num)
test_dicts = gen_test_dicts(num) 

ecdc = EcDcDict(var_list)

def run_test():
    
    ratios = []

    for d in test_dicts:

        json_len = len(json.dumps(d))
        ecdc_len = len(ecdc.encode(d))
        ratios.append(ecdc_len / json_len)
        
    return sum(ratios)/len(ratios)
    
    
print(run_test())