How to clear an io.BytesIO buffer.

[wnelis]

An ESP8266, programmed using microPython 1.22.1 via Thonny 4.1.4, has two sensors attached to it and the measured values are reported via JSON-encoded MQTT messages. A message with the values from one sensor is build in an io.BytesIO() buffer. The basic flow is currently:

  pld= io.BytesIO( 511 )
  pld.write( sensor_0_header )
  pld.write( sensor_0_body )
  MqttClient.publish( topic_0, pld.getvalue() )
  pld.close()
  del pld

  pld= io.BytesIO( 511 )
  pld.write( sensor_1_header )
  pld.write( sensor_1_body )
  MqttClient.publish( topic_1, pld.getvalue() )
  pld.close()
  del pld

  gc.collect()

Buffer pld is thus used once, than deleted and recreated. That is the only way I found so far to clear the content of pld, that is the value of method pld.getvalue() becomes b''. I wonder if it is possible to clear buffer pld without recreating it, as object creation (and thus deletion and thus garbage collection) can be relatively costly on such a small processor.

[bixb922]

I'd use a preallocated bytearray and memoryview, not BytesIO

[jimmo]

Yep, you can use slice-assignment to update your payload bytearray.

# keep this around (global, class member, etc)
buf = bytearray(511)

buf[:len(sensor_0_header)] = sensor_0_header
buf[len(sensor_0_header):] = sensor_0_body
MqttClient.publish( topic_0, buf )

buf[:len(sensor_1_header)] = sensor_1_header
buf[len(sensor_1_header):] = sensor_1_body
MqttClient.publish( topic_1, buf )

note you don't actually need a memoryview here.

We're considering adding something like micropython.memcpy which would make a memoryview useful, because then you could avoid the slicing altogether (e.g. make a memoryview pointing to the region for sensor_0_header, and then memcpy directly into it).

[wnelis]

Unfortunately, an io.BytesIO object does not support slice-assignment when using MicroPython version 1.22.1 on an ESP8266:

>>> buf= io.BytesIO(127)
>>> buf.getvalue()
b''
>>> buf.write( 'Ffo Bar' )
7
>>> buf.getvalue()
b'Ffo Bar'
>>> buf[:3]= b'Foo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'BytesIO' object doesn't support item assignment
>>> 

[jimmo]

@wnelis That's right, you should use bytearray instead of BytesIO.

[wnelis]

As a static pre-allocated bytearray seems to be the way to go, I'll try to use class StaticBuffer, shown below, as a replacement of class io.BytesIO. Class StaticBuffer is (for now) modeled after class io.BytesIO.

class StaticBufferException( Exception ):
  pass

# 0 <= next <= lngt <= size
class StaticBuffer():

  def __init__( self, size ):
    self._bfr= bytearray( size )
    self.size= size			# Allocated size
    self.lngt= 0			# Current length
    self.next= 0			# Ordinal to start next write

  def clear( self ):
    self.lngt= 0
    self.next= 0

  def flush( self ):
    pass

  def getvalue( self ):
    return bytes(self._bfr[:self.lngt])

  def seek( self, ordinal ):
    if ordinal < 0  or  ordinal > self.next:
      raise StaticBufferException( b'Seek ordinal out of range' )
    self.next= ordinal

  def tell( self ):
    return self.next

  def write( self, astring ):
    if not isinstance( astring, bytes):
      raise StaticBufferException( b'Wrong parameter type' )
    eos= self.next + len(astring)	# End of slice ordinal
    if eos > self.size:
      raise StaticBufferException( b'String too long' )
    self._bfr[self.next:eos]= astring
    self.next= eos
    self.lngt= max( self.lngt, eos )
    return len(astring)

A test run, using Thonny 4.1.4, shows that now it is possible to rewrite and to clear the buffer:

>>> from StaticBuffer import *
>>> pld= StaticBuffer(64)
>>> (pld.size,pld.lngt,pld.next)
(64, 0, 0)
>>> pld.write( b'Aap noot Mies' )
13
>>> pld.getvalue()
b'Aap noot Mies'
>>> (pld.size,pld.lngt,pld.next)
(64, 13, 13)
>>> pld.seek( 4 )
>>> pld.write( b'N' )
1
>>> pld.getvalue()
b'Aap Noot Mies'
>>> (pld.size,pld.lngt,pld.next)
(64, 13, 5)
>>> pld.clear()
>>> pld.getvalue()
b''
>>> (pld.size,pld.lngt,pld.next)
(64, 0, 0)

The StaticBuffer object should be created as a global variable. I hope that the overhead of using a pure Python class is less than the overhead of the (avoided) garbage collection.

[bixb922]

Nice! You can save more memory allocations changing return bytes(self._bfr[:self.lngt]) to return memoryview(self._bfr) [:self.lngt]

[wnelis]

How about return bytes( memoryview(self._bfr)[:self.lngt] )? Does the type-casting result in a memory allocation?

[bixb922]

Yes, bytes() will create a readonly copy of the data. I don't know which mqtt client you are using, but normally bytes, bytearray and memoryview will work interchangeably as argument for input binary data

[wnelis]

Module umqtt.simple is in use, which can be found on https://pypi.org/. A small test shows that a memoryview is indeed accepted and results in a nicely formatted MQTT message. Just for fun, one of the two messages, captured with mosquitto_sub, is shown below:

Client (null) received PUBLISH (d0, q0, r0, m0, 'sensor/dhpt/dhpt00', ... (232 bytes))
{"type":"dhpt","identifier":"00","dew point":[8.96,8.92,8.98,"C","dew point"],"humidity":[0.518,0.516,0.519,"","humidity"],"pressure":[99890,99884,99895,"Pa","pressure"],"temperature":[19.13,19.10,19.14,"C","temperature"],"count":0}

Thanks.