Composable Object Streams
In my last post I introduced the pcap-socket module to help test against real, captured network data. I was rather happy with how that module turned out, so I decided to mock out dgram next in order to support testing UDP packets as well.
I almost immediately ran into a few issues:
dgrammodule does not implement a streams2 duplex API. It still provides an old-style “spew-stream”.
- I wanted to share code with
pcap-socketfor parsing ethernet frames and IP headers.
- I also wanted to implement some missing features such IP fragment reassembly. These types of features require operating across multiple packets and therefore fit the streaming model better than the simple utility function approach.
Using the basic rebuffering byte streams provided by the new streams2 API seemed problematic. For one, UDP packets are distinct and shouldn’t be summarily rebuffered. Also, I needed a way to extract packet header information and pass it along with the byte stream.
This seemed to solve a lot of my problems. I could now turn off rebuffering and I could pass arbitrary objects around.
The new object mode, however, did create one new issue.
Now that streams are not just ordered bytes, how can I write general purpose composable streams other people could easily use? If every person uses their own object structure then it could be very difficult to put together separate stream modules in a useful way.
Over time I came up with an approach that seemed to work well for the network protocol domain. Essentially, I structured messages like this:
1 2 3 4 5 6 7 8 9 10
Each message is the combination of some binary
Buffer data and additional
meta-data. In this example I have an ethernet frame that’s been parsed off
the start of the
After a full parsing chain:
1 2 3 4 5 6
out message might look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
This lets us inspect all of the extracted information at the end of the processing pipeline.
This approach also lets different stream implementations work together. For
IpStream can inspect the
msg.ether.type property provided
EtherStream to see if
msg represents an IP packet or not.
This approach also allows streams to avoid stepping on each others toes. Both
src properties, but they don’t
conflict because they are namespaced under
To help solicit feedback on this approach I started a gist that outlines the approach. If you’re interested or have an opinion please check it out. I’d love to know if there is a better, more standard way to build these sorts of object streams.
Oh, and I did finally implement the
dgram mock object. See the
pcap-dgram module for examples. Both it and
pcap-socket are built on
top of the new ether-stream and ip-stream. And
indeed now support fragmentation reassembly.
All of these composable object streams are implemented using a new base class module called object-transform. It makes it fairly easy to write these kinds of transformations.
Of course, this still leaves me with my first issue. The dgram core
module still does not provide a streams2 API. If this message structure makes
sense, however, it should now be possible to provide this API without losing
rinfo meta-data provided in the current
'message' event. UDP wouldn’t
benefit from the back pressure improvements, but this would allow
pipe() into other composable stream modules.