Protocol Buffers

[Xylios]

Hello,

Me and my colleague are using your implementation of Protocol Buffers for Racket.
However we are having some issues using the Serialize and Deserialize commands.
At the moment we are using as output a file, using the binary mode and if something exists on the file we use append.

(define output (open-output-file "D:\\DocumentsD\\test.bin" #:mode 'binary #:exists 'append))

Our idea is to serialize two messages

(serialize (strct* #:x 100 #:s "test1") output)
(serialize (strct* #:x 200 #:s "test2") output)

Followed by two deserialize

(define str1 (deserialize (strct*) input))
(define str2 (deserialize (strct*) input))

Unfortunately, we get the following error
deserialize: missing required fields: (seteqv 1 2)
This happens when we do the second deserialize.

str1 contains the second serialize, x = 200 s = test2

Can you help us find a solution to this problem?

If you would like to see the code, the full source is here.

#lang racket
(require (planet murphy/protobuf:1:1))
(require (planet murphy/protobuf/syntax))
(require (planet murphy/protobuf/encoding))

(define output (open-output-file "D:\\DocumentsD\\test.bin" #:mode 'binary #:exists 'append))

;;The contract
(define-message-type strct ([required primitive:int64 x 1] [required primitive:string s 2]) )

;;Racket sending the message
(serialize (strct* #:x 100 #:s "test1") output)
(serialize (strct* #:x 200 #:s "test2") output)

;;Close output port
(close-output-port output)

(define input (open-input-file "D:\\DocumentsD\\test.bin" #:mode 'binary))

(define str1 (deserialize (strct*) input))
(define str2 (deserialize (strct*) input))

;;Close input port
(close-input-port input)

[murphy]

This is not a bug, you are running into an idiosyncrasy of the protocol buffer format!

Protocol buffers have no delimiters or size information at the top level, hence the deserialization function will always try to read the entire stream. When multiple protocol buffers are appended the way you are doing it, they are effectively combined into one larger message with some duplicate fields.

If you want to add simple length based framing to your stored records, you can use procedures from the encoding layer of my package: Simply replace (serialize STUFF output) by (write-sized serialize STUFF output) and (deserialize TYPE input) by (read-sized (cut deserialize TYPE <>) input), where cut is taken from SRFI 26.