nspyre.data.streaming._pickle

Implement special Pickling functions that can be used for streaming data. Instead of serializing and deserializing an entire object, pickle operations will only serialize/deserialize the differences since the most recent pickle. This can be used to efficiently stream data e.g. over a network connection.

Module Contents

Classes

PickleDiff	Represents a serialized object, where some of sub-objects are special
StreamingPickler	Special pickler for streamed objects.
StreamingUnpickler	Special unpickler for streamed objects.

Functions

streaming_pickle_diff(obj)	Special streaming pickle function. For special Streaming objects,
streaming_load_pickle_diff(stream_obj_db, pickle_diff)	Special streaming unpickle function. For special Streaming objects, only
serialize_pickle_diff(pickle_diffs)	Serialize the PickleDiff object.
deserialize_pickle_diff(pickle_diffs)	Deserialize the PickleDiff object.

class nspyre.data.streaming._pickle.PickleDiff(pkl=None, diffs=None)

Represents a serialized object, where some of sub-objects are special streaming-capable objects. For streaming objects, a set of diffs is used to reconstruct the object.

Parameters:

• pkl (Optional[bytes]) – Byte string from pickle operation.

• diffs (Optional[Dict]) – Dict where keys are uid and values are lists of diffs as dictated by the corresponding class-specific code in persistent_id().

squash(pd)

Merge the given (more recent) PickleDiff into this PickleDiff.

Parameters:

pd – PickleDiff object to squash into this one.

class nspyre.data.streaming._pickle.StreamingPickler(file)

Special pickler for streamed objects.

Parameters:

file – file-like object where the pickle will be saved.

persistent_id(obj)

class nspyre.data.streaming._pickle.StreamingUnpickler(file, stream_obj_db, diff)

Special unpickler for streamed objects.

Parameters:

• file – File-like object where the pickle will be saved.

• stream_obj_db – Dictionary where the keys are an object uid and the value is a Streaming object.

• diff – Dictionary of diffs.

persistent_load(pid)

nspyre.data.streaming._pickle.streaming_pickle_diff(obj)

Special streaming pickle function. For special Streaming objects, extract the differences since the last pickle. These differences are returned separately from the main pickle data.

Parameters:

obj (Any) – The object to pickle.

Returns:

Tuple of the form (obj pickled, diffs dictionary)

Return type:

PickleDiff

nspyre.data.streaming._pickle.streaming_load_pickle_diff(stream_obj_db, pickle_diff)

Special streaming unpickle function. For special Streaming objects, only the differences since the last pickle will be unpickled, then the objects will be updated.

Parameters:

• stream_obj_db (Dict) – Dictionary for storing the special streaming objects. The keys are an object uid and the value is a Streaming object. This will be populated automatically. It must persist between unpickling operations in order to perform the updating.

• pickle_diff (PickleDiff) – PickleDiff object.

Returns:

Unpickled object.

Return type:

Any

nspyre.data.streaming._pickle.serialize_pickle_diff(pickle_diffs)

Serialize the PickleDiff object.

Parameters:

pickle_diffs (PickleDiff) – The differences since the last pickle.

Returns:

Serialized pickle, diffs.

Return type:

bytes

nspyre.data.streaming._pickle.deserialize_pickle_diff(pickle_diffs)

Deserialize the PickleDiff object.

Parameters:

pickle_diffs (bytes) – Dictionary containing the differences since the last pickle.

Returns:

Dictionary containing the differences since the last pickle.

Return type:

Any