From SQLAlchemy to JSON and back again, part 1

JSON is one of the most valuable data interchange format nowadays. It is native for JavaScript, which is a de-facto standart for building rich wed apps. It is simple to implement is other languages when writing a web-app backend. But actually you do not store objects as a JSON string? Yes, there are NoSQL databases that expose data in JSON format, but for some reasons developers still use traditional relational databases, and use ORM to provide object-to-database access.

Okay, let us have a relational database, an SQLAlchemy ORM, and we are writing a web-app backend in Python. In fact, we deal with Python objects, while SQLAlchemy does 99.9% for storing them in DB. So how should we convert these objects to JSON and back again?

Simple Objects

What we will call a simple object? It's a complex datastructure, built of lists, dictionaries, strings, integers and floats, boolean an None, with arbitrary nesting, and the 'top-level' data structure will we a dictionary.

Example (from iPython console):

In [2]: d = dict(type='SimpleObject', id=3, name='Simple object #3', friends=[1,2])

In [3]: d
Out[3]: 
{'friends': [1, 2],
 'id': 3,
 'name': 'Simple object #3',
 'type': 'SimpleObject'}

The reason for that is that we can easily convert this representation to JSON:

In [6]: import simplejson
In [7]: simplejson.dumps(d)
Out[7]: '{"friends": [1, 2], "type": "SimpleObject", "name": "Simple object #3", "id": 3}'

... and back:

In [8]: d1 = simplejson.loads('{"friends": [1, 2], "type": "SimpleObject", "name": "Simple object #3", "id": 3}')

In [9]: d1
Out[9]: 
{u'friends': [1, 2],
 u'id': 3,
 u'name': u'Simple object #3',
 u'type': u'SimpleObject'}

That's great, it's the same! So how do we convert other objects to this simple representation and back?

A SimpleObject Mixin

Let's design and implement a mixin class, that we will use with our SQLAlchemy Base to build entitites.

The prototype

class SimpleObjectMixin(object):
    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        # TODO: build the object
        return obj

    def json(self, attrs=None):
        return simplejson.dumps(self.simple_object(attrs))

The idea: the `simple_object` method will introspect the object and build a simple representation of it. The optional `attrs` argument will be used to filter attributes, if required.

Let's start building. First we will always include the `type` field with the value of class name.

    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        obj = dict(type=self.__class__.__name__)

        return obj

Okay, so we have a dictionary with the class name there. How do we iterate over all attributes in a class?

    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        obj = dict(type=self.__class__.__name__)

        for name in self.__dict__.keys()
            # skip 'private' attributes
            if name[0] == '_':    
                continue

            # skip an attribute, if we have `attrs` passed
            if attrs and name not in attrs:    
                continue

            value = getattr(self, name)

            # a None value does not need any conversion
            if value is None:
                obj[name] = None
                continue

            # Maybe, the `value` object can convert itself?
            try:
                value = value.simple_object()
                continue
            except AttributeError:
                pass

            # No, we do some handling ourselves:
            if isinstance(value, datetime):
                try:
                    obj[name] = time.mktime(value.timetuple())
                except:
                    obj[name] = 0
            elif isinstance(value, dict):
                obj[name] = {k: v.simple_object() for k, v in value.iteritems()}
            elif isinstance(value, list):
                obj[name] = [v.simple_object() for v in value]
            else:
                obj[name] = unicode(value)

        return obj

Updating

What about creating a new or updating existing entity from JSON? In fact, it's not as simple as getting JSON for existing object. But remember, we are creating a mixin class for SQLAlchemy, and Base's default constructor can initialize the entity with provided dictionary. But it has no built-in update method.

This simple code will help us a bit - in fact, it has many caveats and limitations, we will discuss them further.

    def update(self, dikt, set_nones=False):
        """ Update current object from given dictionary
        """
        for k, v in dikt.iteritems():
            if k[0] == '_':
                continue
            if v is None and not set_nones:
                continue
            setattr(self, k, v)

Okay, that's a working alpha code! What it still does not:

• correctly handle SQLAlchemy relations;
• update related object state from SimpleObject/JSON.

To be continued soon.