Have a REST: building a mockup for a large RESTful service, part I.

So, You are familiar with the concept of REST and RESTful web services. You even created some already. But just imagine:

• you are starting a new large project; there are over 20 entitites!
• you have just a week, working with your JavaScript/UI developer, to deliver a mockup to your customer. 

Just one week and you are just two. And the UI guy depends on your backend. You must be fast. You just do not have time to implement alll these 20(30,64) REST handlers!

Have a rest. Im 90% cases, all your entities behave the same way. You already can transform your entities to JSON and back again (I already wrote about it). So why worry?

Okay, You are smart and You use Flask and Flask-RESTful

A universal real-only REST handler

First, we will create a simple readl-only REST controller template, so your UI guy can start.

from flask import abort, request
from flask.ext.restful import Resource, reqparse
from myapp.lib import database
from myapp import model as M

class ReadOnlyRestfulResource(Resource):
    Entity = None

    def get(self, id=None):
        if not id:
            query = database.session.query(self.Entity)
            return [ entity.simple_object() for entity in query ]

        entity = database.session.query(self.Entity).get(id)
        if not entity:
            abort(404)

        return entity.simple_object()

Okay, so we can create a module "views" like this - we just name each restful resource class as it's corresponding model class:

from myapp.lib.restful import ReadOnlyRestfulResource
from myapp import model as M

__all__ = ['Person', 'LegalEntity', 'Contract']

class Person(ReadOnlyRestfulResource):
    Entity = M.Person


class LegalEntity(ReadOnlyRestfulResource):
    Entity = M.LegalEntity


class Contract(ReadOnlyRestfulResource):
    Entity = M.Contract

Now, just import them in your app.py and attach routes:

from flask.ext.restful import Resource, Api

# create RESTfull API
api = Api(app)

# other app details skipped

import views

for view_name in views.__all__:
    try:
        view = getattr(views, view_name)
        view_name = view_name.lower()
        api.add_resource(view, '/' + view_name + '/', '/' + view_name, '/' + view_name + '/')
        app.logger.debug(u"Registered view {}".format(view_name))
    except Exception as e:
        app.logger.warn(e)

Great! Now, accessing /person will provide your UI guy with a list of all persons, and /contract/13 with contract #13, etc. Fill your database with sample data and you almost have your mockup alpha ready. Next you fill your database with 1000+ rows of sample Contract entities... and notice, that a list of all contracts is too large. You only want a contract id, datem and come status in the full list. So let's implement different filtering in 'get-all' and 'get-one'. Remember, our 'simple_object' method accepts a list of fields, so let's use it.

from flask import abort, request
from flask.ext.restful import Resource, reqparse
from myapp.lib import database
from myapp import model as M

class ReadOnlyRestfulResource(Resource):
    Entity = None
    fields_all = None
    fields_one = None

    def get(self, id=None):
        if not id:
            query = database.session.query(self.Entity)
            return [ entity.simple_object(self.fields_all) for entity in query ]

        entity = database.session.query(self.Entity).get(id)
        if not entity:
            abort(404)

        return entity.simple_object(self.fields_one)

And next we add some field filtering, if we want to. We can skip any or both filters:

class Person(ReadOnlyRestfulResource):
    Entity = M.Person
    fields_all = ['id', 'name', 'email']
    fields_one = ['id', 'name', 'email', 'telephone', 'legal_entity_id']


class LegalEntity(ReadOnlyRestfulResource):
    Entity = M.LegalEntity
    fields_all = ['id', 'display_name', 'vat_id']


class Contract(ReadOnlyRestfulResource):
    Entity = M.Contract
    fields_all = ['id', 'legal_entity_id', 'date_signed']

Okay, our output is much better. Next, we want some other filtering on the server-side. The reason is: we have 100+ customers, each has 100+ contracts; requesting all contracts... wow! I just want contracts for this customer!

class ReadOnlyRestfulResource(Resource):
    Entity = None
    fields_all = None
    fields_one = None

    def get(self, id=None):
        if not id:
            query = database.session.query(self.Entity)

            for k,v in request.args.iteritems():
                try:
                    column = self.Entity.__table__.c[k]
                    filter = column==v
                    query = query.filter(filter)
                except:
                    pass

            return [ entity.simple_object(self.fields_all) for entity in query ]

        entity = database.session.query(self.Entity).get(id)
        if not entity:
            abort(404)

        return entity.simple_object(self.fields_one)

Now we can request /contract?customer_id=13 - and we fetch only desired contracts.
At this point, we can grab some Faxe, Spendrups or Lapin Kulta and have a rest, while our UI guy creates his templates, etc. Or we can even help him! :)
What's next?

• implement handling POST, e.g. creating new entities;
• add some advanced filtering;
• implement PATCH for updating existing entities.

So, to be continued!

From SQLAlchemy to JSON and back again, part 1

JSON is one of the most valuable data interchange format nowadays. It is native for JavaScript, which is a de-facto standart for building rich wed apps. It is simple to implement is other languages when writing a web-app backend. But actually you do not store objects as a JSON string? Yes, there are NoSQL databases that expose data in JSON format, but for some reasons developers still use traditional relational databases, and use ORM to provide object-to-database access.

Okay, let us have a relational database, an SQLAlchemy ORM, and we are writing a web-app backend in Python. In fact, we deal with Python objects, while SQLAlchemy does 99.9% for storing them in DB. So how should we convert these objects to JSON and back again?

Simple Objects

What we will call a simple object? It's a complex datastructure, built of lists, dictionaries, strings, integers and floats, boolean an None, with arbitrary nesting, and the 'top-level' data structure will we a dictionary.

Example (from iPython console):

In [2]: d = dict(type='SimpleObject', id=3, name='Simple object #3', friends=[1,2])

In [3]: d
Out[3]: 
{'friends': [1, 2],
 'id': 3,
 'name': 'Simple object #3',
 'type': 'SimpleObject'}

The reason for that is that we can easily convert this representation to JSON:

In [6]: import simplejson
In [7]: simplejson.dumps(d)
Out[7]: '{"friends": [1, 2], "type": "SimpleObject", "name": "Simple object #3", "id": 3}'

... and back:

In [8]: d1 = simplejson.loads('{"friends": [1, 2], "type": "SimpleObject", "name": "Simple object #3", "id": 3}')

In [9]: d1
Out[9]: 
{u'friends': [1, 2],
 u'id': 3,
 u'name': u'Simple object #3',
 u'type': u'SimpleObject'}

That's great, it's the same! So how do we convert other objects to this simple representation and back?

A SimpleObject Mixin

Let's design and implement a mixin class, that we will use with our SQLAlchemy Base to build entitites.

The prototype

class SimpleObjectMixin(object):
    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        # TODO: build the object
        return obj

    def json(self, attrs=None):
        return simplejson.dumps(self.simple_object(attrs))

The idea: the `simple_object` method will introspect the object and build a simple representation of it. The optional `attrs` argument will be used to filter attributes, if required.

Let's start building. First we will always include the `type` field with the value of class name.

    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        obj = dict(type=self.__class__.__name__)

        return obj

Okay, so we have a dictionary with the class name there. How do we iterate over all attributes in a class?

    def simple_object(self, attrs=None):
        """ Returns a Simple Object representation
        """
        obj = dict(type=self.__class__.__name__)

        for name in self.__dict__.keys()
            # skip 'private' attributes
            if name[0] == '_':    
                continue

            # skip an attribute, if we have `attrs` passed
            if attrs and name not in attrs:    
                continue

            value = getattr(self, name)

            # a None value does not need any conversion
            if value is None:
                obj[name] = None
                continue

            # Maybe, the `value` object can convert itself?
            try:
                value = value.simple_object()
                continue
            except AttributeError:
                pass

            # No, we do some handling ourselves:
            if isinstance(value, datetime):
                try:
                    obj[name] = time.mktime(value.timetuple())
                except:
                    obj[name] = 0
            elif isinstance(value, dict):
                obj[name] = {k: v.simple_object() for k, v in value.iteritems()}
            elif isinstance(value, list):
                obj[name] = [v.simple_object() for v in value]
            else:
                obj[name] = unicode(value)

        return obj

Updating

What about creating a new or updating existing entity from JSON? In fact, it's not as simple as getting JSON for existing object. But remember, we are creating a mixin class for SQLAlchemy, and Base's default constructor can initialize the entity with provided dictionary. But it has no built-in update method.

This simple code will help us a bit - in fact, it has many caveats and limitations, we will discuss them further.

    def update(self, dikt, set_nones=False):
        """ Update current object from given dictionary
        """
        for k, v in dikt.iteritems():
            if k[0] == '_':
                continue
            if v is None and not set_nones:
                continue
            setattr(self, k, v)

Okay, that's a working alpha code! What it still does not:

• correctly handle SQLAlchemy relations;
• update related object state from SimpleObject/JSON.

To be continued soon.

Auto-timestamping SQLAlchemy entities

There are a lot of cases when we want to track time when an entity was created or updated. Here is a simple recipe to make some or all of your SQLAlchemy entities auto-timestamping. To achieve this, we will provide a mixin class. Prerequisutes:

from datetime import datetime
from sqlalchemy import Column, DateTime, event

The mixin class:

class TimeStampMixin(object):
    """ Timestamping mixin
    """
    created_at = Column(DateTime, default=datetime.utcnow)
    updated_at = Column(DateTime, default=datetime.utcnow)

    @staticmethod
    def _updated_at(mapper, connection, target):
        target.updated_at = datetime.utcnow()

    @classmethod
    def __declare_last__(cls):
        event.listen(cls, 'before_update', cls._updated_at)

This mixin class adds two columns, 'created_at' and 'updated_at', they are initialized with current timestamp. A static method `_updated_at` just updates the `updated_at` field for the `target` instance it receives. But the real magic is done in `__declare_last__` method. According to SQLAlchemy docs (see http://docs.sqlalchemy.org/en/rel_0_9/orm/extensions/declarative.html#declare-last) it gets called after the mapper is configured. The reason is that we cannot set a listener for an unmapped class, and mixin itself will never be mapped. We could add listeners for each entity directly, but we want some magic. So, this method adds a listener for `before_update` event for each derived class! And finally,

from sqlalchemy.ext.declarative import declarative_base

class Base(TimeStampMixin):
    pass

Base = declarative_base(cls=Base)

Just derive all your model entities from this custom Base class - and magic is here.

The reason and a welcome

So, it's time to blog again.

There are several reasons for that.

1. I have something to share.
2. I would like to share it.
3. Sometimes, I need comments :-)

Still short notes will stay on my G+ feed. This place is for longer posts.

def test():
    pass