Copy-on-write (COW) is a popular mechanism of lazy evaluation, that helps improve running speed and reduce memory requirements by transparently delaying the copying of data. Essentially, this is how it works: When you try to copy an object, you are instead given a fake object. Attempts to read that new object will instead read from the original object. Writing to the object will create a new copy (as originally requested) and refer to it in the future for reads and writes.

It allows to write simpler and safer programs. For instance, a programmer can pass “copies” of his data with minimal performance impact, and not have to worry about others changing his original data.

It’s great, but can it be more?

I present here two proposals for extending this idea for even greater optimization. They are far from my areas of expertise, so I hope they still make sense.

1. Copy-On-Write + Fragmentation

Fragmentation of data is a mechanism that allows different parts (blocks) of data to reside in different locations in memory while appearing intact (that is, sequential).  This mechanism is often accused of  slowing the computer down. However, its a critical feature of virtual memory, which is a basis to all modern operating systems.

Introducing fragmentation into your data structures has many benefits, but let’s discuss the benefits regarding COW, which may be obvious by now: On write, you don’t have to copy all of the data, just the blocks which are being changed. This can be a big difference, if you’re changing a few bytes in a 50mb string.

You still have to copy the meta-data (such as, where are the blocks, what is the next block, etc.), but that’s a small price to pay, and a reasonable design requirement.

Now instead of copying the entire data, you copy only a fragment of it. How big is that fragment? Perhaps a fixed size, such as 64k. But assuming you have no real restriction on the size of these data blocks, the next logical step, in my eyes, is to ask: Why not make it as small as possible? That is, why not intentionally fragment the block into three smaller blocks: Before the area that is to be written, the area that is to be written, and after the area to be written. At this point we continue as we originally planned: We copy only the block which is to be written, which is, of course, exactly as small as it can be.

Eventually, we have a model in which writing n bytes into a COWed data of m bytes takes O(n) time and memory, instead of the original O(m+n) time and O(m) memory. I argue that in the common case, n is significantly smaller than m, and so the win is big.

Of course, fragmentation has a habit of slowing down reading times. When fragmentation is “too high”, it is possible to defragment the memory (an occasional O(m) process). The optimal balance of fragmentation depends heavily on the frequency of reads vs of writes, but I argue that even a sub-optimal, common-case balance, will produce an improvement in performance.

Edit: I’ve been unclear about how it affects look-ups. Fragmentation to blocks of fixed size remains O(1) for getting and setting items. However, for variable-size blocks it’s not so simple. A search tree can achieve a look-up of  O(logn) where n is number of fragments, which is a lot slower than the original array peformance. It is probably only a good idea if you have access to the operating system’s memory management, or if the use of look-ups is rare (and then an occasional defragmentation would still be necessary). Still, fixed-size fragments are good enough, and they can be dynamically resized with little cost, as long as the resize is uniform.

2. Copy-On-Write-And-ReaD

Or in short, COWARD, is a mechanism to even further delay copying, to only after the written data is also read. That is, when the programmer requests to write data, this mechanism will instead journal the data, producing sort of a “diff”. Only when the programmer attempts to read the result, the original data is copied and the diff is applied. A diff structure is provided by any implementation of lazy evaluation, by definition, but perhaps there are other more suitable diff structures for this purpose.

This starts to make more sense with fragmentation: Then the diff can be applied only to the block that is read. And so, a block will be copied only if it is both written and read. In some cases, there may be very little intersection between the two (and so, very little copying).

So basically, COWARD is just a (non-)fancy name for an array of promises (not to be confused with a field of dreams). The (possible) novelty is in the way this array is created and used: transparently, and relatively efficiently. Note that, like the previous proposal, it provides little value in situations where the all of the data is altered or read. However, I argue it will significantly improve performance in cases where only part of the data is read and written.

It can, for instance, be useful in cases where an algorithm works on COWed data (which happens quite often) and provides more processing than the user requires. Using this method, only blocks that the user requests are copied — and if the calculations themselves are lazy — processed. And all of it transparent to both the user and the implementer of the algorithm .

Here’s to lazier COWs!

For a long time now, I’ve been hindered by the issue of utilities, or snippets. These are convenience functions and classes that are too small or too incomplete to justify a library, yet are useful enough to be used.
I’ve posted a few on my blog: Namespaces, X and now FileDict. Others I didn’t post, and include a priority queue, an A* implementation, a lazy-list, an LRU memoizer, etc. You probably have a few of those. I know because I see them on snippet sites.

However, I rarely actually use these in my code. I really want to. But once your code spans more than one file, you usually need to make a proper installation, or at least trouble your “users” a bit more. Usually saving a few lines just isn’t worth the trouble. Copy-pasting the snippet into the file is sometimes the solution, but it really pains me that I’ll have to re-paste it every time I improve my snippet.

I’m sure some of you looked at my snippets, or other people’s, thought “cool”, but never used them, simply because it was too much trouble.

Paradoxically, this is especially true when writing snippets. They are just one small file, and using another snippet would probably make them too hard to distribute. This is a magic-circle, for-ever limiting our snippets to a low level of sophistication, and discouraging re-use.

I want to break that circle. I want to create an economy of snippets, increasingly building on each other, eventually creating a “standard library” of their own. But how would one do that? I have one suggestion, along with a proof-of-concept, which I will present here.

PySnippets

PySnippets is my attempt of making snippets usable. It’s comprised of two solutions – a server and a client.

1. Server - A website for uploading snippets. Simple enough. You can rate them, tag them, discuss them, offer some documentation and of-course post newer versions.
2. Client - A python module that automagically imports snippets from the web. Essentially, it downloads the snippets you request to a cache, imports them if they’re already there, and periodically searches for updates.

The server is structured in a predictable way, so that the client knows how to fetch a snippet just by its name.

The Client

Here’s a usage example with my current client implementation, I creatively call “snippets”:

import snippets
antigravity = snippets.get('antigravity')  # "snippet-import"
antigravity.start(mode='xkcd')

Easy as that!

The snippets.get function looks for the module in the local snippets-cache. If it’s there, get just imports it and returns the module. If it’s not, it queries the server for a snippet called “antigravity” (names are unique), stores it in the cache, and the imports it. What the user notices is a 2-second pause the first time he ever imports that snippet, and nothing else from then on.

You can specify to download a specific version, like this:

filedict = snippets.get('filedict', version=0.1)

Auto-Updating Snippets

The current implementation also includes an “auto-update” feature: Periodically, before importing a module, the client surveys the server for a newer version of it. If a newer version exists, it downloads it to the cache and continues with the import.

Auto-updates can be disabled in a parameter to get.

The Server

The server is yet another service to upload snippets, however it has a slightly unusual design (which no other snippet site I know of has):

• A URL to a snippet is easy to deduce given its name.
• There is a conscious (though simple) support for versions.
• To increase reliability and trust (more on that later), uploaded snippets cannot be altered (but a new version can be issued)

Since I know very little about administration and server-maintenance, I chose wikidot.com to host my POC web-site. They have an elaborate support for permissions and most of the features I need, such as the ability to rate, tag and discuss snippets.

Trust

Perhaps the biggest issue with such a system is trust. Since you’re running code which resides online, you have to trust me not to maliciously alter the snippets, and also you have to trust the author of the snippet not to do so.

Perhaps higher-ups in the Python community would like to take some sponsorship of the project, removing the remaining trust-issues with the administrator (that’s me).

Implications

• To distribute your snippets, all you need is for the reciever to have an internet connection, and the snippets client.
• If you’re sending someone code, you can attach the client (it’s rather small, too), and just import away. The reciever will benefit from improvements and bugfixes to your snippets.
• You can use other people’s snippets just as easily, as long as you trust them.
• Snippets can now build on each other without worrying too much.

What if my user is offline?

Then probably PySnippets isn’t for him.

However, I do have some ideas, and might implement them if there is sufficient demand.

Afterword

PySnippets is my humble attempt at solving the utility/snippet reuse problems. I hope you like it and find it useful.

Please try it!

In my previous post I introduced FileDict. I did my best to get it right the first time, but as we all know, this is impossible for any non-trivial piece of code.
I want to thank everyone for their comments and remarks. It’s been very helpful.

The Unreliable Pickle

A special thanks goes to the mysterious commenter “R”, for pointing out that pickling identical objects may produce different strings (!), which are therefor inadequate to be used as keys. And my FileDict indeed suffered from this bug, as this example shows:

>>> key = (1, u'foo')
>>> d[(1, u'foo')] = 4
>>> d[(1, u'foo')]
4
>>> d[key]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "filedict.py", line 64, in __getitem__
    raise KeyError(key)
KeyError: (1, u'foo')

And if that’s not bad enough:

>>> d[key] = 5
>>> list(d.items())
[['a', 3], [(1, 2), 3], [(1, u'foo'), 4], [(1, u'foo'), 5]]

Ouch.
I’ve rewritten the entire storing mechanism to poll only on hash and compare keys after unpickling. This may be a bit slower, but I don’t (and shouldn’t) expect many colliding hashes anyway.
Bug is fixed.

DictMixin

Under popular demand, I’m now inheriting from DictMixin. It’s made my code a bit shorter, and was not at all painful.

Copy and Close

I no longer close the database on __del__, and instead I rely on the garbage collector. It seems to close the database on time, and it allows to one copy the dictionary (which, of course, will all be always have the same keys, but doesn’t have to have the same behavior or attributes).

New Source Code

Is available here

Python’s dictionary is possibly the most useful construct in the language.  And I argue that for some purposes, mapping it to a file (in real-time) can be even more useful.

Why?

The dictionary resides in memory, and so has three main “faults”:

1. It only lasts as long as your program does.
2. It occupies memory that might be useful for other, more commonly accessed, data.
3. It is limited to how much memory your machine has.

The first can be solved by pickling and unpickling the dictionary, but will not survive an unexpected shutdown (even putting the pickling in a try-finally block won’t protect it against all errors).

FileDict

FileDict is a dictionary interface I wrote, that saves and loads its data from a file using keys. Current version uses Sqlite3 to provide consistency, and as a by-product, acidity.

The result is a dictionary which at all-times exists as a file, has virtually no size limit, and can be accessed by several processes concurrently.

It is meant as a quick-and-simple general-purpose solution. It is rarely the best solution, but it is usually good enough.

Performance obviously cannot compare to the builtin dictionary, but it is reasonable and of low complexity (refer to sqlite for more details on that).

Uses

FileDict can be used for many purposes, including:

• Saving important data in a convinient manner
• Managing large amounts of data in dictionary form, without the mess of implementing paging or other complex solutions
• Communication between processes (sqlite supports multiple connections and implements ACID)

Examples

$ python
>>> import filedict
>>> d=filedict.FileDict(filename="example.dict")
>>> d['bla'] = 10
>>> d[(2,1)] = ['hello', (1,2) ]
-- exit --
$ python
>>> import filedict
>>> d=filedict.FileDict(filename="example.dict")
>>> print d['bla']
10
>>> print d.items()
[['bla', 10], [(2, 1), ['hello', (1, 2)]]]
>>> print dict(d)
{'bla': 10, (2, 1): ['hello', (1, 2)]}

>>> d=filedict.FileDict(filename="try.dict")
>>> with d.batch:  # using .batch suspend commits, making a batch of changes quicker
>>>    for i in range(100000):
>>>            d[i] = i**2
(takes about 8 seconds on my comp)
>>> print len(d)
100000
>>> del d[103]
>>> print len(d)
99999

Limitations

• All data (keys and values) must be pickle-able
• Keys must be hashable (perhaps this should be removed by hashing the pickled key)

Source Code

Is availible in here

Future

Additions in the future may include:

• An LRU-cache for fetching entries
• A storage strategy different than Sqlite

Other suggestions?

More than once I have wasted my time in attempt to introduce graphics into the webpage, some of these attempts are recorded in this very blog: First were my lines in CSS, followed by rotating rectangles. I say “wasted” because I have now found Raphaël, and I’m in love.

Raphaël

What is it? Their site is too modest, claiming “Raphaël is a small JavaScript library that should simplify your work with vector graphics on the web”.
While true, it is also a most ambitious project to change the way the web looks, whether they mean to or not.
All (popular) browsers support vector graphics: IE supports VML, and all other browsers support SVG. By implementing both languages and switching between them, you can create complex vector graphics that work in almost every browser.
This idea was new to me, at least, and I think it’s great. Now Raphaël comes along and does it for you. Nice.

My Prophecy
Vector graphics (VG) will become more popular. As VG libraries mature (and Raphaël is already in good shape), VG on the web would become so easy that anyone could use them, and many would. Soon enough you will have beautiful interactive GUIs. At fiirst snippets and demos, then complete libraries for everyone to use. What more can it do? Maybe interactive animations, maybe just add some spice to websites.
It probably won’t bury Flash, but I expect it would be a competition.

A few days ago I found myself writing pure ansi C code, after over two years of not touching it at all. In fact, these two years consisted mostly of Python, and a little of C#, both very far from it. While coding away, trying to get re-accustumed to mallocs and frees, return values instead of exceptions and pointer arithmatic, I had to perform a memcpy. Very easy, of course, but I could not for the life of me remember how to call it. I remembered rather vividly that it had 3 arguments, named ‘dst’, ’src’ and ‘len’, the first two being void* and the latter.. perhaps plain int? But in which order they came, I could not recall. Now, that is not so evil, of course, because a quick look at the C reference refreshed my memory (until next time). This repeated with other functions (such as memset). But only when it started to happen with my own functions, I realized how unnatural it is for me, to remember things by order and not by name. Names allow us to establish a context, to find reasoning. Also, we are (relatively) good at remembering names.

By using position as a way for transferring parameters, we are essentially calling them “one”, “two”, “three” and etc., which are still names, but are devoid of context or meaning, and are the same for every function. It is not only hard to remember the right order; we can endure it (as reality proves), but it makes our code less readable, less natural.

Also, it complicates changing existing APIs. Move an argument from its position, and you have to change the position everywhere in the code. This can easily lead to subtle bugs (consider switching dst and src, it may be very hard to find where this happened!). So if you append parameters, they must come in the end, which somtimes does not make much sense.

While all programming languages (that I know of) sin in this evil of positionality , one device takes it one step ahead: Regular expressions. Specfically, RE Substitution. Not only do you address its matches using numbers, but actually figuring out which number goes to which match can be confounding. I cannot count how many bugs this must have produced.

In conclusion:

Design your systems differently.
Let computers count. Let humans use names!

My CSS-Parrot gave me an appetite for more in-browser graphics. This, led to my newest creation: A rotating “3D” cube, written with JavaScript.

Check it out here: http://kanelynchsucks.com/sfsr/rot_cube.html

(glory to Tzahi for hosting it)

Please note, this cube is not really 3D.  I just hacked it together using a couple of tricks, which I think are rather clever.

My next post (or more) will explain thoroughly how I made the cube and the tricks that I used. But I highly recommend you, the readers of this blog, to try and figure it out yourself. It can be a fun exercise. The actual code is very readable, and the only mystery is – how did I do it?

My last post introduced the concept of X, a class which “absorbs” operations and behaves like a function.
As many people pointed out, this was merely a syntactic alternative to lambda. You may like it, you may not.
Now, after a rewrite, X can now be pickled. But let me explain first.

Python lambdas cannot be pickled. In fact, python code cannot be pickled.
Pickling an object, aka serializing, is converting the object’s state (that is, its data) to a string, which can at a later time be unpickled to re-create the object with that state. The unpickling process instanciates that class, assuming it has not changed, and updating the new instance’s state to the pickled one. Python code is never stored.

Trying to pickle a class or a function might appear to work, but it does not really pickle it; it simply pickles the reference to it (and its state). Unpickling the string in a new terminal would prove that (as would a quick analysis of resulting string).

Attempts to pickle methods, nested functions or lambdas fail on the spot. That is because a reference to them cannot be kept (Actually, it can be. But they are rather volatile, so it might not be wise). Eventually, python code, or even expressions, cannot be pickled.

This brings me back to X. X allows you to do just that:

>>> expr = 1 + (X + 3) * 4
>>> s = pickle.dumps(expr)

(destory objects, change objects, switch an interpreter, whatever you wish)

>>> expr2 = pickle.loads(s)
>>> expr2(5)
33

By using X, the programmer can blend dynamic code with his data, and still be able to pickle it.
I believe this removes a very big limitation.

Just to be fair, I will note that there is another way to achieve this: Keep your expressions in a string, and eval it when you need it run. I highly recommend not doing it.

X’s new source code (a bit cryptic, but it’s the best I could do. Suggestions for simplification are welcomed) :

"""
x.py

Author: Erez Sh.
Date  : 11/11/2008
"""

import operator

def identity(x):
	return x

class _Return(object):
	"Pickle-able!"
	def __init__(self, value):
		self._value = value

	def __call__(self, *args):
		return self._value

class _Partial(object):
	"Pickle-able!"
	def __init__(self, callable, *args):
		self._callable = callable
		self._args = args

	def __call__(self, *args, **kwargs):
		args = self._args + args
		return self._callable(*args)

class _X(object):
	def __init__(self, func, *args_to_run):
		self.__func = func
		self.__args_to_run = tuple(args_to_run)

	def __getstate__(self):
		return self.__func, self.__args_to_run
	def __setstate__(self, state):
		self.__func, self.__args_to_run = state
	def __reduce__(self):
		#raise Exception("Deprecated!")
		return object.__reduce__(self)

	def __apply_un_func(self, func ):
		return _X(func, _Partial(self))
	def __apply_bin_func(self, func, arg ):
		return _X(func, _Partial(self), _Return(arg))
	def __apply_rbin_func(self, func, arg ):
		return _X(func, _Return(arg), _Partial(self))
	def __apply_multargs_func(self, func, *args ):
		return _X(func, _Partial(self), *map(_Return,args))

	def __call__(self, arg):
		return self.__func(*[x(arg) for x in self.__args_to_run])

	def __getattr__(self, attr):
		return self.__apply_bin_func( getattr, attr )

	def call(self, *args, **kwargs):
		return self.__apply_multargs_func( apply, args, kwargs)

	# Containers
	def __getitem__(self, other):
		return self.__apply_bin_func( operator.getitem, other )
	def __getslice__(self, a,b=None,c=None):
		return self.__apply_bin_func( operator.getslice, other )
	def in_(self, other):
		return self.__apply_bin_func( operator.contains, other )

	# Arith
	def __add__(self, other):
		return self.__apply_bin_func( operator.add, other )
	def __sub__(self, other):
		return self.__apply_bin_func( operator.sub, other )
	def __mul__(self, other):
		return self.__apply_bin_func( operator.mul, other )
	def __div__(self, other):
		return self.__apply_bin_func( operator.div, other )
	def __floordiv__(self, other):
		return self.__apply_bin_func( operator.floordiv, other )
	def __truediv__(self, other):
		return self.__apply_bin_func( operator.truediv, other )
	def __mod__(self, other):
		return self.__apply_bin_func( operator.mod, other )
	def __pow__(self, other):
		return self.__apply_bin_func( operator.pow, other )

	def __radd__(self, other):
		return self.__apply_rbin_func( operator.add, other )
	def __rsub__(self, other):
		return self.__apply_rbin_func( operator.sub, other )
	def __rmul__(self, other):
		return self.__apply_rbin_func( operator.mul, other )
	def __rdiv__(self, other):
		return self.__apply_rbin_func( operator.div, other )
	def __rfloordiv__(self, other):
		return self.__apply_rbin_func( operator.floordiv, other )
	def __rtruediv__(self, other):
		return self.__apply_rbin_func( operator.truediv, other )
	def __rmod__(self, other):
		return self.__apply_rbin_func( operator.mod, other )
	def __rpow__(self, other):
		return self.__apply_rbin_func( operator.pow, other )

	# bitwise
	def __and__(self, other):
		return self.__apply_bin_func( operator.and_, other )
	def __or__(self, other):
		return self.__apply_bin_func( operator.or_, other )
	def __xor__(self, other):
		return self.__apply_bin_func( operator.xor, other )

	def __rand__(self, other):
		return self.__apply_rbin_func( operator.and_, other )
	def __ror__(self, other):
		return self.__apply_rbin_func( operator.or_, other )
	def __rxor__(self, other):
		return self.__apply_rbin_func( operator.xor, other )

	def __rshift__(self, other):
		return self.__apply_bin_func( operator.rshift, other )
	def __lshift__(self, other):
		return self.__apply_bin_func( operator.lshift, other )

	# Comparison
	def __lt__(self, other):
		return self.__apply_bin_func( operator.lt, other )
	def __le__(self, other):
		return self.__apply_bin_func( operator.le, other )
	def __eq__(self, other):
		return self.__apply_bin_func( operator.eq, other )
	def __ne__(self, other):
		return self.__apply_bin_func( operator.ne, other )
	def __ge__(self, other):
		return self.__apply_bin_func( operator.ge, other )
	def __gt__(self, other):
		return self.__apply_bin_func( operator.gt, other )

	def __abs__(self):
		return self.__apply_un_func( abs )
	def __neg__(self):
		return self.__apply_un_func( operator.neg )

X = _X(identity, identity)

Readers, meet X. X is a class I wrote in Python as an alternative to using lambda. It has two main features:

1. It acts as an identity function ( so X(3) == 3, etc. )
2. When performing operations on it, it returns a new class that acts as a corresponding function.

Let me explain. Doing X+2 will return a new class that whenever called with an argument, will return that argument added with 2. So:

>>> map( X+2, [1, 2, 3] )
[3, 4, 5]

>>> filter( X>0, [5, -3, 2, -1, 0, 13] )
[5, 2, 13]

>>> l = ["oh", "brave", "new", "world"]
>>> sorted(l,key=X[-1])
['world', 'brave', 'oh', 'new']

These operations can be chained:

>>> map(2**(X+1), range(10))
[2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]

>>> map( "P" + X[3:]*2 + "!", ["Hello", "Suzzy"] )
['Plolo!', 'Pzyzy!']

Caveats

X has a few limitations.

• Using X twice in the same expression probably won’t work (this can be solved)
• Since calling X evaluates it, it can’t emulate method calls. For that you have to do use call, like: X.upper.call() (‘hello’) –> ‘HELLO’.
• Not all operations can be “captured”. For example, the “in” operator. For that you have to use X.in_( … ), like: X.in_(range(10)) (5) –> True
• Not all attributes will be accessible
• More problems? Likely.

Conclusion

While not innovative nor a complete solution, I believe X can be a useful replacement for some uses of anonymous functions, providing a shorter and simpler syntax which is easier to read and understand.

It is provided here in full, in hope that it will be useful to my readers (Improvements and fixes are welcome):

class _X(object):
	def __init__(self, func):
		self.__func = func

	def __call__(self, arg):
		return self.__func(arg)

	def __getattr__(self, attr):
		return _X(lambda x: getattr(self(x), attr))
	def call(self, *args, **kwargs):
		return _X(lambda x: self(x)(*args,**kwargs))

	# Containers
	def __getitem__(self, other):
		return _X(lambda x: self(x)[other])
	def __getslice__(self, a,b=None,c=None):
		return _X(lambda x: self(x)[a:b:c])
	def in_(self, other):
		return _X(lambda x: self(x) in other)

	# Arith
	def __add__(self, other):
		return _X(lambda x: self(x) + other)
	def __sub__(self, other):
		return _X(lambda x: self(x) - other)
	def __mul__(self, other):
		return _X(lambda x: self(x) * other)
	def __div__(self, other):
		return _X(lambda x: self(x) / other)
	def __floordiv__(self, other):
		return _X(lambda x: self(x) // other)
	def __mod__(self, other):
		return _X(lambda x: self(x) % other)
	def __pow__(self, other):
		return _X(lambda x: self(x) ** other)

	def __radd__(self, other):
		return _X(lambda x: other + self(x))
	def __rsub__(self, other):
		return _X(lambda x: other - self(x))
	def __rmul__(self, other):
		return _X(lambda x: other * self(x))
	def __rdiv__(self, other):
		return _X(lambda x: other / self(x))
	def __rfloordiv__(self, other):
		return _X(lambda x: other // self(x))
	def __rmod__(self, other):
		return _X(lambda x: other % self(x))
	def __rpow__(self, other):
		return _X(lambda x: other ** self(x))

	# bitwise
	def __and__(self, other):
		return _X(lambda x: self(x) & other)
	def __or__(self, other):
		return _X(lambda x: self(x) | other)
	def __xor__(self, other):
		return _X(lambda x: self(x) ^ other)

	def __rand__(self, other):
		return _X(lambda x: other & self(x))
	def __ror__(self, other):
		return _X(lambda x: other | self(x))
	def __rxor__(self, other):
		return _X(lambda x: other ^ self(x))

	def __rshift__(self, other):
		return _X(lambda x: self(x) >> other)
	def __lshift__(self, other):
		return _X(lambda x: self(x) << other)

	# Comparison
	def __lt__(self, other):
		return _X(lambda x: self(x) < other)
	def __le__(self, other):
		return _X(lambda x: self(x) <= other)
	def __eq__(self, other):
		return _X(lambda x: self(x) == other)
	def __ne__(self, other):
		return _X(lambda x: self(x) != other)
	def __ge__(self, other):
		return _X(lambda x: self(x) >= other)
	def __gt__(self, other):
		return _X(lambda x: self(x) > other)

	def __abs__(self):
		return _X(lambda x: abs(self(x)))

X = _X(lambda x:x)

Put it in x.py and import as:
from x import X ]]> http://erezsh.wordpress.com/2008/11/01/fun-while-avoiding-lambda/feed/ 9 erezsh http://erezsh.wordpress.com/2008/09/02/namespaces-in-python-revisited/ http://erezsh.wordpress.com/2008/09/02/namespaces-in-python-revisited/#comments Tue, 02 Sep 2008 12:47:07 +0000 erezsh http://erezsh.wordpress.com/?p=93 ]]>

Taking a short break from my artwork craze (but I promise more is to come), I reviewed some of the new features and changes in Python, brought by versions 2.6 and 3.0. There are many interesting features, but a very specific one caught my eye: the ability to modify existing properties.

From the documentation:

Properties now have three attributes, getter, setter and deleter, that are decorators providing useful shortcuts for adding a getter, setter or deleter function to an existing property. You would use them like this:

class C(object):
    @property
    def x(self):
        return self._x

    @x.setter
    def x(self, value):
        self._x = value

    @x.deleter
    def x(self):
        del self._x

class D(C):
    @C.x.getter
    def x(self):
        return self._x * 2

    @x.setter
    def x(self, value):
        self._x = value / 2

I found this very interesting. If you’re, for some reason, a frequent reader of this blog, you’d remember how in a previous post I suggested a hack to allow namespaces inside Python classes. However, it was largely incomplete because I couldn’t get it to work for properties. Perhaps now I can?

Now, don’t let that pretty decorator syntax mislead you. Doing:

    @x.setter
    def x(self, value):
        self._x = value

is exactly identical to doing:

    def x_set(self, value): # Name changed to preserve 'x'
        self._x = value
    x = x.setter(x_set)

And this is exactly what I was missing to complete my code.

So here is the new make_namespace, applying also to properties:

(This was only tested on Python 3, but should also work for Python 2.6. It will not work on version 2.5 or lower)

import types
def make_namespace(self, ns_cls):
    "This code iterates all functions and properties in ns_cls and binds them to self"
    for attr_name in dir(ns_cls):
        attr = getattr(ns_cls, attr_name)
        if type(attr) == types.FunctionType:
            setattr(ns_cls, attr_name, attr.__get__(self) )
        elif type(attr) ==  property:
            setattr( ns_cls, attr_name, attr
                    .getter( lambda x  : attr.fget(self))
                    .setter( lambda x,v: attr.fset(self,v))
                    .deleter(lambda x  : attr.fdel(self))
                )
    return ns_cls()

Notice that I can chain getter, setter, and deleter, because they all return a property object.

And here is a possible usage:

class YetAnotherGuiWindow:
    def __init__(self, pos, size, ...):
        ...
        self._size = size
        self.spatial = make_namespace(self, self.spatial)

    class spatial:
        ...
        def _get_size(self):
            return self._size
        def _set_size(self, size):
            self._size = size
        size = property(_get_size, _set_size)
        ...

...
# Meanwhile, in my console
>>> g.spatial.size = (100,100)
>>> g.spatial.size
(100, 100)
>>> g.spatial.size = (140,10)
>>> g.spatial.size
(140, 10)
>>> g._size
(140, 10)

I hope you’ll find this useful!