ludovf.net blog

Jfokus 2012

2012-02-21T00:00:00+01:00

Here are my notes after attending the Jfokus in Stockholm this year.

Tradition

Java EE programmers seem to feel most at home in the JEE/patterns paradigms. Injection, repositories, DAOs are everyday fare. The Java worldview has been abundantly shaped by this ‘software engineering’ worldview. Hacking or intellectual sophistication of the mathematical kind come second.

Attendee profile

Striped shirts, higher education. Serious engineering. Big company culture. Mostly male, as always. Why aren’t there more women in software engineering? I highly doubt that innate capabilities are at play.

Buzzwords

Domain driven design, command query separation and Every presentation namedrops ‘command objects’ and separating methods that modify state from those that do not.

Web sockets and interactivity

Flash starts to feel outdated (because it is not supported by Apple?), WebSocket becomes an interesting replacement; some industries like finance and entertainment particularly feel the need for push content. But there are also other applications, like pushing horse race results in real time.

Every second you win on user time to complete a task, multiplied by the number of users, yields savings. Therefore increasing responsiveness makes business sense in any domain.

Functional buzz

There were multiple presentations on Scala and Clojure. Also everyone talks about immutability. Is it the start of a revolution of rigour and mathematics in programming?

Fixtures vs Factories

2012-02-05T00:00:00+01:00

Many web frameworks integrate fixtures: a mechanism to define sample data to be loaded in the database for testing. The format of choice is often YAML, and the frameworks supplies classes for setting up and removing this data.

But all too often you load data into the database when all you really need is to access sample objects in your test code. Inserting data into the database for each run slows down tests considerably. Not to mention that with JPA and probably other ORMs as well, you are forced to load a lot more data than strictly required for your tests just to satisfy some foreign key constraint. You might try to solve this by exporting data from your production database, but you will invariably encounter another headache as the export does not work as expected, or importing fails because some data needed to satisfy a foreign key constraint is missing.

Fixtures are not parametrizable, so you are forced to create an entire new set if for some reason you need a different value for some object property in a certain test.

Finally, accessing the database makes your tests also test the database connection: they are not unit tests any more.

Fixtures are often the default choice for no better reason that there is support in the framework; nothing is preventing you from to writing some classes which can build your test specimens without reading data from the database.

Ruby has got Factory Girl, which provides a very concise DSL-ish API to create your objects, but, whatever the programming language, it is easy to write a set of factory methods that can configure the objects you require for testing.

If you wish your tests will become faster, easier to setup and more suitable for test-driven development, you might want to look at this and other methods to avoid touching the database too much in your tests.

The refactoring dilemma

2012-01-27T00:00:00+01:00

It is an accepted principle that you should leave working code alone. But sometimes you need to modify its behaviour. Now admit that the existing code is fairly opaque, that there are no tests and that the application has never been used nor tested extensively.

In any case, you will need to change. You could either

introduce your change with minimal modifications
refactor the existing code into something more understandable before introducing your change

The second option seems riskier at first thought. On the other hand, the ‘minimal modifications’ could turn out to have unexpected repercussions all through the rest of the application. In that case, the second option would give at least result in clearer code.

To determine which method is more efficient in a particular case, you could ask a programmer to try one approach and then another, but the result would be distorted because the second time round the programmer would have a better understanding of the code base. Alternatively, you could ask two programmers to attempt a different solution each, but it is unlikely that the two programmers would be of equal talent, so you would not get an answer about which method is better, but about which programmer is more experienced.

It is often suggested to add tests before operating on a legacy codebase; that often forces to adopt the second solution, since, in its current state, the application might be impossible to test.

Fluctuating requirements can contribute to the incertitude; but maybe, the whole situation represents an opportunity to take control and put your design hat on. How should this application work? What code is practical to write given the budget or time constraints?

Up and down the stack

2012-01-20T00:00:00+01:00

I am starting to become more and more convinced that to spot issues with a piece of software, you need to go take a wider view of what consitutes ‘the application’. In a typical Java server side application, ‘the application’ does not include only the code running on the JVM.

For instance, looking at a database query log can be very instructive. It is amazing how much the number of generated queries can be reduced in some cases by some simple changes.

But it does not stop at the database. You need to inspect the whole environment the application is deployed in. The web server configuration can be a source of unnoticed bugs, and if logging is not setup correctly you will be left in the dark.

Focus on unit testing is great, but as long as you limit yourself to testing your development configuration, the deployed application could still not function as required after all tests pass.

This raises the very hairy question of configuration management making whole environments reproducible and testable. The issue is further complicated when new programs must be deployed together legacy systems, where the configuration might be undocumented. You are then stuck with an unkown environment that you cannot modify for fear of breaking the legacy application.

How to sell tech books in a physical bookstore

2012-01-16T00:00:00+01:00

I am leaning towards the opinion that it is pointless to keep a tech book assortment in a physical bookstore, as the target audience is going to buy their books online and tech books run out of date so quickly that they probably are an inventory headache.

There is a large general bookstore called Selexyz in Rotterdam. By all appearances, the least likely place where to look for an interesting tech book assortment. But this particular store seems to have given some thought to what they would put in stock.

Here is a sample of what they had on display: the latest edition of a well-known book (Bulletproof CSS by Dan Cederholm! There’s a new edition out! I didn’t know it!), recent shorter O’Reilly books about trendy technologies (Cassandra, Mining the Social Web) and a selection of the weightier Manning tomes on similar subjects (since Manning does not do shorter formats as O’Reilly), like Erlang/OTP, and Lift. Some classics like the Gang of Four book and Martin Fowler’s Domain Specific Languages and even an academic book on Information Retrieval and collection of Donald Knuth’s writings on games.

Thye must have hired someone who knows his or her stuff: while most shops of this kind make a weak attempt at some sort of comprehensive reference (one book about C++, one book about Java, one book about PHP, and so on), here it feels like they are attempting to target Hacker News reader impulse buys. There are a lot new, half of them relatively short books on buzzword topics. You might not have ordered them from Amazon yet or bought the eBook; but if you see them on the shelf you might be tempted to take them home with you.

There is still room for experimenting in physical retail stores. Even if I believe that tech books will shortly be published almost exclusively on the web and tech publishers will become editing consultants, it’s good to remember that the alternative is not necessarily either jumping on the new technology or remaining stuck in the old ways, but there are ways to adapt the old technology to compete with the new.

Mapping a composite key in JPA with @EmbeddedId

2012-01-11T00:00:00+01:00

Why @Embedded objects?

JPA sits in the uncomfortable middle between managing relations stored in the database and providing object-oriented programming facilities. The general rule is that you annotate your classes with @Entity and each class will correspond to a separate table, with while each field of the class will correspond to a column in that table.

The @Embedded and @Embeddable annotation exists to allow you to define two separate classes without using two separate tables in the database. You annotated a class with @Embeddable instead of @Entity, and then add a field of the type of the @Embeddable class to another class annotated with @Entity. Then the fields of the @Embeddable class will be mapped to column in the table associated with the class annotated with @Entity.

The reason that this annotation is not well-known is that it is much more frequent for two related classes to be backed by two separate tables with a foreign key relationship. In that case, the relevant JPA annotations are @ManyToOne and @OneToMany, as you will no doubt know.

@EmbeddedId and composite keys

@EmbeddedId is also used in conjunction with @Embeddable and works exactly the same way as @Embedded, except that if you use @EmbeddedId instead of @Embedded, the columns corresponding to the fields of the class annotated with @Embeddable will be interpreted by a JPA implementation as the composite key for that table.

JPA attempts to offer constructs which make sense both from an object oriented and relational perspective. The more perplexing parts of the JPA specification arise when the correspondence between the object oriented and the relational side breaks down. Using @EmbeddedId just to indicate a composite key forces you to create a new class which is going to be completely useless from the object-oriented programming standpoint.

It can become even more confusing because there is another way to designate composite primary keys in JPA, using the @IdClass annotation, which results in the fields corresponding to the composite primary key being defined twice: once on the main class and once on the class whose purpose is to hold the fields making up the composite key: but let’s not cover that approach in this post.

Hitting JPA’s limits

JPA does allow you to map almost any database design to Java types. In this regard, the specification can be considered a success. On the other hand, the resulting object system can be weird and filled with objects whose purpose is only to map a database table. An example of this is the way composite keys are declared with the @EmbeddedId annotation.

I am often led to wonder whether it is not often preferable to avoid mapping a class to a table: one of the clear alternatives would be to concentrate on operating on SQL statements instead. The most famous library following this approach in Java is MyBatis; but many Scala libraries, e.g. ScalaQuery, Squeryl and Anorm (sorry about the lack of documentation on the last one), are headed in as imilar direction.

Packet Analysis with Wireshark

2012-01-02T00:00:00+01:00

Some months ago I wanted to remind myself of how the Web works. In addition to Computer Networking I went through the second edition of Practical Packet Analysis, by Chris Sanders.

Notwithstanding the general-sounding title, Practical Packet Analysis is very much about Wireshark, which is a GUI tool to analyse network traffic, released under the GPL version 2. On a Fedora Linux machine, running Wireshark is very easy, you just start it up and it does its magic; no configuration required except adding yourself to the wireshark group.

There is a big advantage to working through this book together with a more academic tome such as Kurose and Ross. In theory books, each layer is presented neatly separated from the other, but Wirehsark shows you the the data just as it passes through the wire. This means that you see ARP requests interleaved with TCP handshakes and HTTP and DNS requests. With DHCP thrown into the mix, the number of packets that are exchanged to establish a simple connection in a local network can be quite overwhelming.

Then you start having fun tracking the TCP sequence numbers’ regular increase, and notice that those same numbers really are missing from UDP packets. You do not need to capture live traffic all the time, as the book is full of exercises are based on recorded captures that are downloadable from the book website.

The chapter on wireless networks was a the only slightly disappointing part; it is only a very high level overview of wireless network protocols; one gets the feeling that the topic is very large, since wireless networks need to contend with reliability and security issues that are more complicated than wired ones, but that little of it is covered.

This hiccup is more than made up for by the chapter where you are encouraged to investigate the behaviour of a trojan that downloads itself, takes screenshots of the user’s desktop and uploads them to some shady server. It’s a great exercise to be forced to think about why that particular TCP conversations looks suspicious and, and you even get to use a hex editor (I have used Emacs’ Hexl mode).

Practical packet analysis is an excellent way to refresh your knowledge of the TCP/IP stack. It is a relatively short read for a tech book, and each chapter contains hands on activities which are fun but relatively simple, so you can have some good time reading this book.

Some Perl idioms

2011-12-09T00:00:00+01:00

Recently I have worked with Perl again for some system monitoring scripts. Some people at Lunatech, where I work, are real Perl mavens. Here are some quick reminders

Constants

use constant CONSTANT_NAME => "constant value"; creates a compile time constant. It is mainly useful for clarifying intent, although if the right side is a more complex expression, for instance a calculation, there might be some performance benefit. Learn more about constants in the official Perl docs.

Regular expressions

my @matches = my $variable =~ <regex> if there is a match, @matches will contain the values of the capture groups in the regular expression.
my $string =~ s/<pattern>/<replacement>/ if any substitutions have been made, in addition to modifying $string, will evaluate to true. Handy for use in conditionals.

Classpath trouble

2011-11-17T00:00:00+01:00

Java programmers are haunted by inferiority complexes. Before the functional programming wave, a much ink had been spilled debating the benefits of dynamic languages such as Python and Ruby. While the discussion focused mostly on syntax and the pros and cons of dynamic typing, it recently struck me that the major reason something like Python feels so much more easy to experiment with than Java is that the interpreter manages to import most installed third-party libraries automatically. The popular dynamic languages all have standard package managers (cpan, gem, setuptools) that can install third-party libraries from a standard location on the Net to a standard location on your filesystem. Once this has been taken care of, you can just specify which modules you want to use at the top of your program files, and you're good to go.

Because Java is language of the Web era, where open source libraries have proliferated, a typical Java program relies on many more external dependencies than a C or C++ program, but since it came of age so early, it did not develop tools as sophisticated as those available for the scripting languages. Hence the endless artifact hunting and path setting.

Of course, there are Java solutions to the problem. More or less standard classpaths have been defined for Java: it's one of the features of things like JEE containers. But this goes only half-way: these conventions are only environment specific (and in fact, even vendor specific). There is no standard location to use for that experiment we might want to write to test a technique or a library.

Hence the existence of Maven and more recent competitors like Gradle, which create a common location for all your application dependencies and take care of downloading the stuff from the Web. Although this an improvement over passing options to javac and java, each tool behaves slightly differently and you still need to define a build file for every new project. In Perl, Python and Ruby, once a library has been installed globally, it is automaticaly available in all programs. Of course, global libraries have their own problems, but they are so practical for fooling around! Which is how you start learning everything.

What Rails got Right

2011-09-26T00:00:00+02:00

While getting updated on Rails 3.1 with the latest Manning book on the subject and the thought struck me that where Rails has been the most succesful has been creating a new vocabulary.

A DSL for CRUD Web applications

More than just providing an MVC structure, Rails has created a DSL for Web programming, which expresses most of the common CRUD operations in a few concise lines. The con is that there is a learning curve, but in exchange you get to think in terms of these operations really fast, and begin to type code very fluidly. It also contributes to building the common language of the Rails community.

Enforcing conventions through tools

In Rails there is a generator or a bundled command for everything, from creating files (rails generate) to updating dependencies (bundle). This is more than a convenience: ensures that conventions are respected while relieving the programmer of the burden of enforcement.

Code sharing and conversations between become simple.

Community building

Despite occasional complaints of juvenile behaviour, the Rails community is impressive. It is well known that going through the same motions every day will build a repertoire of common references and in-jokes. So I wonder if the relative rigidity of Rails is not part of the secret here.

Spreading beyond the core community

The proof of the strength of Rails in creating new vocabulary and conventions is that many of these have seeped thorugh in the Node.js and even in the larger web design world (http://compass-style.org/).

In conclusion, the very way Rails is structured as a piece of software has pro bably been instrumental in shaping an original programmers' culture, something that other equally competent frameworks such as Django lack.

EuroSciPy 2011

2011-09-21T00:00:00+02:00

Last month I have attended the Euroscipy conference at the École Normale in Paris. The conference is about scientific computing with Python, in particular the numpy/scipy/matplotlib stack.

Performance

Scientific computation is very CPU intensive; an interesting change from applications where memory and storage are the main bottlenecks.

In the performance talk by Gael Varoquaux, we got to see the demonstration of a computation runnning noticeably slower when iterating through a matrix along one direction than the other. This was due to the way data is allocated in the CPU registers. In one direction, the CPU can find the next number already in its registers, while in the other it contansly need to fetch the data from memory.

GPU computing

The talk on OpenCL was presented by Eiliff Muller from the Ecole Polytéchnique Fédérale de Lausanne. OpenCL is API that allows to perform general computations on the GPU. Again the focus was on the processors, since the interest lies in the different architecture of the GPU, which makes them suitable for highly parallel workloads.

OpenCL requires to program in a C-like special purpose language. PyOpenCL was demontrated integrates OpenCL and Python by embedding OpenCL programs as strings in the Python program. This goes to show that the preferred approach is to write most of the program in mainstream programming language to make it execute on the CPU, and to assign to the GPU only very specific tasks which are suitable.

Machine learning

Finally a particular striking demonstration was the session on Scikits learn led by Fabian Pedregosa. It is fairly amazing what can be achieved today with a freely available library. applications that were presented as cutting edge. We got to see a very impressive demontration of facial recognition. Three

Programming to learn

For scientists, programming is indispensable, but not their main occupation. The problem domains tend to be totally different than with your typical web projects.

But the most important point is that in scientific computing, software development is much more about answering questions than about building a system.

Designers needed

2011-09-21T00:00:00+02:00

Designers needed

Sketching. Thinking out navigation paths. Planning surprises and moments of delight.

We don't need designers meaning 'people who draw pretty pictures.' We need people who know that some problems can't be solved by yet more code and are ready to interview clients, draw on A3 paper, and sit down with the end users to know how they really work and figure out what's really going to make a difference.

Communication

Feedback loops need to be a lot shorter. We need people good at involving the client and communicating the impact of choices. And brainstorming and selling design and the process. That needs to happen a lot more on a lot more projects. We need to help break down the confusions that sometime exist between our client and the client's clients, so that our own clients' investment wil not go to waste.

15 years after the Web burst onto the scene there is still a misunderstanding. As if the introduction of new technology did not mean changing our processes and changing the way we perform so many tasks. It needs deep involvement at many levels in an organization and it needs people who can accompany clients on that journey.

Whether these are people who come from a 'design' or architecture school, or whether that means that some developers and consultant will need to expand their skills and shift their focus. Writing less code, makes no sense if we don't spend more time on perfecting user experiences and improving the delivery pipeline..

Failure nowadays does not happen because of the basic building blocks, open source solves that. It's about the strategy of content, form and function and targeting the right audience, with the right schedule, and we have to deliver that.

the Great JavaScript Closure Pattern explained

2011-03-02T00:00:00+01:00

Most of you will have seen it in jQuery, but also maybe in Mustache.js, or even another library:

(function() {
  })();

You know you need this somehow to protect the global namespace etc., But how does it work again? Follow along.

It’s just a function call

Yep, that’s right. You know that in JavaScript you can define anonymous functions:

function() {

}

Well, you can also call them immediately and that's what the above code does. With that out of the way, let's concentrate on the function body and remember later to see why we would want the function to be executed.

Why you need a function

What does a JavaScript program need like humans need oxygen to breathe? Scope. JavaScript burns scopes faster than a pack of cigarettes in a noir movie, because loops, if/else branches have no scope of their own. The only thing that has a scope is a function and that's why you create functions all the time.

The good thing is that wa can dump pretty much everything in a function body and it will stay there forever in its own parallel dimension. Every object and function defined inside a function body will have access to everything that was in the function body at the moment it was defined. That's the famous closure property. (Hint, hint: and that's why we'll need to execute the function, to get a piece of that parallel dimension into the real, cold, harsh world outside. We need the function to return something or in any case assign something to something).

When you think of it, it's pretty cool. It's like every function was its own little package or module, for languages which have that sort of thing. So just go at it, define whatever variable or and function you want in the closure body.

Moving it outside

Done? But how can I use all of this stuff in my web page?, you will ask. Well, you need to move it outside in a convenient wrapper. The trick is that you only really need to move the interface, the functions that need to be used first hand in the outside world. All the rest can remain forever inside the closure and still be accessible from what you exported.

There are two main techniques being used, but both involve first creating an object to hold your interface.

Building an object for outside

Just do

var MyChampion =  { usefulFunction: function() { // uses things in closure scope};

// other functions }

Inside the braces you can, as already said, access anything that's defined inside the closure. There are a couple of other techniques for building your object, but let's not cover them here. Let's move on to the essential.

Bringing it outside

Finally! As already said, there are two ways to do it. First way: just before the end of your big encircling closure/anonymous function, do

window.whateverNameYouWant = myOutsideMissionary

and your object will be accessible anywhere with all its methods intacts. In this case, you need to execute the anonymous function to perform this assignment.

The other solution is to assign the result of the execution of the anonymous function to a variable outside the closure.

var whateverNameYouWant = (function() {};)();

In that case, the last thing your anonymous function must do is to return the missionary object. So, here you want to execute the anonymous function to actually return that object.

Happy ending

So now our object is out there, making its methods available to the whole world; the global namespace is intact; and you have one thing less to be confused about.

As you explore other JavaScript libraries, you will meet other variations on this pattern. Experimentation is still in full swing in the JavaScript community, but that’s why it’s fun and one day, you too might make the next big invention (but spiky hair is not en vogue anymore).

2011 New Year resolutions

2010-12-31T00:00:00+01:00

2010 has been a great year.

I have almost come to the end of my Mathematics studies. The first programming project which has made be a bit happy, concordar, has got off the ground: it’s just a toy and could be better in too many ways to name, but somehow it fits the vision of what I’d like to do more than anything else I have written before.

I have become more familiar with the skilled jobs market. In the last part of the year I have been happy and grateful to receive some consideration as a prospective graduate from companies. So, what’s cooking for 2011?

First of all, I am looking forward to my internship with Lunatech Research starting February. I also hope to develop my mathematics further, read about fundamental computer science topics and start blogging about mathematics a bit. Finally, I hope to graduate with the best possible result!

Happy 2011 to you, readers!

Hardboiled Web Design review

2010-11-30T00:00:00+01:00

Yesterday I bought a PDF of Andy Clarke's Hardboiled Web Design, and, wow. It is one of the best refreshers on HTML and CSS, and especially a practical implementation guide for HTML5 and CSS3. On top of the gorgeous illustrations.

You can actually use the new <article>, <section>, <footer> elements in most recent browsers, as they in fact render elements even if they do not tecognize them. There is a workaround for Internet Explorer versions lower than 9, which involves adding a minuscule amount of JavaScript.

If you are in doubt on how to use WAI-ARIA and microformats, you will find answers to your questions. I found it particularly useful in the case of microformats, where the official documentation is sometimes a bit hard to navigate.

Finally, the way CSS3 multiple backgrounds, transparency and shadows are demonstrated cleared my personal misconceptions by showing that they can be put to the service of a well organized and efficient design. They are definitely woth more than the bling.

It is obvious that a lot of thought went into editing and publishing this book. It is chock-full of wondeful illustration inspired by the world of hard boiled detective novels and the PDF is available in two different formats, single page and two-page spreads. You also receive a download full of example files. Again, a lot of care went into the examples and they are very fine pieces of web design in themselves.

My only complaint is that sometimes the hard boiled metaphor is a bit far-fetched, but it is a minor stylistic gripe. If you are considering a book to get up to date on the latest web technologies, this one deserves serious attention.

Interesting resources mentioned in the book

What’s C++ like?

2010-09-27T00:00:00+02:00

Whoever said Perl looks like line noise should take a look at ‘modern’ C++.
C++ is still a requirement for many engineering/number crunching jobs, so it's still worth learning.
It mashes so many things together that it is like a huge revision of everything you have ever learnt about programming. The fact that you are encouraged to write your own memory mangement abstractions makes you think hard about resource management; more so than with plain C.
Despite the line noise and the mysterious compiler messages, I quite like generic programming.

Unfortunately, there aren’t that many Open Source projects one can learn ‘modern’ C++ from. Those that do use C++, mostly use the C-like part of the language, with some simple classes or… namespaces! (node.js, mongodb).

Good books to learn C++ from

You cannot make a list of C++ books without mentioning ‘The C++ Programming language’ by Bjarne Stroustrup. I find this book is better as a reference than for learning the language. You might also wait for the fifth edition covering the new standard before committing the shelf space. Meanwhile you can easily browse it on services such as Safari.

Personally, I have started with Accelerated C++ by Andrew Koenig and Barbara Moo. It covers the template library and best practices straight away, and is really an excellent book. It is quite dense though. Don’t let the title mislead you; this is not one of those ‘Learn C++ in 7 days’ books.

The next book could be Effective C++ by Scott Myers. It covers common mistakes and ways to improve your code. The only problem is that if you’ve read through Accelerated C++, you will find Myers’ book covers much of the same ground. Nevertheless, it is a book many have learnt their C++ from and it is good to keep it in mind.

What I have really liked so far is Herb Sutter’s Exceptional C++. I think the title is a pun on making C++ code ‘exception-safe’. The book really illustrates a lot of dark corners of the language and offers advice you will only find there. The material is presented as a set of problems you can try to solve on your own before checking Herb Sutter’s solution. There is also ‘C++ Coding Standards’ by the same author (with Alexei Alexandrescu), but that book covers a lot of language agnostic material (’Use a revision control system’), and is in my opinion less interesting.

Migrating from WordPress to Django

2010-09-24T00:00:00+02:00

This is the system I devised to move my blog posts from WordPress to a custom Django application. It looks a bit intricate, but it did not take more than an evening of coding.

Start from the obvious: WordPress exported XML

The WordPress native export functionality gets you a custom XML file, mixing your post data, with a lot of information you do want necessarily to keep, because it is needed only by WordPress internals (for example, post ids).

Instead of writing a program which would parse and create content for the new system directly from the XML, I thought it would be simpler and more convenient for future use to have two: a program to first output a subset of the XML data into JSON, and another program which interfaces with the Django database API and loads the required data from the JSON.

This way you obtain a clean JSON file, without WordPress-specific cruft, which can also work a sort of intermediate backup.

Python standard lib all the way

I decided the Python standard lib had everything I needed:t etree for XML and the json module. etree is quite an interesting API, though as most XML APIs it seems geared towards extracting a simple piece of information from the document, not trasforming the whole document into something else. I think SAX is still the best for that, once you get the hang of it.

Anyway, this time I had conveniently reduced the task to what the XML library was actually good at, so I had only to target the tag list, the post title and the post slug, create a Python dictionary for each post with this data, and dump that to JSON. The code was not probably very efficient, but there was not a lot of data either.

Then it is only a matter of turning the JSON back into Python (json.load), then running through the dictionary json.load returned and saving objects through the Django db api.

Strings and unicode in Python

2010-09-20T00:00:00+02:00

This post wil be about strings and encodings in Python, with an eye to Python 3 migration. Hopefully, you will find some useful rules of thumbs to to navigate these sometimes perplexing questions.

'String' types

What you commonly call a string can be two different things:

bytes: more low level, it's the actual byte sequence representing the data
unicode: more abstract, it can be thought as the sequence of human meanungful symbols which we use in written communication and that you learn at school.

Now this is extremely rough and even incorrect (unicode is just one standard, there are non unicode encodings) , but you the aim is to get a working mental model. The important thing is that, seen from this very high altitude, both these concepts exist in both versions of Python, but they have changed name.

Encoding/decoding

Where the practical problems arise is the encoding/decoding dance.

To get the unicode from the bytes, you decode. To get the bytes from unicode, you encode. This is confusing, as unicode is a kind of encoding, so you might think that to get the unicode you need to encode. No! You decode the bytes to get unicode! And you encode the unicode to get bytes!

This might be another way of defining bytes and unicode: if it needs decoding to become more abstract, alphabet or ideogram-like, it's a byte. If it need encoding to become low level, it's unicode.

Now, things are much the same in Python 3 and 2, except that in Python 2 what you think of as strings (things between quotation marks) are bytes by default (wihout u in front), while in Python 3 they are unicode by default (and putting an u in front is invalid syntax). It's almost as if the unicode type in Python 2 had become the string type in Python 3, and the Python 2 had simply evaporated (actually not really, we have bytes, which have a much simplified API). The Python 2 strings need to be decoded (they are more like bytes), the Python 3 strings are already decoded (they are unicode).

Examples

In your interactive interpreter:

Python 2:

>>> type('peacock')
<type 'str'>
>>> 'peacock'.decode('utf-8')
u'peacock'
>>>type('peacock'.decode('utf-8'))
<type 'unicode'>
>>>type(u'peacock'.encode('utf-8'))
<type 'str'>

You might have heard that to prepare a migration to Python 3, you can use

from __future__ import unicode_literals

in your Python 2 code.

Now, let it be clear what this does. This does not replace the string type in Python 2 with the one in Python 3. Type str are still encoded (read bytes) strings. What this does, is it changes the type of unadorned string literals, that is things such as 'peacock' which in Python 2 are (Python 2) str, with unicode_literals will be (Python 2) unicode.

>>> from __future__ import unicode_literals
>>> type('peacock')
<type 'unicode'>
>>>type('peacock'.encode('utf-8'))
<type 'str'>

Now Python 3:

>>> type('peacock')
<class 'str'>
>>> 'peacock'.decode('utf-8')

Throws an AttributeError. Even though the type is still named 'str', it is now a unicode string and is already 'decoded'. But:

>>> type('peacock'.encode('utf-8'))
<class 'bytes'>

Now some input/output. Python 2:

>>> with open('LICENSE.txt', 'r') as f:
...     text = f.read()
...
>>> type(text)
<type 'str'>
>>> type(text.decode('utf-8'))
<type 'unicode'>

Python 3:

>>>with open('example.txt', 'r') as f:
...     text = f.read()
...
>>> type(text)
<class 'str'>

>>> type(text.decode('utf-8'))

Throws AttributeError; input/output functions return unicode by default.

Conclusion

To sum up, there are still things called 'str' in Python 3, but in Python 3 it behaves much more like a Python 2 'unicode'.

As a consequence, there is no 'unicode' type anymore in Python 3.

from __future__ import unicode_literals will make quotation marks-enclosed thingies 'unicode' objects in Python 2. It will not transform them into Python 3 'str', nor will it introduce Python 3 'str' anywhere.

In short, if something expects unicode in Python 2, it can take Python 3 str without too many problems. The problem is if something expects Python 2 'str', as Python 3 'str' are a totally different thing. The closest approximation is the Python 3 byte. But it lacks many of the Python 2 'str' methods (which may cause problems, see the WSGI discussion).

Postscript: some more encodings

You might wonder what that 'utf-8' parameter means. Well, as I said at the beginning, there various ways of encoding characters. Most of the time in new programs you will want to use utf-8, but you might also need to handle other encodings. Some are defined in Unicode, most not. It really does not matter from the practical standpoint. For instance, if you work with files originating from Western Europe, you will often be confronted with the ISO-8859-1 (commonly named latin-1) and Windows-1252 (which, characteristically, is almost the same). These are not defined in the Unicode standard, but they are handled in exactly the same way in Python programs. That is, replace 'utf-8' with the appropriate string you can obtain from this handy reference.

Django CSRF verification failed

2010-09-10T00:00:00+02:00

If you migrate to Django 1.2 you might encounter a CSRF Verification failed error when rendering a form.

An interim solution is to enable django.middleware.csrf.CsrfResponseMiddleware:

MIDDLEWARE_CLASSES = (
   # Your other middleware here

'django.middleware.csrf.CsrfResponseMiddleware',
)

CsrfResponseMiddleware is there for legacy compatibility. If you want to use the features introduced in Django 1.2, CsrfViewMiddleware should take care of all CSRF needs at the middleware level. The new CSRF functionality relies on modifying individual views.

Views in need of CSRF protection should provide a special variable in the template rendering context.

The easiest way to provide this variable is to pass a django.template.RequestContext to the template. You instantiate a RequestContext by passing the current request (quite logically):

def some_view(request):

        c = RequestContext(request)

If you are using render_to_response, you can pass the context as an extra parameter after the template variable dictionary:

render_to_response('template.html', data_dict, c)

You can also do everything at once:

render_to_response('template.html', data_dict, RequestContext(request))

Interview with the author of Nginx HTTP server, Clément Nedelcu

2010-08-25T00:00:00+02:00

As promised, here is an interview with Clément Nedelcu, author of the first English book on the Nginx web server, namely Nginx HTTP Server from Packt Publishing.

Hello. Can you introduce yourself to our readers?

Hi Ludovico and thanks for your review. My name is Clément Nedelcu and I am a French native currently residing in China. I have worked in the computer science industry for almost ten years now and I own a large network of popular websites. Although back in the day I was more into programming (Delphi, PHP, C# with .NET), I have progressively switched to Linux system administration applied to the Web. Lately I worked as a technology consultant for a small company called Coforcert during my time in France, running .NET development seminars mainly, and I now teach various subjects of computer science at a Chinese University near Shanghai (Linux technologies, Web design and .NET development) on top of my network management activities.

How did you discover Nginx?

I suppose the same way as most people did. As my websites grew more and more popular, my servers experienced massive slowdowns. Upgrading to more powerful computers didn’t help, so I asked a friend for help. He advised against using Apache and said it was cluttering the servers completely. As a replacement he suggested Nginx, which he had been running on his network for a while. I looked into this little gem, installing it as simple reverse-proxy at first, then I wanted more and replaced Apache by Nginx completely. It has never let me down since, and it even resisted DDoS attacks!

What problem did Nginx solve for you?

Well clearly the page serving speeds. With Apache, it came to a point where people couldn’t load my websites at all. The average load time was over 11 seconds per page, and at peak times the requests downright timed out! I have it under two seconds now which is a miracle considering the amount of resources we have on those pages (images and backgrounds, script files, CSS files…). Additionally, I find the Nginx configuration to be easier to manage than Apache’s, but that is merely a matter of opinion.

What’s Nginx’s killer feature?

The overall design of the application is its killer feature: scalability, performance, efficiency. It does wonders with so little resources! On my most active web server, with 500 active connections the RAM usage is under 20 megabytes, and the CPU usage rarely exceeds 0.3% (8-core Intel CPU). The rest of the resources is used by third-party applications such as PHP.

The book includes a mini guide to the basics of the Linux command line and the Linux filesystem. Are you seeing people migrating away from Windows Server/IIS towards Linux and Nginx?

I have had several people ask me what this first chapter was for. Although the final decision was ultimately mine and mine only, the publisher recommended me to start the book with a few reminders on basic Linux system commands. I don’t think it was a bad idea considering the target audience, but naturally most Linux users won’t waste their time with this chapter. I don’t see Windows Server administrators specifically switching to Linux/Nginx but if they are concerned about performance then why not?

Nginx has a reputation to scale and serve well under high loads. What made you decide to target the book towards beginners?

The reasoning is simple: Nginx is an excellent and easy-to-use web server, so why is it so far behind Apache in terms of web server market share? One of the answers you could provide is the lack of proper English documentation for beginners. Sure the Nginx Wiki is an excellent source of information and there are many well-explained blog articles detailing the process of setting up and configuring Nginx. But there needed to be something straight, detailed and easy enough for beginners to completely understand and come to master. A lot of people (including me) learned about Nginx by reading blog articles that explained how to set it up, and provided example configurations that could be used and adapted without really understanding the functioning of the application. The book helps understanding everything from the start to the most advanced directive. I believe that a book about Nginx will certainly help increase its popularity, that it will help spread the word about this awesome web server. With no other Nginx-related book on the market, it might have seemed strange to target the book at advanced administrators who would already be familiar with Nginx. I wish I could have discussed more advanced topics in the book (particularly load-balancing which I find to be an interesting subject) but I was given a limit of 250 pages, which as you may have noticed I exceeded by far.

What is the part of the book you are most proud of?

Well this was my first attempt at writing and as pointed out by Matthew Helmke in his review, the writing style may seem inconsistent throughout the chapters. But if there’s one thing I can be proud of is that I have (and I encourage readers to continue doing so) participated in porting the information that I have gathered in the book to the Nginx Wiki for everyone to read. Obviously the wiki was an important source of information for the book and I wish to give back everything I can by completing all the missing directives and documentation. The publisher has already granted us the authorization to do so. Gathering information about missing directives, and finding out more about important configuration sections was quite difficult considering the lack of English documentation on the Web. I often had to look into the application source code and even refer to the IRC channel or the mailing lists to ask for information about the more obscure directives. I even got help from Igor Sysoev himself at first but unfortunately he quickly became too busy to help.

Is there anything you would like to tell prospective readers before closing?

I want to tell them that this book is an excellent way to get started with Nginx. As my friend Martin would say: “teach a man to fish!” It is dangerous to start by reading random blog articles, copying and pasting a configuration file without understanding what you are doing, and crossing your fingers hoping for Nginx to work. You’d better understand the fundamentals and learn to create your configuration by yourself rather than using examples that weren’t designed for what you are trying to achieve. The book also describes multiple setups so if you have (like me) already installed Nginx as reverse proxy by fear of breaking your Apache configuration, you should definitely give this a read and figure out how to run Nginx all by itself. Performance-wise, it is well worth it!

Book review: Nginx HTTP Server

2010-08-16T00:00:00+02:00

Packt Publishing has provided me with a copy of their new book, Nginx HTTP Server (this URL contains a tracking code for Packt to monitor their marketing efforts) by Clément Nedelcu, kindly asking me to review it. They sent me a PDF version of the book, which is litte more than 305 pages in length.

You might be aware that Nginx is a very fast and lightweight server developed by Igor Sisoyev, originally for the Russian portal site rambler.ru, and released under the BSD license. Nginx shines for rapidly serving static files an serving as a frontend for other servers.

This book, the first English language book about Nginx, is aimed squarely at beginners: either people migrating from a more traditional Apache/nix setup, or even those totally new to nix, as the first chapter is a brief introduction to the Unix/Linux command line. The book then covers basic Nginx configuration (including virtual hosts), and then running Nginx in tandem with Fastcgi and as a frontend for Apache. As is often the case, the configuration reference contains a lot of material which is almost identical to wiki.nginx.org, so it will mostly appeal to those who are allergic to perpetually loading the documentation in the browser.

The chapters on running Nginx with Fastcgi and as a reverse proxy for Apache are more interesting, as they collect in one place a lot of information which is scattered all over the web. For those new to Fastcgi, there is a very accessible explanation of how the standard evolved from CGI, and how the Fastcgi process communicates with the server. The section on Fastcgi is focused on PHP, also there is chapter on running Django with Flup. Personally, I would not use Flup to run Django, and would prefer Apache with ModWSGI or a dedicated WSGI server.

This brings us to the next chapter. Here the focus is on using Nginx as a reverse proxy in front of Apache, but much would apply if instead of Apache you had any other HTTP server, including Tin (for Rails) or Gunicorn (for Python). Again, the information is very basic, but it will no doubt get you started on the right track. There are also a few lines about using multiple backend servers and round-robin configurations, but the subject does not receive a very in depth treatment.

The last chapter might prove the most popular. It contains instructions for migrating from Apache to Nginx, including detailed translations of the mod_rewrite rules for popular PHP applications such as WordPress and Mediawiki into the Nginx rewrite syntax. It also explains how .htaccess files work in Apache and how to replace them with Nginx.

The book is well written, the explanations are clear and correct, and the reader is mercifully spared the amount of superfluous jokes which plague too many technical books. If you want to give Nginx a try, maybe for saving some RAM on your VPS, and you haven’t got the faintest idea of where to start, this book might be a good choice. Another reason to buy the book would be to use the ready made recipes for running Nginx with popular applications.

Nevertheless, I must confess to be a bit puzzled by Packt’s decision to produce such an introductory book, as I would imagine that people who have heard of Nginx are already quite savvy, and probably expect more advanced information to justify the expense.

If you want to know more about Nginx HTTP Server from Packt Publishing, I may be able to have a short email interview with the author in the coming days.

Django: display the current language in a template

2010-07-23T00:00:00+02:00

It is good practice to use the lang attribute to specify the language of an HTML document

<html lang="en">

Of course, you could hard code the attribute by yourself in the template; but this information is stored by Django in settings.py (as LANGUAGE_CODE): so why not keep it DRY and retrieve it from there?

Unfortunately, there is no default way to access information in settings.py from a template. One solution is to write a custom template tag.

To create a template tag, make a templatetags directory in your app directory. Inside create a file named language_extras.py (or anything else for that matter; the file name has no importance). Template tags are simply functions which are registered with the template system using a django.template.Library instance.

As our template tag is very simple, we can just write a function which returns the value we want to display:

def current_language():
    from django.conf import settings
    return settings.LANGUAGE_CODE

We register it with:

register = template.Library

register.simple_tag(current_language)

Now we can use our tag in any template this way:

{% load language_extras %}

<html lang="{% current_language  %}">

How good is Django, the Python web framework?

2010-07-21T00:00:00+02:00

Ok, as you know, you are never happy until you have written your own blogging software. So I went the easiest way and I did in Django. As you can see, we lost the comment functionality. I hope it will be back soon; but I need a way to protect this blog from those pesky spammers.

Now, for a mini-review of how Django fits this site, and Django compared to a CMS like WordPress or Drupal.

Rewriting this site in Django

The reason I have used Django, besides because it is the most popular Python framework, was to take advantage of the multisite functionality, but in fact you cannot run all sites from the same Django instance because you need to load a different settings file for each one, as the current site is determined once and for all in the settings file. You cannot dynamically load different settings depending on the URL for example.

You must use a different WSGI daemon for every instance. If you run everything on a single machine this wastes memory, and it does not make the sites more decoupled than in Drupal, where you can always copy your settings to a different install on a different machine if need be. A consolation is that working in the Django admin does feel faster than in the WordPress backend.

I had started with building some custom tags for displaying the site name and language in a template depending on the site ID, but then I realized I was not going to use the same template for two different sites anyway, and I did not need to redistribute anything, so I could as well hard code them.

Which brings us to the good thing compared to using a ready-made CMS such as WordPress: the product is custom, so there is no need to bend over backwards to suit the generic and redistributable nature of something ready-made (I find the cognitive load of developing for Django actually less than working on just a WordPress theme).

A Django business?

At the same time, this one-off character is not so good for business, as the only obvious Django related thing you can sell is your labour. There are no ‘easy’ derivatives like templates or modules that often are very popular with large customer segments. You either need to go down the services or the product road. And as Joel Spolski said, services don't scale.

This leaves products. But then, the framework or language do not count as much as that tiny spark of genius.

Django 1.2 and the new syndication API

2010-07-12T00:00:00+02:00

While mainaining backward compatibility, Django 1.2 has introduced a new Feed API which is more in tune with the rest of the framework. This post will to explain how to migrate from the previous Django syndication API to the new one.

The old way

In Django 1.1, an RSS feed was something like this:

from django.contrib.syndication.feeds import Feed
from models import Entry

class LatestEntries(Feed):

title = 'Latest Post'
    link = '/'
    description = 'Latest posts from example.com'
    def items(self):

return Entry.objects.order_by('-date)[:5]

You had to associate the feed to a URL with a dictionary which mapped the feed slug to the class. This made /updates.rss return our LatestEntries feed:

(r'^(?P<url>\w+)\.rss$', 'django.contrib.syndication.views.feed',
    {'feed_dict' {'updates': LatestEntries,}})

The syntax was a bit too verbose, and the feed URL was split between the regular expression and the dictionary key.In addition, just to display the title and description of feed items, you had to create two templates named feeds/(feed_slug)title.html and feeds/(feed_slug)description.html. So your template directory would get filled with files containing in most cases almost no code.

What’s new in 1.2

In Django 1.2, Feeds are regular views, so to use the new syndication API, change the import statement to read:

from django.contrib.syndication.views import Feed

The method to map a feed to a url is now much less verbose. No need for a dictionary; just pass an instance of your Feed subclass:

(r'updates\.rss$', LatestEntries())

Finally, to display the title and description, instead of templates, you can just define two new methods in your Feed subclass. Supposing that the Entry class has a title and content attribute, and that you want to use those for your RSS item title and content, you would write:

class LatestEntries(Feed):

def items(self):

return Entry.objects.order_by('-date)[:5]

def item_title(self, item):

return item.title
def item_description(self, item):

return item.content

The new syndication API in Django 1.2 allows you to use much simpler syntax for url mapping, and cuts back on superfluous templates. But as you are not forced to change your Feed subclasses declaration in any way, it is also easy to switch from the old API.

Introducing the Python defaultdict

2010-06-23T00:00:00+02:00

A defaultdict is a dictionary with a default value for keys, so that keys for which no value has been explicitly defined can be accessed without errors. Let’s see it in action.

Frequencies

Suppose you are given a list of words and you are asked to compute frequencies. You could do something like this:

frequencies = {}
for word in wordlist:
    frequencies[word] += 1

Unfortunately, Python throws a KeyError the first time through, because you cannot increment the the values for the words, as they have never been initialized. A workaround would be to catch the KeyError exception:

for word in wordlist:
    try:
        frequencies[word] += 1
    except KeyError:
        frequencies[word] = 1

Or you could use an if/else block:

for word in wordlist:
    if word in frequencies:
        frequencies[word] += 1
    else:
        frequencies[word] = 1

The defaultdict solution

In Python 2.5 and later, though, the collections.defaultdict class comes to the rescue! A defaultdict is just like a regular Python dict, except that it supports an additional argument at initialization: a function. If someone attempts to access a key to which no value has been assigned, that function will be called (without arguments) and its return value is used as the default value for the key. Clever, right?

Going back to our example, we want the default value to be 0, so we pass the built-in function int()to the defaultdict constructor. When called without arguments, the int() function simply returns 0.

from collections import defaultdict
frequencies = defaultdict(int)
for word in wordlist:
    frequencies[word] += 1

Creating a QToolbar with QT Designer

2010-05-22T00:00:00+02:00

Adding the QToolbar to a QMainWindow

In QT Designer 4.62, do not look in the widget list. Instead right-click on the main window and select Add toolbar from the context menu. To add buttons to the toolbar, you must create some QActions. Select the Action Editor tab (if the tab is nowhere to be seen, go to the view menu and make sure Action Editor is checked), push the create new action button and fill the form that appears. Then drag your newly created action from the list in the action editor to the toolbar.

Adding separators

To add a separator, click on a button (QAction) you just created and select Add separator before… Do not select Append separator, if you do, QT Designer will append a separator to the end of the toolbar, not the button. Finally, remember that toolbars can only be created on main windows (QMainWindow class).

Python profiling with cProfile

2010-05-14T00:00:00+02:00

Compared to Perl's Devel::NYTProf, Python profilers are a bit rough around the edges. The recommended profiler as of 2.6 is part of the standard library and is called cProfile. It works only at the function (as opposed to statement) level and fancy graphical output is non existent. Here's a brief overview of what it offers.

Profiling output

For each function, cProfile makes the following data available:

ncalls: how many times the function is called
tottime: time spent on the function, excluding time spent on calling other functions
percall: tottime divided by ncalls
cumtime: time spent on the fucntion, incuding calls to other functions
percall: cumtime divided by tottime

Two different datasets are headed percall. The data is printed by default exactly in the order given above; except in columns from left to right. Keep that in mind for deciphering the ouput.

There are two ways of obtaining this display. If you run the cProfile on a script without specifying an output file, this data will just be printed to the screen. If you specify an output file, you need to use the pstats module to load the data (the actual output file is in some binary format). pstats also offers some methods to sort and filter the data. So if your program is not very small, you are better off using pstats.

python -m cProfile -o outputfile.profile name_of_your_program

then

import pstats
stats = pstats.Stats('outputfile.profile')

and now you can display the information in a manageble way by calling the appropriate methods on the stats object.

How to write books for the Web?

2010-05-07T00:00:00+02:00

What can you do with a book when it’s not on paper?

Comments

Well of course. Publishers should at least provide some mean of entering comments or reviews on the book site. Also people could publicly annotate the book at specific points, like the Django book (a book about a software development framework)

Continuous updates

We are not pushing the envelope enough in terms of continuous updates, addiction factor: take a cue from technical publishers: O’ Reilly has live books, published drafts, book upgrades. Today it’s all or nothing: Imperial Bedrooms is out on 15 June: then it’ll be available all at once, but until then not at all. Think of movie trailers, game demos. Of course, some read previews in the New Yorker. How many? Like 1% of the target readership? Free give aways, free chapter to circulate in the hope of becoming viral fall into this category.

Pictures!

Remember when you were small? And you had those books full of beautiful pictures? Then they gave you books for grown-ups; no more pictures, only black and white lines of text. Pictures are cheap now! We can make beautiful illustrated books for everybody. We can put any picture we want, photos we shot ourselves, for instance. A slight problem is that e-ink does not support pictures very well at the moment. No way though someone is not going to put something on the market which will show pictures and text well.

Merchandise, obviously

Sell merchandise: t-shirts with quotes from the book, mugs, you name it. Like Chuck Palahniuk (but minus thee annoying, irrelevant ads. Is there no ad server better targeted at this audience?)

Any language you want

The web allows total delocalization. Why should one publisher/platform only concern itself with only one language. Everybody on Earth reads US literature anyway; so it’s not like there are these huge cultural gaps between Pynchon wannabes (about 40% of high-brow authors under 50 in any language today). We can help authors of any language get published on our platform. We can automate/crowdsource translation and get fast turnarounds. Publish translations a bit at a time. No one says you need to get the whole thing at once.

And of course, edit it!

Everybody can edit the book and keep their own version, do whatever they want with it, print it, etc. Which license? It makes no sense to limit the edited copies for private use. You could give complete control of the edited version to the editor, but then it would be easy for anybody to make just a few irrelevant edits, call this version his/her own and undercut you. It also would stifle collaborative drives. You could reserve all rights on the original, then force everybody else to license their version under a CC Share Alike license. But it does not feel right. Paying for the privilege of editing the book? No one would: the only adavantage of paying would be that you could do it in public.

Control, control

The main question seems to be that to do the more interesting stuff you have to relinquish a lot of control on the book, in terms of creation and, to make creation interesting, in terms of distribution. Which is fine, except that it limits support options. Writing a book is cheap compared to shooting a movie. Still, you have to keep at least one person alive for the time needed to research and write it. Not all authors can be amateurs, because few people can commit enough time without financial compensation.

SQL vs NoSQL

2010-04-23T00:00:00+02:00

MongoDB and CouchDB have been gaining traction because of performance and scalability. But are performance problems inherent to the relational model?

Michael Stonebraker argues that traditional database systems performance problems occur in communicating with the applications and logging. Michael Stonebraker, who has worked on Ingres, is one of the fathers of contemporary database systems. In his post he mentions briefly another traditional argument of relational model advocates: SQL as a higher-level abstraction. As any C. J. Date faithful will recognize, the debate on NoSQL solutions is really not about the lack of SQL but certain features of these systems which in theory have nothing to do with SQL or the relational model. SQL and things like JSON and indexes are simply different levels of abstraction. MongoDB and CouchDB interface with programs at a lower level. The fact that CouchDB and MongoDB are tied to a low-level implementation should be a point in favour of SQL.

SQL in the real world

Unfortunately, abstraction and elegance do not sit squarely on the relational side. The offenders are well known:

Multiple SQL dialects
Relational database systems are more than database systems

To illustrate the second point briefly, many working Oracle installations are not limited to data storage functionalities, but contain a lot of PL-SQL code which you would academically consider part of application code. Given the age of the installation, the fact that they were conceived as replacements or complements for existing COBOL systems, they are more of a two tier system , where only presentation is really separate, and there is little distinction between application logic and data storage. So much for all those pretty diagrams.

Relational database systems: the next gen?

But can at least the (perceived) performance problem be overcome? Stonebraker claims that there are implementations of relational engines which surpass the current generation’s performance problems, and there are startups preparing to launch such products right now.

But if the technology is available, why have not any of these products emerged? Why have not these parallel, high performance, relational systems appeared yet? Maybe technology has really nothing to do with it and it’s marketability problems that are slowing things down.

A blog engine that supports the creative process

2010-04-16T00:00:00+02:00

I have already talked about designing personal web publishing sofware. After Drupal and MovableType, like everybody I have been trying out Wordpress. Here are my ideal blog software features.

Obvious features

These are provided out of the box by popular blogging systems or else are available with common plugins.

comments
tags
scheduling
sitemap
related posts
integration with identity systems (Twitter, Facebook, OpenID)
math output

By related posts, I mean suggestions for further reading at the bottom of a post right next to the comment area. Integration With OpenID and similar is available, but I am not convinced by any of the implementations I have known on Wordpress or Drupal; I would like an interface similar to IntenseDebate or Disqus. I do not know whether to generate math with Latex or MathML. The latter seems more lightweight, but it only works natively in Firefox (there is an IE plugin).

The exotic

These are features which do not seem to have any established implementation.

Draft functionality which keeps a draft separate at all times from the post (I want to work to have a sort of brainstorming whiteboard for every post)
Editorial comments, that is comments that are private to the blog authors

These features are directed towards authors, not readers, and towards writing, not impact measurement.

A little authors love?

Most blog software today does not include features to help authors take stock and reflect . It is true that many blogs have far from literary goals; and user metrics are helpful; but blog software could do more to support the revision and refinement work which makes for quality publishing.

Genshi templates basics

2010-04-07T00:00:00+02:00

Genshi is a Python templating system which guarantees correct XML output, and escapes all strings by default.

In exchange, you need to write more code and it is a bit of a pain to work with Unicode characters. Also, there is an extra step in case you do not want your strings to be escaped (i.e. you are passing good XML to the rendering mechanism). Here are some notes on the programming API, covering Genshi 0.5.1.

Streams

Most Python templating system (meaning Mako, Jinja2, Django) involve the use of two objects: some kind of lookup or loading or environment object, which you initialize with things such as the template directory location, and a template object. Basically, the factory pattern. Genshi is a bit more complex as the template object does not return strings but streams. Streams are sequences of events, similar to those defined in a SAX API. The theoretical advantage is that this API allows to chain any kind of filter on these streams (most XML manipulation APIs are built on top of an event based parser). The developers even repurposed the Python bitwise operator to achieve a syntax which looks like Unix pipes (!). There is built in support for selecting subsets using Xpath. When you call render on a stream object, you need to specify a format, by passing a string (one of 'html', 'xhtml', 'xml 'and 'text'). This is where the validation comes in, as there will be an error if Genshi cannot produce a valid document of that type. You might wonder what the difference is between 'xml' and 'xhtml'. From the official docs:

the 'xhtml' option understands the pecularities of producing XML-compliant output that can also be parsed without problems by the HTML parsers found in modern web browsers

Rendering a template

To get a string you can print somewhere, you need to call a method on the stream object with a parameter to specify the kind of output you want. Three objects are involved: a template loader, which has methods that return template objects, templates objects (Template), which have methods outputting streams, and stream objects, which you have a render method. So before you get fit to print somewhere, you need to instantiate three objects. Suppose that your template is in mystuff/mytemplates/template.html. Before you can use a template, you need to instantiate a TemplateLoader

loader = TemplateLoader('mystuff/mytemplates')

Then get a template

template = loader.load('mytemplate.html')

Then get a stream

stream = template.generate(variable1="something", variable2="somethingelse')

Finally you get your string that you can print. Say you know that XHTML should come out:

string = stream.render('xhtml')

Autoescaping

If you try to pass xml to the generate method of a template object, Genshi will escape eveything. To avoid this you need to wrap the string in a genshi.core.Markup instance. The string must again be a utf-8 string (see below)

myhtml = "<h2>Heading</h2> <p>Paragraph brimming with wisdom</p>"

template.generate(section=genshi.core.Markup(myhtml)

UTF-8

If you need to use text which is utf-8 encoded, you need to use unicode strings everywhere, not raw byte strings. So if you read your strings from a file, call decode(utf-8) on the string, otherwise things will blow up.

Conclusion

Genshi looks a bit overkill for run-of-the-mill (read web) application; it seems to have been developed to generate XML documents from other XML documents. If this is not what you want (and workflows based on XML are not that hip anymore, but they are entrenched in many corporate and public sector bureaucracies), you might consider using a text based templating language.

The biggest blog software mistake is…

2010-04-03T00:00:00+02:00

…storing the main content and comments the same way in a relational database.

I have been secretly bemoaning the transition from static sites such as the Linux Documentation Project to blogs. Static sites were very effective, because you could shape the text any way needed, without relying on a complex content type system such as Drupal's. You just wrote in the way that best suited your content each time.

In blogs, you are constrained by the post format. Some try to build structure by breaking up a tutorial, for instance, over a series of posts, but this make navigation annoying.

On the other hand, the great strength of blogs are comments; comments are great at adding insights, corrections or suggestions, harnessing collective intelligence and the power of communities yada, yada, yada (has been repeated to death, but it's still true). So we should really have the best of both; a model could be the MySQL documentation.

Luckily, the coupling between posts and comments is not that tight: a simple many to one relationship from a bunch of comments to the post they refer to, and that's it. Relationships between posts, such as categories or tags, have no bearing whatsoever on comments, and posts need not have access to comments in any way.

In addition, the performance characteristics for posts and comments need not be similar. Posts rarely change, while if there are enough visitors, comments will be updated much more frequently. Posts can do with relatively slow writes if that helps speeding up the reads. The comment system is the only one needing relatively efficient writes.

You can conceive this situation as two parallel collections: a collection of comment documents, and a collection of post documents. In theory, the connection between the two could simply be made at the template level. That's exactly how services such as Disqus or IntenseDebate are implemented, by inserting a javascript tag in your templates.

The big take away is that you can use completely different storage systems for comments and the main content. If you store the posts in an unstructured datastore (be it NoSQL or simply the filesystem) this frees you from having to worry about content types and defining fields each time you want to structure your content differently. Comments can be stored in whatever system you prefer.

I guess that's all at the moment. I just may build a system to put these ideas into practice, once I have looked into the various unstructured storage solutions.

Installing PHP 5 from source

2010-03-31T00:00:00+02:00

PHP 5.3 offers performance improvements and somewhat saner defaults. Unfortunately, neither Debian Lenny nor Centos ship it. If you want to install it you will need to compile it. The process is not totally straightforward, especially if you desire to install php as an Apache module.

Choosing php extensions

After downloading and unpacking php, run ./configure --help to examine the different extensions that you can build. Be careful when compiling you have to select the appropriate extensions. I prefer to enable the bare minimum, to speed up compilation and simplify extension management (if you have not built the extension at all, you do not run the risk of loading it when you do not need it). Many applications still do not make use of PHP 5-only feaures features. If you are using Wordpress or Drupal 6, for instance, you can disable pdo, even if from the description it may look like essential. Traditionally, the mysql and mysqli extensions needed to be compiled against the mysql client headers, as they had to use the MySQL client library to communicate with the server. Starting with PHP 5.3, a new native library has been introduced, called mysqlnd. Mysqlnd should offer better performance and allows you to avoid installing the mysql headers just to compile PHP, so you might want to enable it with these configure options:

--with-mysql=mysqlnd --with-mysqli=mysqlnd

Make and make install

Make itself runs without problems most of the time. Running make install may cause problems if you are installing the Apache module. Apache uses an utility called apxs during the install process. Apxs might fail once it attempts to configure the module, bringing the whole installation to a halt. You will get a message error containing the line

apxs:Error At least one LoadModule directive already has to exist

This problem affects distributions which split the Apache config files, so that the main Apache configuration file does not contain any LoadModule directive. This thread suggests adding a dummy LoadModule directive, but really, it's fine force the install to continue with

make install -k

then add the LoadModule directive yourself. The sample php module configuration does not contain the recommended Apache directives. The correct directive is AddHandler, not AddType.

Wrap up

Now run

/usr/local/bin/php -i

to check that the configuration is what you expected. This gives the same information as the phpinfo() function, but you do not need to use a web browser to make it readable. Among the listed information, you can find the location of the parsed php.ini files. A sample php.ini will have been installed, which configures the apc extension, but apc has to be installed first through pecl, so php will complain. Comment out the line activating apc in php.ini to resolve this problem (of course, you can also install apc if you wish so, in fact using an opcode cache is a huge boost to php performance). Now you should be all set to go. If you experience any difficulties, feel free to comment.

Revenue and social networks

2010-03-25T00:00:00+01:00

Facebook ads often hang in the awkward place between dating sites and online lotteries.

At South by Southwest 2010, Danah Boyd explained how the web mashes up social networks which rarely intersect offline. They mix up people who just ended up in your email address book, family, colleagues, friends from different situations and stages of life.

Conclusion? Maybe Facebook-like social graphs are a mediocre indicator of tastes, or more to the point, of the kind of content a person is willing to accept at that precise moment in her online activities. Accurately targeting an audience is difficult. It is not by chance that Farmville, Mafia Wars have a generic, blockbuster-ish vibe.

Being a mass market negates some of the advantages of the web, except that now you can target the mass market more cheaply. But if you are after ad revenue, you will encounter the same difficulties as TV prime time advertisers.

Target, target, target

In his keynote interview, Spotify CEO Daniel Ek mentioned how Spotify increases conversions by studying user music listening habits. Some campaigns have reached rates of 2% (still apparently not enough for some music labels).

The obvious suggestion is that narrow aspects of individual behaviour (musical tastes), match more closely what people like to see (and hence, act upon).

We all suffer from multiple personality disorder. You need to focus on one personality at a time.

Build on top of what's already out there

This does not mean, though, building a slew of niche communities from scratch.

Rather, sift through the data from established communitites and use API tools to focus on subcommunities. Build your 80s metal community on top of Spotify (or LastFm, or Facebook). Now you can serve the subcommunity targeted and profitable ads, while the main social network earns money by licensing the subcommunity management.

Unfortunately, the tools are available only in embryonic form. The licensing model does not exist anywhere that I know of. Nevertheless, it may be a good time to experiment. For instance, geographic location is a perfect example of vertical market (which will allow you to tap into the money of local businesses), but existing apps are not integrating very well with the big networks (location aware iTunes, anyone?)

Monetizing the data flood and the future of culture at SXSW 2010

2010-03-20T00:00:00+01:00

Data on the web proliferates faster than anywhere else, ever; at the same time it has ever been so easy to access.

Three challenges have arised. How can we help people put to use this army of tweets, Youtube videos, blog posts? How do we make money in the process? How will cultural authorities (academics, film critics, publishers) grapple with fundamental changes in their work as mathematical tools become indispensable to recognize and present noteworthy content?

Vertical markets

In Monday's keynote, Twitter co-founder Evan Williams' announced a monetization strategy for Twitter which consists in making tweets available for third parties to organize and analyse. I know you will say Duh!, but here is what I believe is a point worth considering. We use to think of large scale data analysis as data mining: typically confined within the walls of a research or consulting firm, and with the purpose of tuning a separate product or a marketing campaign. But, I believe, this is not what Evans had in mind. Instead, the data will be reformatted and organized and directly fed back to customers in vertical markets (say doctors, or graphic designers, or real estate agents). Large scale data analysis is used to create value for the consumer directly and build a product around this value; just as Google or Bing today. Also think of the way Bing has claimed to focus on its health, travel and shopping verticals.

The future of curators

The second issue is what will happen of traditional authorities, be it academics, Criterion Collection editors, book publishers. These are traditional guides through the cultural landscape, helping to focus the public's attention on what films, music, research papers have the most chance at not being a complete waste of time. This year, many SXSW panels (Nobody Wants to Watch your Film, New Publishing ad Web Content, Universities in the Free Era) have placed a great deal of emphasis on the transition from authorities to humble curators: for example, professors will give up exclusive claims to authority and instead assemble the information lying around and present it to the attention of their students, without worrying about the institutional provenance of the data. Although dissolving traditional auras is a huge concession, there is another, more radical, change at hand. Is it really possible for a single person to operate a meaningful selection among the heaps of data available? Or will not the sample he or she will manage to access be so small that the selection will almost be random? Curation cannot be thought anymore as the work of a single individual, or even a team, leafing through whatever articles, pictures or soundfiles can be found. There is just too much and it is updated too often. Curation will mean more and more the design and implementation of data analysis and presentation tools. Contrary to third-party Twitter plans, such systems will not all need to turn huge profits, nor will they able to. They might focus on long tail niches (60s French film lovers, neurobiologists, electric guitar geeks). They will need to rely on individuals who are not always highly trained statisticians, and they will need to place a big emphasis on fostering and empowering a community around the data.

Fun ahead

Monetization techniques of popular web applications such as Twitter converge with answers to our need to keep on building reference and meaning in the data flood. We can expect disruption in this area to continue for a long time, as profits are pursued and answers to the navigation problem are formulated. And solutions will have to be found to allow relatively unexperienced people to manipulate large data sets. This will probably mean creating another abstraction layer on top of products such as Apache Mahout (itself an abstraction layer on top of Apache Hadoop), to make them usable for people wih low programming backgrounds.

Configure Nginx to proxy virtual hosts to Apache

2010-02-26T00:00:00+01:00

You want to take advantage of nginx's speed serving static files, while making use of the .htaccess files bundled with applications such as Wordpress or Drupal, which contain Apache-specific configuration directives.
Fastcgi is too unstable for you. Fastcgi sometimes requires careful tuning to avoid things such as processes unexpectedly dying. Apache is quite good at managing processes. but you want to avoid the overhead of running the PHP (or or Ruby, or Python) interpreter to only serve static files.

Solution? Use nginx as a proxy in front of Apache. Match any static files with an nginx location directive and serve them directly. Pass all other requests to Apache.

Example configurations

We start with Apache and make it listen on a port other than 80 and only on the local address (so no one can access it directly from outside):

Listen 127.0.0.1:8000

Then configure nginx to serve static files directly, but pass any other request to Apache with the proxy_pass directive:

listen 80;
location /static/ {
  alias /webroot/mysite/static/;
  expires 30d;
}
location / {
  proxy_pass http://127.0.0.1:8000;
}

You do not need to have your static files in a separate directory. You can also use regular expressions to match static file extensions:

location ~* "\.css$"{
  do something;
}

In addition to the proxy_pass directive, other directives allow you to pass various information about the original request to the upstream server for logging purposes or similar. Refer to the nginx documentation for the proxy module.

What about Apache name-based virtual hosts?

Since Apache sees requests as emanating from nginx, you need to use the proxy_set_header directive in nginx's config to tell Apache about which hostname was requested originally. You need an nginx virtual host corresponding to every Apache virtual host to match every incoming request and pass down a the appropriate hostname. For example, if you have this in your httpd.conf:

<VirtualHost *:8000>
  ServerName example.com
  …
</VirtualHost>

 <VirtualHost *:8000>
  ServerName fghrt.com
  …
</VirtualHost>

…you will need this in your nginx.conf:

server {
  server_name example.com;
  proxy_pass http://127.0.0.1:8000;
  proxy_set_header Host $host;
}

server {
  server_name fghrt.com;
  proxy_pass http://127.0.0.1:8000;
  proxy_set_header Host $host;
}

Now you can enjoy the speed of nginx without translating every Apache configuration directive, and you can also let Apache manage the interpreter for the dynamic language of your choice.

Installing Arch Linux on a Dell Inspiron 1564

2010-02-17T00:00:00+01:00

The Inspiron 1564 is a new laptop from Dell, sporting an Intel Arrandale cpu. The install goes smooth except for the following:

the integrated wireless card, which is a Broadcom and therefore has worse support under Linux than, say, Intel. Here is a quick breakdown of the install steps under Arch Linux.
you will need the most recent kernel you can get at the time of writing (2.6.32), which isn't yet shipped by default in any major distro to get the screen working at native resolution and to get open source wireless drivers.

Let's get started.

Broadcom wireless card

The wireless card is a Broadcom 4312. Broadcom has been notorious for its bad Linux support, recently things have started looking up. Broadcom has provided its own hybrid drivers for this card, and the now the free software b43 driver in the most recent kernel (2.6.32) supports it, provided you supply it with the appropriate firmware. Unfortunately I've experienced frequent connection drops with the b43 . So I've resorted to the proprietary Broadcom driver. Download the dource from Broadcom, then do the usual make dance. Place the resulting wl.ko file where it belongs, in /lib//modules/$(uname -r). Then run

depmod -A

to regenerate the new modules.dep file (the -A switch tells it to only look for recent changes). Now test your new module.

modprobe -r  b43

to unload the b43 module, then

modprobe wl

to load your new driver. Do a quick

iwlist scan

to see if you detect any wireless networks. If you experience weird drops, check that you are not pressing the activate/deactivate button (F2) by mistake. If you are satisfied you can then blacklist the other modules in /etc/modprobe.d/modprobe.conf.

Video card (Intel HD Graphics)

Works flawlessly for me with kernel 2.6.32 and the 2.9. intel drivers. Phoronix reports crashes with OpenGL. As I do not game or do any 3D work I cannot say much about this.

Final notes

Function keys work perfectly out of the box, including activating/deactivating wireless, setting screen brightness. I have not tested the audio yet. All in all, a pretty good Linux laptop if you are willing to put up with the semi-proprietary Broadcom drivers.

Education to Free Software

2010-02-09T00:00:00+01:00

While I was volunteering at the Info desk at Fosdem, I was approached by a visitor looking for information on how to get started with Linux. The man had already installed various distros, and his daytime job was as a mainframe programmer. He knew COBOL and assembly. Initially, I was a bit surprised that he felt he needed help. He was looking for courses, starting with the command line, through programming linux applications. I thought everybody was learning with forums and how-tos on the web. There seem to be hundreds of those on the web. For example, linuxcommand.org. He answered he did not learn well that way.

How do you learn Free Software?

I ended up recommending a mixture of Evi Nemeth’s book on Unix administration, Linux from scratch and a new Open University course. But I felt he was still somehow disappointed. He was not interested in becoming a system administrator, or adept at that or other task (setting up Apache, Samba, compiling a kernel, ifconfig and tcp wrappers, python gui programming and whatnot). He was looking something along the lines on how to become a free software hacker.

Becoming a hacker

You will object there are thousands of ways of becoming a hacker, and one of the nice things is that there is no one true path. Nevertheless, some technologies are more important than others in the current ecosystem. Think of bash + C + python. So there probably could be some course/book/workshop along the lines of: master key free software technologies. And attendees need not be n00bs: it could be people proficient with other technology. It could be programmers familiars with other environments, scientists, statisticians. Community is important. Our visitor expressed an explicit interest in helping and participating. This course should preferably be in presence, or at least contain a sizeable online participation component, and embed interacting with distributions and collaborating on code all along. A great way to have packaging and version control there from the start.

Lightweight Free Software workshops

There are commercial training offerings, but they are mostly geared towards companies. This means expensive, and in some cases outright rip-off, as is usual when the people learning are not the same as those paying for it. Also, these events can be very dry and rigidly programmed. So I think it would be interesting to organize with a different pedagogical formula, with a lower price, with a more informal feel but a serious program, geared towards individuals. Think aquarelle evening courses, but more geeky. It could last a month, two evenings per week, from 18 to 20, say. What do you think?

ludovf.net blog

Jfokus 2012

Tradition

Attendee profile

Sponsors

Buzzwords

Web sockets and interactivity

Functional buzz

Fixtures vs Factories

The refactoring dilemma

Up and down the stack

How to sell tech books in a physical bookstore

Mapping a composite key in JPA with @EmbeddedId

Why @Embedded objects?

@EmbeddedId and composite keys

Hitting JPA’s limits

Packet Analysis with Wireshark

Some Perl idioms

Constants

Regular expressions

Classpath trouble

What Rails got Right

A DSL for CRUD Web applications

Enforcing conventions through tools

Community building

Spreading beyond the core community

EuroSciPy 2011

Performance

GPU computing

Machine learning

Programming to learn

Designers needed

Designers needed

Communication

the Great JavaScript Closure Pattern explained

It’s just a function call

Why you need a function

Building an object for outside

Bringing it outside

Happy ending

2011 New Year resolutions

Hardboiled Web Design review

Interesting resources mentioned in the book

What’s C++ like?

Good books to learn C++ from

Migrating from WordPress to Django

Strings and unicode in Python

'String' types

Encoding/decoding

Examples

Conclusion

Postscript: some more encodings

Django CSRF verification failed

Interview with the author of Nginx HTTP server, Clément Nedelcu

Hello. Can you introduce yourself to our readers?

How did you discover Nginx?

What problem did Nginx solve for you?

What’s Nginx’s killer feature?

The book includes a mini guide to the basics of the Linux command line and the Linux filesystem. Are you seeing people migrating away from Windows Server/IIS towards Linux and Nginx?

Nginx has a reputation to scale and serve well under high loads. What made you decide to target the book towards beginners?

What is the part of the book you are most proud of?

Is there anything you would like to tell prospective readers before closing?

Book review: Nginx HTTP Server

Django: display the current language in a template

How good is Django, the Python web framework?

Rewriting this site in Django

A Django business?

Django 1.2 and the new syndication API

The old way

What’s new in 1.2

Introducing the Python defaultdict

Frequencies

The defaultdict solution

Creating a QToolbar with QT Designer

Adding the QToolbar to a QMainWindow

Adding separators

Python profiling with cProfile

Profiling output

Links

How to write books for the Web?