PGXN is on CPAN! http://search.cpan.org/search?query=PGXN

Release most of the toolchain to CPAN.

The second piece of the PGXN infrastructure, after PGXN Maanager, is the PGXN API Server. I’ve just finished the API documentation, which covers both the lightweight static file API provided by mirrors and the superset provided by the API server. So now seems like a good time to talk about the design of the API server and how it works.

At its core, the PGXN API server is just another mirror. It has an hourly cron job that rsyncs to the master mirror, updating the mirror. But then it iterates over the rsync log and transforms some things. Here’s what it does:

• Unpacks each distribution in a directory for browsing. Here, for example, is where one can browse the semver 0.2.1 sources.
• Searches for a README file and any files recognized by Text::Markup and converts them to sanitized HTML with a table of contents. Such files can then be used to display the README on the distribution page and to display individual documentation files.
• Merges the distribution metadata file with the latest stable release META.json generated by PGXN Manager. For example, as of this writing, the API server’s semver 0.2.1 META.json and the unversioned semver.json are identical. Effectively, this format has all the metadata from the META.json as well as a list of all releases of the distribution from the semver.json. This is useful for displaying all the data on the distribution page by fetching the data in a single API request.
• Updates all other versions of the META.json file. For example, if you look at the semver 0.0.0 META.json, you’ll see that it includes 0.2.1 in its list of releases, even though 0.2.1 was released after 0.2.0. This allows semver 0.2.0 page on the main site to have a select list of version to choose from, including versions released later, with a single API request.
• Adds additional metadata to the extension JSON file for all extensions in the distribution. The added data includes release dates for the list all distributions providing the extension, as well as an abstract and doc path for the latest stable release. To see the differences, compare the mirror semver.json to the API semver.json.
• Adds an abstract for each distribution listed in the user’s JSON file and all tag JSON files. Compare, for example, the mirror theory.json to the API theory.json and the mirror data types.json to the API data types.json. This allows the user page and tag pages to include the abstract in the list of distributions released by the user or associated with a tag.
• Adds records to a Lucy-powered full text search index.

All of this merging stuff came out of my thinking following the discussion of the PGXN API RFC. The decision to use Lucy instead of PostgreSQL’s full-text search followed rather naturally from this, as I quickly realized that there was no other driving need for a relational database behind the API at all. The only dynamic API is the search API. Everything else is just static files. And given the performance issues of in-database search, as well as the desire to have fewer outside dependencies, made the decision a natural one.

Beyond the syncing, there is a very simple web server providing the HTTP REST interface to the static JSON files and the full-text search. That’s it, really. The API server is really just another mirror on steroids. The nice thing is that it allows an interface, such as WWW::PGXN or the new PGXN client to work with either interface, just failing gracefully when API server APIs are unavailable.

If you want to learn more about the specifics of the REST API, the API documentation has all the details. Really, it’s quite comprehensive!

I actually consider the API to be 1.0-complete at this point, unlike PGXN Manager. The only thing I want to add is JSONP support for static JSON files (right now it’s only for search results) and might tweak a few things here and there, but otherwise I think it’s in pretty good shape.

Longer term, though, it might be worthwhile to add some other features to enhance the value of PGXN overall. Some ideas:

• Distribution and/or extension ratings (reviews, Like/Dislike, stars, or something).
• Diffs to compare changes between versions.
• A test reporting infrastructure with result matrices (á la CPAN Testers.

But I think we need to build up some momentum on the foundation that’s in place. Have you submitted your extensions, yet?

By Daniele Varrazzo.

PostgreSQL 9.1 is quickly rolling to the beta phase, and the release will bring a new useful feature: extensions. Well, actually extensions have always existed, but they didn’t have an identity before: what you used to get was a set of loose objects and mysterious functions in your namespace, typically getting in the way in your dump and restore. Now here it is: CREATE EXTENSION, in all its uppercase glory!

So, an extension will exist in your database, and this finally gives the possibility to organize them, this is what PGXN is for: a repository for metadata, documentation and code, neatly packaged and asking to be picked off the shelf. I’ve recently written an extension myself so I got some original material to play with PGXN, which has resulted in quite a pleasant experience: the infrastructure is well done thanks to David’s design, and the API is ridiculously easy to interact with: I’ve literally explored it using curl.

There was a big piece missing in the jigsaw that’s being composed: a client to make easy for database developers and administrators to download, build and install extensions, and with PGXN getting more mature and people starting packaging their work it was really begging to be written! So here it is: pgxn.client pgxnclient, a command line tool to interact with PGXN. There is still a lot to be developed, but the idea is to release early, get feedback for it and improve it quickly to complete the extensions picture. So it’s already available on PyPI and very easy to start with it: just use

$ sudo easy_install pgxnclient

and in a few seconds you will have a script called pgxn, offering several commands that can be displayed with pgxn --help (and currently this is the most up-to-date documentation you can get, as new commands are getting added very quickly).

Were you looking for a solution to store hashes? You may try:

$ pgxn search hash
sha 1.0.0
session_hash_tools 1.0.0
semver 0.2.1

Uhm, is sha a possible solution?

$ pgxn info sha
INFO: best version: sha 1.0.0
name: sha
abstract: This module provides datatypes for storing SHA-1,
SHA-2 and MD5 hashes
maintainer: Alexey Klyukin <a...@commandprompt.com>
license: postgresql
release_status: stable
version: 1.0.0
date: 2011-03-16T10:33:00Z
sha1: 0187b0d261d302605bf8d0a15cdbd809deb245dd
provides: sha: 1.0.0

Let’s say it is exactly what we were looking for (we could also display the readme with pgxn info --readme). Shall we give it a try?

$ sudo pgxn install sha
INFO: best version: sha 1.0.0
INFO: saving /tmp/tmpj8G6kM/sha-1.0.0.zip
INFO: unpacking: /tmp/tmpj8G6kM/sha-1.0.0.zip
INFO: building extension
[some compiler log]
INFO: installing extension
[files being copied]

Now the code is in the right place in the database directory (a specific pg_config can be specified to choose which one): If you want it in a specific database:

$ pgxn load -U postgres -d test sha

This will result in CREATE EXTENSION being invoked, if the target database supports it, or in the loading of the provided sql file for PostgreSQL versions up to 9.0.

Easy, wasn’t it?

There are still a lot of features to add:

• drop extension from a database and uninstall from the system
• select the target schema
• work with locally downloaded packages

but as pgxnclient uses the PGXN server on a side and the PGXS build infrastructure on the other, the features will be easy to add and the client work is actually an easy one.

Give it a try if you want: your feedback is very welcome. Meanwhile we will keep on adding features to cover the entire extensions life cycle.

Edit: the program has been renamed to pgxnclient: the relevant URLs have been updated.

The PGXN API documentation is complete. https://github.com/pgxn/pgxn-api/wiki Have a look!

Just deployed new versions. Download files are now .zip instead of .pgz. http://pgxn.org/

The PGXN API Documentation is up! I’ve just finished writing the docs for the lightweight REST API provided by all PGXN mirrors. It also documents how the same APIs differ when provided by the API server. Their are four more documents to write still, APIs provided by the API server but not the mirrors (including full-text search); I should be able to get those done tomorrow.

So if you’re interested in use the API for various things, please have a look! I’d appreciate any feedback or corrections.

Speaking of users of the API, Daniele Varrazzo has started writing a PGXN client app in Python. This is so awesome! It makes me happy that people are able to just get going on this. And Daniele did it before I’d written the docs, just from reading the PGXN source code. Nice!

Anyway, getting these docs written was the last barrier I had to releasing various pieces of PGXN on CPAN and generally being able to get on with my life. I wanted to write these docs first so that I would have a little more freedom to change things if something struct me as especially stupid. Fortunately, as I’ve written the main site as a thin client for the API, most of the lameness has already been excised. I think the only change I’d like to make is to change the {char} URI template variable for the userlist API to {letter}, because only lowercased ASCII letters a-z are allowed. It’s more accurate.

Another thing I think I’ll change is the file name suffix used for the distribution download files. Currently it’s .pgz, but after Daniele complained about it, I asked around and the concensus seems to be to change it to .zip. I’ll likely do that tomorrow, too. Fortunately it’s pretty easy: Just edit the configuration file and rename the files on the master mirror. At least it should be that easy!

There was one other thing in the APIs the felt a bit silly, but I don’t remember what it was, so screw it.

Anyway, feedback on the API docs would be greatly appreciated. And — get hacking!.

I pushed out new versions of PGXN::Manager, PGXN::API, and PGXN::Site today, all with the aim of resolving some issues with case-sensitivity. I was alerted to this when Daniele Varrzzo released italian_fts with the Italian tag and the API sync blew up. Oops.

So here’s the deal. The following objects have case-insensitive names on PGXN:

• Distributions
• Extensions
• Versions
• Users (nickname)
• Tags

These names are case-preserving, so they should show up with the proper capitalization in JSON files and whatnot. But to avoid issues with URLs, I decided that all file names should be lowercase only. Fortunately, nearly everything was lowercase already: only a few nickname JSON files were not lowercased, plus the files for pgTAP. So once I got the code updated, I renamed those files appropriately. In the case of the pgTAP download, I actually re-indexed it, so that the prefix in the zip file would properly be “pgtap-0.25.0” instead of “pgTAP-0.25.0”, since the file name is now pgtap-0.25.0.pgz.

So why make these objects case-insensitive? Well, because I tend to think of the names of things as having meaning, and really, there is no difference in meaning between “pgtap,” “pgTAP,” or “PGTap.” I’ve relased pgTAP; I think it would be as confusing as hell if someone else released a completely different distribution named “PGTap.” (And I went through a bit of hell when the Apache::test CPAN module was replaced with Apache::Test, a completely new module. It really messed things up on the case-insensitive file system I use).

But as I said, the case is preserved in the name I use, just not in the names of files. So if you look at the pgTAP distribution page, you’ll see that “This Release” is labeled “pgTAP 0.25.0” (I need to fix the <h1> element still).

New site deployed with case-sensitivity issues fixed. http://pgxn.org/

Would appreciate feedback on the format of the API docs. So far only “index” and “download” are documented. https://github.com/pgxn/pgxn-api/wiki