by Dickson S. Guedes

Hello everyone!

I’m proud to tell you that a new version of pgxn_utils was released!

In this version some errors with OpenSSL was fixed (thanks @theory to report then), and now you can release an extension to PGXN in five steps even using Ruby 1.8!

Another change was the executable’s name that changed from pgxn_utils to pgxn-utils for a close integration with next version of PGXN Client, but some work need to be done, yet.

I used pgxn-utils to release itself to PGXN!

$ pgxn-utils release pgxn_utils-0.1.3.zip 
Enter your PGXN username: guedes
Enter your PGXN password: ***********************
Trying to release pgxn_utils-0.1.3.zip ... released successfully!
Visit: http://pgxn.org/dist/pgxn_utils/0.1.3/

Cool, eh? So, since the PGXN’s mirrors were synced and you have pgxn client, you could install pgxn_utils using:

pgxn install pgxn_utils

If you don’t have pgxn client you can install it using rubygems

gem install pgxn_utils

Have fun!

by Dickson S. Guedes

Hello everyone!

I’m proud to tell you that a new version of pgxn_utils was released!

Now you can release a distribution to PGXN in five steps!

First, install it:

gem install pgxn_utils

Second, create your extension, optionally overwrite some defaults:

mkdir $HOME/extensions
cd $HOME/extensions
pgxn_utils skeleton my_extension --maintainer "Dickson S. Guedes"

Third, code!

Fourth, bundle it:

pgxn_utils bundle my_extension
Extension generated at: /home/guedes/extensions/my_extension-0.0.1.zip

Fifth, release it:

pgxn_utils release my_extension-0.0.1.zip
Enter your PGXN username: guedes
Enter your PGXN password: ******
Trying to release my_cool_extension-0.0.1.zip ... released successfully!
Visit: http://manager.pgxn.org/distributions/my_cool_extension/0.0.1

Ah, you can export PGXN_USER and PGXN_PASSWORD if you are tired to type your username and password everytime.

Check this out!.

by Dickson S. Guedes

Hello everyone!

This was a productive weekend that allowed me to work on some new features in pgxn_utils and I’m proud to tell you that a new version was released!

Trying to simplify your extension-development life I’ve added two tasks to pgxn_utils: change and bundle. The first one is just a convenient way to change META information about extension, incrementally, the second one is an easy way to archive your extension in a zip file well named.

To install it just type:

gem install pgxn_utils

Or, if you don’t want to install it yet, see it in action on this screencast [3:05].

Work in progress…

I’m working now to simplify the release, creating a task to send bundled file to PGXN.

There are a lot of work to do yet so, please, tell me if you found a bug or have suggestions.

Have a nice code! “:)

by Dickson S. Guedes

Do you ever have problems with copy and paste? I often did, and that is why I create custom templates for often used files that match certain patterns.

With files from the structure of PostgreSQL’s extensions was the same thing.

I was tired of creating the files and edit the META, controlfile, READMEs, etc. every time I start a new extension and felt that I need something that made me more productive, so I decided to create an automatic generator and share it with the world.

I called it pgxn-utils, and you should give it a try: it is easy to install, easy to use and will help you to start hacking quickly!

How?

1. First install it:

   gem install pgxn_utils
   

2. Then start a new extension:

   pgxn_utils skeleton my_cool_extension
   

Thats all! It will create the initial skeleton for you and you can start coding! But, if you don’t want to install it, see it in action

Good hack!

Following my post outlining a possible network directory structure, Aristotle Pagaltzis saw fit to bug me via email about a different approach. I couldn’t understand WTF he was talking about until today. Then it lit my brain on fire. As a result, I now think that there is a much better way to organize the metadata files for the PGXN — one that happens not to include any symbolic links (which is something that Andreas König has been flagging, via email, as a possible bottleneck).

First, the /dist directory will be the same as before. Releases of pgTAP would be in:

dist/p/pg/pgtap/pgtap-0.23.pgz
dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.23.readme
dist/p/pg/pgtap/pgtap-0.24.pgz
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.24.readme
dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme

The only change is that the pgtap.json symlink is gone.

Now, the new stuff. In the root directory will be a file, index.json, that contains templates for URIs. It will look something like this:

{
    "dist":   "/dist/$a/$ab/$dist/$dist-$version.pgz",
    "readme": "/dist/$a/$ab/$dist/$dist-$version.readme",
    "meta":   "/dist/$a/$ab/$dist/$dist-$version.json",

    "by-dist":      "/by/dist/$a/$ab/$dist.json",
    "by-extension": "/by/extension/$a/$ab/$extension.json",
    "by-owner":     "/by/owner/$a/$ab/$owner.json",
    "by-manager":   "/by/manager/$a/$ab/$manager.json",
}

The PGXN client will always fetch this file before it does anything else, because the file tells it how to find stuff. The advantage here is that the client doesn’t have to know anything about how the directory is actually organized, just what the template variables might be. They are:

• $dist: A distribution name
• $version: A version number
• $extension: An extension name
• $owner: An owner’s name
• $manager: A release manager’s name (managers are the people who upload distributions to PGXN)
• $a: The first letter of a distribution, extension, owner, or manager name.
• $ab: The first two letters of a distribution, extension, owner, or manager name.

I’m not thrilled about using prefix-staggering to avoid having too many files in a directory. But the truth is that this approach allows me to punt. I could also make sure the client supports, for example, $bc and $cd, so that one could stagger things differently. And then the nice thing is that I don’t have to use those at all. The templates will tell the client exactly how to construct the URIs for things, and the templates needn’t include those staggering variables if they’re not appropriate. The client won’t care because it will have no built-in knowledge of how things are organized. It will have to find out from index.json.

From the URI templates, you can now see where the other metadata will be stored. For extension names, a hypothetical pgTAP distribution with two extensions will have a JSON file for each extension:

/by/extension/p/pg/pgtap.json
/by/extension/s/sc/schematap.json

The pgtap.json file will look something like this:

"stable":   "0.25.0",
"testing":  "0.26.0b1",
"unstable": "0.30.0u",
"versions": {
    "0.26.0b1": { "dist": "pgtap", "version": "0.26.0b1", "status": "testing" },
    "0.30.0u":  { "dist": "pgtap", "version": "0.30.0u",  "status": "unstable" },
    "0.25.0":   { "dist": "pgtap", "version": "0.25.0",   "status": "stable" },
    "0.24.0":   { "dist": "pgtap", "version": "0.24.0",   "status": "stable" },
    "0.25.0":   { "dist": "pgtap", "version": "0.23.0",   "status": "stable"  }
}

Right at the top, it would always list the most recent stable, testing, and unstable version number, and then it would have a list metadata for all versions. Said metadata would include the associated distribution name, version, and release status.

Here’s how it would work. Say I ask the client to install pgtap:

PGXN> install extension pgtap

The client would first fetch /index.json, then look for the URI template for “by-extension”, which is /by/extension/$a/$ab/$extension.json. Filling in the template, it would know to request /by/extension/p/pg/pgtap.json. With that file, it would see that the most recent stable version is in the “pgtap” distribution version 0.25.0. Using the dist URI template, which is /dist/$a/$ab/$dist-$version.pgz, it would then fetch /dist/p/pg/pgtap/pgtap-0.25.0.pgz.

The advantage here is that there are no symbolic links and no knowledge of the directory structure built into clients. The client just knows to fetch /index.json and then to use the templates in that file to fetch other information. That’s the whole interface. Very RESTful.

The structure of the other /by files would be similar. For

PGXN> install dist pgtap

the client would use the “by-dist” URI template to construct the URL /by/dist/p/pg/pgtap.json. That file would have something like:

"stable":   "0.25.0",
"testing":  "0.26.0b1",
"unstable": "0.30.0u",
"versions": {
    "0.26.0b1": "testing",
    "0.30.0u":  "unstable",
    "0.25.0":   "stable",
    "0.24.0":   "stable" ,
    "0.23.0":   "stable"
}

So then the client would know that “0.25.0” was the most recent version, and use the dist URI template to request /dist/p/pg/pgtap/pgtap-0.25.0.pgz.

If The client command had been:

PGXN> readme dist pgtap

It would use the readme URI template. And the command:

PGXN> meta dist pgtap

Would use the meta URI template to fetch the metadata for the distribution.

If the client had requested a specific version:

    PGXN> install dist 0.23.0

It could either use the by-dist URI template to download the list of all versions to see if 0.23.0 was valid, or just use the dist URI template to try to download the distribution itself.

And finally, the owner and manager JSON files, such as

/owner/t/th/theory.json

Would look something like:

"full_name": "David Wheeler",
"email": "theory@pgxn.org",
"uri": "http://justatheory.com",
"distributions": {
    "pgtap": [ "0.25.0", "0.24.0", "0.23.0" ]
    "pair": [ "0.2.0", "0.1.0", "0.0.5" ]
}

With that, the client can be asked to fetch metadata for a given owner name and use it to figure out what distributions and versions the the owner, um, owns. One could then fetch the metadata, readme, or distribution file for any of those distributions and versions.

Overall, I think that this is a much better solution than I outlined before. If only I could figure out something more elegant that the prefix-staggering/hashing stuff, it would be just about perfect.

Thoughts?

I’ve posted a draft of the “PGXN distribution metadata specification,” or PGXN Meta Spec.” This document specifies the structure and format of the META.json file that PGXN will require in every distribution. In fact, this is the only required file in a distribution. Its job is to describe the distribution, its extensions, and its dependencies, among other things. This file is key to the whole thing.

To create it, I’ve ported the CPAN Meta Spec, version 2, but with all deprecated fields removed, and some of the more complex stuff taken out. I also made a couple of the “required” fields “optional.” At its simplest, the file might look something like this:

{
  "name": "pgTAP",
  "abstract": "Unit testing for PostgreSQL",
  "version": "0.25.0",
  "owner": "David E. Wheeler <theory@pgxn.org>",
  "license": "postgresql",
  "meta-spec": {
    "version": "1.0.0",
    "url": "http://github.com/theory/pgxn/wiki/PGXN-Meta-Spec"
  }
}

Not too bad, eh? The URL for the spec might change (might move it to the main site and/or the mirrors), but otherwise, I think this is pretty solid. Not too much work to deal with, and reasonably easy to create by hand (which is likely how we’ll all start out).

And additional key that will be really important for the PGXN client is prereqs. This key allows you to identify the prerequisites (from PGXN or the PostgreSQL core contrib extensions) required to build, test, and/or use a distribution. For example, if I were to release an ordered pair extension, it of course would include tests written with pgTAP. So I’d have something like:

{
  "name": "pair",
  "abstract": "An ordered pair data type",
  "version": "0.1.0",
  "owner": "David E. Wheeler <theory@pgxn.org>",
  license: "postgresql",
  "meta-spec": {
    "version": "1.0.0",
    "url": "http://github.com/theory/pgxn/wiki/PGXN-Meta-Spec"
  },
  "prereqs": {
    "runtime": {
      "requires": {
        "PostgreSQL": "8.0.0"
      },
      "recommends": {
        "PostgreSQL": "8.4.0"
      }
    },
    "test": {
      "requires": {
        "pgTAP": 0
      }
    }
  }
}

That’s saying that the “pair” distribution requires PostgreSQL 8.0.0 or higher and any version of pgTAP to run the test suite. I’ve also recommended PostgreSQL 8.4, as that’s where it will run best.

Of course, to get the real power of PGXN, you’ll also want to use the provides key, which allows you to identify the extensions included in your distribution. Say that I finally got around to breaking out the schema testing assertions from the logical testing assertions in pgTAP. I might call the second module “schematap.” So to spell it out, I’d add this to the first example above:

  "pgtap": {
    "file": "sql/pgtap.sql.in",
    "version": "0.25.0"
  },
  "schematap": {
    "file": "sql/schematap.sql.in"
  }

So now the indexer will know that the “pgtap” extension is in sql/pgtap.sql.in and the “schematap” extension is in sql/schematap.sql.in. This is important because it allows other distributions to specify “schematap” as a prerequisite. It also means that, in the PGXN client, you can type something like:

PGXN> install schematap

And, because “schematap” will have been indexed on the network, the client will be able to find the pgTAP distribution and install it, complete with the “schematap” extension.

A final note. Version numbers in the Perl community are a disaster. Wanting to avoid that whole morass, I had originally intended to require numeric version numbers. But David Golden — the current maintainer of the CPAN Meta Spec — pointed me to Semantic Versioning, a version number specification by GitHub’s Tom Preston-Werner. This style of version numbering is great for PGXN for a few reasons:

• It closely matches how PostgreSQL itself is versioned.
• It’s very easy to compare version strings.
• Someone else has already dealt with the pain of writing a spec

So this is the standard that PGXN will require. Every version number will be dotted-integer with three integers (X.Y.Z) and an optional ASCII string at the end. That’s it. PGXN won’t invest any special meaning in the version string the way CPAN does. It will just compare version numbers.

Anyway, please review the spec itself and leave any comments or questions below. I expect to start hacking on this stuff this week!

Update 2010-08-24: I’ve just updated the spec to change “owner” to “maintainer.” I think that the latter term is much better for this purpose. And then I can use “owner” in PGXN to identify the person who uploads a distribution.