With files from the structure of PostgreSQL’s extensions was the same thing.

I was tired of creating the files and edit the META, controlfile, READMEs, etc. every time I start a new extension and felt that I need something that made me more productive, so I decided to create an automatic generator and share it with the world.

I called it pgxn-utils, and you should give it a try: it is easy to install, easy to use and will help you to start hacking quickly!

How?

1. First install it:

   gem install pgxn_utils
   

2. Then start a new extension:

   pgxn_utils skeleton my_cool_extension
   

Thats all! It will create the initial skeleton for you and you can start coding! But, if you don’t want to install it, see it in action

Good hack!

In essence, it’s really simple: Just create a META.json and upload. But to get the full benefit, there are quite a few other recommendations. Already familiar with it? Here’s the checklist:

• Create a META.json
• Create a control file
• Create a Makefile
• Implement the code in the sql and src directories
• Write tests in the test directory
• Write a Text::Markup-recognizable README
• Write Text::Markup-recognizable documentation in the doc directory
• Consider including other files: Changes, LICENSE, INSTALL, COPYING, AUTHORS
• Release it!

Be sure to read the howto for details. Got feedback or suggestions? Leave a comment!

Oh, and check out pgxn-utils and simplify your extension-development life.

Search Sites and APIs

Well, sort of. First of all, I’ve decided that the “search site” should not be a separate thing. The main site will be the search site. This is following the example of JSAN, as well as feedback from Graham Barr, who created and maintains CPAN Search. Apparently people are often confused that search.cpan.org is separate from www.cpan.org. No point in adding in confusion from the beginning. And besides, now that the PGXN fund-raising is over, I don’t know what else would go on the home page.

The other thing that’s happened is, just as I was getting my butt in gear on this stuff, a new CPAN search site came to my attention, μετα CPAN. This is an interesting project. What they did instead of creating a monolithic HTML search site is to create a simple API that serves nothing but JSON. It has search and displays metadata for CPAN objects (distributions, maintainers, modules, etc.). The search site, then, is not really a site at all, but a pure JavaScript application. Once you load it, it just uses the API server to get all the data. There are a few tricks server-side to proxy the API server so as to avoid cross-site scripting issues. But otherwise it just works in the browser.

Now I’m not sure I’ll do the same thing, exactly, but there’s a lot of appeal in creating a RESTful API server that’s independent of the search site, and then building the search site to use it. It also has the advantage of being useful for other projects to just use. Want to create a PGXN search widget for your blog? Yeah, there’s an API for that.

A Super RESTful Directory

Of course, thanks to the “RESTful Directory” design for the mirrors (described here and revised here), any mirror is a lightweight API already. There’s a lot of metadata one can get just from the static JSON files it generates. The design is flexible–but designed with a command-line client in mind. As such, many commands executed in a command-line client would likely requires multiple requests to a mirror. For example:

> install pgtap

This would request /by/extension/pgtap.json from the server. It would then parse that file and see that the latest stable version of pgTAP is in the distribution “pgTAP” at version “0.25.0”. So it would then download /dist/pgTAP-0.25.0.pgz to install.

This is great for a command-line client, but wouldn’t be so great for a search site to be responsive. Ideally, a site should send a single request to get all the data it needs for a particular page.

So here’s what I’m thinking for a PGXN API server: It will offer a superset of the functionality of any other PGXN mirror. That is, all the JSON files in a mirror will be present, but many of them will have more information than they would on other mirrors. And then, of course, there will be other URIs to offer additional API calls.

Details

So what does that look like? Let’s take the pgTAP distribution, which I released on PGXN earlier this week. To find the pgTAP distribution, one requests:

/by/dist/pgTAP.json

From that, one can see that the latest table release is 0.25.0, and so one can then request

/dist/pgTAP/pgTAP-0.25.0.json

to get all the metadata for that particular release. What I propose, to avoid the two requests, is to include the contents of the second file in the first. That would then have all the data necessary to generate the pgTAP distribution page on the PGXN site.

The API would offer similar supersets of data for the extension , owner , and tag metadata files, to have the data necessary for the design of the corresponding extension, owner and tag layouts of the site.

Additional Resources

In addition to adding metadata to the existing mirrored JSON files, there would be other resources available for request from the API server. They would include:

• Extension Documentation. Each distribution may include documentation for included extensions in the doc subdirectory. These will go under the directory for a specific distribution such as /dist/pgTAP/pgTAP-0.35.0/doc/pgtap.html. The latest version of each document would also be available under /by/extension, as in /by/extension/pgtap.html. This requires that the documentation file have the same base name as the extension file itself.

• Other documentation. I’d like to support arbitrary documentation, such as for included binary executables, HOWTOs, etc. The canonical copies will go under the versioned distribution URL, of course, but I’m not sure about permalinks. That might require an extension of the Meta Spec; I haven’t quite figured that out, yet.

• Source code. There will be an interface to browse an unpacked copy of any distribution as plain text. This will be under /src, as in /src/pgTAP/pgTAP-0.35.0/.

Search API

Of course. This is the big one, really. I think it makes sense to have the /by URI respond to search requests. Thus, a request for

/by?q=testing

would search everything. If you only want to search a certain category of object, you’d hit the appropriate URI:

/by/dist?q=tap
/by/owner?q=clochard
/by/tag?q=test
/by/extension?q=gis

The nice thing about this is that it retains the existing entity URLs. The directory level determines which entities you get.

Your Thoughts?

So that’s my thinking on the search API. I’m going to start hacking on it in earnest tomorrow, and perhaps next week I can get a very early version out (basically just another mirror to start with).

But what do you think? Seem like a sane approach? Am I missing anything obvious or doing anything clearly stupid? Please let me know in the comments!

Well, back to work a bit, anyway. Tomorrow is Thanksgiving in the U.S., so I’m not likely to do much more until next week. But I’ll be heads down on it, then, working to get the search site done.

Before I talk about that, though, I realize that I forgot to post the link to my slides from PGWest. I actually posted them before the talk, I just never got round to mentioning it here. So, here it is:

I’m pretty happy with how it came out. The organizers shot video of it, too; I’ll post a link or embed when it drops.

Meanwhile, I have returned to work on the search site design. How do you like it? The pages I’ve completed the design for are:

• Search/Home Page
• Search Results
• Extension documentation
• Distribution information
• Owner information

Imagine if all open-source PostgreSQL extensions were posted on this site. You could search through them all, including their documentation, browse their docs, find related extensions, and see what else the extension owner has worked on. All in one place, without downloading anything. That’s the plan for this site. Notice anything missing you think should be there? Let me know.

So I’m going to start work on the app to create this now, but before I do, I wanted to point out the PGXN project wish list wiki. Folks have been having all sorts of ideas for features that would be great for PGXN, and I don’t want to lose track of them. So if you’ve had any ideas at all, no matter how big or small, please feel free to add them to the wiki. As the project progresses, some I might be able to just build in, but others I might want to see go onto a project road map. What do you want to see on the road map?

Oh, and if you want to see your extensions on this site, please do register a PGXN Manager account. Once your account is approved (and we’re pretty quick to do so), have a look at the how to and release your extensions on PGXN today!

• PostgreSQL itself
• Bundled procedural languages (plpgsql, plperl, etc.)
• All contributed modules

Basically, anything that an extension might want to declare as a dependency, but that isn’t on PGXN itself.

There are a number of ways to do this. Which do you think would be the best approach?

1. Hard-code a list into the source code
2. Include an editable list in the runtime configuration file
3. Add a database table reserved for them and an API to edit it
4. Create a “pgdev” user and upload a distribution declaring all those extensions in a META.json file
5. Same as 4, but actually upload the PostgreSQL source

I’m leaning towards #2, perhaps having it automatically maintain a list in the database and a metadata file on the mirrors.

But what do you think? Opinions wanted!

Can you help us get to our goal in time for PgWest 2010, which is November 2-4 in San Francisco? I’ll be giving a talk there, “Building and Distributing PostgreSQL Extensions Without Learning C”, in which PGXN will of course be featured. Would be great to announce that the fundraising was successful.

As for the time I’ve put in so far, I’m happy to have PGXN Manager up and working, but of course it has taken more hours than I expected. 76.5 so far. I’d estimated 40. Meanwhile, the database design is up to 43 hours from the estimated 24. And that doesn’t count the hours I spent chasing shiny yaks and shaving them, like SemVer and Router::Resource. Those of you who estimate development projects, take heed! I clearly need to double all my estimates before I submit them.

Still, with the fundraising nearly done, I’m committed to finishing this project. I view it as a project budget, and so that’s what it will be, whether or not it takes me twice or four times as many hours as I’d estimated.

That said, you could help. Right? PGXN Manager is in good shape, but it’s not done. If you’d like to roll up your sleeves and contribute some code, please fork it, build it, and hack what you can. A few things on the to-do list:

• Add user admin interface. Should include checkbox to make and remove administrative permission
• Add mirror admin interface
• Add extension ownership permissions interface

The database API is there for these bits already; the code would mainly be in Perl. Hit me on #pgxn on Freenode if you want to help.

If documentation is your thing, contributions there would be appreciated, as well. In particular, the About PGXN page is a bit thin. Other interfaces will need help, too. More on that as we add users.

Thanks everyone for your support!

So tell me what you think about the alternative plan I’ve come up with.

I’ll have two plack apps, one mapped to /auth and one mapped to /no-auth. The former will require authentication and the latter will not. They’ll have separate dispatch tables, of course. Then I’ll have the SSL site proxied to /auth and the non-SSL site proxied to /no-auth. Makes sense, right?

The only hangup I can see (though maybe you can see others?) is that my current method of generating URIs knows nothing about proxies. So If I link to /auth/account, when requests come through the proxy, it should actually create a link to /account. Does it make sense to use relative links instead of absolute links for all links to avoid this issue? I think it might be kind of annoying, because not all the code is aware of the current URI, though there are ways to deal with that.

Thoughts?

I guess I could stick with absolute URLs, and then have the uri_for use $req−>uri−>path for a proxied request and $req−>path_info for a non-proxied request.

Sure would be nice if there was some way to tell from the environment that a request was forwarded from an SSL connection, though. Alas, there are only three extra environment variable set by the proxy server:

• HTTP_X_FORWARDED_FOR
• HTTP_X_FORWARDED_HOST
• HTTP_X_FORWARDED_SERVER

No HTTP_X_FORWARDED_PORT or HTTP_X_FORWARDED_SSL or SSL_ENABLED or anything like that. Maybe I’m missing something in my reading of the mod_proxy documentation?

Architecturally, I’ve gone for a very minimal Plack-based app. No Catalyst, Jifty, or even Dancer. Just a very simple Plack app that uses Router::Simple::Sinatraish to route URIs to the appropriate controller actions (which are just class methods). The controller just dispatches to Template::Declare-based templates for the HTML rendering. I guess I’ve kind of created my own framework here, but really, there ain’t much to it. This app is simple enough that I couldn’t see the use of adding all the overhead of a framework.

Meanwhile, I’ve been hacking on the distributon class. This will be the core class of the app. It takes a Plack upload object and a username and does all the rest, analyzing an uploaded archive, normalizing it if necessary, registering it with the database, and indexing it by updating all the appropriate JSON files on the mirrors. It’s nearly finished, but I have one other thing to do in the database, first.

In my last update, I asked for advice on whether or not PGXN should allow an extension with the same version number to appear in multiple distributions. And thanks to a comment from Aristotle, I’m changing it to allow that. But it also means that my original extension json spec needs to change.

Here’s an example of what I’m thinking. Say that there are three versions of an extension named “trip”, and that they appear in distributions as follows:

trip 0.2.6
  pair-0.3.0
trip 0.2.5
trip-0.2.2
pair-0.2.2rc
pair-0.2.1
trip-0.1.1
trip 0.2.4
pair-0.1.1rc
pair-0.1.0

So sometimes it’s in the “trip” distribution and other times it’s in the “pair” distribution. My thought is that, for a given version, it would list the distributions it’s in in reverse chronological order (by upload date). So the format would be:

{
   "latest": "stable",
   "stable": { "dist": "pair", "version": "0.3.0" },
   "testing": { "dist": "pair", "version": "0.2.2rc" },
   "distributions": {
      "0.2.6": [
         { "dist": "pair", "version": "0.3.0" }
      ],
      "0.2.5": [
         { "dist": "trip", "version": "0.2.2" },
         { "dist": "pair", "version": "0.2.2rc", "status": "testing" },
         { "dist": "pair", "version": "0.2.1" },
         { "dist": "trip", "version": "0.1.1" }
      ],
      "0.2.4": [
         { "dist": "pair", "version": "0.1.1rc", "status": "testing" },
         { "dist": "pair", "version": "0.1.0" }
      ]
   }
}

This way, every distribution it’s included in is listed, and clients can quickly tell where to find the latest stable, testing, and unstable versions, and which of those is the most recent. This is a bit more convoluted than the original, but I think is a good choice, in that it’s comprehensive but also easy to figure out what’s the latest.

Unless you can think of a better format, this is what I’m going with. Comments?

Look for a post next week announcing an alpha program!

Let me tell, you, that’s a lot easier than it sounds. After browsing through images returned for queries such as “pixin,” “pig skin,” “zen,” “zen stones,” “pixies,” and god knows what else, I finally focused on “database icon.” This was much more useful, as I hit on images such as the one to the right. Pretty standard database icon, right? Three disks stacked up. I’m not sure why this is the classic graphical abstraction for a database, but I’ve seen it a zillion times. So often that it’s a cliché.

Much more interesting was stumbling onto the icon to the left. I love how three disks have been broken down to their simplest possible representation. Because I’m familiar with the cliché, I can see at a glance exactly what this icon is supposed to be. So I thought that might be useful to use.

Searching for “extension icon” was a bust. Most of the relevant images, like the one to the right, seemed to be related to Adobe Extension Manager (try to find a link for that!). I didn’t want to go near that, of course. But, inspired by the CPAN logo, I searched for “library icon.”

Bingo! I hit on the amazing icon at right. Of course! Gears make a great icon for extensions, don’t you think? This folder is like an extension library folder. So then I thought, how can I combine these two icons, the abstract database and the gear?

Well it’s simple, isn’t it? The database icon can be the holes in the extension gear. It’s punched out, right? PGXN will offer loads of database extensions to download (we hope), and when you use one in your database, PostgreSQL will, you know: turn the gear. It just fits.

Strongrrl created the logo and type treatment based on these ideas, adding the tilt and the figure/ground reversal, and I really love it. It’s a really great logo for a database extension site. I imagine I can use variants of this throughout the final design.

1. Allow things to scale.
2. Try to put things in intuitive locations.
3. Allow for metadata to be stored to act as a Web API.

The first two are a bit mutually-contradicting. Thinking scalability mainly means minimizing the chances that too many files can be put into a single directory. CPAN started out with all author directories in a single directory. Given the sheer number of CPAN contributors, this quickly got to be a bottleneck. To correct for that, they started “hashing” the first two letters of author names to create subdirectories. My CPAN directory, for example, is D/DW/DWHEELER. That doesn’t quite make for an intuitive location, but it’s not bad, and the tradeoff seems sufficient to keep things sane.

The storage of metadata is important, too, as I plan to have the client send requests for JSON files to mirrors in order to find distributions. Basically, this came down to a naming convention, as well as a recipe for how to find metadata files for people and extensions. What I’ve come up with is three directories:

• meta will contain PGXN metadata
• dist will contain distributions
• by will contain query-able JSON files

The meta directory will contain PGXN metadata (mirrors.json, timestamp, other detritus). If I ever create an index file that lists all distributions, it would go there, too. I’m not going to discuss it any further here, except to note that it already has mirrors.json, which clients will be able to use to present users with a list of mirrors to choose from.

The dist directory will be organized with directories named with “hashes” of the first two letters of distribution names. Let’s say that I’m releasing pgTAP 0.25 on PGXN. To distribute it, the management application will create the directory (if it doesn’t already exist):

dist/p/pg/pgtap/

Then, for the 0.25 release, it will add three files to that directory:

dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme

The pgtap-0.25.pgz file will contain the zipped distribution ready for download. pgtap-0.25.json will contain metadata about the distribution, such as the owner’s name, the manager’s name (more on these folks below), list of included extensions, location of the .pgz and .readme, and its SHA1. pgtap-0.25.readme will of course contain the README for the distribution (if it has one).

Every release of pgTAP will have these three files, so after several releases, the pgtap directory might have these files:

dist/p/pg/pgtap/pgtap-0.23.pgz
dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.23.readme
dist/p/pg/pgtap/pgtap-0.24.pgz
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.24.readme
dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme
dist/p/pg/pgtap/pgtap.json

The last file there, pgtap.json, will actually be a symlink to the JSON file for latest production release of pgTAP. In this case, it would link to pgtap-0.25.json. The nice thing about this is that, if a client wants to find the information about the latest release of the pgtap distribution, all it will have to do is send an HTTP GET request for dist/p/pg/pgtap/pgtap.json to any mirror.

The by directory will also contain JSON files for clients to request. The idea is that you want to find information “by” something. To start with, there will be three subdirectories:

by/extension/
by/manager/
by/owner/

The first directory, by/extension/, will contain links to JSON files for extensions. Say that the pgTAP distribution offers two extensions to PostgreSQL named “pgtap” and “schematap”. The links would be:

by/extension/p/pg/pgtap/pgtap-0.23.json
by/extension/p/pg/pgtap/pgtap-0.24.json
by/extension/p/pg/pgtap/pgtap-0.25.json
by/extension/p/pg/pgtap/pgtap.json

Each of these will simply be symlinks pointing to the appropriate distribution files:

dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap.json

Yes, the last one is a symlink to a symlink. Similarly, these files for “schematap”:

by/extension/s/sc/schematap/schematap-0.23.json
by/extension/s/sc/schematap/schematap-0.24.json
by/extension/s/sc/schematap/schematap-0.25.json
by/extension/s/sc/schematap/schematap.json

Point to exactly the same files. Essentially, this is a way for extensions to point to the distributions that contain them. by/extension/s/sc/schematap/schematap.json points to dist/p/pg/pgtap/pgtap.json, which contains path to the .pgz file to download (and lots of other metadata, too).

The idea is that, whatever the name of the extension you want, the client will be able to easily find the metadata file that tells it where to find the distribution.

The by/manager and by/owner directories, on the other hand, contain JSON files with information about managers and owners and their distributions. Definitions:

An “owner” is someone who owns a distribution. This will often be the original author of an extension, but may be someone else if maintenance has been passed on. Basically, the “owner” is the person who should be contacted with bug reports and the like

A “manager” is someone who manages the release process. This is the user who will log into the PGXN management application and upload a distribution for release.

These two people will often be the same person, but not always. I’ve avoided the term “author” (the term used by CPAN) because the author of an extension may no longer maintain it. “Owner” seemed like a better choice (individual distributions are free to describe their contributors however they wish).

As the owner of a few extensions on PGXN, I’d have this file:

by/owner/t/th/theory.json

This file would contain a list of my distributions and perhaps some other information (like my full name and blog URL).

As the manager of extensions on PGXN (that is, I actually uploaded them), I would also have:

by/manager/t/th/theory.json

This file would contain a list of the distributions I’ve released on PGXN. This might be exactly the same as the list in my owner file, but may not be. Perhaps for one release of pgTAP, say 0.26, Duke Leto uploaded a release. In that case, 0.26 would probably be in my owner file, but not in my manager file: it would be in Duke’s manager file, instead.

So these are the basics of the directory structure for the networked mirrors. Note that I haven’t thought much about everything that will go into the JSON files (or whether or not they’d be versioned). That will likely depend quite a lot on what the management database ends up looking like. I’ll be working on that next.

But other than that, comments? Questions? Criticisms? Recommendations? Leave a comment and let me know!

Since then, we’ve done quite well, and are getting closer to our goal every day. PostgreSQL Experts soon committed to cover $5,000 in expenses, Dalibo pledged €5,000 last month, and just yesterday Etsy promised $1,000. We’ve also received contributions from:

• Richard Broersma ($250)
• TigerLead ($250)
• Hubbell Group Inc. ($100)
• John S. Gage ($100)
• Damian Soto-Renou ($100)
• Thom Brown ($100)
• Kineticode, Inc. ($25)
• CxNet (Chile) ($25)
• Schemaverse ($25)

So far, we’ve raised $18,359. Yay! I’m really happy with how well this is going. But we still need to raise another $6,641 to meet our goal. I’m hoping to get that done in the next couple of weeks. Can you help out?