JSON

Using JSONP Callbacks

2026-10-07T16:13:48Z

Using the new JSONP callback support in the PGXN API, I just added a sidebar to my personal blog that shows a list of all my PGXN distributions. The code is simple:

<div class="links">
    <h2>PGXN Codeh2>
    <script type="text/javascript">
    function pgxn_distros(data) {
        document.write('');
        for (dist in data.releases) {
            document.write(
                ' + dist +
                '>' + dist + '
' +
                '' + data.releases[dist].abstract + '
'
            );
        }
        document.write('
');
    }
    script>
    <script type="text/javascript"
        src="https://api.pgxn.org/user/theory.json?callback=pgxn_distros">
    script>
div>

And the output looks like this:

Not bad, eh? AS you can see, JSONP is dead easy to use with the JSON files served by the JSON API. Try that, CPAN!

About the Infrastructure: PGXN Manager

2026-10-07T16:13:48Z

The PGXN infrastructure is currently made up of four parts:

The network of mirrors, derived from the master mirror, and synchronized on various schedules via rsync (host a mirror).
PGXN Manager (code) is the core of the network. It provides the interface for users to upload releases, processes those releases, indexes them, and puts them on the master mirror. Details below.
PGXN API (code) is a mirror server with benefits. Once an hour, it rsyncs from the master mirror, and does extra processing of new and modified files, notably full-text indexing. I’ll write up some details next week.
PGXN Site (code) powers the main site. It’s a thin wrapper around the API server, using WWW::PGXN to fetch JSON files and convert them to HTML. I’ll write more about this bit next week, too.

Some details on Manager. This is by far the most complicated part of the system. Which is funny, because I hadn’t anticipated that when I started work on PGXN (I’d estimated half as many hours as for implementing the site). But as I worked through design issues and wrote the code, the need for that complexity became apparent – and not just because it’s the only part that offers authentication. The main reason it’s complex is so that no other part needs to be.

Allow me to explain. People can upload almost anything as a distribution. So long as the META.json adheres to the spec, the rest can be just about anything. But the upload doesn’t necessarily end up on the network unmodified. Sure, if you follow the guidelines of the HOWTO what you uploaded will be exactly what ends up on the network. But I didn’t want to be that strict about PGXN Manager would accept. So in addition to verifying the structure of your META.json file, PGXN::Manager::Distribution also:

Extracts the archive. If it’s not a zip file, a zip archive is created from the contents of the uploaded file. You can upload any kind of archive readable by Archive::Extract. The currently supported formats are: .tar, .tar.gz, .gz, .Z, tar.bz2, .tbz, .bz2, .zip, .xz,, .txz, .tar.xz and .lzma. By always converting to a zip file, PGXN client apps can be quite simple, not having to worry about the archive format.
Validates the META.json. This part isn’t perfect, yet. I fixed a bug today where it would die and return a 500 on a missing version number. It’d probably be worthwhile to adapt CPAN::Meta::Validator to validate PGXN META.json files at some point, both for Manager and for developers wanting to validate before uploading.
Normalizes all version numbers in the META.json into semantic versions. You can specify the distribution version, prerequisite versions, and extension versions as simple numbers and they’ll be converted to semantic versions. A version like “1.20”, for example, becomes “1.2.0”. See the SemVer documentation for details on how versions are normalized via the declare() method. This normalization is done so that client applications will get known valid semantic versions to compare when determining dependencies. However, it’s best that they be semantic versions to begin with. Normalized versions will be written back to the archive META.json file (with the “generated_by” key updated to reflect that PGXN Manager regenerated the file). If no versions need validating, the archive META.json will be left alone.
Makes sure the zip archive has a directory prefix named "$dist-$version/". If the archive has no directory prefix, or if the prefix is not "$dist-$version/", the archive is rewritten with that prefix. This ensures that the archive will always extract into a directory with the same name as the archive and not spray files all over your desktop when you unzip it.
Copies or writes out a new zip file named "$dist-$version.pgz". Think of .pgz as “PostgreSQL Zip” or, if you’d rather, “PGXN Zip”. Either way, it’s just a zip archive.

That processing done, with a good META.json and zip archive, the JSON, username, and SHA1 of the zip archive are handed off to the database for more processing. The add_distribution() database function does all the heavy lifting here. It:

Parses the JSON string, validates that all required keys are present, and normalizes version numbers. Yes, this is redundant, but I don’t think I need to lecture the reads of this blog about database integrity. :-)
Creates a new metadata structure and stores all the required and many of the optional meta spec keys, as well as the SHA1 of the distribution file, the date, and the user’s nickname.
Sets the “release_status” to “stable” if there was no status in the original JSON.
Adds a “provides” section to the metadata if none was included in the original JSON. In such a case, it assumes that the distribution contains one extension and that it has the same name and version as the distribution itself.
Validates that the uploading user is owner or co-owner of all provided extensions. If no one is listed as owner of one or more included extensions, the user will be assigned ownership. If the user is not owner or co-owner of any included extensions, an exception will be thrown.
Records the distribution, extensions, and tags in the database.

Once all this work is done, add_distribution() returns all the JSON that needs to be written to the mirror. These files make up the “index” on the network, and include:

The META.json file (example). This file is derived from the META.json included in the archive (example), but reflects all the normalization changes and added keys outlined above.
A short JSON file summarizing all releases of the distribution (example).
JSON files describing all releases of all extensions included in the distribution (example).
A JSON file for the user, which includes data about all releases made by that user (example).
JSON files for each tag associated with the distribution. Each of these files lists all the distributions the tag is associated with (example).

It also returns JSON for network statistics files. These are updated every time a new release is uploaded:

dist.json lists the 56 most recent releases and has a count of all distributions and of releases of those distributions.
extension.json lists the 56 most recent extension releases and a count of all extensions on the network.
user.json lists of the 56 most prolific users (based on the number of distributions) and a count of distributions and releases for each.
tag.json lists the 56 most popular tags (measured by the number of distributions they’re associated with) and a count of all tags on the network.
summary.json has basic summary information about the network, which is just counts of distributions, releases, extensions, users, tags, and mirrors.

If you think that’s a lot of data to be updated, you’re right! But since releases are relatively infrequent (a couple a day at the moment), it’s best to generate all this stuff as static files that are rsynced to all mirrors. In this way, any mirror can function as a very simple, lightweight REST API. And indeed, that’s just how the planned PGXN client will behave. The WWW::PGXN already provides the interface it will use.

I guess that was a lot of information. Let this be a reference document for interested hackers, then. The core functionality is all there, but there’s a lot more to be done:

Add a UI for users to assign extensions to co-owners (when more than one user is allowed to upload an extension) and to transfer ownership (#6).
Add a user administration interface (#7).
Add a UI where admins can transfer ownership of extensions when an owner disappears (#8).
Add a callback architecture to trigger other actions on successful upload. These would include tweeting new releases (currently just handled in the controller) and running arbitrary command-line utilities (such as syncing the PGXN API server so it can be up-to-date as quickly as possible) (#9).
Add support for “instant mirroring” via rrr (#10).
Add an Atom feed for recent releases (basically a dupe of the distribution stats file) (#17).
Add a command-line interface to the API. The HTTP server is written in such a way that it can act as an API serving JSON, but it’s not well-tested and nothing uses it yet (that I’m aware of) (#1).
Add UI for re-indexing a distribution (#11).
Add ability to delete distributions (#12)?
Add an email gateway so that email addressed to $nickname@pgxn.org will be forwarded to a user’s actual address. This would allow us to remove literal email addresses from the JSON files and the site (the site obfuscates them, but still…). Anyone got some good postfix chops for this? The users table is quite simple (#13).

Want to help out? Fork PGXN Manager and have at it. Hell, at this point I’d really appreciate a code review, as I’m pretty sure there’s only been one set of eyes on this code so far.

Next week, I plan to blog about

Project status: where the hours went and where they’re going.
URI templates and how they provide a method index for the mirror API
Mirrors and mirroring via rsync
PGXN API: how it provides a superset of the mirror API, including full-text search.
PGXN Site: how it’s just a very thing wrapper over the API (but can read the API directly from the file system!)
Plans for the PGXN client

But given how these things go, and how I need to start writing mirror API and API server documentation, it might take me a longer to get to them all.

What Media Types for Browsing?

2026-10-07T16:13:48Z

Thanks to a comment from “Anon,” I’m auditing the media types used to serve files from the “Browse” links on the site. The browse interface is managed by Plack::App::Directory, which gets its media type mapping from this file. What PGXN::API does is simply change the mappings of some of those types to text/plain. As of this commit, files are served as plain text if:

The strings “html”, “x-c”, “xml”, “calendar”, or “vcard” appear in the default media type; or
The file name extension is one of .bat, .css, .eml, .js, .json, .mime, or .swf.

Are there other media types that should be disabled for safe browsing of user-submitted content?

PGXN API RFC

2026-10-07T16:13:48Z

Things slowed up a bit over the last couple of months, I admit. There are any number of reasons for that, not the least were the intrusion of the holidays and a little side project I’ve been hacking on after-hours (and sometimes during-hours). But I’m ramping things up again now and need your feedback on my current plans. Here’s what I’m working on: the search site.

Search Sites and APIs

Well, sort of. First of all, I’ve decided that the “search site” should not be a separate thing. The main site will be the search site. This is following the example of JSAN, as well as feedback from Graham Barr, who created and maintains CPAN Search. Apparently people are often confused that search.cpan.org is separate from www.cpan.org. No point in adding in confusion from the beginning. And besides, now that the PGXN fund-raising is over, I don’t know what else would go on the home page.

The other thing that’s happened is, just as I was getting my butt in gear on this stuff, a new CPAN search site came to my attention, μετα CPAN. This is an interesting project. What they did instead of creating a monolithic HTML search site is to create a simple API that serves nothing but JSON. It has search and displays metadata for CPAN objects (distributions, maintainers, modules, etc.). The search site, then, is not really a site at all, but a pure JavaScript application. Once you load it, it just uses the API server to get all the data. There are a few tricks server-side to proxy the API server so as to avoid cross-site scripting issues. But otherwise it just works in the browser.

Now I’m not sure I’ll do the same thing, exactly, but there’s a lot of appeal in creating a RESTful API server that’s independent of the search site, and then building the search site to use it. It also has the advantage of being useful for other projects to just use. Want to create a PGXN search widget for your blog? Yeah, there’s an API for that.

A Super RESTful Directory

Of course, thanks to the “RESTful Directory” design for the mirrors (described here and revised here), any mirror is a lightweight API already. There’s a lot of metadata one can get just from the static JSON files it generates. The design is flexible–but designed with a command-line client in mind. As such, many commands executed in a command-line client would likely requires multiple requests to a mirror. For example:

> install pgtap

This would request /by/extension/pgtap.json from the server. It would then parse that file and see that the latest stable version of pgTAP is in the distribution “pgTAP” at version “0.25.0”. So it would then download /dist/pgTAP-0.25.0.pgz to install.

This is great for a command-line client, but wouldn’t be so great for a search site to be responsive. Ideally, a site should send a single request to get all the data it needs for a particular page.

So here’s what I’m thinking for a PGXN API server: It will offer a superset of the functionality of any other PGXN mirror. That is, all the JSON files in a mirror will be present, but many of them will have more information than they would on other mirrors. And then, of course, there will be other URIs to offer additional API calls.

Details

So what does that look like? Let’s take the pgTAP distribution, which I released on PGXN earlier this week. To find the pgTAP distribution, one requests:

/by/dist/pgTAP.json

From that, one can see that the latest table release is 0.25.0, and so one can then request

/dist/pgTAP/pgTAP-0.25.0.json

to get all the metadata for that particular release. What I propose, to avoid the two requests, is to include the contents of the second file in the first. That would then have all the data necessary to generate the pgTAP distribution page on the PGXN site.

The API would offer similar supersets of data for the extension , owner , and tag metadata files, to have the data necessary for the design of the corresponding extension, owner and tag layouts of the site.

Additional Resources

In addition to adding metadata to the existing mirrored JSON files, there would be other resources available for request from the API server. They would include:

Extension Documentation. Each distribution may include documentation for included extensions in the doc subdirectory. These will go under the directory for a specific distribution such as /dist/pgTAP/pgTAP-0.35.0/doc/pgtap.html. The latest version of each document would also be available under /by/extension, as in /by/extension/pgtap.html. This requires that the documentation file have the same base name as the extension file itself.
Other documentation. I’d like to support arbitrary documentation, such as for included binary executables, HOWTOs, etc. The canonical copies will go under the versioned distribution URL, of course, but I’m not sure about permalinks. That might require an extension of the Meta Spec; I haven’t quite figured that out, yet.
Source code. There will be an interface to browse an unpacked copy of any distribution as plain text. This will be under /src, as in /src/pgTAP/pgTAP-0.35.0/.

Search API

Of course. This is the big one, really. I think it makes sense to have the /by URI respond to search requests. Thus, a request for

/by?q=testing

would search everything. If you only want to search a certain category of object, you’d hit the appropriate URI:

/by/dist?q=tap
/by/owner?q=clochard
/by/tag?q=test
/by/extension?q=gis

The nice thing about this is that it retains the existing entity URLs. The directory level determines which entities you get.

Your Thoughts?

So that’s my thinking on the search API. I’m going to start hacking on it in earnest tomorrow, and perhaps next week I can get a very early version out (basically just another mirror to start with).

But what do you think? Seem like a sane approach? Am I missing anything obvious or doing anything clearly stupid? Please let me know in the comments!

First Upload to PGXN

2026-10-07T16:13:48Z

Last night I deployed PGXN::Manager v0.2.1 and uploaded the first distribution, pair. If you follow that link you’ll see three files:

pair-0.1.0.json is metadata file generated by PGXN manager to describe the distribution. Most of its data is taken from the META.json included in the uploaded zip file, but a few keys, like “sha1”, are generated, and others, like “release_status” are added if they’re not in the included META.json.
pair-0.0.1.readme is a copy of the README file distributed with pair.
pair-0.1.0.pgz is the distribution zip file. That’s the file you want to download, unzip, and build and install.

Following the spec I previously wrote up, there are a number of other files that get created when a new distribution is uploaded to PGXN. For the pair extension, we got:

by/dist/pair.json, which will be updated with information for every release of the “pair” distribution.
by/extension/pair.json, which will be updated for every upload containing the “pair” extension.
by/owner/theory.json, which will be updated every time I upload a distribution.
Files for every tag listed in the metadata are also created. In this case, that includes:
Each of these files will be updated every time a distribution is uploaded containing the relevant tag.

You’ll soon be able to upload your own extension distributions to PGXN. If you’re interested, please subscribe to the mail list, where I’ll soon be inviting folks to get an account and start uploading.

But first, a blog post on how to create a PGXN-friendly distribution archive. Coming up shortly.

A RESTful Directory

2026-10-07T16:13:48Z

Following my post outlining a possible network directory structure, Aristotle Pagaltzis saw fit to bug me via email about a different approach. I couldn’t understand WTF he was talking about until today. Then it lit my brain on fire. As a result, I now think that there is a much better way to organize the metadata files for the PGXN – one that happens not to include any symbolic links (which is something that Andreas König has been flagging, via email, as a possible bottleneck).

First, the /dist directory will be the same as before. Releases of pgTAP would be in:

dist/p/pg/pgtap/pgtap-0.23.pgz
dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.23.readme
dist/p/pg/pgtap/pgtap-0.24.pgz
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.24.readme
dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme

The only change is that the pgtap.json symlink is gone.

Now, the new stuff. In the root directory will be a file, index.json, that contains templates for URIs. It will look something like this:

{
    "dist":   "/dist/$a/$ab/$dist/$dist-$version.pgz",
    "readme": "/dist/$a/$ab/$dist/$dist-$version.readme",
    "meta":   "/dist/$a/$ab/$dist/$dist-$version.json",

    "by-dist":      "/by/dist/$a/$ab/$dist.json",
    "by-extension": "/by/extension/$a/$ab/$extension.json",
    "by-owner":     "/by/owner/$a/$ab/$owner.json",
    "by-manager":   "/by/manager/$a/$ab/$manager.json",
}

The PGXN client will always fetch this file before it does anything else, because the file tells it how to find stuff. The advantage here is that the client doesn’t have to know anything about how the directory is actually organized, just what the template variables might be. They are:

$dist: A distribution name
$version: A version number
$extension: An extension name
$owner: An owner’s name
$manager: A release manager’s name (managers are the people who upload distributions to PGXN)
$a: The first letter of a distribution, extension, owner, or manager name.
$ab: The first two letters of a distribution, extension, owner, or manager name.

I’m not thrilled about using prefix-staggering to avoid having too many files in a directory. But the truth is that this approach allows me to punt. I could also make sure the client supports, for example, $bc and $cd, so that one could stagger things differently. And then the nice thing is that I don’t have to use those at all. The templates will tell the client exactly how to construct the URIs for things, and the templates needn’t include those staggering variables if they’re not appropriate. The client won’t care because it will have no built-in knowledge of how things are organized. It will have to find out from index.json.

From the URI templates, you can now see where the other metadata will be stored. For extension names, a hypothetical pgTAP distribution with two extensions will have a JSON file for each extension:

/by/extension/p/pg/pgtap.json
/by/extension/s/sc/schematap.json

The pgtap.json file will look something like this:

"stable":   "0.25.0",
"testing":  "0.26.0b1",
"unstable": "0.30.0u",
"versions": {
    "0.26.0b1": { "dist": "pgtap", "version": "0.26.0b1", "status": "testing" },
    "0.30.0u":  { "dist": "pgtap", "version": "0.30.0u",  "status": "unstable" },
    "0.25.0":   { "dist": "pgtap", "version": "0.25.0",   "status": "stable" },
    "0.24.0":   { "dist": "pgtap", "version": "0.24.0",   "status": "stable" },
    "0.25.0":   { "dist": "pgtap", "version": "0.23.0",   "status": "stable"  }
}

Right at the top, it would always list the most recent stable, testing, and unstable version number, and then it would have a list metadata for all versions. Said metadata would include the associated distribution name, version, and release status.

Here’s how it would work. Say I ask the client to install pgtap:

PGXN> install extension pgtap

The client would first fetch /index.json, then look for the URI template for “by-extension”, which is /by/extension/$a/$ab/$extension.json. Filling in the template, it would know to request /by/extension/p/pg/pgtap.json. With that file, it would see that the most recent stable version is in the “pgtap” distribution version 0.25.0. Using the dist URI template, which is /dist/$a/$ab/$dist-$version.pgz, it would then fetch /dist/p/pg/pgtap/pgtap-0.25.0.pgz.

The advantage here is that there are no symbolic links and no knowledge of the directory structure built into clients. The client just knows to fetch /index.json and then to use the templates in that file to fetch other information. That’s the whole interface. Very RESTful.

The structure of the other /by files would be similar. For

PGXN> install dist pgtap

the client would use the “by-dist” URI template to construct the URL /by/dist/p/pg/pgtap.json. That file would have something like:

"stable":   "0.25.0",
"testing":  "0.26.0b1",
"unstable": "0.30.0u",
"versions": {
    "0.26.0b1": "testing",
    "0.30.0u":  "unstable",
    "0.25.0":   "stable",
    "0.24.0":   "stable" ,
    "0.23.0":   "stable"
}

So then the client would know that “0.25.0” was the most recent version, and use the dist URI template to request /dist/p/pg/pgtap/pgtap-0.25.0.pgz.

If The client command had been:

PGXN> readme dist pgtap

It would use the readme URI template. And the command:

PGXN> meta dist pgtap

Would use the meta URI template to fetch the metadata for the distribution.

If the client had requested a specific version:

PGXN> install dist 0.23.0

It could either use the by-dist URI template to download the list of all versions to see if 0.23.0 was valid, or just use the dist URI template to try to download the distribution itself.

And finally, the owner and manager JSON files, such as

/owner/t/th/theory.json

Would look something like:

"full_name": "David Wheeler",
"email": "theory@pgxn.org",
"uri": "https://justatheory.com",
"distributions": {
    "pgtap": [ "0.25.0", "0.24.0", "0.23.0" ]
    "pair": [ "0.2.0", "0.1.0", "0.0.5" ]
}

With that, the client can be asked to fetch metadata for a given owner name and use it to figure out what distributions and versions the the owner, um, owns. One could then fetch the metadata, readme, or distribution file for any of those distributions and versions.

Overall, I think that this is a much better solution than I outlined before. If only I could figure out something more elegant that the prefix-staggering/hashing stuff, it would be just about perfect.

Thoughts?

RFC: The PGXN Metadata Specification

2026-10-07T16:13:48Z

I’ve posted a draft of the “PGXN distribution metadata specification,” or PGXN Meta Spec." This document specifies the structure and format of the META.json file that PGXN will require in every distribution. In fact, this is the only required file in a distribution. Its job is to describe the distribution, its extensions, and its dependencies, among other things. This file is key to the whole thing.

To create it, I’ve ported the CPAN Meta Spec, version 2, but with all deprecated fields removed, and some of the more complex stuff taken out. I also made a couple of the “required” fields “optional.” At its simplest, the file might look something like this:

{
  "name": "pgTAP",
  "abstract": "Unit testing for PostgreSQL",
  "version": "0.25.0",
  "owner": "David E. Wheeler ",
  "license": "postgresql",
  "meta-spec": {
    "version": "1.0.0",
    "url": "https://github.com/theory/pgxn/wiki/PGXN-Meta-Spec"
  }
}

Not too bad, eh? The URL for the spec might change (might move it to the main site and/or the mirrors), but otherwise, I think this is pretty solid. Not too much work to deal with, and reasonably easy to create by hand (which is likely how we’ll all start out).

And additional key that will be really important for the PGXN client is prereqs. This key allows you to identify the prerequisites (from PGXN or the PostgreSQL core contrib extensions) required to build, test, and/or use a distribution. For example, if I were to release an ordered pair extension, it of course would include tests written with pgTAP. So I’d have something like:

{
  "name": "pair",
  "abstract": "An ordered pair data type",
  "version": "0.1.0",
  "owner": "David E. Wheeler ",
  license: "postgresql",
  "meta-spec": {
    "version": "1.0.0",
    "url": "https://github.com/theory/pgxn/wiki/PGXN-Meta-Spec"
  },
  "prereqs": {
    "runtime": {
      "requires": {
        "PostgreSQL": "8.0.0"
      },
      "recommends": {
        "PostgreSQL": "8.4.0"
      }
    },
    "test": {
      "requires": {
        "pgTAP": 0
      }
    }
  }
}

That’s saying that the “pair” distribution requires PostgreSQL 8.0.0 or higher and any version of pgTAP to run the test suite. I’ve also recommended PostgreSQL 8.4, as that’s where it will run best.

Of course, to get the real power of PGXN, you’ll also want to use the provides key, which allows you to identify the extensions included in your distribution. Say that I finally got around to breaking out the schema testing assertions from the logical testing assertions in pgTAP. I might call the second module “schematap.” So to spell it out, I’d add this to the first example above:

  "pgtap": {
    "file": "sql/pgtap.sql.in",
    "version": "0.25.0"
  },
  "schematap": {
    "file": "sql/schematap.sql.in"
  }

So now the indexer will know that the “pgtap” extension is in sql/pgtap.sql.in and the “schematap” extension is in sql/schematap.sql.in. This is important because it allows other distributions to specify “schematap” as a prerequisite. It also means that, in the PGXN client, you can type something like:

PGXN> install schematap

And, because “schematap” will have been indexed on the network, the client will be able to find the pgTAP distribution and install it, complete with the “schematap” extension.

A final note. Version numbers in the Perl community are a disaster. Wanting to avoid that whole morass, I had originally intended to require numeric version numbers. But David Golden – the current maintainer of the CPAN Meta Spec – pointed me to Semantic Versioning, a version number specification by GitHub’s Tom Preston-Werner. This style of version numbering is great for PGXN for a few reasons:

It closely matches how PostgreSQL itself is versioned.
It’s very easy to compare version strings.
Someone else has already dealt with the pain of writing a spec

So this is the standard that PGXN will require. Every version number will be dotted-integer with three integers (X.Y.Z) and an optional ASCII string at the end. That’s it. PGXN won’t invest any special meaning in the version string the way CPAN does. It will just compare version numbers.

Anyway, please review the spec itself and leave any comments or questions below. I expect to start hacking on this stuff this week!

Update 2010-08-24: I’ve just updated the spec to change “owner” to “maintainer.” I think that the latter term is much better for this purpose. And then I can use “owner” in PGXN to identify the person who uploads a distribution.

Thoughts on the Network Directory Structure

2026-10-07T16:13:48Z

I’ve been thinking about the arrangement of stuff to be distributed to the mirrors. In doing so, I’ve kept three goals in mind:

Allow things to scale.
Try to put things in intuitive locations.
Allow for metadata to be stored to act as a Web API.

The first two are a bit mutually-contradicting. Thinking scalability mainly means minimizing the chances that too many files can be put into a single directory. CPAN started out with all author directories in a single directory. Given the sheer number of CPAN contributors, this quickly got to be a bottleneck. To correct for that, they started “hashing” the first two letters of author names to create subdirectories. My CPAN directory, for example, is D/DW/DWHEELER. That doesn’t quite make for an intuitive location, but it’s not bad, and the tradeoff seems sufficient to keep things sane.

The storage of metadata is important, too, as I plan to have the client send requests for JSON files to mirrors in order to find distributions. Basically, this came down to a naming convention, as well as a recipe for how to find metadata files for people and extensions. What I’ve come up with is three directories:

meta will contain PGXN metadata
dist will contain distributions
by will contain query-able JSON files

The meta directory will contain PGXN metadata (mirrors.json, timestamp, other detritus). If I ever create an index file that lists all distributions, it would go there, too. I’m not going to discuss it any further here, except to note that it already has mirrors.json, which clients will be able to use to present users with a list of mirrors to choose from.

The dist directory will be organized with directories named with “hashes” of the first two letters of distribution names. Let’s say that I’m releasing pgTAP 0.25 on PGXN. To distribute it, the management application will create the directory (if it doesn’t already exist):

dist/p/pg/pgtap/

Then, for the 0.25 release, it will add three files to that directory:

dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme

The pgtap-0.25.pgz file will contain the zipped distribution ready for download. pgtap-0.25.json will contain metadata about the distribution, such as the owner’s name, the manager’s name (more on these folks below), list of included extensions, location of the .pgz and .readme, and its SHA1. pgtap-0.25.readme will of course contain the README for the distribution (if it has one).

Every release of pgTAP will have these three files, so after several releases, the pgtap directory might have these files:

dist/p/pg/pgtap/pgtap-0.23.pgz
dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.23.readme
dist/p/pg/pgtap/pgtap-0.24.pgz
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.24.readme
dist/p/pg/pgtap/pgtap-0.25.pgz
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap-0.25.readme
dist/p/pg/pgtap/pgtap.json

The last file there, pgtap.json, will actually be a symlink to the JSON file for latest production release of pgTAP. In this case, it would link to pgtap-0.25.json. The nice thing about this is that, if a client wants to find the information about the latest release of the pgtap distribution, all it will have to do is send an HTTP GET request for dist/p/pg/pgtap/pgtap.json to any mirror.

The by directory will also contain JSON files for clients to request. The idea is that you want to find information “by” something. To start with, there will be three subdirectories:

by/extension/
by/manager/
by/owner/

The first directory, by/extension/, will contain links to JSON files for extensions. Say that the pgTAP distribution offers two extensions to PostgreSQL named “pgtap” and “schematap”. The links would be:

by/extension/p/pg/pgtap/pgtap-0.23.json
by/extension/p/pg/pgtap/pgtap-0.24.json
by/extension/p/pg/pgtap/pgtap-0.25.json
by/extension/p/pg/pgtap/pgtap.json

Each of these will simply be symlinks pointing to the appropriate distribution files:

dist/p/pg/pgtap/pgtap-0.23.json
dist/p/pg/pgtap/pgtap-0.24.json
dist/p/pg/pgtap/pgtap-0.25.json
dist/p/pg/pgtap/pgtap.json

Yes, the last one is a symlink to a symlink. Similarly, these files for “schematap”:

by/extension/s/sc/schematap/schematap-0.23.json
by/extension/s/sc/schematap/schematap-0.24.json
by/extension/s/sc/schematap/schematap-0.25.json
by/extension/s/sc/schematap/schematap.json

Point to exactly the same files. Essentially, this is a way for extensions to point to the distributions that contain them. by/extension/s/sc/schematap/schematap.json points to dist/p/pg/pgtap/pgtap.json, which contains path to the .pgz file to download (and lots of other metadata, too).

The idea is that, whatever the name of the extension you want, the client will be able to easily find the metadata file that tells it where to find the distribution.

The by/manager and by/owner directories, on the other hand, contain JSON files with information about managers and owners and their distributions. Definitions:

An “owner” is someone who owns a distribution. This will often be the original author of an extension, but may be someone else if maintenance has been passed on. Basically, the “owner” is the person who should be contacted with bug reports and the like

A “manager” is someone who manages the release process. This is the user who will log into the PGXN management application and upload a distribution for release.

These two people will often be the same person, but not always. I’ve avoided the term “author” (the term used by CPAN) because the author of an extension may no longer maintain it. “Owner” seemed like a better choice (individual distributions are free to describe their contributors however they wish).

As the owner of a few extensions on PGXN, I’d have this file:

by/owner/t/th/theory.json

This file would contain a list of my distributions and perhaps some other information (like my full name and blog URL).

As the manager of extensions on PGXN (that is, I actually uploaded them), I would also have:

by/manager/t/th/theory.json

This file would contain a list of the distributions I’ve released on PGXN. This might be exactly the same as the list in my owner file, but may not be. Perhaps for one release of pgTAP, say 0.26, Duke Leto uploaded a release. In that case, 0.26 would probably be in my owner file, but not in my manager file: it would be in Duke’s manager file, instead.

So these are the basics of the directory structure for the networked mirrors. Note that I haven’t thought much about everything that will go into the JSON files (or whether or not they’d be versioned). That will likely depend quite a lot on what the management database ends up looking like. I’ll be working on that next.

But other than that, comments? Questions? Criticisms? Recommendations? Leave a comment and let me know!