RJBS Advent Calendar 2009

One Last Present, All Boxed Up

2009-12-26T00:00:00-05:00

It's Boxing Day!

Boxing day, the day after Christmas, is so called because it is the time of year when you take all the gifts you received and liked and put them in a box to keep. The rest you give to someone who couldn't be bothered to show up for Christmas with the rest of the family. It is also known as St. Stephen's day, in honor of my brother.

Perl! Perl is in the box!

I maintain a box of my favorite presents from the CPAN. Some of them are my own donations, already mentioned this month. Plenty of them were shared with the CPAN by other contributors. This box is Task::BeLike::RJBS. It's a Task distribution, which means that it doesn't actually do anything on its own. It's just a pile of prerequisites. Instead of installing each one, you can install the task and the rest are installed for you.

When I set up a new workstation, the first thing I do is compile my own recent perl to avoid any weird vendor packaging issues. The next thing I do is install Task::BeLike::RJBS. Whether or not you want to be like me, I'm stuck with it, so I try to do it as efficiently as possible.

If you're interested in the modules that I consider essential to getting work done, check out the task and its contents.

Building Your Own Box

Task::BeLike::RJBS is built with Dist::Zilla -- specifically with Dist::Zilla::Plugin::TaskWeaver, a plugin for building tasks. To use TaskWeaver, you just write a module with some Pod like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	`=pkgroup Simple Modules (Wednesdays) =pkg Number::Nary 0.108 has robust unique-decodability detection =pkg Data::Section =pkg String::RewritePrefix 0.005 provides "rewrite" export =pkg Sort::ByExample`

You'll get output Pod like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15: `	`=head1 TASK CONTENTS =head2 Simple Modules (Wednesdays) =head3 Number::Nary 0.108 Version 1.08 is required because: has robust unique-decodability detection =head3 Data Section =head3 String::RewritePrefix 0.005 Version 0.005 is required because: provides "rewrite" export =head3 Sort::ByExample`

You'll also get all those packages, with the versions given (if any) listed as prerequisites, so installing your task will get all those prerequisites as well.

Have fun, make some bundles, share, and enjoy.

You Can Keep the Calendar

2009-12-25T00:00:00-05:00

Merry Christmas! Advent is over.

It's a common misconception that Advent calendars include today, Christmas Day, but traditionally they stop yesterday on the 24th. Still, who am I to argue with the Perl Advent Calendar traditions? Here's my compromise: today, I'll just give you the calendar itself: WWW::AdventCalendar.

In which our author fails to steal from his peers...

On November 18th, I decided it might be fun to write a series of Advent Calendar articles for just my own code, published along side the other calendars. I asked one or two people if they thought this was a terrible idea, and they said it was not, so I decided I'd better get to work as quickly as possible. That means not worrying about writing any software to produce the article, but just to write articles. I would steal someone else's code!

First, I looked for whatever powered the Perl Advent Calendar. I figured with new CSS it would be fine. I found Pod::Advent, but something about it made me decide to keep looking. I have no idea what anymore. Then I went to CatalystAdvent, which powers the Catalyst Advent Calendar. I liked its look out of the box, so I started to install it, but it needed things no longer on CPAN. I hacked around that and then started to wonder whether I wouldn't be happier just publishing static files. Why? Who knows.

I try pretty hard not to re-invent solutions to solved problems, and (believe it or not) I succeed more often than not in quashing my urge to rewrite perfectly good systems. Here, though, it seemed like I'd be happier with something I wrote for myself. In the end, I think I was.

Anyway, I mostly stole the CSS and templates from CatalystAdvent, and I'm pretty happy that, too.

WWW::AdventCalendar

WWW::AdventCalendar is, as you might expect, pretty stupidly simple. Right now it only works for Advent, although I'd like to make it better at doing longer spans of time. Could I use a "real" blogging system for this? Yes. Am I likely to? No.

There's just a program to run that looks for your articles, decides how many to publish, and writes out the article HTML, the index, the Atom feed, and a few other files.

  ~/code/advent$ ./bin/advcal -c advcal.ini --today 2009-12-08
  processing article for 2009-12-01...
  processing article for 2009-12-02...
  processing article for 2009-12-03...
  processing article for 2009-12-04...
  processing article for 2009-12-05...
  processing article for 2009-12-06...
  processing article for 2009-12-07...
  processing article for 2009-12-08...

The input files are Pod documents where the non-pod content is an email document. Alternately, the input files are email documents where the body is Pod. What?? Well:

`1:  2:  3:  4:  5:  6:  7: `	`Title: You can take the calendar with you when you go... Package: WWW::AdventCalendar =head1 Merry Christmas! Advent is over. It's a common misconception that Advent calendars include today, Christmas Day,`

I originally made the body markup type pluggable, but then I remembered how much I love Pod and I dropped that feature. This let me use one set of syntax semantics for everything. For example, I could write:

`1:  2:  3:  4:  5:  6:  7:  8: `	`=begin vim lisp (format nil "~4,'0d-~2,'0d-~2,'0d" 2005 6 10) ; ==> 2005-06-10 (format nil "~:(~a~)" "tHe Quick BROWN foX") ; ==> "The Quick Brown Fox" (format nil "~:r" 1234) ; ==> "one thousand two hundred thirty-fourth" (format nil "~:@r" 1234) ; ==> "MCCXXXIIII" =end vim`

...and get some nicely colorized Lisp code. (The color scheme, by the way, is based on my Vim colorscheme. The code uses Vim (the editor) to colorize my code, but for Perl I decided to use PPI::HTML, which was faster and had a bit more control. I could just write =begin perl but that got tedious, so I added another transformation so that by writing...

`1:  2:  3:  4: `	`but that got tedious, so I added another transformation so that by writing...   #!perl   my $x = { foo => 10 };`

The indented block's shebang would be noticed, and instead I'd get:

`1: `	`my $x = { foo => 10 };`

The Pod Transformers

Syntax Highlighting

So, those bits of code are the useful bits, unless you really want to run an advent calendar just like me. For now they're being released with WWW::AdventCalendar, because they're just a bit too hacky to be for general use yet, I think. Still, here's what they do:

I'm using Pod::Simple::XHTML to convert from Pod to HTML. I'd started with Pod::Xhtml, but it was buggy and unmaintained. Pod::Simple, on the other hand, is buggy and maintained. Or, at least, it's maintained. It can be hard to tell whether it's buggy or just incomprehensible. Anyway, I can have a =begin html block that lets me provide literal XHTML rather than Pod. This is perfect for providing marked up, colorized code blocks. All I had to do was get those XHTML blocks produced before Pod::Simple::XHTML did its thing.

Obviously, this was a job for Pod::Elemental, the engine behind Pod::Weaver! I wrote two libraries, Pod::Elemental::Transformer::PPIHTML and Pod::Elemental::Transformer::VimHTML, both of which composed the Pod::Elemental::Transformer::SynHi role. All that left for the transformers to do was have a method that rewrote plain text into marked up HTML.

I'm hoping that these will make it easy to write other syntax highlighting engines for Pod-to-HTML conversion in the future -- possibly engines that don't require spawning Vim instances!

Shorthand Lists

I mentioned using Pod::WikiDoc syntax for lists when talking about Pod::Weaver. WikiDoc regions are really powerful, and make writing lists really easy, but they do much more than lists and I didn't want them to! For example, look at this region:

`1:  2:  3: `	`=for wikidoc * L<Acme::ProgressBar> * L<Games::Nintendo::Mario> - it's I<super>`

It doesn't do what I want, because it doesn't expect formatting codes (things like L<...>. What I have to write is:

`1:  2:  3: `	`=for wikidoc * [Acme::ProgressBar] * [Games::Nintendo::Mario] - it's ~super~`

This isn't so bad, but it's not what I'm using in the rest of the document! To get lists that were as easy to write as those in Pod::WikiDoc, but still Pod, I wrote Pod::Elemental::Transformer::List, which lets me write

`1:  2:  3: `	`=for :list * L<Acme::ProgressBar> * L<Games::Nintendo::Mario> - it's I<super>`

The colon in :list says that the contents of the region are really Pod (even though they're re-interpreted into more Pod by the transformer. It's easy to write nested lists, too:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17: `	`=begin :list * glossary These words are useful to know: =for :list = advent the four weeks leading up to Christmas = noel the two weeks beginning on Christmas eve * glossy These words mean shiny: ... =end list`

Same Code Next Year

I'll definitely be using this code again next year... if I decide to try to repeat this undertaking, which is a big if. I'm hoping to make the syntax stuff a useful tool for other projects, and I really want to make the advent calendar work across arbitrary date ranges. Still, it was fun to write, easy to use, and I hope somebody else gets some kind of benefit from it in the future.

Until then... Merry Christmas!

Sending Email Simply

2009-12-24T00:00:00-05:00

The Biggest Box Under the Tree

Of everything I did in 2009, I think Email::Sender is the most important and the one I most hope to see adopted. For better or worse, I often found myself looked at as "that guy who does the email code," and people ask me what library they should use for this or that. I end up shrugging, most of the time, and saying, "What I look for in an email module is not the same as what you will care about. I'm a weirdo. I have very peculiar needs."

Email::Sender is not like that. If you are going to send email from any code that matters, you should use Email::Sender. Let's see why.

Sending Email Simply

`1:  2:  3: `	`use Email::Sender::Simple qw(sendmail); sendmail($email);`

That's it. For most values that you're likely to throw into $email, that will work. Either it will be delivered for all recipients, or it will throw an exception. It will decide whether to use sendmail or SMTP based on your operating system and the availability of the sendmail program. It will decide on the envelope sender and recipient based on the message headers. If you don't know what an envelope is, you don't have to. It will do what you expect.

If you do know what the envelope is, you know how important it is to be able to control it. It lets you manage bounces, perform BCC or blind distribution list, and do all sorts of other really important things. It's easy:

`1: `	`sendmail( $email, { to => \@recipients, from => $sender } );`

The success value returned or the failure exception thrown both can be inspected for more information, but that's rarely useful for common cases; you just want to know whether it worked or not.

Specifying a Transport

Normally, Email::Sender::Simple can pick what transport to use for you. You can always specify one by hand, though.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15: `	`my $transport = Email::Sender::Transport::SMTP->new({   host => 'smtp.pobox.com',   ssl => 1,   sasl_username => $username,   sasl_password => $password, }); sendmail(   $email,   {     from => $sender,     to => \@recipients,     transport => $transport,   }, );`

Now we'll send that message through the configured SMTP relay. This can be useful for unusual cases, but the same kind of feature can be vastly more useful in testing your code. You can override the transport with the environment (%ENV), and that will affect every use of Email::Sender::Simple, whether or not an override is given in the call to sendmail itself. That means you can very easily test every email your program will send like this:

`1:  2:  3:  4:  5:  6:  7: `	`$ENV{EMAIL_SENDER_TRANSPORT} = 'Test'; test_app(MyAppCmd => [ sendreports => @args ]); my @deliveries = Email::Sender::Simple->default_transport->deliveries; # ...and now we inspect the mail we would have sent out...`

Setting that environment variable behaves as if you'd created a single instance of Email::Sender::Transport::Test and passed it as the transport parameter for every call to sendmail everywhere.

This works very well for testing things. I have many times turned up unexpected emails from some deep subsystem by using this. Sometimes they've been reports of encountered but handled exceptions going to the admin staff -- so by consistently testing all the email we'd send, I could find all kinds of weird problems.

This is great for those tests, but lots of our tests fork. Once our test process forks, storing all its deliveries in what is effectively a global variable, scoped per process, isn't very useful. Instead, we test like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	`$ENV{EMAIL_SENDER_TRANSPORT} = 'SQLite'; test_app(MyAppCmd => [ sendreports => @args ]); # here there be forks my $rows = Email::Sender::Simple->default_transport->dbh->selectall_arrayref( # ... ); # ...and now we inspect the mail we would have sent out...`

We've trivially collected all the mail sent by any forked child -- and by any spawned subprocess that uses Email::Sender.

More Testing Tools

Want to run your program and look at the results yourself? You can set the environment to send all the output mail to your mailbox. You can send it all to an mbox or Maildir on disk. You can route it all to /dev/null or to the screen. These things are all easy. Really, they're trivial. You just set the EMAIL_SENDER_TRANSPORT environment variable and exactly what you want happens.

What's less trivial -- but immensely powerful for testing -- is the Failable transport wrapper.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18: `	my $sender = Email::Sender::Transport::Test->new; my $failer = Email::Sender::Transport::Failable->new({ transport => $sender }); my $i = 0; $failer->fail_if(sub {   return "failing half of all mail to test" if $i++ % 2;   return; }); {   my $result = eval { $failer->send($message, { to => [ qw(ok@ok.ok) ] }) };   ok($result, 'success'); } {   eval { my $result = $failer->send($message, { to => [ qw(ok@ok.ok) ] }) };   isa_ok($@, 'Email::Sender::Failure', "we died"); }

See what we did there? We told our program to send mail through a transport that is guaranteed to fail every other delivery. We could also have failed on deliveries to specific domains, over a given size, or almost anything else. This makes it easy to test how our program behaves when its email-sending routines fail, which is an often ignored code branch -- because it's traditionally been so hard to test. Making all your mail go through that kind of transport is easy: you throw the failure conditions in a package and set your environment variables accordingly.

Seriously, Use It

Because of the global effects of the testing environment settings (and other related tools not seen here), the more code that uses Email::Sender::Simple, the more testable everything becomes. It's easy to extend to new kinds of transports and messages. Even if you don't plan on testing your code's email sending, by using Email::Sender::Simple, you'll make it possible for anyone else to do so second-hand. That can be your gift to them: an application that's easier to maintain and test.

Sorting Automatically, By Hand

2009-12-23T00:00:00-05:00

Here's a nice easy one: sorting stuff by providing a pre-sorted set of data.

The problem with custom sort functions is that sometimes it's stupidly easy to explain how you want to sort something to a person, but once you try to explain it to the computer, it becomes a real chore. Here's an example: we've got a bunch of presents to delivery, and usually we deliver them by the destination's latitude, northerly destinations first, but we have a few VIP users who always get first service. Their presents get delivered first, then everybody else's.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13: `	use Sort::ByExample qw(sbe); my $sorter = sbe(   [ 'Klortho the Magnificent', 'Mrs. Claus', 'Cold Miser' ],   {     xform => sub { $_[0]->recipient },     fallback => sub { $_[3]->degrees_north <=> $_[2]->degrees_north                    \|\| $_[2]->submission <=> $_[3]->submission     },   } ); my @jobs = $sorter->( @presents );

Now all deliveries will be sorted by destination, but presents for our three VIPs will be pulled out of order: first, all of Klortho's gifts (by submission time), then Mrs. Claus's, then Cold Miser's. The xform callback is used to perform a Schwartzian transform, so that the sort routines will see the recipient's name rather than the present object. fallback provides a callback to sort items not sorted by the example, or that have the same position in the example. Note that the fallback routine looks at the third and fourth elements rather than the first two. These give it access to the value before transformation.

Finally, you can mark items as "ties" in the example sort by using a hashref as the example, with the values indicating rank. By marking two as having the same value, they'll get send to the fallback to break the tie:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14: `	my $sorter = sbe(   {     'Klortho the Magnificent' => 1,     'Mrs. Claus' => 2,     'Cold Miser' => 3,     'Heat Miser' => 3,   },   {     xform => sub { $_[0]->recipient },     fallback => sub { $_[3]->degrees_north <=> $_[2]->degrees_north                    \|\| $_[2]->submission <=> $_[3]->submission     },   } );

Sort::ByExample is not intended to be blazing fast, but it should be fairly efficient. It also isn't the Swiss Army chainsaw of sorting. I tried to use that at first before giving up and writing this code.

Sort::ByExample solves a pretty specific problem, and one that could always be solved by writing just a few more lines of Perl. Still, not having to write those lines is a pretty nice perk that leaves more time for solving more pressing problems.

Object Marginalia

2009-12-22T00:00:00-05:00

Scribbling Notes on an Object

Mixin::ExtraFields is hard to summarize, but here's a whack at it: it's a system for storing extra data related to an object and creating methods for accessing and altering that state.

Here's a really simple example to start with. We have an object that's a blessed hashref and we want to have a bucket for arbitrary state:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10: `	`package Holiday; use Mixin::ExtraFields -fields => {   moniker => 'tradition',   driver => 'HashGuts', }; sub new { ... } sub id { ... }`

Now we've got this bucket for extra values in our Holiday objects which we can use in a few ways, like:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16: `	my $holiday = Holiday->new('Christmas'); $holiday->set_tradition(colors => [ qw(red green) ]); $holiday->set_tradition(music => 'Unbearable'); $holiday->set_tradition(icon => 'Santa Claus'); # ... die "nothing to eat!" unless $holiday->exists_tradition('food'); for my $thing ($holiday->get_all_tradition_names) {   print stringf "The traditional %s for %i is %r.\n",     $thing,     $holiday->id,     $holiday->get_tradition($thing); }

If we use Data::Dumper on our object, we might see something like this:

`1:  2:  3:  4:  5:  6: `	`$VAR1 = bless( {   id => 'Christmas',   'Mixin::ExtraFields::Driver::HashGuts=HASH(0x100864e18)@0' => {     foo => 10,   }, }, 'Holiday');`

That horrible key is generated as a place in our object to put a reference to its associated data. That's only one reference, though. We keep track of data for that object externally, too. That means if we create another object with the same id, it will have the same traditions.

This means we can also have objects share their extra fields:

`1:  2:  3:  4:  5: `	`package Holiday; use Mixin::ExtraFields   -fields => { moniker => 'tradition', driver => 'HashGuts' },   -fields => { moniker => 'month_info', driver => 'HashGuts', id => 'month' };`

Then we can say:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10: `	`my $christmas = Holiday->new({ id => 'Christmas', month => 'Dec' }); my $kwanzaa = Holiday->new({ id => 'Kwanzaa', month => 'Dec' }); $christmas->set_tradition(color => 'green'); $christmas->get_tradition('color'); # ==> green $kwanzaa->get_tradition('color'); # ==> undef $christmas->set_month_info(length => 31); $christmas->get_month_info('length'); # ==> 31 $kwanzaa->get_month_info('length'); # ==> 31`

This can be a quick way to add state to a class you don't control for the sake of debugging:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13: `	use Weird::Object; use Mixin::ExtraFields   { into => 'Weird::Object' }, # add methods to that class, not this one   -fields => {     -prefix => '__', # make our methods very private     driver => 'HashGuts',     id => undef, # they have no id-like method; use the refaddr     moniker => 'debug',   }; my $object = Weird::Object->new( ... ); $object->__set_debug(status => 'brand new');

Alternate Sets of Methods

It's easy to provide methods other than the get_extra and set_extra (and the half dozen related methods) that we saw above. For example, you could use Mixin::ExtraFields::Param instead and get a method like CGI.pm's param:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15: `	`package Holiday; use Mixin::ExtraFields::Param -fields => {   moniker => 'tradition',   driver => 'HashGuts', }; # ... $holiday->tradition(   color => [ qw(red green) ],   food => 'sugarplum',   tree => 'artificial', ); $holiday->tradition('food'); # ==> sugarplum`

Or, my favorite, Mixin::ExtraFields::Hive to get a Data::Hive-based hierarchical-looking set of methods. The hive makes it look like you have a deep storage structure, but flattens everything out to key/value pairs for storage. It lets us do tricks like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15: `	`package Holiday; use Mixin::ExtraFields::Hive -hive => {   moniker => 'tradition',   driver => 'HashGuts', }; # ... $holiday->tradition->floral->tree->decoration->SET('tinsel'); $holiday->tradition->mood->malice->DELETE; if (my $phrase = $holiday->tradition->exclamation->GET) {   say $phrase; }`

The fact that Data::Hive takes this apparently structured data and stores it in a flat table is extremely useful, because it keeps to a known quantity the number of methods that alternate drivers have to provide.

Alternate Storage Drivers

Alternate storage is why we had to keep saying we wanted the HashGuts driver, above. If you wanted, you could write a driver that stored all attributes only in some otherwise inaccessible hashref. You could store attributes by calling another set of methods on the object being altered. There are lots of possibilites, but the one I've used the most is storing the table of extra values in a database table.

We can decorate objects in our program with these extra fields, storing any set values in a database and retrieving them back from the database. So we can write a class like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21: `	package Holiday; use Calendar::Schema; use Mixin::ExtraFields::Hive -hive => {   moniker => 'tradition',   driver => {     class => 'DBIC',     schema => sub { Calendar::Schema->connect(...) },     rs_moniker => 'HolidayTradition',   }, }; sub new {   my ($class, $holiday) = @_;   bless { id => $holiday } => $class; } sub id {   my ($self) = @_;   return $self->{id}; }

It's about the simplest object you could get, with just one attribute stored in a plain old hash, but then it has this "hive" mixed in that persists things in your DBIx::Class database schema. You can turn your little holiday objects into flyweights for accessing (apparently) structured data like:

`1:  2: `	`printf "Tonight we will eat %s!",   Holiday->new('Christmas')->tradition->food->dinner->entree->GET;`

...and we'll pull that out of the database from the initial load of all our favorite holiday traditions. (To keep things easy for you, the DBIC driver can actually set up the DBIC resultsource for you, so your HolidayTradition source above, need only be two or three lines of code.

Sometimes you might just want to stick your extra data into an object's guts, and sometimes you might just want to use a full database layer to map your whole object to SQL, and sometimes you might want a full meta-programming layer for adding attributes to your classes. Other times, though, you need a quick way to add data to objects and share it correctly without having to think about it too much. That's exactly when I start mixing this library in.

Gimme Everything You Got

2009-12-21T00:00:00-05:00

Priming Your New Workstation's ~/code

...or ~/src or whatever you use. Personally, I have ~/code/hub/, in which I have a clone of all the repositories that I host on GitHub. If you checked, you saw that I have a lot of repositories there. This can be a real pain when I set up a new workstation. (That doesn't happen often, but if Santa wants to bring me a brand new MacBook, I will leave out extra milk and cookies.) It's also a big pain to get all my repos up to date, which I like to do before travel, or as a backup procedure. I wanted one command that would ensure I had every repo cloned and up to date. That command is git megapull. When run, it finds all your remote repositories and, for each, checks whether you have a matching directory in cwd. If you don't, the repository is cloned. If you do, its origin is fetched and the master branch is merged. It also tells you if you have directories lying around that don't correspond to remote repositories.

There are three switches to git megapull. -c will only clone repositories; existing clones will not be updated. -b will create bare repositories when cloning, which is great for making backups.

The third switch is -s or --source. It tells megapull how to get a listing of your remote repositories. Git::Megapull comes with a source for GitHub, so you can run git megapull -s Github (or put "GitHub" in the megapull.source entry of your ~/.gitconfig) to find and clone all your repositories from GitHub using their API.

Providing Other Sources

Once I had this tool working on my laptop and a few other workstations, I wanted it at work, where we use gitosis. This was pretty easy to throw together; a source just needs to return a hashref of names and cloneable URIs.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22: `	use strict; use warnings; package Git::Megapull::Source::Pobox; use base 'Git::Megapull::Source'; sub repo_uris {   my ($self) = @_;   my @repos     = `ssh git.pobox.com 'gfind ~git/repositories -type d -mindepth 1 -maxdepth 1'`;   chomp @repos;   s/\.git//g for @repos;   s{\A.+/repositories/}{}g for @repos;   my %repo_uri;   $repo_uri{ $_ } = "git\@git.pobox.com:$_.git" for @repos;      return \%repo_uri; } 1;

...aka Config::RJBS::AndNobodyElse::KnowsWTFItDoes

2009-12-20T00:00:00-05:00

I'm not apologizing!

This is Advent, not Lent! It's a time for giving wonderful things, not for feeling stupid about the dumb things we did. I've published plenty of stupid unneeded code, and maybe someday I'll make a big list of my screwups, but for now it's not happening. I'm gonna shove this present down your throat and you're gonna love it.

Ho ho ho. Now I have chainsaws, too.

dD»»» dD»»» dD»»» dD»»» dD»»»

It's Called "Config::MVP"

There are a lot of config-reading libraries. There are hundreds of them. There are even some that I like, so what the heck was I thinking? I was thinking, "using any of those other hundreds of modules is going to suck for what I want to do." What I wanted to do, at the time, was screw around with my procmail settings, but that's another story.

I had a few major goals:

make "configuration for a plugin" also mean "load the plugin"
allow plugins to help validate and contextualize their own configuration
allow any given plugin to be loaded multiple times
force plugins to be kept in a specific order

I ended up using the INI format because it was so easy to work with, and because I'd already started down that road before formulating what I really wanted. (That's how Config::INI was born, which is also another story.) So, here's an example of all of the above features in use:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23: `	`[@CorePrep] [Name] [Version] [Region / prelude] [Generic / SYNOPSIS] [Generic / DESCRIPTION] [Generic / OVERVIEW] [Collect / ATTRIBUTES] command = attr [Collect / METHODS] command = method [Leftovers] [Region / postlude] [Authors] [Legal]`

Hey, look! It's configuration just like Pod::Weaver! Dist::Zilla and Pod::Weaver both use Config::MVP to great effect. It works like this: every block of the configuration is associated with a plugin -- we use String::RewritePrefix to expand the short form into a package name. If there's something after the slash, we use that as the plugin's name. That way we can have (for example) multiple Generic sections like we see above. We just can't have two sections named OVERVIEW.

Packages can provide methods like mvp_multivalue_args and mvp_aliases to tweak how its inputs are handled. For example, one plugin might let you provide multiple entries for author while another would not. These methods aren't required, so any package can be treated as a plugin by Config::MVP.

The Config Sequence

There are a few ways to read in MVP-style config, but as far as I know, everything serious right now uses Config::MVP::Reader::INI. Any reader, though, will return a Config::MVP::Sequence object. It's easy to deal with -- it lets you get at your plugin configuration as a list of plugins or by name. For example:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28:  29:  30:  31:  32: `	package Christmas::Todo; sub read_plugin_config {   my ($self) = @_;   my $seq = Config::MVP::Reader::Finder->new->read_config({ ... });   die "you must have a Chimney" unless $seq->section_named('Chimney');   for my $section ($seq->sections) {     my $plugin = $section->package->new( # the payload is a hashref of the actual       # vars set in the section $section->payload     );     $self->add_plugin( $section->name => $plugin );   } } # Later... sub decorate {   my ($self) = @_;   my @plugins = grep { $_->does('Christmas::Todo::Decorations') }                 $self->plugins;   for my $plugin (@plugins) {     die "error when decorating " . $plugin->name       unless $plugin->string_up and $plugin->plug_in and $plugin->turn_on;   } }

By establishing an explicit order, rather than using a hash, the user who writes the config file can always know in what order his plugins will act. By requiring unique names, we know that we can always get at any plugin by name and that we can identify sources of trouble unambiguously when things go pear shaped.

Reading Configuration

Although the INI file reader is the only serious reader right now, writing more is made trivial by using the Config::MVP::Assembler system... but let's not focus on that. INI files are fine. Still, since there might be other really great offerings, there's something like Config::Any for MVP: Config::INI::Reader::Finder, which looks for known types of MVP config files and tries to read any of them:

`1:  2:  3:  4: `	`my $sequence = Config::MVP::Reader::Finder->new->read_config({   root => "$ENV{HOME}/.christmas", # I know, I know, unix-o-centric   basename => 'config', });`

This will look for any kind of plugin it knows how, using the associated extensions. It will look for config.ini, but it could also look for config.json or config.pl if readers were written.

Config::MVP doesn't make sense when a simple hashref is really all you need, but for many more complex systems it can save lots of time and make plugins much easier to manage. I'm looking forward to hearing about other people making MVP an important asset in their applications in the coming year.

Restoring The Traditions of our Forefathers

2009-12-19T00:00:00-05:00

The War on "Finger"

I'll admit it: I am a traditionalist. I believe in honoring the protocols that were left to us by our forefathers. Maybe that makes me "old-fashioned" or "short sighted" or "terminally irrelevant," but I see no reason I should have to abandon technologies that have existed for 30 years just because they are "horrible" and "obsolete" and "full of security holes." What does HTTP really give us over Gopher? Why should I use BitTorrent instead of FTP? How is Wave any better than Ytalk?

...and then there's finger. People try to replace finger more than anything else. First "the home page" and then "MySpace" and "Classmates" and now I think the hot new thing is "Orkut." Well, the .plan file will live forever in my heart.

I wanted to provide a suitable tribute to my favorite ancient protocol, and what better way than to pair it up with one of the trendy technologies that I do enjoy. Git might only be relevant for a short while, but it's giving us all kinds of good ideas to backport to Arch.

Finger My Git

Git::Fingerd is a Net::Finger::Server-powered repository information delivery platform that allows you to easily access information about a server's git repositories from any finger-enabled operating system like Mac OS X. Here you can see the result of fingering one of my git servers:

  ~$ finger @git.codesimply.com
  [zodiac.codesimply.com]
  Repository                          Description
  Acme-Canadian                       Canooks in your code, eh?
  Acme-Lingua-EN-Inflect-Modern       modernize Lingua::EN::Inflect rule&#39;s
  Acme-ProgressBar                    a simple progress bar for the patient
  Acme-Studly                         convertBetween various_well_known Id...
  Amce-CNA                            a moer tolernat verison of mehtod lo...
  App-Addex                           generate mail tool configuration fro...
  App-Addex-AddressBook-Abook         use the &quot;abook&quot; program as the addex...
  [ ... ]

If I want to learn more about a repository, it's easy:

  ~$ finger Rx@git.codesimply.com
  [zodiac.codesimply.com]
  Project  : Rx
  Desc.    : simple, extensible schemata
  Clone URL: git://git.codesimply.com/Rx.git

  [heads]
  master          = 38633c138f2c149ccbb7434a711975173f51a6ca
  struct-fail     = 4c8449e0693c4bac3033c8f093be6bf26a54ac34

  [tags]
  0.003           = b0ecc50fed4e6e84c6350cd12d9be33de926bebb
  0.004           = e625d75cfae4f43f51e43491b2b184a8396b8d50
  perl-0.001      = 602b746868d227e0a72144e6677b0d4a14a668cd
  perl-0.002      = 4b3102379609957b0ca2202f47a7a050ad6c42ee

  [README]
  Rx is really cool, yadda yadda, whatever...

The Future!

As time goes on, we can forwardport more ancient protocols into the future, making sure that the future is just as technologically advanced as the past! Imagine using Net::GitHub and Gopher::Server to create a Gopher interface to Github! git push via UUCP, or git clone across telnet! The future will be so bright that we'll have to wear Wayfarers.

Glory to Pod in the Highest

2009-12-18T00:00:00-05:00

Glory? To Pod?

Oh, I don't know. I have really mixed feelings about Pod. It's really easy to write, most of the time, and the toolchain for it is pretty fantastic. You can convert Pod to text, man pages, HTML, LaTeX, PDF, and all kinds of other formats. Then again, there's too much vertical whitespace. Lists are a colossal pain to write. Beyond the annoyances of Pod itself, the Pod of Perl modules uploaded to the CPAN is expected to contain a bunch of boring filler that gets copied around.

A number of other documentation systems tempted me with the idea of writing less and getting information, like "this bit of documentation associates with this method," giving you things like Python's help builtin. I set about finding a way to write a lot less Pod that would tell me a lot more about my code and look indistinguishable to consumers. There were a few stops and starts and a lot of spun-off code, and a very motivating grant from The Perl Foundation. The end result was Pod::Weaver, which is something like a templating system for Pod.

Pod Templates with Pod::Weaver

Like Email::MIME::Kit, though, Pod::Weaver doesn't just take a document and fill in strings. It takes a description of the goal output and tries to build something matching that description. For example, here's roughly the default template for Pod::Weaver:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23: `	`[@CorePrep] [Name] [Version] [Region / prelude] [Generic / SYNOPSIS] [Generic / DESCRIPTION] [Generic / OVERVIEW] [Collect / ATTRIBUTES] command = attr [Collect / METHODS] command = method [Leftovers] [Region / postlude] [Authors] [Legal]`

This describes the document to be created; every section of the configuration will try to produce a section in the output Pod, except for @CorePrep, which is a plugin bundle, which could provide a bunch of sections or other kinds of plugins. In the case of CorePrep, we're adding plugins to ensure that we have an object tree that's something like what we expect from a Pod document, as Pod::Elemental starts with a very bare-bones structure that can be tedious to work with.

After that, we try to make sure there are NAME and VERSION sections, then we look for prelude region, and so on. The input document might look like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28: `	`package Tree::Christmas; # ABSTRACT: a binary tree decorated with baubles and popcorn =begin :prelude =head1 WARNING This code may cause your home to smell like pine needles. =end :prelude =head1 DESCRIPTION This library provides a framework for hanging tinsel, popcorn, and other forms of garland. In some locales, pickles may also be nested in a Tree::Christmas. =method ignite_lights This method... =method new ... =attr pickle_jar This is an arrayref of pickles.`

If we want to include things like a name, version, license, and author in the output, we'll need to provide those. We'll do that with something like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11: `	`my $input_pod = Pod::Elemental->read_string( $the_sample_above ); my $weaver = Pod::Weaver->new_with_default_config; my $output_pod = $weaver->weave_document({   authors => [ 'Ricardo Signes <rjbs@example.com>' ],   license => Software::License::Perl_5->new({ ... }),   version => '1.234',   pod_document => $input_pod,   ppi_document => $input_ppi, # (don't worry, keep reading) });`

That gets us a new Pod::Elemental::Document which, when turned into a Pod stirng, looks something like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28:  29:  30:  31:  32:  33:  34:  35:  36:  37:  38:  39:  40:  41:  42:  43: `	=head1 NAME Tree::Christmas - a binary tree decorated with baubles and popcorn =head1 VERSION version 1.234 =head1 WARNING This code may cause your home to smell like pine needles. =head1 DESCRIPTION This library provides a framework for hanging tinsel, popcorn, and other forms of garland. In some locales, pickles may also be nested in a Tree::Christmas. =head1 ATTRIBUTES =head2 pickle_jar This is an arrayref of pickles. =head1 METHODS =head2 ignite_lights This method... =head2 new ... =head1 AUTHORS Ricardo Signes <rjbs@example.com> =head1 COPYRIGHT AND LICENSE This document is copyright (C) 1991, Ricardo Signes. This document is available under the blah blah blah.

Not bad! We avoid typing nearly half of the output document, by lines -- and that's only part of the benefit. If we have five modules in our distribution, all that boilerplate is added everywhere for us, so multiply that savings across all our modules. Then, if ever we want to change our license or add an author or even change the overall layout of the output document, we only need to change our configuration in one place.

If you're wondering about the ppi_document input above, it's there because some sections can be built by analyzing the Perl content of the document, so you can automatically document things like included packages, defined subroutines, and so on. (You could also write something to compile the code and look at the symbol table or meta layer, although no one has done that yet!) Because "load a .pm file, break it into Pod and Perl, rewrite the Pod, and write it back out" is such a common pattern, there's even Pod::Elemental::PerlMunger, which takes care of everything but the "rewrite the Pod" step for you, leaving you free to focus on configuring your weaver or other rewriting tool.

That's the interface on which Dist::Zila::Plugin::PodWeaver is built. It gets all its inputs from your Dist::Zilla configuration and then rewrites all your Perl modules with Pod::Weaver. If you're using Dist::Zilla, all you need to do to get all the rewriting seen above is add the line [PodWeaver] into your dist.ini.

Extending Pod::Weaver

Pod::Weaver tries to make it easy to add new behavior. For example, a simple plugin to add an "Acknowledgements" section is only about a dozen lines of code. A plugin to integrate Pod::Weaver with Pod::WikiDoc is only about six lines of code. This allows the transformation of the easy to type:

`1:  2:  3:  4: `	`=for wikidoc * bell * book * candle`

...into the grumble-inducing:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15: `	`=over 2 =item * bell =item * book =item * candle =back`

A recurring theme of this set of articles has been my hatred for needless typing and boilerplate content, and Pod::Weaver has gone a long way to letting me avoid both. Hopefully you can use it to avoid them now, too, and spend more time solving problems.

Rewrite Email Without Worrying

2009-12-17T00:00:00-05:00

Encoding and Email

Talking about encoding is sort of tiresome, and I think everybody has heard enough of it. Talking is about email is pretty tiresome, too, and I know I have had enough. So, here's a very, very brief overview of why the two things are annoying together.

There are many systems for picking a set of symbols and assigning them numerical values. These are character sets. Unicode is the current favorite. In unicode, for example, the character numbered 0x2603 is a snowman (☃). There are also systems for writing out strings of characters to bytes so you can put them in a file system. These are encodings. UTF-8 is a popular way to do this for unicode, but not the only way. Some character sets already fit in 8-bit bytes, so they are their own encoding. Latin-1 is a good example. MacCyrillic is also an example.

Rougly speaking, in email, character set and encoding are conflated and called charset. If you declare your message's charset to be UTF-8, it means that it's UTF-8 encoded Unicode. In general, this conflation does not lead to problems. The problem is that email has to be encoded to 7-bit octets -- that is, bytes where the high bit is always off. The process by which this is done is called encoding (content transfer encoding).

So, say you're operating a mailing list and you need to attach footers to a message. First you have to decode from 7-bit to 8-bit and then from 8-bit to characters. Then you can tack on your footer, then you have to do the reverse encoding dance.

Hiding the Encoding Process

MIME::Visitor does its best to get rid of the whole mess.

`1:  2:  3:  4:  5:  6:  7:  8: `	`# This example might be a bit naive, but it should make the point pretty # clearly: my $mime_entity = get_email; MIME::Visitor->rewrite_parts($mime_entity, sub {   my ($content_ref, $part) = @_;   $$content_ref .= footer_for( $part->mime_type ); });`

So, with just a few lines of code, we hide all the annoying bits of encoding and decoding, letting you just focus on the content you want to play with. It has a number of other ways to handle messages, and it does work to ensure that it never rewrites parts that haven't been changed.

Unwrapped to code you'd have to write, the above is something like:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28:  29:  30:  31:  32:  33:  34:  35:  36:  37:  38:  39:  40: `	my $mime_entity = get_email; my $walk; $walk = sub {   my ($root, $code) = @_   if ($root->is_multipart) {     $walk->($_, $code) for $root->parts;     return;   }   return unless $_[0]->effective_type =~ qr{\Atext/(?:html\|plain)(?:$\|;)}i;   my $charset = $entity->head->mime_attr('content-type.charset')              \|\| 'ISO-8859-1';   $charset = 'MacRoman' if lc $charset eq 'macintosh';   Carp::carp(qq{rewriting message in unknown charset "$charset"})     unless my $known_charset = Encode::find_encoding($charset);   my $changed = 0;   my $got_set = Variable::Magic::wizard(set => sub { $changed = 1 });   my $body = $known_charset            ? Encode::decode($charset, $entity->bodyhandle->as_string)            : $entity->bodyhandle->as_string;   Variable::Magic::cast($body, $got_set);   $code->(\$body, $entity);   if ($changed) {     my $io = $entity->open('w');     $io->print($known_charset ? Encode::encode($charset, $body) : $body);   } }; $walk->($mime_entity, sub {   my ($content_ref, $part) = @_;   $$content_ref .= footer_for( $part->mime_type ); });

MIME::Entity!?

The most glaring problem with MIME::Visitor is that it works with MIME::Entity objects instead of Email::MIME objects, which are generally more common in my toolchain. When MIME::Visitor was written, access to get and set the Unicode content of a text part wasn't very good. It's better now, so this may get fixed soon!

More Stuff I Stole from Catalyst (that they stole back)

2009-12-16T00:00:00-05:00

I Hate Typing

It's true. The only time that I like typing is when I'm playing Typing of the Dead. (I try to get in a big dose of zombie video games every Yuletide; maybe I'll bring the old Dreamcast up again this winter.) Apart from shooting zombies by typing quickly, typing stinks. It hurts my hands, and I live in perpetual dread of the day when I must find a new profession because my fingers no longer work. That's why most of the code I write is meant to save keystrokes for when I write other code.

Catalyst lets you write something like this in your config for Holiday::Web:

`1:  2:  3:  4:  5: `	`my $config = {   'Model::GreetingCard' => { ... }   'View::TextTemplate' => { ... }   '+Catalyst::Plugin::Kwanzaa' => { ... } }`

The first two get rewritten to "Holiday::Web::Model::GreetingCard" and "Holiday::Web::View::TextTemplate" respectively. In the third, the + means that it should be used literally -- so just the plus sign is dropped. There's also a ~ prefix for slightly more arcane uses. Obviously, this can save a lot of typing and, just as importantly, a lot of reading. I liked this!

I really wanted to use this kind of rewriting for Dist::Zilla, and to make sure I could re-use this pattern whenever I wanted, I wrote String::RewritePrefix. Catalyst's behavior could be written as:

`1:  2:  3:  4:  5:  6: `	`use String::RewritePrefix rewrite => {   prefixes => {     '' => 'Holiday::Web::',     '+' => ''   }, };`

That's using Sub::Exporter, of course, so we can import a few rewriting strategies:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14: `	use String::RewritePrefix   rewrite => { -as => 'holiday_px', prefixes => { '' => 'Holiday::Web::', '+' => '' } },   rewrite => { -as => 'num_px', prefixes => { '-' => 'negative ', '+' => 'positive ' } }; my @numbers = num_px( qw( +40 -10 +18 0 ) ); # ==> (positive 40, negative 10, positive 18, 0) my @plugins = holiday_px( qw(   Model::GreetingCard   View::TextTemplate   +Catalyst::Plugin::Kwanzaa )); # ==> (Holiday::Web::Mode::GreetingCard, # Holiday::Web::View::TextTemplate, Catalyst::Plugin::Kwanzaa)

There isn't much more to it, apart from one or two weird options. Still, think of all the typing and squinting you save in your config files!

The prefix rewriting in Catalyst::Utils, which inspired me to write String::RewritePrefix, now uses String::RewritePrefix! It's always a joy to find out that other people benefit from my attempt to be lazy and type less. It means they have more time for Typing of the Dead, too.

Like FUSE, but Stupider

2009-12-15T00:00:00-05:00

From Filename to Thing

A few years ago, my coworker Dieter and I were working on improving the componentization of some of our internal web applications. We wanted it to be easier to say "put a paginated table thing here" or "give me a link here." We were pretty happy with HTML::Widget::Factory for low-level use, and the rest could be done with Mason components. We just didn't like the way that you'd have to know whether a component was a method call on a widget factory object or a component call in Mason. I said something like, "I wish we could make it all look like Mason." Totally blowing my mind for all time, Dieter said, "We can" and made it happen seamlessly.

The thing is that Mason's system for resolving path-like things into a component is itself a thing that can customized or replaced. It's the resolver, and you can write a new one.

Many times after seeing the elegance of that solution, I wanted to be able to replace parts of all sorts of code that expected filenames. I looked a number of "emulate a filesystem in Perl" libraries, but they wanted to provide all kinds of stuff I didn't care about -- like statting deleting files. Heck, I just wanted an entity!

Finally my will broke and I wrote the crappiest thing I could think of that would still work. That was Path::Resolver version 0, and now we're up to 3. It's still pretty stupid, and that's a feature. The simplest use is something like this:

`1:  2:  3:  4:  5:  6:  7:  8: `	`my $resolver = Path::Resolver::Resolver::FileSystem->new({ root => '/tmp' }); my $content = $resolver->entity_for('/baz/bar/quux.txt')->content; # or ... my $resolver = Path::Resolver::Resolver::Archive::Tar->new({ archive => 'foo.tar.gz' }); # or ... my $resolver = Path::Resolver::Resolver::DistDir->new({ dist_name => 'Christmas-App' });`

The resolver is, of course, the thing that goes from a filename to an entity. The definition of an entity is pretty vague: it's just a thing. Resolvers know what kind of things they'll return, and you can give them a converter that will expect that kind of thing and return another kind. For example, the FileSystem resolver resolves things to absolute filenames. Then it can be converted to a "SimpleEntity," which basically is just a thing with content. The tarball resolver produces SimpleEntities natively. By knowing what kind of entity things will find, and by having a means to convert them, it's easy to put together a bunch of different kinds of resolvers and multiplex them.

Multiplexing Resolvers

Multiplexing resolvers? It's really simple. "Mux" resolvers just say "I have a bunch of resolvers that all return the same kind of thing, and I have a strategy for picking one of them." There are two important muxers that come with Path::Resolver: Ordered and Prefix. The ordered resolver lets you set up things like the $PATH environment variable. Imagine you did this:

`1:  2:  3:  4:  5:  6:  7: `	`my $resolver = Path::Resolver::Resolver::Mux::Ordered->new({   resolvers => [     Path::Resolver::Resolver::FileSystem->new({ root => '/bin' }),     Path::Resolver::Resolver::FileSystem->new({ root => '/usr/bin' }),     Path::Resolver::Resolver::FileSystem->new({ root => '/usr/local/bin' }),   ], });`

Now you'd have something resembling the way your shell finds commands to run. Of course, there's no reason that all the resolvers in there have to be filesystem resolvers. Maybe your web app installs its own templates, but they can be overridden by the user.

`1:  2:  3:  4:  5:  6: `	`my $resolver = Path::Resolver::Resolver::Mux::Ordered->new({   resolvers => [     Path::Resolver::Resolver::FileSystem->new({ root => './templates' }),     Path::Resolver::Resolver::DistDir->new({ dist_name => 'My-WebApp' }),   ], });`

If the file isn't found in the ./templates directory, we'll look in the files we installed when your webapp was installed.

The prefix muxer is more complex. It represents a virtual directory where each subdirectory is handled by a different resolver. This is used in Email::MIME::Kit::KitReader::SWAK to make Email::MIME::Kit easer to use. It sets up resolvers like this:

  file.txt      - looks in the mkit directory
  /kit/file.txt - looks in the mkit directory
  /dist/My-WebApp/header.html      - looks in dist ShareDir for My-WebApp
  /fs/usr/share/common/header.html - looks in /usr on the filesystem

Putting it to Use

A nice thing about Path::Resolver is that in general, you can use it without worrying about how it will get used later. If you have some part of your program that reads stuff from disk, you can just say, "Whatever, I'll use a Path::Resolver here." Then whenever anyone wants to make it do something wild and crazy, it's much, much simpler for them to do so -- they just supply a new Path::Resolver object to replace your stock FileSystem resolver.

First-Class CLI Applications

2009-12-14T00:00:00-05:00

Scripts are Our Friends

In my experience, it's pretty common to find a large piece of functionality built into a command-line program (or "script" (and I am totally not going to get into the "script" vs. "program" debate here)) and then to find that the program isn't tested at all. When you say, "For the love of God, why are you not testing this vital program?" the answer is, "Well, scripts are really hard to test!"

It's true. Maybe they're not as hard to test as people think, but it's still a pain. They're also full of too many standards for getopt or error messages. People just do whatever gets work done, and then later have to pay the price for making crazy decisions.

A Simple App::Cmd Program

App::Cmd is a simple framework for writing command-line applications that are easy to test, that have powerful and easy to use standard tools, and that can be extended easily.

For example, here's a simple command we might write, in two parts. First, the script that we put in our path, then the library that implements it.

./bin/christmas:

`1:  2: `	`use Christmas::App; Christmas::App->run;`

./lib/Christmas/App.pm:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27: `	package Christmas::App; use base 'App::Cmd::Simple'; # ABSTRACT: an app for managing our christmas shopping sub opt_spec {   return (     [ 'nice\|n' => 'list only nice people' ],     [ 'all\|a' => 'list even people for whom shopping is done' ],   ); } sub validate_args {   my ($self, $opt, $args) = @_;   $self->usage_error("no args expected") if @$args; } sub execute {   my ($self, $opt, $args) = @_;   my @presents = Christmas::Presents->get( # In other words: use switches to build query: ($opt->all ? () : (done => 0),     ($opt->nice ? (nice => 1) : ()),   );      print $self->_list_presents(@presents); # implementation left to imagination }

So far, we've only added a little structure to our code, but it's already a big help. The opt_spec routine uses Getopt::Long::Descriptive to not only process command line switches (with quite a lot of power), but also to generate helpful usage messages like:

  Usage: christmas

  christmas [-an] [long options...]
    -n --nice   list only nice people
    -a --all    list even people for whom shopping is done

We also get a phase before execution but after argument processing to decide whether the arguments we were given make any sense -- here we just ensure that we didn't get any!

Putting it to the Test

One of the big reasons to use App::Cmd was supposed to be its testability, so let's see how that works.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28: `	use Test::More; use App::Cmd::Tester; use Christmas::App; {   my $result = test_app('Christmas::App' => [ qw(--all) ]);   is($result->stderr, '', "nothing to stderr");   like($result->stdout, qr/Ricardo Signes/, "rjbs is in our list");   like($result->stdout, qr/Violet Beauregarde/, "violet is, too"); } {   my $result = test_app('Christmas::App' => [ qw(--all --nice) ]);   is($result->stderr, '', "nothing to stderr");   like($result->stdout, qr/Ricardo Signes/, "rjbs was nice");   unlike($result->stdout, qr/Violet Beauregarde/, "Violet was naughty"); } {   my $result = test_app('Christmas::App' => [ qw(--nice) ]);   is($result->stderr, '', "nothing to stderr");   unlike($result->stdout, qr/Ricardo Signes/, "no rjbs w/o --all");   unlike($result->stdout, qr/Violet Beauregarde/, "Violet still naughty"); }

This should be fairly straightforward: test_app runs the application, using the passed arrayref as the value for @ARGS. It doesn't run in a subprocess, so there's no weird issues with interprocess communication. Also, because it runs in process, you can replace hunks of the app with mocks if you want, and you'll have them available for inspection after testing.

Organizing Complex Interfaces

I didn't write App::Cmd for simple programs, though, I wrote it for complex ones. I wanted to write programs that behave like svn or git, where the first thing you tell the command-line program is which of its subcommands you want to run. So, maybe the program we wrote above is meant to be run as christmas list. We also want to have christmas music to control our MP3 player and christmas cards to assemble and send off some mkit Christmas cards.

This is easy, we do it like this:

rename Christmas::App to Christmas::App::Command::list
replace its use base 'App::Cmd::Simple' with use Christmas::App -command
create a new Christmas::App (shown below)
create Christmas::App::Command::music and ::cards

Christmas::App is easy to write; here it is in its entirety:

`1:  2:  3: `	`package Christmas::App; use App::Cmd::Setup -app; 1;`

Extra commands just need those three original methods, for example:

`1:  2:  3:  4:  5:  6: `	`package Christmas::App::Command::music; use Christmas::App -command; sub validate_args { ... } sub opt_spec { ... } sub execute { ... }`

That's it. Now the new Christmas::App will be run, it will find all the command classes we've written, and it will decide which one to execute based on the first argument to the christmas command.

Other Cool Stuff

When you write a "full" App::Cmd program -- that is, one that uses App::Cmd and not App::Cmd::Simple -- you get a bunch more features for free. For one thing, you get commands for commands and help that can list and describe the other available commands. (commands is what happens by default if you were to run christmas with no arguments, but you can change that by writing a default_command method.)

You get access to the plugins system, which is woefully underdocumented, but allows you to set up easy to use routines in all your commands, so that you could say, in Christmas::App:

`1:  2:  3:  4: `	`package Christmas::App; use App::Cmd::Setup -app => {   plugins => [ qw(App::Cmd::Plugin::Prompt) ], };`

...and then all your commands could use routines like prompt_yn or prompt_any_key without you having to waste keystrokes on ugly method calls to some object delegate.

Valid Data to All Nations

2009-12-13T00:00:00-05:00

Params::Validate is Okay

The CPAN has quite a few data validation libraries on it. There's Params::Validate, Data::FormValidator, and probably at least fifty others. Maybe a lot more. I've used a dozen or so of them. Some of them stink, some of them are pretty good. They are, unsurprisingly, pretty Perlocentric. It's useful, in Perl, to be able to say "I'm expecting an instance of a class that does this interface" or "this part of the data should be a regular expression" or "test this data with a custom callback."

The problem with this kind of validator is that it leads to business rules that are only expressible in Perl, or that require substantial post-implementation analysis to translate them into other languages. A while ago, I started to really want to valdiate data in both Perl and JavaScript, specifically data that I was handing back and forth between a Perl web service and two clients, written in Perl and JavaScript. I wanted to be able to validate the I/O in JavaScript, but writing all my validation twice looked like a huge pain that would lead to not wanting to validate at all. I decided I'd look for schema languages.

I found XML Schema and RELAX NG and a few others meant for XML, but they really didn't make a lot of sense for the data structures common to the batch of dynamic programming languages I wanted to support. They were very XML oriented, which is fine, because they were supposed to be. I also found Kwalify and JSON Schema, neither of which seemed really well done to me, for one reason or another. After some token pleading with friends ("Please talk me out of writing a schema system!") I decided to write my own.

Rx Schema

The Rx Schema system is very simple. It's meant to be easy to extend with your own datatypes, but to guarantee only a small set of core types that map to core datatypes in many common languages. We can start with this data structure (shown here as Perl):

`1:  2:  3:  4:  5:  6:  7: `	`my $data = {   'Armand Assante' => { disposition => 'nice', present => 'Wii' },   'Ricardo Signes' => { disposition => 'nice', present => 'Sleep' },   'Violet Beauregarde' => { disposition => 'naughty' },   ... };`

We'll make a slightly tortured schema for this, just to show off a few things. Here, I'm representing the schema in YAML for readability, but they're really just data structures. You could write it in Perl. I usually use JSON.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11: `	`--- type: //map values:   type: //any   of :     - type: //rec       required: disposition: { type: //str, value: 'naughty' }     - type: //rec       required:         present: //str         disposition: { type: //str, value: 'nice' }`

In other words, we require a hashref. Each value is a hashref with either one or two entries. One must be disposition. If it's naughty, there will be no other entries. If it's "nice" there must be an entry for present, which must be a string.

It's easy to use this schema to validate things:

`1:  2:  3:  4:  5:  6:  7: `	`use Data::Rx; my $schema = Data::Rx->new->make_schema( $that_structure_above ); unless ($schema->check($input)) {   die "invalid input!"; }`

But the whole point was to have this work in many languages, and it does:

`1:  2:  3:  4:  5:  6: `	`var rx = new Rx; var schema = Rx.makeSchema( structure ); unless (schema.check( input )) {   throw "invalid input!"; }`

...or...

`1:  2:  3:  4:  5:  6:  7:  8: `	`require 'Rx' rx = Rx.new schema = Rx.make_schema( structure ) unless schema.check( input )   throw "invalid input!"; end`

There are other implementations, too.

The Rx schema system is easy to extend. If you want to use regular expressions you can use the PCRE extension, which many language will be able to implement, then you can write:

`1:  2:  3: `	`--- type: /pcre/str regex: '\A867-[5309]{4}\z'`

Coming Features (or: Missing Features)

Right now, the various releases of Rx can only check whether or not something is valid. Obviously it's much more useful to be able to know why something isn't valid, if it fails. This is being implemented (in the struct-fail branch), although development has been on hold while I work on some other projects. Once that works, Rx Schema will be much more useful.

I've often talked about other ideas that could built atop Rx like RPC, URI templating, and API library autogeneration. Although I'd love to work on them, I think it's time to admit that I don't see them happening in the near future unless somebody decides to really push for it. For now, a portable schema language is all I'm aiming for -- and I think that's enough for me!

E-mail Made Simple

2009-12-12T00:00:00-05:00

Okay, I Lied

I got a decent amount of angry comments on Thursday's post about Email::MIME::Kit. It seems people are sick of my insistance that e-mail is really complicated and they want me to publicly admit that it's simple.

I admit it. E-mail is pretty much just a dumped hash, then a bunch of lines, and maybe some attached files at the end. I've written some code to show off how easy things can be, if you just accept that e-mail is simple. This code is the acme of simplicity and the acme of utility.

It doesn't use any objects or special variables or functions. It just lets you put data into a structured variable, and then you can send it.

E'Mail::Acme

Setting up the Header

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12: `	`use E'Mail::Acme; my $e_mail = E'Mail::Acme; # Now we have our e-mail variable, so we can put some headers in it. # Headers are just a hash, remember? $e_mail->{From} = q<Ricardo Signes <rjbs@example.com>>; # ...and we can get multiple headers easily enough, just assign a few times: $e_mail->{To} = q<Santa Claus <santa@north-pole.mil>>; $e_mail->{To} = q<Kris Kringle <kk@macys.com>>;`

So, easy! The header is just a hash, like you'd expect.

Setting up the Body

The body is just a bunch of lines of text. In other words, an array. We can add lines easily enough.

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	push @$e_mail, "Dear Santa,"              , ""              , "Here are some things I want:"              , @things_i_want              , ""              , "Please gimme."              , ""              , "-- "              , "rjbs";

Great. Of course, that looks pretty ridiculous. Probably we wanted to just do this:

`1: `	push @$e_mail, `cat /tmp/mutt-terrordome-2608-9063-429`;

And just to show that we're not limited to the world of boring "single part" e-mails, we'll attach a little incentive.

`1:  2:  3:  4:  5:  6:  7: `	# first, we make an attachment my $attach = E'Mail::Acme; $attach->{'content-type'} = 'image/jpeg'; push @$attach, `cat amazon-giftcert.jpg`; # then we attach it push @$e_mail, $attach;

Sending our Message

Sure, we could bring in some modern, best-practice library like Email::Send:

`1: `	`Email::Send->new->send("$e_mail");`

...but we'd just be back to these ridiculous object-oriented interface that clutter up our simple e-mail handling. We set up our e-mail object, and now we want to do something with it. There's only one thing we do with e-mail, right? We send it! All we have to do is tell Perl we want to do (that thing with) the e-mail:

`1: `	`do &$e_mail;`

We could also have written $e_mail->(), but the arrow notation unnecessarily reminds me of objects, so I try to avoid it.

Sane, Simple, Sensisble

So, there you have it. E-mail handling made simple and straightforward, just like you'd like it. Use it everywhere!

Giving the Gift of Code

2009-12-11T00:00:00-05:00

I Hate Wrapping

I like to give people things that they like. It's nice for someone to open a present and find just the thing they wanted -- or better, something that they didn't know they wanted, but that they're elated to receive. It's a nice feeling. I just don't like wrapping presents. I have to pick wrapping paper, measure it, cut it, tape it, and try not to make it look like a five year-old did it. I generally fail at all of these tasks and hand people presents that are balled up in tape and crumpled paper.

It's the same feeling I get when I release code to the CPAN: I'm excited that people might find it useful, even if they hadn't realized they'd wanted something like that -- but I'm annoyed at the idea of writing up all the stupid boilerplate: generic hunks of Pod, build files, the same tests I use everywhere else, license documents. It's just a pain, and I still end up with a big red "You forgot the README!" light flashing at me on CPANTS. At least, I used to.

First, to fight this problem, I started using Module::Starter, and I contributed a number of changes to make it easier to extend. I didn't get it very easy to extend though, just slightly more than it had been. In the end, too, it really only got you started, as the name suggested. Every time you wanted to make a new release or add a file or change something you'd put in all the files, you realized that Module::Starter stopped being useful well before your first release.

I tried to use Module::Install to deal with this problem, and it helped a little. It could add a few files that I'd need generated like license documents or tests. It just didn't do enough -- and it was such a pain to extend, being built with weird fundamental design decisions atop the ancient and insane ExtUtils::MakeMaker.

My Own Private Gift Counter

Finally I decided that all I really wanted was a way to make make install way, way more powerful and hookable. I made a little flowchart of the steps I wanted to perform and then build a little skeleton of code to hang those steps on, and that skeleton was Dist::Zilla. Dist::Zilla, as the name suggests, is an absolute monster. It only needs to be run by maintainers of CPAN distribution, not installers, so it doesn't shy away from demanding all sorts of hefty prerequisites.

Here's how I use Dist::Zilla, more or less, although there's more than one way to do it:

  ~$ cd code
  ~/code$ dzil new My-Project
  will create new dist My-Project in /Users/rjbs/code/My-Project

  ~/code$ cd My-Project
  ~/code/My-Project$ vi dist.ini

Most of my dist.ini, the master configuration file, is set up for me already -- it's only about five lines -- but I edit it a bit to make it look like this:

`1:  2:  3:  4:  5:  6:  7:  8: `	`name = My-Project author = Ricardo Signes <rjbs@cpan.org> license = Perl_5 copyright_holder = Ricardo Signes [@RJBS] [@Git] [AutoPrereq]`

Now I'm ready to start coding. I'll make a t/my-tests.t and lib/My/Project.pm. My module might look something like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19: `	`use strict; package My::Project; # ABSTRACT: our top-secret project for playing bowling against WOPR use Games::Bowling::Scorecard; use Games::War::Nuclear::Thermonuclear::Global; use Path::Resolver 2.012; =method play_a_game   $project->play_a_game($num_of_players); This method starts a game. It's a strange game. =cut sub play_a_game { ... } 1;`

You can load up your Changes, too, just adding the details of your release so the file looks like:

  Revision history for {{$dist-&gt;name}}

  {{$NEXT}}
            first release
            doesn&#39;t handle zero players
            tends to destroy civilization

That's it.

Now, when you run dzil release, this happens:

abort if you're not all committed to git
determine all prereqs by finding "use" and similar statements
write out a README, LICENSE, Makefile.PL, MANIFEST, META.yml and META.json
decide on the next release's version number
defined $VERSION in all your packages
add a NAME, VERSION, LICENSE, and AUTHORS section to all the Pod
turn all =method commands into grouped "normal" Pod
update the changelog with the new version and the current date/time
build a tarball and upload it to PAUSE (submit it for CPAN index)
commit the changes to the changelog file
tag the release
push the changes and tags to the remote git repository

I've actually left out a few steps that are very useful but harder to explain. All of those steps happen for each release, so you can reconfigure them all each time without having to update a lot of embedded boilerplate code -- the kind you'd end up with, with other distribution-creation systems.

Finally, all those steps are configurable. I described the configuration that I showed, but you could use fewer plugins, or more plugins, or just different plugins. If you want to maintain your own list of prereqs, you can. If you want to write your own MANIFEST, you can do that -- and there's even a plugin to sanity-check it for you when you try to build a release.

By making it so easy to wrap up and deliver your code, Dist::Zilla lets you just enjoy giving the gift itself -- so you'll be eager to do it all the more often!

MIME, Dreaming of a White Christmas

2009-12-10T00:00:00-05:00

Making an Email

I maintain a lot of Perl email code, and one of the most popular modules I maintain is Email::Simple. It was written to make a very simple class for looking at email documents. It didn't let you do much, and its implementation assumed that you wouldn't even go very far using the interface it provided. That's fine: it said, "here is a very simple tool for performing very simple operations on this very complicated thing, email." Unfortunately, sometimes we latch onto the dream -- the other way to read Email::Simple's name -- that email, itself, is simple. Sadly, it just isn't.

One of the most common kinds of email to send is a form letter. You want to send an invoice or welcome message, and you want to fill in the details for this purchase or customer. This is a simple thing to think about doing, and it's something we do a lot, so it seems like it should be really straightforward. Frustratingly, this is about as simple as it gets:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28:  29:  30:  31:  32: `	my $html_template = <<'END_HTML'; <h1>Welcome to Yoyodyne, [% customer.realname %]!</h1> <p>You signed up with the following address:</p> <pre>[% customer.address.envelope_style %]</pre> END_HTML my $text_template = <<'END_TEXT'; Welcome to Yoyodyne, [% customer.realname %]! You signed up with this billing address: [% customer.address %] END_TEXT my $email = Email::MIME->create(   header => [     From => 'Customer Service <noreply@example.com>',     To => $customer->realname . " <" . $customer->email . ">",     Subject => 'Welcome to Yoyodyne!',   ],   { attributes => { content_type => 'multipart/alternative' } },   parts => [     Email::MIME->create(       attributes => { content_type => { 'text/plain' } },       body => render_tt($text_template, { customer => $customer }),     ),     Email::MIME->create(       attributes => { content_type => { 'text/html' } },       body => render_tt($html_template, { customer => $customer }),     ),   ], );

Yow! That's a lot of code, and a lot of it looks stupid and boring. Also, there are a bunch of bugs. If the customer's real name contains special characters (and I mean more than just 8-bit), the headers will be illegal. We probably made a mistake by stringifying the customer's "address" attribute in the plain text template (but not in the HTML template). We didn't specify any kind of encoding for the text parts, so there might be more bugs there.

Email::Stuff is designed to make this a bunch simpler, replacing the Email::MIME->create call with:

`1:  2:  3:  4:  5:  6:  7: `	my $email = Email::Stuff           ->to($customer->realname . " <" . $customer->email . ">")           ->from('Customer Service <noreply@example.com>')           ->subject('Welcome to Yoyodyne!')           ->text_body(render_tt($text_template, { customer => $customer }))           ->html_body(render_tt($html_template, { customer => $customer }))           ->email;

That's a lot better! Still, we're going to end up with encoding bugs, our templates diverged, and there are other more subtle problems. For example, what if we accidentally pass in the wrong kind of object for $customer? It might throw an exception when we try to call methods on it setting the to header, but it might not happen until the Template Toolkit template, so that exception might just translate to missing data. It's a mess.

Making Email Simple

Email::MIME::Kit is designed to make it easy to write templates that get constructed into one email, abstracting the boring details and making it as easy as possible to add new features.

Here's an example of a fully-loaded message kit (or mkit (pron. "em kit"), as we call them at work), both on disk and in the code that sends it. We'll make a directory, welcome.mkit, which is the mkit itself. In it, we'll stick a bunch of files, starting with manifest.yaml (often manifest.json instead), which configures the kit as a whole:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13: `	`validator: Rx renderer : TT assembler: - Markdown - html_wrapper: wrapper.html   text_wrapper: wrapper.txt header: - From: Customer Service <noreply@example.com> - To : '[% customer.realname %] <[% customer.email %]>' - Subject: Welcome to Yoyodyne! path: body.mkdn`

This file explains how the message will be built. It says we'll validate the input using Rx, render templates using TT2, and then construct the message using the Markdown-style assembler. The Markdown assembler is the least straightforward bit of that, so let's look at it first.

It says: We've got a file, body.mkdn, which is Markdown. We'll render that Markdown as a TT2 template, wrap it up in some boilerplate, and use that as the plain text part. Then we'll take that Markdown and render it into HTML, wrap that in some boilerplate, and use that as the HTML part.

In other words, we'll get a easy to read plain text part and a pretty HTML part, and they'll have identical content. So, for the message we were sending, above, we might have this Markdown file:

`1:  2:  3:  4:  5: `	`# Welcome to Yoyodyne, [% customer.realname %]! You signed up with the following address:     [% customer.address.envelope_style %]`

Unless told otherwise, Email::MIME::Kit assumes you're working in Unicode with files stored in UTF-8 and takes care of encoding for you. Now you've got one template, identical across both alternatives, and all that remains is to validate the input. We declared we'd use the Rx schema system, and we'll put a schema in rx.json in our mkit directory.

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	`{   "type": "//rec",   "required": {     "customer": {       "type": "/perl/obj",       "does": "Yoyo::Customer"     }   } }`

Now we'll refuse to even build an email unless we were given a valid customer object. How do we actually build the message? It's easy.

`1:  2: `	`my $kit = Email::MIME::Kit->new({ source => './welcome.mkit' }); my $email = $kit->assemble({ customer => $customer });`

At this point, we've written fewer total lines of code than we did in the very first example and have gotten a far superior result: fewer bugs are possible and all the "this email as a template" data are stored safely out of the way. This is also just the tip of the iceberg: mkits make it easy to write much more powerful templates and to share common components. They can also be used with fewer of their features, getting you fewer bug-preventing measures, but still plenty of benefit over other techniques.

In almost all non-trivial cases, mkits are a win over other methods of sending form letters -- and Email::MIME::Kit is still getting better.

Like <DATA>, Only Less Stupid

2009-12-09T00:00:00-05:00

First, An Alternative

Data::Section was actually written when I realized I couldn't use Inline::Files. I'd wanted to use that ages ago, but it had a number of little problems. The biggest problem, unfortunately, was that it just tried to be way, way too powerful.

I like overpowered modules as much as the next guy, but here's about where I draw the line: Inline::Files is a form of source filter, and includes this warning:

It is possible that this module may overwrite the source code in files that use it. To protect yourself against this possibility, you are strongly advised to use the -backup option described in "Safety first".

The DATA Section

Data::Section (and Inline::Files) are meant to make the __DATA__ section both more powerful and more reliable. For those of you who haven't encountered it, Perl lets you access a virtual filehandle that reads non-code content at the end of your program file. In other words, if you write this program:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14: `	`my $total = 0; while (<DATA>) {   chomp;   printf "Got number: %u\n", $_;   $total += $_; } printf "Total: %u\n", $total; __DATA__ 2 4 6 8`

...then your program would output:

  Got number: 2
  Got number: 4
  Got number: 6
  Got number: 8
  Total: 20

Why Data::Section Beats DATA

No Need to seek

There are a few problems with DATA, though. The most annoying is that you can't easily seek on it. Its read position is actually relative to the whole source file, so you'd need to try to tell it first, then save that, and things get messy.

Data::Section caches your data for you, so you can reread it any time you want.

Multiple "Files" Per Module

The DATA filehandle is one big entity, but Data::Section lets you provide multiple virtual files in your data:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22: `	`package Your::Package; use Data::Section -setup; my $sections = Your::Package->merged_section_data; for my $filename (keys %$sections) {   printf "== %s ==\n%s\n", $filename, Your::Package->section_data($filename); } __DATA__ ___[ so-far.txt ]___ Sub::Exporter Number::Nary Email::Pipemailer::DieHandler Pod::Coverage::TrustPod Games::Bowling::Scorecard __[ coming-up.txt ]__ Mixin::Linewise App::Cronjob String::Formatter Data::Section Email::MIME::Kit`

You'll get two "filenames" and each one will have five lines in it.

(By the way, that use Data::Section line? Yeah, that's all you have to do. Obviously, Data::Section uses Sub::Exporter, so you can rename its methods, and there are a few parameters you can use to customize them.)

Inherited Files

If you're wondering what's "merged" in the merged_section_data method, it's inherited files. In other words, if you wrote a subclass of the Your::Package code above, you could add new files in its DATA section, but the old files would still be accessible -- unless you added a so-far.txt definition to your subclass. It behaves very much like an overridden method, in that way.

Practical Uses

When is this useful? It's great to use instead of here-docs for a lot of cases. It's also a nice way to avoid having to install and manage text files. A package can store its own bundle of configuration or template files, and it will have methods to access them. Then if you want, you can specialize your pack of template files by subclassing the original one.

Even without all its new features, the simple ability to never re-seek DATA is a big one. (Please note that Data::Section caches the contents, it doesn't magically take care of re-seeking. That ends up being a real problem.) DATA is a global variable, and if code ever tries to read from it more than once, the second time will fail, and that is a really annoying bug to track down -- trust me, I've had to do so many times. So, forget about plain old DATA sections and start using Data::Section. Your future self will thank you.

%{adj}s %s{holiday}s!

2009-12-08T00:00:00-05:00

`sprintf` is My Favorite Chainsaw

There are a lot of horribly overcomplicated builtins, not just in Perl, but in many languages. Some people like Lisp's format best -- and why not? Like...

`1:  2:  3:  4:  5:  6:  7: `	`(format nil "~4,'0d-~2,'0d-~2,'0d" 2005 6 10) ; ==> 2005-06-10 (format nil "~:(~a~)" "tHe Quick BROWN foX") ; ==> "The Quick Brown Fox" (format nil "~:r" 1234) ; ==> "one thousand two hundred thirty-fourth" (format nil "~:@r" 1234) ; ==> "MCCXXXIIII"`

Right? That's pretty awesome. Then there's Perl's pack. It's sort of like a sprintf designed by an assembly programmer who hates you:

`1: `	`pack 'n/a* w/a', 'hello,', 'world'; # gives "\000\006hello,\005world"`

There are actually people who like that, and I have seen, with my own eyes, a guy who defended format:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10: `	format STDOUT_TOP =                         Passwd File Name Login Office Uid Gid Home ------------------------------------------------------------------ . format STDOUT = @<<<<<<<<<<<<<<<<<< @\|\|\|\|\|\|\| @<<<<<<@>>>> @>>>> @<<<<<<<<<<<<<<<<< $name, $login, $office,$uid,$gid, $home .

Well, sprintf isn't as awful as any of these, and that's why I love it... at least if you ignore things like:

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	`# Vectors! printf "%vd", "AB\x{100}"; # prints "65.66.256" printf "version is v%vd\n", $^V; # Perl's version # Put an asterisk "" before the "v" to override the string to use to # separate the numbers: printf "address is %vX\n", ":", $addr; # IPv6 address printf "%2\$*3\$d %d\n", 12, 34, 3; # will print " 34 12\n"`

Enough! Settle down!

Sorry. I got distracted.

sprintf is the sweet middle ground between a double-quoted string and Template Toolkit. If you stick to the basics that most people use, it's a nice simple way to put placeholders in a string, then stick values into it, doing just a little formatting as needed. Here's how people actually use sprintf:

`1:  2: `	`printf "Product %s (%s) added for \$%0.2u.\n",   $product->id, $product->name, $product->price;`

Nice and simple, right? What I really wanted was a way to use these kinds of templates as templates, outside of code. So, for example, I could store the above string in a configuration file without worrying about the order of arguments:

`1:  2: `	`[Purchasing] format = Product %{id}s (%{name}s) added for $%0.2{price}f`

Then my program could call:

`1: `	`printf $config->{purchasing}{format}, $product;`

String::Formatter is all about building routines that make this possible. It's very flexible, and can probably be tortured into doing all kinds of horrible things (although I hope nobody ever reimplements all of Perl's sprintf). Here are just a few simple examples of how you can put String::Formatter to use.

object-specific conversions

You can set up format codes that correspond to methods, making a formatting routine that helps convert given objects to strings.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11: `	`use String::Formatter method_stringf => {   -as => 'datetimef',   codes => {     f => 'strftime',     c => 'format_cldr',     s => sub { "$_[0]" },   }, }; print datetimef("%{%Y-%m-%d}f is the same as %{yyyy-MM-dd}c.", DateTime->now);`

generic method-calling conversions:

You could write a formatter that just calls any method you want:

`1:  2:  3:  4:  5:  6:  7:  8: `	`use String::Formatter stringf => {   -as => 'objectf',   string_replacer => 'method_call_replace', # imaginary but easy   codes => { s => sub { "$_" } } }; print objectf("Product %{id}s (%{name}s) added for $%0.2{price}s", $product);`

custom conversions

You could write something meant to be used just like sprintf, but with different conversion routines:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17: `	use String::Formatter stringf => {   codes => {     s => sub { $_ }, # string itself     l => sub { length }, # length of input string     e => sub { /[^\x00-\x7F]/ ? '8bit' : '7bit' }, # ascii-safeness   }, }; print stringf(   "My name is %s. I am about %l feet tall. I use an %e alphabet.\n",   'Ricardo',   'ffffff',   'abcchdefghijklmnñopqrrrstuvwxyz', ); # Output: # My name is Ricardo. I am about 6 feet tall. I use an 8bit alphabet.

...and you can make it do lots more.

String::Formatter does its job in a few phases, and you can replace each or all of those phases per formatter, so it's easy to write very powerful custom formatting routines. Any time you need to a teeny tiny templating language, consider whether a slightly-improved sprintf would be enough. You might be surprised how often it will be just right.

59 23 24 12 * /bin/celebrate

2009-12-07T00:00:00-05:00

Solaris Ain't So Bad

We've been using Solaris for most of our servers for quite a while at work now, and there's no question that it's made our sysadmin's life easier. Everybody wants the BOFH to be happy, but there's often this idea that Solaris will make you, the programmer, miserable. Given the choice between a miserable me and a miserable sysadmin, I know which one I will pick. After all, if worse comes to worst, you can just replace the sysadmin with some more Perl, right?

(I'm just kidding, Bryan, you're great.)

In reality, Solaris is fine. It almost never gets in my way, and that's because one of the first things that happens to every new Solaris box we build is the installation of the GNU tools. I mean, ZFS and DTrace and zones are great, but Solaris grep is like a trip back to 1982. The one tool that just wasn't quite easy enough to replace, though, was Solaris cron. In the beginning, as we got more and more servers onto Solaris, more and more of my cron mailbox started to look like this:

  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command
  Subject: Output from &quot;cron&quot; command

Yeah. That's just fantastic. Thanks.

Keep the crond, just add more Perl

So, as I said, replacing crond wasn't as simple as we wanted it to be, so instead we wrote a wrapper and used Puppet to make sure that every single cron job used it. The wrapper is cronjob, which you get when you install App::Cronjob.

You just replace "some-job" in your crontab with:

  /usr/local/bin/cronjob -s &#39;nightly llama maintenance&#39; -E -c some-job

By default, cronjob will get a lock for the job. Unless you tell it not to lock, no two jobs with the same subject can be run at once. The default subject is the command to be run, but we've overridden it with the "-s" switch, above. If it gets a lock, it runs the job. Just like normal cron, it will send you the output of the command -- but sometimes that's really annoying, so we've used "-E" to tell it to only send mail if the command exits with an error condition.

In addition to sending you the output of the command, it will include a summary of how the job went:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13: `	From: "cron/lambda.example.com" <cron@lambda.example.com> To: root@lambda.example.com Date: Thu, 19 Nov 2009 03:00:09 -0500 Subject: FAIL: nightly llama maintenance In-Reply-To: <73d2be0cc900096550bc3b159b4d86d4@lambda.example.com> Command: some-job Time : 47.5708s Status : {{{"core": 0, "exit": 2, "signal": 0, "status": 512}}} Output : rm: cannot remove `/var/ram/foo@example_com_.3H6D8RyK': No such file or directory

We can see how long it took, what happened, and its output. The "FAIL" in the subject also lets us pick out jobs that failed from jobs that were merely noisy.

Finally, see that message-id in the In-Reply-To header? That will be the same for every job with the same subject on the same host. That means your cronjob reports will form threads, and that means that you can tell mutt (or Thunderbird or Mail.app, etc) to collapse all your threads and skim over the different error reports and their counts, rather than just seeing and endless stream of the same flapping service over and over.

That's it! App::Cronjob can help hide the pain of Solaris cron, but it isn't just for Solaris. After all, cron might be better on most Linux systems than on Solaris, but its reports still can't compare with App::Cronjob's!

Less Boilerplate, More Ice Cream

2009-12-06T00:00:00-05:00

Just a Little Less Work

Let's take it easy, it's Sunday. This library is just a little timesaver, but it saves you from the worst kind of work: boring, stupid, repetitive work.

Here's some code I have written longhand way too many times:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27: `	use strict; package Character::Count; sub read_file {   my ($self, $filename) = @_;   open my $fh, '<', $filename or die "drat: $!";   return $self->read_handle($fh); } sub read_handle {   my ($self, $fh) = @_;   local $/;   my $string = <$fh>;   return $self->read_string($string); } sub read_string {   my ($self, $string) = @_;   my %result;   $result{$_}++ for split //, $string;   return \$result; }

Except when I wrote it the first way too many times, each implementation was a little different. Sometimes I forgot to check the result of opening a file. Sometimes I duplicated bits of code and only fixed bugs in one or the other. In the one above, I boiled everything down to working on a string, which is going to stink if my input is really big and now I'm holding it all in memory at once and copying it around... and all that boilerplate for just two useful lines of code!

Mixin::Linewise deals with this by letting you just write the filehandle-based iterator:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14: `	`use strict; package Character::Count; use Mixin::Linewise::Readers -readers; sub read_handle {   my ($self, $fh) = @_;   my %result;   while (<$fh>) {     $result{$_}++ for split //;   }   return \$result; }`

That cuts the amount of code roughly in half and ensures that any bugfixes apply to all your modules everywhere instead of just the methods in the modules you remember to go find and update.

There's also a Mixin::Linewise::Writers, which does just what you think: you write write_handle and it gives you write_file and write_string. (write_string returns the string it has produced.)

...and because it's Sub::Exporter...

Mixin::Linewise uses Sub::Exporter, which means you can parameterize and rename its exports, which means you can do this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12: `	`use strict; package ETL::System; use Mixin::Linewise::Readers   -readers => { -suffix => '_ini', method => 'read_handle_ini' },   -readers => { -suffix => '_dat', method => 'read_handle_dat' }; sub read_handle_ini { ... } sub read_handle_dat { ... } # ...and then somewhere else ... my $result = ETL::System->read_file_dat('datasource.dat');`

In other words, you can build a bunch of file and string reader methods with different names built on top of distinct handle-readers.

So, Mixin::Linewise won't revolutionize how you write Perl, and it won't be useful all the time. It will be useful once in a while, and will let you stop writing boring boilerplate code and, what's better, stop having to test it, too.

The Other Kind of Turkey

2009-12-05T00:00:00-05:00

It's Elf Bowling Time Again

Every year, as Christmas approaches, it's time once again to update our email filters to bitbucket all the "elf bowling!!!" emails from friends and family. I've never understood the appeal of ELFBOWL.EXE, but I guess I can say one thing for it: it gets me thinking about bowling, again. I used to really enjoy bowling, but I haven't gone out to the lanes in ages. Living in New England made me stop thinking about it much, because almost all bowling up there is of a strange, abominable variation.

When the Nintendo Wii came out, though, I got hooked on bowling again, and realized that although I had always enjoyed the game, I really never understood how to score it. What better way to learn than to implement a CPAN module for tracking bowling scores? Probably many, but that's the one I picked!

Here's the sort of scorecard I might get to fill out on a good day. (I said I liked bowling, not that I was good at it!)

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17: `	`use Games::Bowling::Scorecard; my $card = Games::Bowling::Scorecard->new; $card->record(6,1); # slow start $card->record(7,2); # getting better $card->record(10); # strike! $card->record(9,1); # picked up a spare $card->record(10) for 1 .. 3; # turkey! $card->record(0,0); # clearly distracted by something $card->record(8,2); # amazingly picked up 7-10 split $card->record(10, 9, 1); # pick up a bonus spare say $card->score; # 156`

If you're a much better bowler, you might get to make this excellent invocation:

`1:  2:  3:  4:  5:  6: `	`my $card = Games::Bowling::Scorecard->new; $card->record( (10) x 12 ); say $card->score; # 300 $alley->trigger_balloon_drop;`

Displaying Score Cards

Of course, printing a number to the screen isn't very satisfying. Part of the joy of bowling is watching all the bizarre math and symbols get drawn on the big overhead screen. We could tack this onto my game above...

`1:  2:  3: `	`use Games::Bowling::Scorecard::AsText; print Games::Bowling::Scorecard::AsText->card_as_text($card);`

And we'll get this extremely satisfying output:

  +-----+-----+-----+-----+-----+-----+-----+-----+-----+-------+
  | 6 1 | 7 2 | X   | 9 / | X   | X   | X   | - - | 8 / | X 9 / |
  |   7 |  16 |  36 |  56 |  86 | 106 | 116 | 116 | 136 |   156 |

Practical Uses..?

If you know how to bowl, you probably know how to keep score, so you're not going to be saving a lot of time by keying in your scores to a Perl program. You could easily enough throw together a ReadLine-based program to keep track of turns and scores, but I think those uses are never going to be big wins over transparencies and grease pencils.

No, I think that the big hot application for Games::Bowling::Scorecard will come with the growing resurgence of SDL Perl. Pretty soon, dozens of graphical bowling games written in Perl will be in the works, and they'll all be easier to write thanks to Games::Bowling::Scorecard. I look forward to the day when the ELFBOWL2.EXE that I angrily delete is compiled from Perl, using my own code.

In Pod We Trust

2009-12-04T00:00:00-05:00

Testing Pod Coverage

Andy Lester's fantastic Test::Pod::Coverage makes it easy to check whether you've covered all your code. The standard usage is to make a file named something like t/pod-coverage.t, with these contents:

`1:  2:  3:  4: `	`use Test::More; plan skip_all => "Test::Pod::Coverage 1.08 required for this test"   unless eval "use Test::Pod::Coverage 1.08; 1"; all_pod_coverage_ok();`

The test will find all the Perl libraries in your distribution, find all the public subroutines or methods, and then try to ensure that they're documented. It does this by finding all defined subroutines, ignoring ones that it thinks are private (anything starting with an underscore and Perl-wide things like import or DESTROY). Then it looks for =headn or =item entries. There's even a mechanism to supply your own coverage_class to, for example, allow documentation in parent classes to suffice -- that way you don't have to document new in every subclass you write.

Sometimes, though, you want to note that a symbol doesn't need to be checked. Maybe it's documented without having its own header, or maybe it's from a documented source more complex than an is-a relationship, like a role or trait. With Test::Pod::Coverage's default behavior, this is sort of a mess. You can either specify global options:

`1: `	`all_pod_coverage_ok({ trustme => [ qw( weird_doc_sub ) ] });`

...but this will trust that symbol to be documented in all your modules. So instead, you could test libraries one by one, expanding that all_pod_coverage call into a bunch of single-module calls with the right arguments:

`1:  2:  3: `	`pod_coverage_ok('Library::Alpha', { trustme => [ qw( weird_doc_sub ) ] }); pod_coverage_ok('Library::Beta', { trustme => [ qw( weird_doc_method ) ] }); pod_coverage_ok($_) for qw(Library::Gamma Library::Epsilon);`

...but this is a bunch more work, and you won't realize until it's too late that you forgot to add Library::Delta to the list of "no special treatment" routines.

Putting Trust Where it Belongs

Pod::Coverage::TrustPod is built on the idea that the best place to make a note about things already documented is in the documentation itself. You don't need to write a bunch of documentation for weird_doc_method, but you can make a note about it in the Pod. That way, every file has its own list of routines to consider already documented, but you can still call all_pod_coverage_ok.

When you write your library, you'd write something like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20: `	`package Library::Alpha; use parent 'Library::Common'; =for Pod::Coverage   new   weird_doc_method =cut sub new { ... } sub weird_doc_method { ... } =head2 documented_method This method is really documented! =cut sub documented_method { ... }`

And you'd write your t/pod-coverage.t file like this:

`1: `	`all_pod_coverage_ok({ coverage_class => 'Pod::Coverage::TrustPod' });`

That's it! Now all your documentation's coverage is easy to test, and exceptions are easy to maintain by putting the exceptions right where they belong: in the Pod document itself.

A final request...

Having Pod coverage tests is great. It helps you make sure that you documented things. What isn't great is when you let those tests run on the computer of anybody who's installing your code and they fail for some stupid reason. It prevents the user from installing your code just because it thinks you forgot to write enough documentation.

So, always write this t/pod-coverage.t

`1:  2:  3:  4:  5:  6: `	`use Test::More; plan skip_all => "author test only" unless $ENV{RELEASE_TESTING}; plan skip_all => "Test::Pod::Coverage 1.08 required for this test"   unless eval "use Test::Pod::Coverage 1.08; 1"; all_pod_coverage_ok({ coverage_class => 'Pod::Coverage::TrustPod' });`

That way, the tests will only be run when you're testing your distribution before release.

The Little Deferral Boy

2009-12-03T00:00:00-05:00

Avoid Unwanted Returns

I write a whole lot of code that takes a piece of mail and does something with it. Some of these are SMTP servers, some process mailboxes on disk, but at least half of them are pipemailers. A pipemailer reads a message on STDIN and exits with a code that lets the calling program know what happened. To grossly simplify, the possible exit values are:

0  - everything is okay; the message has been handled
75 - we couldn't handle the message right now, try again later
anything else - something is wrong; report failure and never try again

Why does this matter? Well, say you're using Email::Filter or Mail::Audit or some other Perl library for delivering your mail. I mean, you're not using procmail are you? One day you decide you're going to start checking your mail with SpamAssassin, so you copy this blurb from the documentation:

`1:  2:  3: `	`$mail−>exit(0); $mail−>simple(Email::Simple−>new($mail−>pipe("spamassassin"))); $mail−>exit(1);`

It's in the documentation, so it should be fine... but you've copied and pasted from perldoc and you're getting bitten by its stupid use of <U+2122> instead of a hyphen in the arrow operator. (This gets me about once a week.) Unfortunately, you don't notice, and now your program dies because perl has no idea what to do with that fake hyphen. Your program exits 9 (or something else seemingly random) and your mail server starts bouncing absolutely all your mail.

A test suite would help, and so would lots of other things, but Email::Pipemailer::DieHandler is dead simple, and a good first line of defense. When you install its "SIGDIE" handler, any fatal exception outside of an eval will be logged and will cause the program to exit 75. Your mail might not get delivered, but at least it won't get bounced. Hopefully you'll notice before it all times out!

Email::Pipemailer::DieHandler is a pretty tiny module. It's so short that I can reproduce all of its effective code right here:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16: `	package Email::Pipemailer::DieHandler; sub import {   my ($self, $install, $arg) = @_;   return unless $install and $install eq '-install';   $arg \|\|= {};   my $logger = $arg->{logger} \|\| sub {};   $SIG{__DIE__} = sub {     return if $^S; # don't interfere with evals     my ($e) = @_;     defined $^S and eval { $logger->($e); };     $! = 75;     die $e;   }; }

It might be the shortest module I've published on the CPAN, but don't let that fool you: this module has saved me quite a lot of embarassment by converting stupid bounces into errors that only I needed to see.

Making a List, Checking it 0b10 Times...

2009-12-02T00:00:00-05:00

Beyond Hex and Octal

It's sort of easy to convert between decimal, hexadecimal, and octal in Perl. sprintf can produce decimal, hexadecimal, octal, or binary digit strings, and the oct built-in can read those strings back into numbers. For example, to convert from hex to binary strings, we might do the following:

`1:  2:  3: `	`my $hex = '99'; my $num = oct("0x$hex"); # or we could use hex($hex); my $bin = sprintf '%b', $num; # ...and we get 10011001`

This is a little inconsistent, but fast and easy. Unfortunately, sometimes we need to work with really weird digit sets. Fortunately, Number::Nary makes these really easy to work with. For example, if we need to convert from sexagesimal to trinary (presumably because we are writing a Star Trek script) we can just create routines using Number::Nary like this:

`1:  2:  3:  4:  5:  6:  7:  8:  9: `	`use Number::Nary   -codec_pair => {     -suffix => '_3',     digits => '012',   },   -codec_pair => {     -suffix => '_60',     digits => [ map { sprintf '%02u', $_ } (0 .. 59) ], # 00 through 59   };`

Now we have four routines for converting between our numbering systems. Given the number 102012012 in trinary, we can easily convert it to sexagesimal:

`1:  2:  3: `	`my $trinary = '102012012'; my $number = decode_3($trinary); my $sexa = encode_60($number); # "021559"`

Neat, but is it ever practical? Yes! For one thing, it can be used to fix numbers into weird character sets. I've used it to cram several large numbers into email addresses by using the valid email address characters as a digit set. There's also Jesse Vincent's Number::RecordLocator, which "encodes integers into a short and easy to read and pronounce locator strings." Using Number::Nary, we can re-implement that module in about four lines:

`1:  2:  3:  4:  5:  6:  7:  8: `	`use Number::Nary -codec_pair => {   digits => [ 2 .. 9, 'A', 'C' .. 'R', 'T' .. 'Z' ],   predecode => sub { my $s = $_[0]; $s =~ tr/01SBsb/OIFPFP/; return $s }, }; # We silently replace some digits with others to keep things easy to read. ok(decode('1234') == decode('I234')); # 513430 ok(decode('10SB') == decode('IOFP')); # 491554`

Number::Nary is slower than sprintf and oct are, but since they can only handle a few different digit sets, they're not always going to solve the problem at hand. Having Number::Nary in the wings when you need it can be a real time-saver.

Finally, here's a quick bit of Christmas spirit:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19: `	use Number::Nary 0.107 -codec_pair => {   digits => [ 'ho ' ],   postencode => sub {     (my $s = shift) =~ s/ \z//;     "Merry Christmas, $s!"   },   predecode => sub {     my $s = lc shift;     $s =~ s/\Amerry christmas, //;     $s =~ s/!\z/ /;     $s;   }, }; # Appropriate: say encode(3); # Not: say encode(1);

Built in Our Workshop, Delivered in Your Package

2009-12-01T00:00:00-05:00

Exporting

In Perl, we organize our subroutines (and other stuff) into namespaces called packages. This makes it easy to avoid having to think of unique names for functions you write, and it means you don't have thousands of core functions to remember. Traditionally, exports were provided by the Exporter module, included with perl5 since its first release.

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10: `	`# You use Exporter in your module... package Gift::Dispatch; use Exporter qw(import); our @EXPORT_OK = qw(wrap label send); our %EXPORT_TAGS = (all => \@EXPORT_OK); sub wrap { ... } sub label { ... } sub send { ... }`

Now when someone says use Gift::Dispatch ':all' the three exported routines become available in their package. This is a useful tool, and most Perl programmers interact with Exporter all the time. Unfortunately, it's not very flexible. For example, here's our attempt to use Gift::Dispatch:

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12: `	`package Gift::Giver; use Gift::Dispatch ':all'; # for physical presents use Email::Send; # for online gift codes for my $gift ( @physical_presents ) {   send($gift, { carrier => 'UPS' }); } for my $gift ( @gift_codes ) {   Email::Send->new->send( email_for($gift) ); }`

Email::Send exports a send routine by default, which will clobber the one from Gift::Dispatch. This is obviously a contrived example, but this problem happens in real life, too, and generally ends up being more obnoxious to track down. Subroutines installed by Exporter also stick around forever, so when somebody ends up trying to call Gift::Giver->send, forgetting that the right method is give, they get one of your imported routines instead of a "no such method" exception.

Sub::Exporter and Naming

Sub::Exporter (sometimes affectionately referred to as S'Ex) makes it easy to fix this problem. First, we update Gift::Dispatch:

`1:  2:  3:  4:  5:  6:  7:  8: `	`# You use Exporter in your module... package Gift::Dispatch; use Sub::Exporter -setup => [ qw(wrap label send) ]; sub wrap { ... } sub label { ... } sub send { ... }`

And then we update Gift::Giver to muck around with the names (unrelated code omitted):

`1:  2:  3:  4:  5:  6:  7: `	`package Gift::Giver; use Gift::Dispatch ':all' => { -prefix => '_', -suffix => '_gift' }; for my $gift ( @physical_presents ) {   _send_gift($gift, { carrier => 'UPS' }); }`

Sub::Exporter and Code Customization

We got to muck about with the names under which things are imported. This is only the tip of the iceberg. Sub::Exporter can let you customize not just the names of imported routines, but also other facets of their operation. For example, we might not want to require the "carrier" parameter for our shipment above, if we can provide a default. It's easy to let Sub::Exporter handle an import like this:

`1:  2:  3:  4:  5:  6:  7: `	`package Gift::Giver; use Gift::Dispatch send => { -as => '_send_gift', carrier => 'UPS' }; for my $gift ( @physical_presents ) {   _send_gift($gift); }`

With arguments like that, we can build routines with built-in defaults, overridden behavior, and all kinds of customization, wrapped up in a simple name that we choose.

Over next few weeks, quite a few of the libraries that will be discussed will either use Sub::Exporter to provide exported functions or will use Sub::Exporter to provide powerful customizable interfaces. To really learn how these work, you should go to the Sub::Exporter::Tutorial, but here's a quick non-trivial sample using the canonical closure example of a "counter":

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14:  15:  16:  17:  18:  19:  20:  21:  22:  23:  24:  25:  26:  27:  28: `	package Counter; use Sub::Exporter -setup => {   groups => [ counter => \'_gen_counter' ], }; sub _gen_counter {   my ($class, $name, $arg) = @_;   my $cb = $arg->{callback};   my @log;   my %sub;   $sub{record} = sub {     my ($value) = @_;     $cb->($value) if $cb and @_;     push @log, $value;   };   $sub{list} = sub {     return @log;   };   $sub{count} = sub {     return scalar @log;   };   return \%sub; }

Then we put it to use...

`1:  2:  3:  4:  5:  6:  7:  8:  9:  10:  11:  12:  13:  14: `	`use Counter   -counter => {     -suffix => '_receipt',     callback => sub { print "sending thankyou note for $_[0]\n" },   },   -counter => {     -suffix => '_gift',   }, ; record_receipt($_) for @presents_we_got; record_gift($_) for @presents_we_gave; die "life isn't fair" if count_gift > count_receipt;`

Sub::Exporter is easier to use than Exporter and in its most trivial configuration provides much more utility. Learning how to make the most of Sub::Exporter can make very complex code generation quite simple and can reduce the amount of code you must write significantly.

RJBS Advent Calendar 2009

One Last Present, All Boxed Up

It&#39;s Boxing Day!

Perl! Perl is in the box!

Building Your Own Box

See Also

You Can Keep the Calendar

Merry Christmas! Advent is over.

In which our author fails to steal from his peers...

WWW::AdventCalendar

The Pod Transformers

Syntax Highlighting

Shorthand Lists

Same Code Next Year

See Also

Sending Email Simply

The Biggest Box Under the Tree

Sending Email Simply

Specifying a Transport

More Testing Tools

Seriously, Use It

See Also

Sorting Automatically, By Hand

See Also

Object Marginalia

Scribbling Notes on an Object

Alternate Sets of Methods

Alternate Storage Drivers

See Also

Gimme Everything You Got

Priming Your New Workstation&#39;s ~/code

Providing Other Sources

See Also

...aka Config::RJBS::AndNobodyElse::KnowsWTFItDoes

I&#39;m not apologizing!

It&#39;s Called &quot;Config::MVP&quot;

The Config Sequence

Reading Configuration

See Also

Restoring The Traditions of our Forefathers

The War on &quot;Finger&quot;

Finger My Git

The Future!

See Also

Glory to Pod in the Highest

Glory? To Pod?

Pod Templates with Pod::Weaver

Extending Pod::Weaver

See Also

Rewrite Email Without Worrying

Encoding and Email

Hiding the Encoding Process

MIME::Entity!?

See Also

More Stuff I Stole from Catalyst (that they stole back)

I Hate Typing

See Also

Like FUSE, but Stupider

From Filename to Thing

Multiplexing Resolvers

Putting it to Use

See Also

First-Class CLI Applications

Scripts are Our Friends

A Simple App::Cmd Program

Putting it to the Test

Organizing Complex Interfaces

Other Cool Stuff

See Also

Valid Data to All Nations

Params::Validate is Okay

Rx Schema

Coming Features (or: Missing Features)

See Also

E-mail Made Simple

Okay, I Lied

E&#39;Mail::Acme

Setting up the Header

Setting up the Body

Sending our Message

It's Boxing Day!

Priming Your New Workstation's ~/code

I'm not apologizing!

It's Called "Config::MVP"

The War on "Finger"

E'Mail::Acme

Like <DATA>, Only Less Stupid

Multiple "Files" Per Module

`sprintf` is My Favorite Chainsaw

Solaris Ain't So Bad

...and because it's Sub::Exporter...

It's Elf Bowling Time Again