This is just a quick post to provide a landing page for a new symfony plugin I’ve created: sfFilesystemFixturesPlugin. I’m not going to explain the plugin in detail here, all the information you need is on the plugin’s readme page.

In a nutshell, the plugin lets you populate a symfony project with filesystem data like images and thumbnails in much the same way as you would populate a database using regular symfony fixtures.

Feel free to leave questions, comments, and bug reports in the comments.

UPDATE:

I have now packaged this script into a symfony plugin called sfEnvironmentFixturesPlugin. Check it out and hit the ‘I use it’ link if you find it useful!

Feel free to leave questions, comments, and bug reports in the comments.

…

Symfony‘s fixtures are a great way to set up initial data and/or test data for a system. Whenever I create a new symfony project however, I always find myself wishing there was an easy way to specify which fixtures contain data required for production systems (the ‘initial’ data) and which contain data that I only need when developing (the ‘test’ data).

I recently started such a project and came across a reasonably elegant solution. With a simple YAML file placed in your fixtures directory, you too can have such a set up if so desired.

The beauty of symfony’s fixtures (at least in the case of Doctrine, not sure about Propel) is that they are interpreted as PHP. This allows us to do nice things like generating large amounts of test data by using iteration instead of writing out each entry individually. It also allows us to exploit it with a small script that can pull in other fixture files.

And now for the script itself:

byEnv.yml:

Place this script in your fixtures directory, then create a fixtures_dev sub-directory in your project’s data directory. Any fixtures placed in this directory will be included when you run the relevant symfony task using the dev environment.

You can use this system for any environment, including custom ones. For example, to load the fixtures needed for a production environment you could run the following:

symfony doctrine:data-load --env=prod

To load initial data, and test data stored in the development fixtures folder:

symfony doctrine:data-load --env=dev

To load initial data, and special data for the foo environment:

symfony doctrine:data-load --env=foo

IMPORTANT!

Be VERY careful when using this in a production environment. The default environment for the doctrine:data-load task is dev. This means if you don’t specify an environment, all your test data will be loaded.

The script includes some nice features too. It will search through directories recursively, just like the original fixture loading system, so you can organise your fixtures how you please. The script will also output messages to the console to notify you of which directories have actually been processed.

Hopefully someone will find this as useful as I do. Enjoy!

Subversion operations on the project in question were lightning fast. I could make a code change, and the commit operation would take only the blink of an eye. Likewise, updating other working copies with my changes was similarly instantaneous.

Then I included a large dependency as a Subversion external. From that point on, Subversion operations were very sluggish and it soon became a chore to update my working copies.

It quickly occurred to me that it was the externals definition that was the cause of the slowdown. Essentially, including another working copy as an external causes a huge amount of excess work for Subversion each time you perform an operation, particularly updates.

The problem

When you run an update, Subversion will contact the server hosting the external repository, asking for any updates, downloading them, and applying them as necessary. If you watch your Subversion client when you do this, it takes a good few seconds for Subversion to make the initial connection even when there are no changes. In addition to this, if you have a large number of directories and files in your external, the time it takes to traverse these can also have a sizeable impact on Subversion’s responsiveness.

So what are the benefits to externals?

Well, for one thing, if a third party wants to include your code in their own project, they ensure that all the dependencies are automatically satisfied upon checkout. But if you’re not writing a framework or some other kind of software library intended to be included within another system, then this is not such an important point. In fact if you’re like me and usually make one-time projects, the only time you would even use externals is during development, since it’s generally considered best practice not to use Subversion in a production environment at all.

The solution

Here is what I do in my projects to keep Subversion responsive and fast. I would typically have a directory for my dependencies; let’s use the symfony convention and call it lib/vendor:

In a typical Subversion externals situation you might set the following for the svn:externals property of the vendor directory:

Instead of doing the above, I would simply set the svn:ignore property of the vendor directory by running the following command (from the root of the project):

…and then run the following commands from the root of the project to create the individual working copies for each dependency:

The end result is largely the same, except that Subversion will completely ignore your dependencies when performing an update, resulting in a drastically reduced amount of time needed to perform the operation.

Of course there is a slight disadvantage here, in that you must manually update all your dependencies when required. Realistically, this is not a problem most of the time unless you are including a software library that undergoes rapid changes. With that said, I usually write a small script to handle the work for me, like so (example in PHP):

update_repos.php:

repos.php:

Place these two files in the root of your project, configure repos.php and you’re all set. You’ll never need to use an externals definition again! Well, maybe…

The script below uses PHP’s Imagick extension to create a thumbnail from pretty much any source image. It can handle RGB & CMYK images, TIFFs, PNGs, JPEGs, GIFs, BMPs and just about any other format in just about any size, and it can also handle images with embedded colour profiles (this is very helpful when converting images created in Adobe software).

You’ll firstly need to grab the standard sRGB colour profile from the International Color Consortium (ICC) website. Place this somewhere accessible to your script and you’re ready to go.

I recently started learning how to use the forms system in symfony 1.2 and for the most part I was impressed with its simplicity and flexibility. Unfortunately I was thrown in the deep end somewhat when a project I was working on had a requirement for an “expanding” form – that is, one where a certain section of a form should be able to be repeated as many times as necessary to accommodate the user’s needs.

In the case of the project in question, the specific situation was an application form where the user could fill out any number of team members involved in their project. Each team member consisted of a name and a role, both free-form text fields. Team members were to be stored in the database, linked to the application in a one to many relationship.

Circumstances conspired that this must be achieved in a single form, so full of confidence in symfony’s design decisions regarding the forms system, a bit of light Googling revealed the answer – symfony’s embedded forms system. Great, so you just plonk in a few embedded forms and away you go, right? Wrong.

The good news is, I managed to find a solution that satisfied the requirements, eventually. The bad news is that it is not easy, elegant, or particularly reusable.

What follows is a loose guide to implementing such a system. Being a relative newcomer to symfony I cannot guarantee that this is the best way to do things, so tread with caution, and if you think of a better way to achieve the same outcome, please please let me know! Read on if you dare…

Introduction

I’m going to assume the following in this guide:

• You have a solid working knowledge of symfony, PHP and Doctrine.
• You have a blank symfony 1.2 project, set up to use Doctrine rather than Propel (although I’m sure the same could be achieved with Propel with a bit of tweaking).
• You have the symfony executable in your system path or in the root of your project.
• You are using a relational database such as MySQL to store your data.
• You have entered your database connection details into your config/databases.yml file and set up all the appropriate database permissions.
• You have completed any web server setup as required to access the project.
• You are aware that you need to clear the symfony cache (by running symfony cc) when new classes are created and when configuration changes are made.

I’m also going to move fairly quickly as there is a lot to cover, so please feel free to ask for any clarification in the comments and I will update as necessary.

For help with installing and setting up symfony 1.2, check out Getting started with symfony. I chose to use Doctrine 1.1 for this tutorial, however the version that comes with symfony should also be fine. For instructions on upgrading to Doctrine 1.1 see Using Doctrine 1.1 with Symfony 1.2.

You can also see a live demo and get an idea of what we are trying to achieve by visiting http://expandingform.sf.ezzatron.com/.

Step 1: Setting up

Here is the Doctrine schema we will be using. I went for something a bit more cheery than the above situation, namely bands and band members:

Place this in your config/doctrine/schema.yml file, and then create a data fixture so we have some test data to work with:

Place the above in data/fixtures/fixtures.yml and run:

This will generate all the model classes and forms we will need, create the database, and populate it with some test data for us. Next we need to create an application, and we’ll use the standard symfony app name “frontend”:

Replacing “randomstring” with whatever you like. Now let’s generate a module from our model classes as a base to start from:

Now if you visit http://<project url>/frontend_dev.php/band/edit/id/1 you should be presented with the following:

We now have a basic form for entering band details. Thus concludes step 1. Moving on to…

Step 2: Building the form

OK let’s dive right in. We are going to embed forms for editing band members inside of the band form itself. To add a single member form you would simply add:

to our lib/form/doctrine/BandForm.class.php‘s configure() method. But what we want is to structure the form in such a way that the form data, when posted to the update action, will contain an array of band members. The way to do so in the symfony forms system is to wrap the embedded forms in another form, then embed this wrapper form itself into the main form. Confused? Don’t worry, everything will become clearer once we actually create the forms.

Firstly, we need to make a small modification to the band member form to allow it to slot into our band form better. The generated form includes a field for the related band ID, which we will not need as all band members will relate to the band represented in the parent band form.

We could modify the BandMemberForm class itself, but it’s possible we may want to use the form outside of a band context, so let’s harness the power of the forms system and extend the form instead. Create a file at lib/form/doctrine/EmbeddedBandMemberForm.class.php with the following class definition (PHP opening tag excluded for brevity):

Next, let’s create the “container” form. Create a file at lib/form/doctrine/BandMembersForm.class.php containing the following class:

This creates three blank forms for inputting band members. Now to actually embed the container form into our main band form, it’s actually very simple. Simply edit lib/form/doctrine/BandForm.class.php so that it looks like the following:

Now when we reload our band form (did you remember to clear your cache?), we will see the following:

Looks promising, and that wasn’t too painful so far, right?

There next obvious step would be displaying the existing band members. We want to be able to see and edit the existing band members and still have the option of inputting new members also. To do this we need to pass the Band object to our container form so that it can access the existing members. Let’s update the container form so that it looks like:

Now we have to change the main form so that we pass the Band object when we instantiate the container form:

Reloading the form should now show the existing members:

So now we have a form that can display the band with all of its members and still provide room for new ones. Time for the next step.

Step 3: Getting the form to save

Try saving the form without making any changes to the data. You should get validation errors because the band member name is a required field:

This won’t do at all because we want to be able to leave a band member blank and have the system ignore it. We also want to be able to delete an existing member by clearing their name and role. So let’s remove the requirement for now, and we’ll add it back in later. Edit your EmbeddedBandMemberForm class so that it looks like this:

Now when we try to save the form again, we get an exception thrown:

As you can see, Doctrine is not happy with us, but why? The answer is that symfony is trying to create new band members from the blank band member forms, but it has no idea which band the new members should belong to. We’ll deal with this issue shortly, but in the meantime let’s start by making symfony ignore any band member forms which are left blank.

Firstly, how do we decide if a band member form is “blank”? Let’s write a couple of custom methods in our form classes. Some of this functionality is reusable, so we’ll place the main body of the code in lib/form/doctrine/BaseFormDoctrine.class.php which is the base class for all of our generated forms:

These two methods essentially traverse an array recursively looking for any non-blank values, and returns false as soon as one is found. The method needs to be recursive because some fields like dates are passed as an array. Now we can write a simple method in our EmbeddedBandMemberForm class to check whether the fields that constitute a band member have been filled out:

We are simply grabbing the field names directly from our model class, and excluding the ones we don’t care about. This is more robust that explicitly specifying name and role as the important fields, as it allows for further fields to be added to the band member form as requirements change.

Now we need to actually change the way the form is processed by overriding some methods in our main band form:

These methods doubtless require some explanation. Firstly let’s look at updateObjectEmbeddedForms(). This method uses the custom method we wrote to check whether the band members passed to the form for processing are blank. If a blank band member is encountered, the related form is removed from the forms array to prevent symfony from attempting to update the related band member object.

The next method, saveEmbeddedForms() works in tandem with the above. It runs through each of the forms and checks whether the related object has been modified. If an unmodified object is encountered, the related form is once again removed from the forms array to prevent symfony from attempting to save an unmodified object.

OK so now we should be able to save our form without any validation warnings or exceptions cropping up. If you can’t, it’s probably time to double check everything, and perhaps clear your cache. On to the next step…

Step 4: Adding and deleting members

Our form now saves without errors, but what does it actually do? Not much at the moment. Let’s try adding another band member. Ladies and gentlemen, may I present Steven on second guitar!:

What happens when we save the form? Ah, it seems we have the same issue we encountered before:

As we found out in the previous step, this is caused by symfony not knowing to link our new band member to the band we are editing. We can fix this by making a simple change to our BandForm class, in the saveEmbeddedForms() method:

Now when we try again, the new member is saved correctly and appears in the list of band members as expected:

Awesome! We now have 2 guitarists and all the glory that comes with duelling guitar solos. But wait; oh no! Steven has decided to get married and move to Norway! Well, it’s time to remove him from the band I guess, so let’s try setting his name and role to blank and see what happens when we save…

Oh dear, it seems Steven is not keen on leaving without a fight. He still appears in the list of band members. So what’s happening here? In short, we simply haven’t told Doctrine to dispose of the unwanted band member yet! Let’s go ahead and do that now by modifying the BandForm class’ updateObjectEmbeddedForms() method:

Now we can try again to delete Steven, and it seems to have worked:

So now we have a fully functional form, which is great! But there are some important things we still need to consider, which leads us to the next step…

Step 5: Validation

That’s right, you forgot, didn’t you? Back in step 3 we disabled important validation for the band member form, which means, for example, if you try to enter a band member with a role but no name, you get this nasty validation exception:

NOTE: You may find that you do not encounter this exception, there seems to be a bug somewhere where the “notblank” validator we specified in the schema is being ignored. Nevertheless, we still need to prevent blank names from being entered.

We can’t leave things as they stand, so let’s create a custom wrapper for the validator schema that we can use to validate band members that have not been left blank. Create a file at lib/validator/EmbeddedBandMemberValidatorSchema.class.php and enter this class definition:

All this class does is check whether the passed VALUES
are considered “blank” before applying the existing validation rules. This increases robustness by allowing us to keep all of the validation rules defined in the BandMemberForm class, and freeing us from having to duplicate their logic in the EmbeddedBandMemberForm class.

Now it’s simply a matter of making a small change to the EmbeddedBandMemberForm class’ configure() method to allow us to take advantage of our new wrapper class:

With those changes made, we can try again to save a band member with only a role and this time we are presented with the appropriate form errors:

OK, that’s validation taken care of, let’s make some usability improvements.

Step 6: Usability

We’re going to add extra functionality to our form for users with JavaScript support, namely the ability to add additional spaces for band members without the need to save the form. What’s more we’re going to do it without preventing non-JavaScript enabled users from using the form. This is known as progressive enhancement and is generally considered a best-practice in web development.

The first thing we need to do is set up an action to render the additional band member forms so that we can fetch them via an AJAX call. Open up your apps/frontend/modules/band/actions/actions.class.php file and add the following method:

The template for this action is very simple. Create a file at apps/frontend/modules/band/templates/memberFormSuccess.php with the following contents:

You may have noticed the call to BandForm::setNumMembers() in the above action. We still need to add this method, so let’s do that now. Edit your BandForm class and add the following method definition:

This in turn means a change to the BandMembersForm class, which should be edited to look like this:

What the above changes do are allow us to insert extra blank band members as we please. We then use this to recreate the “extra” band member form exactly as it would appear if it were rendered as part of the whole form.

All that is left to do is create the AJAX handler and some unobtrusive JavaScript to fetch and insert the new member form. Open apps/frontend/modules/band/templates/_form.php and update the contents to the following:

It’s important to note that this code is somewhat ugly because I wanted to avoid manually changing the form’s layout. In a production situation you could implement a much more elegant solution by customising the form layout or writing a custom form renderer.

This point aside, your form should now look like the image below when JavaScript is enabled (when JavaScript is disabled a message will be displayed informing the user of how to add additional members):

We’re now able to add additional slots for new members dynamically using the “Add member” button. We’re facing one last hurdle unfortunately; when we add a new member slot and hit the save button we’re presented with a form error:

Thankfully this is fairly easy to remedy. The issue is that when the update action handles the posted data, it creates a fresh instance of BandForm which has no knowledge of our “extra” band member form. To fix this we can override the BandForm class’ bind() method to detect the number of band members passed and expand the form as needed:

And hey presto, our form now works perfectly!

Conclusion

If you got this far I’m sure you’ll agree that this was not by any means an easy exercise. The symfony forms system is powerful and flexible, but it seems like perhaps this type of situation was not really brought into consideration when it was designed. I would love to see the symfony architects consider ways that the implementation of expanding forms could be simplified, or at least better documented.

Once again if you have any suggestions on how to improve this implementation I’d really love to hear them. I’m sure there are aspects of the symfony forms system that I don’t understand fully which could make this whole exercise somewhat simpler, and that would certainly be valuable knowledge.

If you are interested, there is a guide to implementing a similar system here, however it seems to deal with the symfony admin generator specifically, and it also seems to have the disadvantage that the parent object must exist before child objects can be added.

Good luck with your own implementations, let me know how they go!

Source code

• _form.php
• actions.class.php
• BandForm.class.php
• BandMembersForm.class.php
• BaseFormDoctrine.class.php
• EmbeddedBandMemberForm.class.php
• EmbeddedBandMemberValidatorSchema.class.php
• fixtures.yml
• memberFormSuccess.php
• schema.yml

This is very handy, for example, when developing an applet that requires permission to access the file system. This requires the applet to be “signed” using the jarsigner utility, and this quickly becomes a tedious step when testing changes made in your applet in the browser itself.

I’m assuming that since you’re reading this article you have basic knowledge of Java and the keytool and jarsigner utilities. If not, there is a good introduction here.

Please note that this how-to is intended for a development environment situation only. Before deploying to a production environment you should read up on the best practices and security implications of the .jar signing process.

Step 1: Setting up your keystore

For this step we will be using the keytool utility distributed with your JDK. This will most likely be in the bin directory of your JDK installation.

Assuming keytool is in your system path, run the command:

Replacing <alias name> with a suitable alias, and <keystore path> with the filename you’d like for your keystore.

You will then be prompted for a number of inputs. For a test setup, use the following answers:

This will create a keystore file at <keystore path> which we will use in the next step to sign .jar files. Remember the passwords you used in this step as you’ll need them very soon.

The -validity 36500 section simply specifies that we want this key to last for a very long time (100 years to be exact). This is fine in a development environment as you probably don’t want to have to regenerate new keys all the time, but in a production environment this would obviously be much shorter. The -validity value is specified in days.

Step 2: Configuring your NetBeans project

In this step we will be editing two configuration files from your NetBeans project; project.properties and build.xml.

The project.properties file should be located in the nbproject directory of your NetBeans project, and the build.xml should be in the main project folder itself.

You can edit these files by hand, or you can do so in NetBeans itself by switching to the Files window.

Add the following lines somewhere in your project.properties file, substituting the relevant paths etc. (hint: your jarsigner utility should be in the bin directory of your JDK installation):

Add the following inside the <project> block in your build.xml file:

And that’s it. The next time you build your project, a signed copy of the .jar file should be created automatically at the path you specified. You should also see a confirmation message in the build output.

Enjoy!

Why implement it in PHP code rather than make a change to the syntax itself? Well, the reason is fairly simple; having too many syntaxes for the same thing can make code very difficult to read and understand unless the reader is a guru in that particular language. Anyone who has been forced to use Perl will know what I mean.

The above comment suggested the use of the eval() function. Whilst this would work, I don’t particularly like the idea for a number of reasons:

• If there is an error in the eval’d code, this will show up in a back-trace as occurring in eval’d code rather than pointing to the actual problem itself – messy.
• If like me, you are used to a nice, cushy IDE, you’ll know that almost all of your code completion / highlighting immediately goes out the window inside of a string.
• I try to avoid eval() as a rule of thumb. It just makes me feel dirty.

I thought I might try a similar solution however, using PHP 5.3′s new closures feature. Closures (A.K.A. anonymous functions) will most likely be familiar to anyone who has used JavaScript in any serious fashion, so I will not go into them here. If you are unfamiliar with the concept, please RTFM.

In order to make this work there are a few things we need:

• An array to iterate over
• A condition of some sort to evaluate on each loop
• A callback to execute if the condition evaluates to true
• Some measure of control over variable names passed to the callback would also be nice

The bottom line is that it is possible, but it’s nowhere near as nice, or elegant as I had hoped.

Method 1: using closures

Using nothing but the new closures syntax I was able to come up with the following reasonable-looking foreachif() function:

This seemed fine until the time came to actually use the thing. Reproducing one of the examples from the aforementioned article resulted in this mess:

Not particularly readable huh? Downright horrible is more like it.

Method 2: mixed closures and eval’d strings

The above mess made me think perhaps I could just use eval() for the condition – perhaps this would help remove some clutter from the statement? Well, yes it does, but it also presents its own problems. Here is the new function:

In order to retain flexibility in naming the variable used to store the current iteration value, I was forced to pass the variable name as a string. I didn’t like this at all, and the situation would only get worse if you wanted to be able to choose the name of the array key inside the condition also.

For the sake of completeness, here is an example the utilises the above function:

Then suddenly I realised I was forgetting a very, very important point; variable scope.

The big problem

What neither of the above implementations give you is access to any variables defined outside of the closure / eval’d code itself. Granted this could be worked around by grabbing every defined variable and re-defining them within the scope of the loop content, but this would almost certainly have major performance issues when used in any real-world situation.

At this point I basically abandoned the search for a function-based implementation and started considering PHP syntax alternatives, which turned out to be a much more elegant solution in the end…

Method 3: using existing PHP syntax in alternative ways

Starting with the ultra-obvious solution gives us the following:

There is also PHP’s alternate syntax for these structures to consider, and I think the following actually comes out looking quite nice:

The main area in which I intended to use this structure was in native PHP templating. In line with this thinking, here is the above restated as I would present it in a template:

As you can see, these last three examples are actually quite readable (the last one as much so as native PHP templating can be). The big bonus of course is that they still allow access to all variables in their scope and have no significant performance impact (provided of course that your condition is not intensive on resources).

There are so many reasons why this should not go ahead. I won’t go into detail here, but you should learn more if you don’t know what I’m on about. Try Horus Kol’s article, or No Clean Feed for some good information.

Please use this nifty form to show your support for the fight against mandatory internet censorship:

In addition, there is another petition you can sign at TakingITGlobal.

If you missed part 1 (you probably did), you can read it here.

“What the hell is a reverse combined operator?” I hear you scream. Well, technically they probably don’t exist yet, so I’m sure a better name could be found, but bear with me and I’ll try to explain…

If you’ve ever done any programming, then you’ve almost certainly used operators whether you realise it or not. Basically, they are some of the most basic building blocks of a programming language, things like +, -, = and == are all operators. For a more detailed description, see the PHP manual.

One of the most commonly used operators in any language is the assignment operator (=). This operator is used to assign a value to a variable. Here is a simple example:

In the above example, PHP first uses the addition operator to get the sum of 1 and 1. It then stores the result in $variable using the assignment operator.

Now consider the following situation; you have a variable that you want to add a certain amount to. Rather than writing the whole expression out like this:

PHP allows you to shorten the above to just:

This is called a combined operator because it is effectively a combination of the + and = operators. This combining of operators is only ever done with the assignment operator and one other operator. Here are all the combined operators that I am aware of:

What I would like to see are reverse combined operators. Essentially these would be the same as regular combined operators, except with the order of operands reversed. This would produce the following options:

Note that not every combined operator would imply the need for a reverse version because the order of operands has no effect on the result in some cases.

I’ll now provide you with a couple of examples where something like this could come in handy.

Inverting a number:

Controlling which array takes precedence when merging arrays using the array union operator (basically just the + operator, but when used on arrays you get the union of the two):

I’m sure if you racked your brain you could come up with dozens of other handy little uses for this feature. Let me know if you can think of any others!

Sometimes though, I do wish for something more. Never enough to switch teams (sorry Python), but there are a few points I keep coming back to. This, in fact, is at the present moment the sole reason for this blog’s existence – I simply don’t have anywhere else to voice my frustration.

So, without further ado I give you part 1 of my PHP wish list: the foreachif() construct!

The foreach() construct is a feature of many modern programming languages, including PHP. In PHP’s case, there are two syntaxes:

foreach (array_expression as $value)
    statement
foreach (array_expression as $key => $value)
    statement

Basically it will iterate over every value in an array but it has advantages over a regular for() loop in cases where the array keys are non-sequential (and it can also iterate over an object’s member variables).

Whilst foreach() is a very useful construct in itself, I too often find myself wrapping everything inside the foreach() loop in one big if() statement:

Now, imagine if there were a new construct formed by combining the above foreach() with PHP’s basic if() construct…

if (expr)
    statement

…to form a new construct, foreachif() with the following syntaxes:

foreachif (array_expression as $value; expr)
    statement
foreachif (array_expression as $key => $value; expr)
    statement

Where statement would be executed for each value in array_expression where expr‘s Boolean value equates to TRUE.

The expr part of the syntax would have to be evaluated last so that the logic of expr could involve the use of $value (and $key in the second form).

For example, the following code (which will echo only even number VALUES
from an array):

Could be written much more concisely using foreachif():

Of course some people would call this lazy, but to me it feels natural. In any case, don’t hold your breath, due to a clear lack of any real need for this feature it’s doubtful this will ever make into PHP 6. Perhaps PHP 10?