As I was clearing out my RSS reader a few days back, I stopped to read a post on the Flickr dev blog about the “new” 640px image size. In the article they lament that they wished they had made “Large” copies of all uploaded images because it would make generating the new 640px size easier.

This got me thinking, what if they did want to convert every image ever uploaded to this new size and cache it? How would they do it? How much would it cost? The more I thought about it, the more curious I became, so I launched a little research project to see what I could find out, and to show how I would work through the problem.

Approach

How to resize?

I tried searching to find out exactly how Flickr is resizing their images, but it is apparently a well kept secret. There was a fair amount of speculation that they use Lanczos resampling, and my own research suggested it was a strong choice for quality resizing, so I chose to use it for the purposes of this experiment.

I wrote a very quick PHP script that uses ImageMagick with the Lanczos “filter” to do the work. This resizer was built with this particular experiment in mind, but it is fairly flexible. If you POST an image, and a value “largestDimInPx” it will resize the image and send it back as the response to the HTTP POST, thus the “RESTFul” name. The resizer also accepts an array of parameters which it sends back in the HTTP header. So if you were actually taking on a mass resizing project like we’re proposing, you can pass the unique ID of the photo in your system and get it back when the resize is complete.

You can find the code for it over on my GitHub.

Just how many images we talkin’?

Another carefully guarded secret it seems, is the total number of images that Flickr is currently hosting. I did a lot of searching and tried a few naughty tricks to see if I could get the API or website to tell me, but it was to no avail.

Some previous “official” numbers were as high as 6 billion images. So for the purposes of my test I assumed that they had 10 billion images that they wanted to resize to this new 640px image size.

CPU Cycles – Pick your poison

I have to assume that Flickr has a pretty impressive infrastructure built out, and that they have a huge amount of processing power at their disposal. Still, to accomplish something like this it’s unlikely that they’d reallocate existing resources. The task is too monumental to just have a couple servers work at it.

I also have to assume that they’re carefully optimizing their servers so that they don’t have to build and maintain more than necessary. So that begs the question, how do you get the processing horsepower necessary to convert all of these images?

Since I’ve recently been engrossed in creating and refining reusable RightScale ServerTemplates for our own servers in the AWS cloud, my thoughts for a solution instantly gravitated toward launching a bunch of servers in the cloud for this purpose then terminating them when the task was done. What I didn’t know is if this made more sense than, say, buying a bunch of servers and configuring them to do the same task.

In order to do some tests, I created ServerTemplate which can be launched on AWS using RightScale that will immediately be able to start resizing images posted to it using my aforementioned PHP resizer.

The experiment

What I learned in short was… This is a HUGE undertaking! I also found that while any approach would probably be cost prohibitive, performing this resizing task using on-demand servers in the cloud is still the cheapest route by far.

Assumptions

First, some of the assertions for this exercise.

• The server(s) performing the resize task will be of the m1.large type from AWS
• Flickr has 10 billion images to convert
• Those images are (on average 12MP and about 4MB
• The images are being resized so that the largest dimension is 640px

You’ll also find that as I describe the results, and the costs they represent, I conveniently exclude data transfer from my numbers. This is actually intentional. My goal is to compare just CPU time.

Yes, bandwidth has a tangible cost when you use it in the cloud. But the same is true for a bare metal server doing the same task, not to mention the networking hardware which must be acquired, configured, and maintained. So for the purposes of this experiment, only the cost of CPU cycles is really being examined.

Findings

With my new m1.large instance launched in the AWS cloud, and exposing the resizer service, I did a few tests to get a feel for just how long it would take to do all of this resizing. What I found out pretty quickly was that the actual task of resizing the image actually only took around 1 second, sometimes less! It should come as no surprise that the real time consumer in the process was I/O bandwidth.

Over my home cable connection (measures at 3.5Mbps up and 11.83Mbps down), using Firefox to post and FireBug to measure, it took an average of 12 seconds to complete the process of posting the image and receiving the response. If I removed the overhead of Firefox and simply used a bash script and cURL, that dropped considerably to 8 seconds.

In an effort to get a “best case scenario” measurement, I copied the file to be resized onto the AWS instance, and ran the same cURL command locally. On what can be effectively called “infinite” bandwidth, the process of posting, resizing, and fetching the result took about 2 seconds.

The tortoise and the hare

One very tangible benefit to using the cloud for this purpose though is the time to market for such a huge task. You can imagine that if Flickr did decide to go ahead with this, they wouldn’t really want to announce that they’ve started but that the process will take many months, or even years. Assuming again that the per image resize time is 2 seconds, it would take one server just over 633 years to convert all of the images. Something tells me that wouldn’t exactly go over well.

Say that they wanted to convert all of the images in one week, at 2 seconds per image for 10B images, it would require just over 33,000 servers to convert that many images in the cloud. What’s more is that it would take effectively “no time” to deploy that many servers, use them for a week, then terminate them. Try rolling out that many real, bare metal servers in that timeframe!

What’s it gonna cost?

You might look at those results and think that these times are fairly reasonable. However, if you take these figures and start multiplying them by ludicrously large numbers (like 10 billion for example), those reasonable numbers start to look very unreasonable.

Using these results, and (as I stated before) only accounting for CPU time, it would cost approximately $1.9M for Flickr to convert all of their images in the cloud. That is assuming that they’re connecting to the servers with effectively infinite bandwidth, ensuring that the upload and download transfer times for the image were less than 1 second (combined!). Seems like a lot of money, but then it’s a lot of images!

Compare that though, to building out a farm of bare metal servers to do the same task. We’ve already established that we’d need a little over 33,000 servers to be able to convert all of the images in a week. If we just take the lowest configuration Dell rack mounted server, the R-415 at a retail price of $4531, it would cost just over $149M, just for the hardware. Worse still, the Dell doesn’t actually have as much processing horsepower as an AWS m1.large does, so you’d probably significantly effect the time per image.

Conclusion

At the end of the day, any approach for converting all of those images would be cost prohibitive, and I’m betting Flickr won’t be taking on this project. Still, if I had to accomplish this task for real, I think I’ve demonstrated fairly convincingly that doing it in the cloud with on demand computing power is totally the way to go.

I’m not a mathematician or a statistician, and I did paint this topic with very broad strokes. You can check my work with the simple Google Spreadsheet I created to track my results. If this were an actual task, I’d be a bit more precise, but I wanted to show the differences between the approaches, without getting into too much minutia.

That said, discussion about factors I may have missed are welcome, I think you’ll find that the delta in price for just the CPU horsepower between the cloud and a bare metal solution more than absorb any real differences I may have excluded.

Over the past few weeks I’ve been spending most of my spare time developing a couple of web sites.  These sites are based on the Drupal CMS written in PHP.  And I’ve taken a bit of a crash course in Drupal module design.

When I set out developing these, I faced a challenge that I’d been aware of, but managed to avoid in the past.  How do I work on these projects in a predictable, constant environment, and still bring my code with me to work on my home machine, work machine, laptop, and anything else I happen to sit down in front of?

This blog post is about how I’ve accomplished this so far.  Read more after the break…

Portable Storage Device

Your first inclination will probably be to install these tools on a USB thumb drive that you can easily store in your pocket.  And you’d be in good company, this was my first choice too.  However, after getting everything going, I quickly realized that it simply wasn’t fast enough.  Others have reported different results, and a lot has to do with the particular device, and the file system (NTFS is best).

That said, I chose to go a different route, with fewer unknowns, and picked up a Seagate FreeAgent Go 250GB USB harddrive.  I picked it up with a docking port from our local Costco for $80 and it eliminates concerns I had about performance.

File Structure

Throughout the rest of this article, I’m going to refer to files in my file structure, so I may as well show it to you so that these references can make sense.  This probably isn’t the best layout, but it’s what works for me.

• FreeAgent Go Root
  • ampp
    • lampp
    • wampp
    • shared
      • htdocs
        • <code>
      • mysql
        • data
          • <databases>
  • Portable
  • cygwin
  • libs
    • java-runtime
      • jre1.6.0_06
  • Program Files
  • 7-ZipPortable
  • eclipse
  • FirefoxPortable
  • junction
  • Rapidsvn
  • soapui-2.5.1

Foundation (AMPP)

The first thing we’re going to needed is a good Apache, MySQL, PHP setup.  For this I did some quick googling and stumbled upon XAMPP.  This is an Apache Friends project that packages all the server side components I need.  You can check them out, and download it over at their site.

I just downloaded the zip file, and extracted it to my drive at \ampp\wampp.  If you don’t unzip this project to the root of your device, you’ll need to run the “setup_xampp.bat” batch script supplied.

Shared Source & Database Files

For the sake of organization, and the ability to run this system on both Windows and Linux boxes (see notes below about linux), I’m keeping my htdocs and MySQL database files outside of the normal directory structure of the XAMPP installation.

See my File Structure above..

This way all of the web files for my individual websites and projects go in the “htdocs” directory, while the database data files go in the “mysql\data” directory, making them easy to find.

Of course, this presents a problem since the XAMPP install won’t be looking in those directories for the databases and code.  So, to solve this I’ve written another small batch script which lives in the “shared” directory.  This script uses the microsoft system tool Junction.exe to create links of my shared directories in the location that XAMPP is looking.  The junction.exe file is in an “Installers” directory on the drive, though I should probably move it somewhere more logical.  Here’s the script that deletes old junctions, and creates new ones.  You’ll want to run this when you move from computer to computer, since the drive letter of the USB drive will likely change, and the junctions are to absolute paths.  I put this file in \ampp\shared and it uses relative paths, so that’s important.

refresh-junctions.bat

[text toolbar="false"]for /d %%A in (htdocs\*) do "..\..\Portable\Program Files\junction\junction.exe" -d "..\xampp\%%A"
for /d %%A in (htdocs\*) do "..\..\Portable\Program Files\junction\junction.exe" -s "..\xampp\%%A" "%%A"

for /d %%A in (mysql\data\*) do "..\..\Portable\Program Files\junction\junction.exe" -d "..\xampp\%%A"
for /d %%A in (mysql\data\*) do "..\..\Portable\Program Files\junction\junction.exe" -s "..\xampp\%%A" "%%A"
[/text]

The requirement here of course is that you always create new sites by creating a directory in \ampp\shared\htdocs, and that any time you create a new database, you must move it’s directory from \ampp\wampp\mysql\data\<database> to \ampp\shared\mysql\data\<database> then re-run the refresh-junctions.bat script while the MySQL server is not running.  But it’s a fairly workable solution.

Integrated Development Environment (IDE)

Okay, so we’ve got a nice predictable, repeatable hosting environment.  Now to  write some code!

My personal choice of IDE is Eclipse with the PHP Development Tools (PDT).  Eclipse is by it’s nature portable since it’s a Java application, and the installation for the PDT is just to unzip it to a drive.  There are a couple of pitfalls though, and here’s what I’ve done to overcome them.

First, since Eclipse does require the Java runtime you have two choices.

1. Make sure that a current JVM is installed on any machine you intend to run your portable environment on.
2. Carry a copy of the JVM with you!

I chose #2, and used instructions found here to do it.  Basically, store a copy of the JVM on your disk and edit the Eclipse startup batch script to use your portable copy.  In my case, the JVM is in the Portable/libs/java-runtime directory as shown in my File Structure above.

Secondly, I thought it would be a good idea to just store my workspace on the portable drive as well, so I could keep all my settings with me.  Thus far I haven’t found an elegant way to do this, since it apparently uses absolute paths for projects.  So instead I keep a workspace on each system I work on that has the settings.  I hope to find a better solution to this, and if I do I’ll post it here.

Portable Firefox Browser & Plugins

Alright, so now we’ve got a web server to show our code, a nice IDE to write code in.  What we need is a predictable way to view it all.  You could install your favorite browser on every computer you interact with, or you could bring it with you!  In my case, Firefox is my very favorite browser so I used Firefox Portable so that I can take it and all my configuration preferences as well as Firefox Add-Ons with me.  My core set of Add-Ons which I find invaluable are..

• Firebug
• IE View
• Web Developer

Portable *NIX tools with Cygwin

So we’ve written some code in our portable IDE, we’ve tested it out using our portable web server and Firefox browser and we’re satisfied with the results, and want to upload them to our production server.  My very favorite method of doing so is over RSYNC so I transfer only what’s changed, and it’s secure.  In order to do that, I usually use cygwin.

On the surface it seems like it’d be easy to just install cygwin to a directory on your portable drive.  And in part, it is.  You can simply install to that directory, and you’re all set.  Problem is as soon as you move it to another computer, and that computer assigns your drive a different drive letter, everything breaks down real quick.  Some quick google searching revealed a way to make this work.  I had to make some minor tweaks in order to fit within my File Structure but you can find the inspiration for my changes here.

My changes are to the X.bat and uninstall.bat files.  I instead called them ~cygwin-install-X.bat and ~cygwin-uninstall.bat and put them in the cygwin install directory at Portable\cygwin.  Here they are.

~cygwin-install-X.bat

[text]for /F %%A in (‘cd’) do set WD=%%A
bin\mount -m | bin\sed s/mount/"%WD%\/bin\\/mount"/ > tmp\mount.log
bin\umount -c
bin\umount -A
bin\mount -bfu %WD%/ /
bin\mount -bfu %WD%/bin /usr/bin
bin\mount -bfu %WD%/lib /usr/lib

set path=%path%;%WD%\bin;%WD%\usr\X11R6\bin
start bin\rxvt.exe -title "" -bg "#fafad2" -fg "#000040" -color10 green4 -color14 brown -fn "Lucida Console-14" -geometry 80×58+0+0 -sl 4000 -sr -tn rxvt -e /bin/bash –login -i
set DISPLAY=localhost:0.0
run usr\X11R6\bin\XWin -multiwindow -emulate3buttons 200
[/text]

~cygwin-uninstall.bat

[text]bin\umount -c
bin\umount -A
bin\bash tmp\mount.log
bin\rm tmp\mount.log
[/text]

Portable Source Control (SVN)

Lastly, you’ll probably want to submit your changes to some sort of source control.  In my case I’m using Concurrent Versioning (CVS) or Subversion (SVN) for my source control, and both are handled nicely by the portable version of RapidSVN.

So there you have it, my end to end solution for doing web development from a portable flash drive on any windows box you happen to come across.  My original intention was to be able to just plug this drive into any windows OR linux box and work, but I have some kinks to work out with the linux solution.  Watch this space for more on that as I develop it.

Hopefully this helps you if you’re trying to setup a nice portable webdev environment!

Okay, so I’ve been a bit preoccupied with a couple projects which have slowed my efforts at blogging etc.

In both cases, I’ve been working on websites that leverage the PHP based Drupal CMS.  One of those projects is over at http://www.chasejarvisshoeproject.com.  That site is basically a small community forum for some folks that are sending a pair of shoes around the world, and taking pictures of them in various locations and situations.  Pretty cool.  The result of this effort is a Flickr photo pool and a good time.

So, of course a website devoted to a group on Flickr, is going to require some integration with Flickr.  Enter the Drupal Flickr Module.  When I started using it, this module was still in either an alpha, or beta (can’t seem to remember) and has released it’s 1.0 as of the 25th of March.  Throughout the time I’ve been using it however, there’ve been a few functions of the Flickr API which I needed, and the module didn’t support.  Those were.

• flickr.groups.pools.getPhotos
• flickr.urls.getGroup
• flickr.urls.getUserPhotos

Since the framework of the module did the hard work of making the calls, it was easy to write my own methods which performed these Flickr API functions for me.  And as the module developers released new versions, I just copy/pasted my code back into the new file, and went about my business.

Well when 1.0 was released, I took a closer look.  It turns out they’d implemented the call to get photos from a group pool, but still nothing for the URL’s.  So I figured I’d contribute and write a patch for the two calls I was still in need of.  And that, is what you see here.

569,597d568
< }
<
< /**
<  * @param $nsid
<  *   nsid of the group whose pool url will be returned
<  * @return
<  *   The URL portion of the response from the flickr method flickr.urls.getGroup
<  */
< function flickr_urls_get_group($nsid)
< {
<   $response = flickr_request('flickr.urls.getGroup', array('group_id' => $nsid));
< if($response)
<     return $response['group']['url'];
<
<   return FALSE;
< }
<
< /**
<  * @param $nsid
<  *   nsid of the user whose photostream url will be returned
<  * @return unknown_type
<  */
< function flickr_urls_get_user_photos($nsid)
< {
<   $response = flickr_request('flickr.urls.getUserPhotos', array('user_id' => $nsid));
< if($response)
<     return $response['user']['url'];
<
<   return FALSE;

As you can see, pretty basic but I figured I'd contribute.  I posted this as a feature request/issue over on the Flickr Module's issue queue, but have yet to receive a response.  If nothing else, I can use this to patch the next version.  Hopefully it helps someone else as well.