Webtropic

This post is an open deliberation on the best usage strategy for the new EBS offered by Amazon Web Services.

Last Thursday Amazon launched a much anticipated new service: Elastic Block Store (:EBS).
EBS offers persistent storage (something that was missing from EC2) at a price of 10 cents per Gb per month. In addition, one has to pay another 10 cents per million of I/O requests.

I was quick to jump and test the new offering. As a matter of fact, I had to, because I was about to transfer a customer’s server to EC2 on Friday anyhow. It took me a few hours to accomplish the task, most of them spent in moving data from server to server.

The setup and use of a new EBS volume is fairly simple and with the aid of Elasticfox, is even simpler.

While the technicalities of the migration did not pose a serious obstacle, I realized that all of a sudden there was too much storage available and I needed a strategy for making the best and most cost effective use of it.

Let me explain.

The site I was migrating was a small site, so my choice of the EC2 instance was obvious: the smaller one.
The small instance comes with 160 GB of storage, split in two: 10G for the operating system (only a fourth of it actually used) and the rest completely free.

Now, the problem with the EC2 instances is that once shut down, storage and data evaporate to thin air. Frequent backing up is, therefore, a must.

Before the introduction of EBS, one could back up the data to S3 (but the process was rather tedious), or to another server.

With the EBS, one can decide how many Gigabytes of persistent storage he needs, create a volume, attach it to its instance and make the necessary changes so that all data, from then on, is stored in the EBS. In the case that something goes wrong with the instance, all data stay intact.
For the record, I created a 50Gb volume, so the storage available was now on 210 Gb

EBS offers a kind of backing up to S3 called snapshot. With Elastic fox it is a piece of cake to take a snapshot of EBS and store it to S3, and while EBS risk of failure is 1/10 of a normal hard disk, in the case that it happens, the data can be restored from the latest snapsot to S3.

Now, this is all well, but isn’t it too much of redundant storage there? And doesn’t this mean a waste of money for the AWS customers?

My first thought was to place the database files in the EBS too, to make them safe from failure. The uploaded files had to go there as well, for the same reason.
The application itself, not a really gigantic one, stayed in the 10Gb partition. And since it is not going to change that often, I built an instance image with the application included and stored it to S3.

What else is left to store? Not much. See the problem? The instance now has 150Gb of storage completely unused.

On second thought, I decided to move the database back to the EC2 instance from the EBS. The reason? Well, the database is responsible for most of the I/O and I/O, as we said, is a cost factor.
Having the database in the instance does not consume too much space. A cron script can back it up at regular intervals and save the backup to EBS. In case of instance failure, the data between two consequent saves are jeopardised though.

EBS cost issues aside, there is always this 150Gb left unused that puzzles me. Assuming most AWS customers will play safe and utilize EBS, what are they supposed to do with the storage of their instances?
Either they have to come up with a reasonable use, or Amazon needs to change the instance policies. What do you think?

Photo courtesy of Flickr user skimaniac

Back in the end of 2007, half a year after Google Reader had launched the sharing feature, I had an idea of a new service that would aggregate all the shared items and sort them according to the number of times one post was shared.

As it usually happens with the new ideas, somebody else had it too, and, most importantly, made it real before I had even started coding (actually, I had, but just a few lines). In a short while, a second similar aggregator appeared and, today, we are fortunate to have ReadBurner and RssMeme.

The two services, both dear to me,  have a lot in common with one notable exception: RssMeme employs a kind of spider to find and aggregate shared items  while ReadBurner is an opt in service.

In due course, other feed readers were added as sources: Bloglines, Netvibes, Newgator etc. and RssMeme went  a bit further querying known services to find out whether an article had been bookmarked in any way.

The idea that what one shares through his feed reader is actually a vote or a recommendation is pretty solid, and, once a big number of sharers is reached, the power of statistics comes to play: the articles that emerge to the top are the ones that people truly feel are important. Isn’t this the essence of social bookmarking? And isn’t it also true that this essence is actually gamed in the digg like sites by a rather small group of people, despite the huge influx of traffic these sites enjoy?

One short  visit to Readburner or RssMeme reveals though, that the articles that rise to the top, have been shared by such a small number of people that, with  equal diggs, they would never see the light of day in digg.

Which leads to the conclusion that either the people who share are not that many, or they have not been included in the two aggregators yet.

Speaking of numbers, how many people really use Google Reader? I tried to google the question but came with no answer. I tried to google also the ‘google reader market share’, but came with no recent data either.

Without an idea of how many people use feed readers and share, it is  pretty hard to make any predictions or recommendations. Yet, if we assume that it is only because it is too early  (less than a year) that the sharing culture hasn’t spread and that it, eventually, will, we can fantasize one implication:

Some clever engineer will think of incorporating the share-votes into digg: a little bit of matching (is the sharer a digg user, and has the shared post been dugg already etc) and there you go.

But would that be a good thing?

Yes, it would. Because it would instill the democratic element of Readburner/RssMeme into digg. And, democracy is a good thing, isn’t it?

Related articles by Zemanta

• A Directory of Google Reader Shared Pages?
• Google Reader comes to its sharing senses, lets users pick friends

In his blogpost “A brilliant idea at Harvard“, Dave Winer says (bold is mine):

Before I started blogging, I held many if not most of my good ideas in reserve because I thought some day I might do them as products. But as you get older, you realize that most of the things you think of are going to be outside your grasp, you’re not going to get to do them, so rather than hold on to them, it’s better to let them go. Maybe someone else will do them, and at least you’ll have the pleasure of using the product before your time is up.

How true this is!

And it is  true on more general circumstances than simply  getting old.

Our personal powers, will and resources are far more limited than the things we can imagine of making. It is only a few ideas that are truly dear to us, and only a fraction of them that we are willing to pursue whatever it takes. And unless we pursue an idea whatever it takes, it has very few chances of becoming a reality. What about the rest?  What is the purpose of holding back ideas that we can’t turn them into reality? Why let a good idea die into oblivion, locked into our mind?

A few weeks ago, I read a little book by Paul Arden called “It’s not how good you are, it’s how good you want to be“. It is the kind of book that offers some recipies for success and I am always too suspicious with such books. Nevertheless, if certain passages from the book stick to your mind, be them even just one little sentence, then the book has some merit.

Here is the one that stuck to my mind:

“Do not covet your ideas. Give away everything you know and more will come back to you”.

An idea can be one’s way from rags to riches. But it is hardly ever sufficient in itself. Lots of other factor need to be in place. If they are not in place, nor can they be soon enough, then part with the idea. Give it away for free. Whatever comes back from it, it will be more than zero.

I don’t remember how many years back was it when I first saw the original nigerian scam mail. You know, the type of ‘I have so many million dollars and I want to get them out of the country etc’.  If there was a spam hit parade this would be Nr 1 for years.

It is so legendary that it does not even annoy me anymore to receive such spam.

Perhaps Google’s gmail antispam filter feels the same way, otherwise it wouldn’t let pass this one. There is a slight variation here. The originator is from Ghana, not Nigeria, but, what the heck, it is pretty close. I copy it for you below (syntax and spelling  are intact):

from Ben Goodman <ben_goodman20gh@yahoo.com>

reply-to ben_goodman25@hotmail.com

to ben_goodman20gh@yahoo.com

date Thu, Aug 7, 2008 at 3:36 PM

subject Regards

mailed-by yahoo.com

From: Mr. Ben Goodman
Accra, Ghana.
Hello Dear
I got your contact during my search for a reliable,honest and a trust
worth person to entrust this huge transfer project with My name is  Mr. Ben. k .Goodman, Branch manager of a financial institution here in Ghana. I am a Ghanaian married with two kids.
I am writing to solicit your assistance in the transfer of $ 7,597.864.00 Million United States dollars only. This fund is the excess of what my branch in which am the manager made as profit duringthe 2005 financial year.
I have already submitted annual report for that year to my head officehere in Accra Ghana as I have watched with keen interest as they will never know of this excess I
have since, placed this amount of $ 7,597.864.00 Million United States dollars only to an Escrow Coded account without a beneficiary (anonymous) to avoid traceAs an officer of the bank, I cannot be directly connected to this money due to civil service code which formits civil servants from owing or operating foreign currency account coupled
with the fact thatthe fund is huge thus I am impelled to request for your assistance to
receive this money into your bank account on my behalf I intend to part 30% of this fund to you while 70% shall be for me.
I do need to stress that there are practically no risk involved in this. It’s going to be a bank-to-bank transfer. All I need from you is to stand as the original depositor of this fund so that the fund can be legally processed to your name and be transferred to your account. If you accept this offer, I will appreciate your timely response to my private mail: ben_goodman01gh@hotmail.com
With Regards,
Mr Ben Goodman

I like the name particularly: Goodman. An attempt to appeal to the subconscious as a  good man.

More puzzling than ‘how did gmail let this pass’ is ‘how can there be such spam today’. It seems that there is no shortage of fools, nor there is going to be anytime soon…

Smashing magazine is running a contest to find new professional writers and is giving a MacBook Air as a prize.

While I was contemplating whether it would make sense for me to participate in the contest, I started thinking about the general issue that a high profile/traffic blog or site encounters: how to find and recruit new talented writers. And then an idea came to me that might prove to be a solution completely in sync with the current trend of ‘social everything’: crowdsource blogpost writing.

How on earth would that work? Well, not much to wonder about, because it actually works.

Let me explain:

Blog posts and the comments they attract have long been considered in unity. The post might be correct or not, might be complete or not, might be original or not. Regardless of what it is aspiring to be, comments tend to ‘correct’ it. Commentators might have more information than the writer and they can contribute it. They might also pin point errors, either in the thinking or in the referenced material. They might add insights to aspects the author has never considered etc.

If you have even a small blogging experience  then all the above is common ground to you. But how does this relate to new blog posts?

Simple. One has to initiate the same process before a post is published.

How would that work in detail?
One option that I considered is this: the publisher (i.e. the blog or magazine owner) makes known his intention to utilize crowdsourcing, and invites interested writers to contribute.
Writers register (if they haven’t already) in a private wiki and offer whatever they think fit on a case by case basis. Contributions can take many forms:

• Submitting an idea for an article
• Submitting a draft article
• Correcting an article
• Complementing and or documenting an article
• etc

Fees for the work contributed can be based on the words submitted that were included in the final post. We are not talking a lot of money here, but we are not talking a lot of work per individual either.
A bonus system that will increase the per word remuneration based on the amount of past contributions can work out the incentives needed to keep people hooked in. The remuneration of the submission of ideas has to be different though. Could be a flat fee or a percentage of the total article cost.

If you like the idea, well, let’s crowsource this post and make it better. No money will be involved though, just the glory :)

Photo by emdot