data processing workflows

if you are faced with data processing workflow which requires to process / transform a huge amount of data in a limited amount of time this can end up in pretty complex implementations. if you have enough hardware to do the job you need a infrastructure makes use of the hardware.

using hardware for a limited time today isn't a big issue. the cloud infrastructure out there (e.g Amazon ec2) is perfect if you have to process a huge amount of data in limited amount of time for a limited duration. you are able to scale the usage of the required hardware for the time they are required and just pay for the required duration.

now you also need a ready to use software infrastructure to implement the processing workflow. MapReduce is a software infrastructure for such kind of problems.

Apache hadoop implement this MapReduce but it lacks of easy of use means is a pretty low level infrastructure and of course lacks of higher level workflows which is not defined by MapReduce.

Cascading closes the gap. based on "stream processing" the MapReduce pattern are applied and used. it is not too complicated within one day i was able to create a simple applicaton which convert 20 TB of svg data to jpg and doing some transformation in between using batik and 20 concurrent hardware nodes.

for some use cases the power of cloud is easy to tell....

directory of services in the cloud

directory of available API's in the cloud are available here:

http://www.programmableweb.com/apis/directory

the list shows two things

1. there are lot of services out there many of them can be used free of charge
   
   but
   
   
2. the stability of usage is a huge problem
   - few listed services are moved or removed completely
   - few service definition often changed without providing a sufficient version management

Buzzwording continues?

the IT industry is very innovative in creating and consuming buzzwords. from year to year at least one new trend alongside with one or more buzzwords are created.

main reason for that success is the corresponding visibility and based on that the opportunity to get budget. the main characteristic of such terms is that there are no formal definition of what is really the essence / definition of such term but on the other hand everybody seems to have a clear and complete understanding and definition for the term / buzzword.

second characteristic of such terms is that a common trend is associated with those terms.

and last but not least the life cycle of such trends are pretty similar, approx. 1/2 year until everybody is aware of it (through publications, blog posts, articles), 1 year highest awareness incl. associated investments and at the end the trend will be replaced by next one.

that looks pretty similar to fashion industry and in my point of view there is not too much structural differences between a new fashion trend and a IT trend.

just a small list of buzzwords from the last few years:

• ....
  
• SOA (Service Oriented Architecture)
• xxx 2.0 (esp. Web 2.0)
• Semantic Web
  
• SaaS (Software as a Service)
• PaaS (Platform as a Service)
• Cloud computing
• .....
  

why is it possible to make money with those trends? because all of them promising to solve real existing problems in real industry. if we use the trends mentioned above their main focus is to

• consistent access to required information at the right time at the right place
  
• get rid of increasing IT complexity
• get rid of proprietary vendor driven information silos
• reduce Total cost of ownership for hosting the available information within a company
• improve collaboration between different business groups
• improve adaptability to changing business requirements
  
• ....

if we take the essence of the mentioned buzzwords and collect them again than the following is really promising

• usage of dedicated and well defined services for business automation
• pay for usage of a defined service level instead of paying for hardware / software and corresponding maintenance (what really cares is the service that automates a certain business step)
• architecture that adapts fast and controlled to change of business requirements (changed SLA) and not to changed IT requirements
  
• .....

the trends above of course drives the creation of standards, software and services makes those requirements easier to fulfill but still it still hard work and more than just using those nice buzzwords.....