Big Data for analytics is a paradigm that became popular in the previous decade. A lot of innovation was done for research projects. New technology enabled researchers in many different domains to capture data in a way they had never been able to do before. In agriculture, for example, ploughs would get sensors that would send little bits of information to a central system (over satellite). Every couple of feet these sensors would measure what’s in the ground (minerals for example), how humid the ground is etc. Based on that, large agriculture companies would then be able to make better decisions on where to grow which crop.

The problem was that traditional systems to store this massive amount of small data (relational databases) were no longer adequate to store this information. Systems like MapReduce and Hadoop were created as an alternative and would store these massive volumes of files as concatenated “Big” files. Big Data was born, Big Data for semi-structured data.

Today we are seeing a similar trend with unstructured data. Studies show that data storage requirements will grow with over 30% over the next decade. 80% of that data are large files: office documents, movies, music, pictures. Similar to databases in the previous decade, traditional storage – file systems – is not the best way to store this data. File systems will not scale sufficiently and actually become obsolete as applications will take over the role of the file system.

A nice example is what Google Picasa does for us: in the old days we would store pictures nicely organized in a file system (hopefully with some backups). One folder per year, one per month in each year, one per holiday or party. Today, we just dump all the pictures in one folder and Picasa will sort them for us based on date, location, face recognition (!) or other metadata. With an intelligent query, we can display the right pictures very fast, much faster than browsing the file system. We don’t even have to worry about backups as we can store copies in the cloud automatically.

The new paradigm that will help us store these massive amounts of unstructured data is Object Storage. Object Storage systems are uniformly scalable pools of storage that are accessible through a REST interface. Files – objects – are dumped into the pool and an identifier is kept to locate the object when it is needed. Applications that are designed to run on top of object storage will use these identifiers through the REST protocol. A good analogy is parking your car Valet vs. self park. When you self park you have to remember the lot, the floor, the isle etc (file system); with Valet you get a receipt when you give your keys and you will later use that receipt to get your car back.

So what is needed to build an object storage system? Basically just lots of disks, a REST API and a way to provide durability. This could be done with traditional systems like RAID but the problem is that RAID requires a huge amount of overhead to provide acceptable availability. The more data we store, the more painful it is to be needing 200% overhead as some systems do. The smarter way to provide durability for object storage is erasure encoding.

Erasure encoding stores objects as equations, which are spread over the entire storage pool: Data objects are split up in sub-blocks, from which equations are calculated. According to the availability policy, an overhead of equations is calculated and the equations are spread over as many disks are possible, also policy-defined. As a result, when a disk breaks, the system will always have sufficient equations to restore the original data block. If a disk is broken, the system can re-calculate equations as a background task to bring the number of available equations on a healthy level again. A pioneer of this technology is Amplidata, who use low power Atom processors in their hardware to reduce power costs. As the entire system, all storage nodes, can recalculate missing equations as a background task, Amplidata figured out it was not necessary to use the high-end nodes that RAID systems need (to speed up restores and avoid performance losses).

Apart from providing a more efficient and a more scalable way to store data, erasure coding based object storage can save up to 70% on the overall TCO thanks to reduced raw storage needs and reduced power needs (less hardware + low power devices save on power and cooling). Also, uniformly scalable storage systems with an automated healing mechanism drastically reduce the management effort and cost.

So what are the use cases for object storage? As data needs grow, object storage will become the storage paradigm of choice in more and more environments, but already today we see the need in a number of situations:

Building live archives

Object storage enables companies to re-activate their data. Currently, most companies see data more as a burden than anything else: the data will never be used again but needs to be archived for a whole lot of reasons. But this data actually has a lot of value. By using live archives, employees have faster access to older data and they can use those valuable resources. With traditional storage it would never be achievable to build disk based archives for this purpose as the overhead would make this too costly.

Online applications

Most of the data-intensive online – cloud – applications are built on public clouds such as Amazon S3, which are early implementations of Object Storage. The benefits for the application providers are plenty: a simple programming interface, low cost and fast time to market. As their data sets grow, those companies might move to private Object Storage implementations to reduce costs even more.

Media and entertainment

Traditionally, the M&E industry has been very much file-oriented but we’re seeing a growing interest in object storage to optimize efficiency and reduce costs, but also because this industry is already hitting the limits of their file systems.

These are just a few examples of Object Storage implementations for Big Unstructured Data. Object Storage was not built to replace any of the current storage architectures. Very much like NAS filers were designed in the 90ies because block storage (SAN was designed when databases were king) was not optimized for Unstructured Data, Object Storage will find it’s place next to those two for Big Unstructured Data.

Gartner recently launched iPaaS, Integration Platform as a Service. iPaaS is defined as “a platform for building and deploying applications within the cloud and between the cloud and enterprise”. It enables developers to create integration flows that connect applications that run in the cloud or on-premise, and to deploy them without installing or managing any hardware or middleware.

iPaas delivers where PaaS came short: where most PaaS offerings are limiting developers to one single cloud platform, iPaaS is designed to give access to a choice of platforms. iPaaS also provides integration flows, the development and life cycle management of integrations, and the management and monitoring of application flows.

One interesting iPaaS platform is AppStack by Appcara, who aim to bring application development to a whole new level by “elevating the abstraction level from the servers/infrastructure to the application layer”. With this approach, Appcara brings all the benefits of PaaS to any IaaS environment, so the developer has the choice of platforms, even if a mix is required. The developer can now also migrate operational applications to different cloud stacks; there is no more lock-in.

To better understand the need and potential of iPaaS, we should probably do a quick analysis of what cloud computing has brought us over the past 4 years: 2008 was the year when the cloud paradigm shift started. It was also the year when the three fundamental layers in cloud computing were defined for the first time: IaaS, PaaS and SaaS. The whole infrastructure stack, including hardware, hypervisors and orchestration tools (cloud platforms), was named IaaS, infrastructure as a Service. “As a Service” implied that compute, network and storage resources could be consumed – and paid for – in an opex model, rather than purchasing costly hardware (capex).

Service providers saw great potential in this new sales model. They found a lot of interest for their “public cloud” services from developers who could now quickly set up development and test environments, and decommission them when their projects were finished. But a few companies saw more potential in this new computing model. Instead of just offering scalable compute instances and storage, those companies created an additional layer on top of IaaS. This layer would be called Platform as a Service (PaaS) and would provide all of the facilities required to support the complete life cycle of building and delivering online applications. The latter would later become the third “layer”, SaaS: Software as a Service. Applications in a SaaS model run mostly online and do not require local installations.

Cloud computing has brought a lot of benefits for suppliers, developers and end users. It has also fostered a lot of innovation, mostly on the IaaS level. Hypervisors have become a commodity and there are many (cloud/orchestration) platforms available that enable companies to build private or public clouds. On top of that, a number of “Cloud (Server) Managers” make it possible for users to deploy cloud instances over multiple private and public clouds. In spite of all that, it is still complex to develop and maintain applications in the cloud.

While the XaaS stack has been quite generally accepted, the real-world situation is different. Few public and private cloud offerings provide PaaS and the few PaaS offerings on the market do not disclose all that much about the underlying layers. So the XaaS stack actually looks like this:

Application developers have two options when building a cloud application: use one of the available PaaS platforms, or go build a development environment on one of the public clouds. The benefits of a PaaS platform are obvious: you get an out-of-the-box application hosting, deployment, testing and development environment, complete with libraries, etc. The provider will provide extensive integrated scalability, maintenance, and versioning services so that the developer can fully focus on the application and its features. The problem is that PaaS platforms are very much closed shops: once you pick your platform, you have to stay with it. The provider can change their terms and as a developer you have little or nothing you can do about it.

For this reason, many developers choose to build their own environment in a public cloud of choice. It requires a lot more preparation but with the help of Cloud Server Managers, but it’s not all that bad. The benefits are limited mobility and flexibility, but the problems come at a later stage: lifecycle maintenance is a pain, migrating to a different cloud is not all that easy and few applications are truly designed as cloud applications. Most of the cloud applications that are developed today have a similar architecture as traditional applications, with VM’s rather than dedicated servers. Cloud Server Managers do not help developers to change the way they think.

iPaas platforms will enable true cloud mobility and resiliency.  iPaaS eliminates service offering lock-in and limitations that make Cloud architectures truly resilient. Platforms such as Appcara provide developers of cloud applications with a richer set of resources and give access to multiple cloud and virtualization technologies. iPaaS works on a higher level than IaaS providers, only the orchestration interface point is important; the choice of hypervisor is not important.  Over time, we have become less interested in the brands of hardware and that same phenomenon now is moving up the stack.

One use case that I found particularly interesting, is that of UEFA: someone who used to work at EUFA until very recently explained to me that the Union of the European Footbal Associations has built a 100% disk based archive that replaces the tape libraries they were operating before. According to my source, for EUFA there is no such thing as dead data so they kicked out their tapes altogether. The archive will contain over 200000 hours of footage, lots of it HD – my guess is this could easily be 50PB. Unfortunately it’s not AmpliStor storage (yet), but who knows, someone at UEFA might want to improve the efficiency and increase the availability one day).

So why is tape no longer the medium of choice for EUFA? Because of the opportunities tapes bring. Imagine a EUFA game being broadcast on tv. At one point, a camera registers Michel Platini in the audience: the editor can at that point search the archive on keywords Platini and goal, he or she will immediately get an overview of all the available footage where Platini was scoring goals during his career as a player. Within minutes, a short fragment of that can be shown, e.g. during the preparation of the next free kick.

Another use case for the EUFA archive is to make the footage available to tv-stations across the world for documentaries, talk shows in preparation of a game etc. Technically this would be possible with tape libraries but think of how much more easy it is when tv stations can just search the archive online.

Or how about the clubs? Think of how valuable the EUFA archive is for scouting purposes, training aids, production of club fan DVDs etc. The online archive can give access to any club or any company contracted by a club.

While tape as a medium has plenty of benefits, think about tape not consuming power when lying on the shelf, I found this an interesting use case of disk-only archives. Sometimes tape is just too slow and disk is the only option. As these disk archives keep growing (think of the number of EUFA games each year), cost-efficient and highly reliable disk storage archives such as Amplidata’s AmpliStor will become a standard in the media industry.

To Webinar

The Webinar is hosted by George Crump of Storage Switzerland.

I highly recommend attending the webinar!

The fundamental concept of digital storage has not changed a lot since we first used computers: to keep data on a medium to be able to access it at a later time. The requirements related to storage and the technologies that make it possible to meet those requirements have evolved a lot more, however. For requirements, think of the size of storage environments, availability requirements, compliance regulations, virtualization support etc. On the innovation side, think of RAID schemes, SAS and SATA disks, SSD and so on.

So let’s not try to define traditional storage but immediately focus on the requirements (read “needs”) and technology that brought us to the largest innovation we’ve seen in the storage industry since … uhm … floppy disks.

 More data, larger files

The most important driver towards cloud storage is the amount of (unstructured) data that needs to be stored. We live our lives in a very digital way and produce content constantly. Think of the pictures and home movies everyone is producing everyday, or the quality of movies that are being produced for our entertainment. Hollywood productions are now multi-petabyte projects.  Industry analysts expect digital storage consumption to increase with 3000% over the next decade, but storage budgets will hardly increase. Also, enterprises are not planning to drastically increase the staff that manages storage environments so every storage administrator will have to manage a lot more storage in the future.

The kind of data we store has also evolved: the 80ies were all about structured data (databases, block based storage, SANs) and the 90ies were about unstructured files (think of the growing number of word and excel documents we were all producing and the success of Netapp’s NAS systems for file-based storage).  But the properties of the content created today (larger files) require the data to be treated as objects, large objects. File systems either do not support the number of files/objects we store or are just obsolete; think of how picture libraries like Picasa replace the file system.

 Always available, we never delete

We live our lives online and want our data to be accessible all the time, synched to several devices, backed up to several geographic locations and available to friends, family and colleagues.

We also seldom delete any data. Why would we? We might need it one day, put it in the archive. Only a small amount of data is deleted because of regulations. And those archives need to be accessible as well. In many companies access latency of days or hours is no longer acceptable. Use disks, forget about tape!

Cheap, damn cheap

Finally, storage needs to be cost-efficient. Since the budgets don’t increase, we have to save on the energy bill, real estate and management.

Conclusion

Today’s storage requirements are all about availability, scalability and efficiency, both for consumers and enterprises. The only way to meet those requirements is by building storage infrastructures that are popularly called Cloud Storage:  large storage environments that are always online and available over the internet, which support many and large objects and which scale … infinitely.

My next post will explain how to build such infrastructures

Hope to find the time to post something completely irrelevant here some time soon!

Cloud computing has helped creating a lot of opportunities. The drivers for this were the many innovations that have been brought to the market over the past three years. Those innovations range from full cloud platforms to online end user applications. There are about a dozen IAAS, or cloud infrastructure platforms on the market, and many more cloud-enabling technologies such as server and network virtualization products, cloud governance solutions, metering and billing applications etc.

An area where innovation has not kept up is storage. Most cloud infrastructures either run on very expensive storage systems that will not be sustainable from a price point of view or on very basic infrastructures that do not provide the required availability or scalability as required by standard Cloud SLA’s. The few “big” cloud providers have created their own storage infrastructures and it remains to be seen whether those infrastructures will meet the requirements as they grow.

To make things worse, recent evolutions in disk technologies (bigger disks at cheaper prices) have made it even harder to meet the storage requirements in cloud environments. Due to larger disks, RAID – the defacto standard protection scheme – has lost a lot of its advantages: restores take too long and leave the system less protected, RAID scales badly and it is not cost efficient due to the overhead.

Reliable storage is a fundamental requirement in providing online services, and quite a few storage system providers have been working on technologies to solve the shortcomings of RAID. The solutions can be put in three categories: new RAID schemes, software layers on top of traditional RAID and distributed storage.

A lot of research has been done to create new RAID schemes. RAID was a great technology so why not keep it and improve it? Unfortunately new RAID schemes do not solve the fundamental problems with RAID: long restore times, poor reliability scores, limited scalability etc. Other suppliers are trying to improve storage reliability and efficiency by putting a software layer on top of traditional RAID. While some of these solutions might give a benefit on the short term and reduce the management cost to some extent, this is not a viable long-term solution.

A more recent innovation for large-scale storage is distributed storage. The concept is simple: reliability & availability are guaranteed by spreading data over the entire storage system, i.e, over “all” the available disks. Using erasure coding, data blocks are split in sub blocks, which are spread over different disks, appliances etc. according to spread policies. The codec only needs a selection of the sub blocks to reconstruct the original data block. One vendor compares it to a Sudoku puzzle and that is a strong conceptual analogy.

While we are still early on the adoption curve, distributed storage is being recognized as the logical next step. A few solutions are commercially available and are seeing a growing success.  Efficiency and reliability claims vary but look very promising. One vendor claims they need as little as 40% overhead to provide better reliability than RAID6 and replication  (numbers depend on the size of the infrastructure and the exact desired reliability). Some solutions feature the possibility to spin down disks or even full appliances. This has a great impact on the energy bill.

The architecture of distributed storage systems makes it particularly easy to scale the infrastructure. As a matter of fact, distributed storage systems become more efficient and provide better reliability “as the infrastructure grows”: the more appliances and disk an infrastructure has, the wider the spread can be.

Performance numbers again vary for the different solutions but most products provide local caching (usually a mix of SSD and SATA) to improve read and write speeds. With online applications and virtual environments in mind, distributed storage systems typically support standard interfaces such as ISCI, NFS, CIFS etc. Other typical features are out of band optimizations for data integrity verification and assurance, zero copy snapshots and cloning.

Distributed storage is obviously a very new concept. The widely adopted RAID will not disappear overnight. But as storage needs (especially online storage needs) grow, solutions based on distributed storage will see a fast growing interest from the market.

An interesting Distributed Storage implementation to have a look at: Amplidata

Nothing new, nothing interesting. So what’s wrong with this? They just try to get to the ultimate decision maker. Fine. Just one thing: the person they try to pass in this very arrogant way might be exactly the decision maker for this kind of purchases. The CEO often doesn’t want to spend any of his time on things like that. A CEO does like to be involved in business critical purchases, but not in things like advertising.

The result? A CEO who feels his time was wasted because the person who passed the call didn’t do his job. The person who passed the call – say the marketing manager – feels bad about (1) being passed and (2) the CEO not being happy. For the sales puppy the result is that he or she can forget about selling to the company ever again.

Quite an interesting company with a few things to think about:

(some answers might already be given by others but I still have to start my actual research)

The fact that AWS loses some of their key people, does that indicate there was disagreement? I always thought they should have done more for private clouds.

Amazon and EC2-like is used a lot on the website – does Amazon still have a stake in this?

It does kind of explain Martin Buhr’s relative silence on Twitter the past few months compared to the year before …

VMware is an investor – Diane Green is a board member: is this how VMware innovates? Is VMware’s cloud portfolio not as complete is I thought it was? Isn’t this partly competitive with VMware? Why no further integration with VMware?

What’s the TAM?

Any ex-Sun guys involved?

Anyway, looks like we finally have an interesting Cloud startup to follow …