Refactoring life. Mercilessly. >> OpenStack

There will be no reliable cloud (part 3)

2013-04-12T00:00:00+02:00

How I stopped worrying and love the cloud

If you have read part 1 and part 2 you may thing all hope is lost and this cloud cannot work at all. However, there is actually a way to create fault tolerant, resilient applications on top of clouds. Actually if the application is created in a distributed, fault tolerant and resilient fashion, it will have no problem at all to run on a cloud infrastructure.

As mentioned in part 2: The main thing to get away from, when thinking about this, is the idea of "a server needs to be fault tolerant". Forget the server. It's about the end service. If your body is a few years old (which it is, if you can read this), I guess that not a single cell of your body is the same as when you where born. Yet, you are still alive. That's the kind of thinking that has to be applied to a service in a cloud setting.

Green field cloud native apps

This behaviour is actually not that hard to achieve and there is a lot of documentation about this. There's a AWS Whitepaper which is AWS specific but most of the ideas apply to any cloud setup. Also Netflix share a lot of their ideas and implementations of their architecture on Sildeshare and on github, one very recent and very good talk is this Netflix Architecture talk.

The main idea is actually pretty simple:

Identify and tear apart statless and stateful parts of the application
make the stateful parts redundant using real, distributed data stores (as mentioned in part 1: Riak, Cassandra, Mysql Galera, etc.)
Make sure that the data store parts are distributed across failure domains (e.g. Availability Zones, Regions, etc.)
Make sure that the dependencies of your system are known to you and designed in a way that reduce the likelihood and impact of failure
Using a Micro-Services approach will help you to get these dependencies to be explicit and it let's you scale the individual parts independently as needed. You can also make the best fault behaviour decisions on a very fine granular level, so that a degraded operation and partial failure is not a big problem

Netflix for example uses Cassandra as their data store. That does not mean that they have one huge Cassandra cluster spanning 100s of nodes. They have 100s of Cassandra clusters with a few (about 5-50) nodes that are completely independent and each service has their own cluster (which is distributed over AZ or regions as needed)

One interesting part about the distributed data store part is that failure or hand-over of data or requests is a totally normal part of the day to day operation of the system. So adding a new node will lead to rebalancing, as would a failure of a node. A system that does these things all the time will work better than some kind of hot-standby system that will "fail-over" when something fails which maybe happen only once in a while.

While this kind of application design puts some kind of burden on the developer to deal with all these failure conditions that have been abstracted way (or to put to more realistic: ignored) through "highly available backends", the running application is the best place to decide what to do from a business logic point of view, if some kind of failure occurs. For example, if your real time web chat system does not work at the moment, you could just sent the message in an asynchronous fashion via the non-real time part of the system (IIRC Facebook does it that way). You can hardly put this kind of logic into your HA-failover scripts.

Another great talk by Michael Nygard about Stability Anti-Patterns shows that design for failure is inevitable for the kind of environment were deploying applications in today. And this isn't even cloud specific!

Even though complexity is not mentioned explicitly, the "reduce integration points"-message is exactly that: Reduce complexity!

The video also contains a fun anecdote about TCP (and how firewalls violate it). Now keep that example in mind regarding the cloud analogy of TCP vs. UDP and think about how this situation would have been different if UDP had been used and the retransmit etc. logic would have been at the application layer…

If you want to make your application more reliable it is actually a good idea to poke it all the time and try to make it fail. This is basically what Netflix does with their chaos monkey approach. The idea behind this is Antifragility. Through the interaction of your software system with the developers and the feedback you get from failures, you can basically turn the complex system that consists only of the software application into a complex adaptive system (consisting of the running(!) software and the interactions of the users and operators), that can respond to change and become more resilient.

Other benefits of apps in the cloud that help with reliability

.. and don't need any special reliable backend. Using a cloud setup you can basically test and deploy in ways that are not - easily - possible in a traditional world (like Blue/Green Deployment), spinning up 100s or 1000s of instances to do load and failure testing. And you can do that with up to 100% of the same environment (architecturewise, operating system, number of services etc.) as production.

But I have a legacy system and want to put it into the cloud

The main thing to remember is not to confuse "cloud instance" with "server". So if you need to roll your classic HA-DB Setup in the cloud, do not put it in one Avaibility zone on two instances. Use to zones that are guaranteed to be independent.

There is actually a good overview - although very old school in nature (Who would have thought - it's from Oracle!) - Mysql Reference Architectures if you choose the failure domains as communicated by the cloud provider there should not be a big problem really.

If your cloud provider (or internal cloud) cannot do that, you can still plan to reduce your MTTR. This is a very good idea anyway! I'm always surprised how little thought seems to be put into reducing MTTR. "We have a HA system, so we're good". No! Think about what happens after failure and about the impact! You cannot predict the probabilities but you can predict impact of failures pretty well!

Of course you can split up workloads in different cloud environments - latency might become a problem. So YMMV.

Another problem with non cloud-native apps that run on more than one server at some point most of them use something like NFS as a shared file-based datastore. Of course you can run NFS on some cloud instance, but then NFS itself is not really distributed and prone to catastrophic failure. Also, the filesystem abstraction is really broken over the network. You might get away with it, if you know and control the network itself. In a large scale cloud network... Not the best idea (failure semantics for file systems are really very different than network protocols...).

So if you want to run this kind of applications in the cloud, you have to think about the tradeoffs: Maybe you don't really need a shared-filesystem backend. Maybe you can easily change the application to use HTTP-based Object-Storage. Or you could run some distributed NFS-like replacement such as GlusterFS or xtreemfs on top of cloud infrastructure (again: use more then one zone). As always: it depends.

Closing thoughts

This is certainly not all I have to say about this topic (expect more posts ...), but instead of collecting more and more links and ideas, I wanted to get this out there to fuel the discussion about these ideas. I think "the cloud" as in "compute infrastructure" is mostly still misunderstood. It seems to be something we have had for a long time. It's "just VMs in the internet behind an API" or "Virtualization 2.0". It's not. It's so much more! It's different. If you think differently about some things...

There will be no reliable cloud (part 2)

2013-04-09T00:00:00+02:00

In while the first part was more basic information and technical, the second part will be about why I think it is impossible and not viable business wise to aim for high availability of the cloud in the infrastructure layer. Part 3 will then go into why this actually isn't that bad an we can still use this "crappy infrastructure" to build systems that are available to the end user.

I just want to make the following point:

Complexity + Scale => Reduced Reliability + Increased Chance of catastrophic failures

This is all this post is about.

Complexity

Almost any software system is a complex system. A software system that has networking components certainly is complex and if you put it on a cloud infrastructure, I think there is no argument that this system is not complex.

Complex systems fail in certain ways. If they are poorly designed these failures are catastrophic, meaning the whole system is taken down. In terms of the cloud this means: All software systems that run on the infrastructure are not available any more and may not recover ever.

This is very abstract, so where does complexity show in a cloud infrastructure?

Failure domains

Let's look at a basic cloud infrastructure setup (say a compute infrastructure implemented with OpenStack nova):

We have a controller and four compute nodes (Hypervisors). Now the controller is certainly a Single point of failure, so let's add another one (HA!). You can immediately see how this increased the complexity! And if the failover fails you gained nothing.

Compare that with the approach of - instead of adding another HA-style controller - creating another zone which itself is as simple (and unreliable) as the basic example.

If you just look at one zone it is unreliable and does not seem to make sense. But if you look at the whole system consisting of two zones you can actually see that you gained something: You have two independent(!) systems with known failure semantics: Failure of one controller in zone 1 will never lead to a failure of zone 2.

Of course you have to build your system in a way to make use of this. This is topic of part 3.

Why is this approach useful? Remember: Almost any public cloud provider takes this approach. And please don't start to argue that one controller per zone is not a good idea and you still need HA for that. This is a pattern. Of course you should make the controller in a zone as reliable as possible - taking into account the trade-offs of zone failure and costs.

At the controller level you can actually make that piece of the infrastructure more reliable with very small costs. In the OpenStack case: Just add a MySQL galera cluster, UCARP virtual IP in front and you're basically done. 1 changed component and 1 additional component is a small price to pay for the gain.

The main idea here is about the "failure domain" pattern which splits and reduces complexity instead of increasing it using dependent systems like HA-pairs.

Some math - or not

Ever since statistics class in school, I've had the gut feeling that using probabilities in the real world is somehow wrong. I was thrilled to read my feeling was right. There is actually a good video about the whole topic by Michael Nygard Reliablity engineering matters except when it doesn't. Please watch the whole thing or don't watch it at all. If you stop half way, you'll get everything wrong.

"It is often the failover mechanisms themselves that generate the failure" - Michael Nygard @ 20:20 in "Reliablity engineering matters except when it doesn't"

The basic heuristic for reliability of systems is:

The higher the number of dependent components => the lower the overall availability and the bigger the impact of failure

And this is not a linear dependency. As soon as you introduce a dependency like a central service of any kind (think network filesystem, SAN, the network itself) you add a dependency.

So if you consider this and now think of a cloud system that uses say 1000 compute nodes which, for normal operation, could be rather independent. Then you add some crazy HA-failover logic to your cloud management software that is supposed to fail over VMs via live migration from any node to any node. Well, congratulations, you just decreased the reliability of your overall system and increased the risk and impact of failure (total system failure!) by orders of magnitude because you just tied all compute nodes to each other.

An example that shows up again and again are failures of AWS EBS as a common source of AWS AZ failure, like the one around christmas last year. The Elastic LB had a dependency on EBS. EBS fails. Now the whole region is down.

I actually did not provide any calculation here - because I think you can only miscalculate here and come to the conclusion that "it's actually not that bad". It is important to get the connections and implications between dependencies, complexity and how scale increases the impact and likelihood of failure (even if you cannot say by how much).

Another thing to consider when using the math-toolkit: Most of the reliability calculations that are used, come from mechanical engineering where things actually behave in a somewhat predictable manner. When you're dealing with just bare metal servers this may apply. If you add virtualisation you now have software added to the stack. And software failure characteristics are very different from hardware.

Also complexity and non-independent system change the calculations drastically - and you still have to predict probabilities. System failures are no dice rolls!

If you're interested in the math (and where it works and where it doesn't), watch the video mentioned above and read this paper if you think you can somehow predict stuff in a real, complex system.

Problems with failures in big complex systems

Another problem with complexity + scale is the type of failures you get. Most of the failures start local and small but then turn into cascading failures like this Google App Engine failure.

There are many ways this can happen and a few ways you can protect against this - some of which are shown in Release It!.

The general approach here is: make failure as local as possible and if something starts to go wrong, make it fail hard and small and contain the failure. Again: Defined small, failure domains and independent systems.

If you have a lot of dependencies in the infrastructure you're not only increasing the likelihood of failure but also make recovery harder. The Thundering herd problem comes to mind.AWS EBS has been hit by this more than once.

The importance of partial failure

Even though the system design that is communicated to the outside is "expect one complete cloud zone failure" it is important to design the system so that it can actually fail partially and work in a degraded mode. So to have small failure domains (e.g. compute nodes) that fail in a predictable manner (e.g completely and only in hardware) helps with that. So a degraded operation might be: 20 of 100 nodes failed but the other 80 are fine and don't care about the failure.

One of my favourite examples of partial failure/degraded operation is the Airbus flight control software with its different flight control laws. A lot of stuff can fail and the plane is still maneuverable. As far as I know nobody ever died from a failure of this system.

You have to design for failure anyway

There's a great post about UDP clouds vs. TCP clouds clouds. I like the analogy. I disagree with Massimo's conclusion: He puts it as if with TCP you'd never really expect a "connection reset" or anything like that because TCP is reliable. The truth is: You have to design for failure anyway! It does not matter how reliable your underlying infrastructure is. And you can actually implement some very performant and reliable applications on top of UDP… ask some of the IT guys at Wall Street.

Business side of things

There are also costs to provide reliability at cloud scale. One of the best papers on this topic certainly is The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. It basically comes to the conclusion that Software reliability is cheaper than hardware reliability at scale because the additional costs of a software deployment are basically zero.

One thing to consider here is also this scenario: Most web scale systems consist of a large stateless part and a small stateful part. The stateless part is easily scalable via scale out and a single component does not have to be reliable. Using this setup, it does not make sense to host the stateless part of the system on a highly reliable cloud system (if we pretend that it exists). You don't want to cast pearls before swine, do you?

The main idea of a compute cloud is to be a general purpose compute environment. To make it as flexible and cheap as possible for customers to use, it does not make sense to provide a super reliable infrastructure.

Now what?

I hopefully showed why a single compute node or zone of the cloud will never be reliable. This "reliable cloud" problem really only exists if you think about compute instances in the cloud of "physical servers". They really aren't servers. So the real question really becomes: WHAT does have to be reliable and does it have to be a certain part of the infrastructure? Everybody seems to have accepted that hard disks will fail all the time and we find ways to design around it. I think the same is true for "the server" in a cloud setting or better "the cloud compute instance".

So if we accept this reality and move on, it turns out we can actually build reliable applications on top of such infrastructure. Mission critical apps in the cloud? Well, NASA does it and they don't seem to use their rocket scientists to solve that problem.

There will be no reliable cloud (part 1)

2013-04-03T00:00:00+02:00

Stop wasting your time trying to find one. Stop wasting your time (and money) trying to build one. If you find a service provider that claims that they have it: Maybe question their understanding of cloud - and business.

With all that free time, start to build reliable systems on top of unreliable clouds.

After all these bold claims I'll convince you that this is a - some will say - sad but still valid fact of life.

The main issue here is scale. Things (very generally) work very, very different at scale. And cloud infrastructures are all about scale. Keep in mind that complexity of systems does increase exponentially and thus the things that work fine with small systems might completely fail with bigger systems.

Let's look at the different approaches to reliability that are out there and how they map to the cloud space. I start with the "building blocks" at the lowest layers and then move up to a whole cloud infrastructure (based in OpenStack) and some example services on top - because cloud for cloud's sake is a bit boring.

High Avaiblity vs. Service resiliency

The "HA" term seems to be prevalent with current system design. You just add an "HA"-pair to your system and your safe. At least that's how vendors seem to pitch this kind of design.

There is actually a very good presentation on the matter which goes into the differences of HA vs. resiliency by Randy Bias and Dan Sneddon of Cloudscaling.

I was in the audience at that presentation and was very excited to hear all the things that are problematic with the HA-pair-approach of doing things: HA-pairs fail in a very catastrophic way, they don't really scale (out), etc.

And I was very disappointed to hear that the actual examples in the presentation were only about resiliency of stateless services.

Why does this matter? Because making stateless services resilient/available is indeed not the domain of HA-pairs. It's the poster child of scale-out architectures like the Web or the internet routing backbone at layer L3.

Side note: One property of resilient systems that surfaces here is the client knowledge of more than one endpoint. Think of multiple DNS entires for a domain that hosts a webpage. In the routing example this is not that obvious but if you look at IPv6 you can see multiple routing entries on the client side.

So making stateless services resilient just means: replicate all the data and serve it from multiple endpoints and let the client know about multiple endpoints. There are several possibilities on how to do that from an architectural standpoint. Choose the one you like and you're done. Easy.

The interesting part - which was left out of the presentation - is resilience of stateful services. And - while most services can actually be designed to be stateless - you have to store your data somewhere and be able to change it. Otherwise this whole information business would be kind of boring and useless.

For stateful services you basically have two options to make it "HA":

take a non-distributed base system and tuck HA on top
take a distributed system and make the right tradeoffs

Let's look at them in detail.

Non-distributed base system with HA on top

This is the classic "HA" case: Take some stateful service that is not distributed in itself like NFS (which is not distributed on the server side) or MySQL add some Pacemaker magic with some DRBD mixed in and you're good. Or miserable.

If you look into the details, most of the time your basically cheating your way out of the CAP theorem by denying the existence of network partitions through a second network/heartbeat link. Also these kinds of setups are cause of failures more often than not. For example several github outages were caused by these kind of HA-failures: Github Mysql failover failure, Github MLAG failure.

"Cluster Software" causes more system outtages than hardware failures or software bugs. (See Martin Thompson's presentation on "Event Sourced Architectures for High Availability". Around 7:30)

Another thing to consider with the traditional approach: This approach in itself does only try to limit MTBF. MTTR can be considered but this is much harder to do as - from the system design standpoint - the expected failures of such a system are catastrophic. They are catastrophic, because a system that is not designed to be distributed and then IS distributed can never take distributed failure conditions into account and the best thing that can happen in case of failure is complete failure. You don't want the two HA-heads failing only "half". In this case one is enforced via STONITH or if something goes wrong further: Failure of both heads would still be considered better than a split brain scenario.

Distributed System and the right tradeoffs

In a distributed system the components of the system have some knowledge about the "distributed-ness" of the whole system and can therefore accommodate certain kinds of failures. Depending on the system they can actually work with partial failures (like network partitions or outtage of several different components) or handle byzantine failures.

Examples of these kinds of database systems are Percona Xtradb Cluster (MySQL with distributed backend), Riak (distributed database where you can make CAP-tradeoffs at the request level), Google Spanner, Cassandra

The database Datomic is interesting in this regard. The design actually considers the state at the smallest possible level: The transaction level. You get Web-scale like Read-Scalability with stateless semantics on the read side and limited write scalability on the write side.

Btw. this is not about SQL vs. NoSQL. There are distributed versions of both "camps" available - with different tradeoffs. I won't go into detail here.

There are also block level and filesystem level systems available that are distributed form the ground up: Ceph, Xtreemfs, Glusterfs.

I'll cover some of the tradeoffs later, when we talk about "layers of the cloud cake".

So these two approaches are fundamentally different. While the "let's accept distributed systems as a fact" approach is harder, because you actually have to make tradeoffs. The "classic" approach tries to hide the "distributedness" of the system and abstract it away. This actually does work - at a certain scale for certain types of systems. Even up to pretty large ones - if you put in enough effort and money.

Definition of availability / reliability?

As we move to the whole picture view of things, let's think about what availability and reliability actually mean. In the and the end user cares about the overall availability of "the system". No user actually cares about MySQL or some Webserver. They care about the service they are using.

Also availability is not only about hardware either: It's also about software failures (See Joe Amstrong's Thesis: "Making reliable distributed systems in the presence of software errors" ). HA systems that need to go down for "maintenance"/software updates or fixes are a kind of a joke. They are "highly available, as long as you exclude things that would bring down availability like updates".

Another thing to consider is the definition of availability at the different service layers: Is a service that is 2 seconds not available, still available? Is it ok, if i just don't loose a request? Or is 0.5 seconds ok, but I might drop requests.

If you think about it, you really have to do MTBF/MTTR considerations at the request/transaction level. "Is it ok, if I drop a request if no answer is there for 3ms? Try another endpoint then. If I get an answer within another 2ms, I'm fine => available" - or "I do a 'stat' system call and it's ok, to wait 2 minutes, but do not ever let that call return with a failure".

I'll get back to that picture in the "layer" discussion. But one spoiler: Most of the really highly reliable services actually solve these kinds of problems very, very high in the stack and do not care that much about reliability of the lower layers...

We have all the ingredients, so let's build that reliable cloud!

What does this whole HA/resiliency thing have to do with a reliable (or not) cloud?

With the text so far you could come to the following conclusion:

A cloud infrastructure is a distributed system, it has some stateless and stateful components (I did not explicitly mention that. But just look at this picture. If that's not distributed… I don't know what is ;-) )
We use the stateless approach for stateless parts (as shown in the Cloudscaling presentation)
We throw in some distributed data store for the stateful parts (You could use the "classic" approach for that but why bother, if there are options like distributed MySQL servers available and arguably better)
While were at it, put everything stateful on the VM side (e.g. base images) on a distributed datastore like Ceph, Xtreemfs or Glusterfs

We have all the ingredients, so let's build a reliable cloud, already. It cannot be that hard!

Well, if we look at the current approaches that are out there of the big cloud providers (Amazon, Google, Microsoft, HP Cloud, etc.), we can see that they follow this model only up to a point. And we can see, that this "distributed stateful part" on the backend side (in AWS terms: EBS) is one of the main causes for outages...

So the idea of "let's just make that part more reliable - and get rid of this insane availability zone business while were at it to use just ONE big reliable backend" somehow seems to be wrong.

I'll show you why this approach won't make sense (business and availability wise) in the next part. Feel free to comment, question and fight my thoughts and ideas. They can only get better by attacking them!

Read part 2

Deploying a multi node setup of OpenStack Folsom on Ubuntu 12.04.1 LTS with one command

2012-12-01T00:00:00+01:00

We're going to cook some OpenStack today, so get your ingredients ready:

vagrant
git
A Hosted Chef Account (You can use your own chef server, of course. Installing a chef server e.g. via knife-server is out of scope of this tutorial, though)
librarian-chef - gem install librarian
spiceweasel - gem install spiceweasel

To be honest, if you count the environment set up commands it's not one command but about five for the whole process. But after the environment set up you can repeat the process using the one setup command over and over again. This is pretty useful for environment specific testing, CI, etc.

Instead of using vagrant with VirtualBox you can of course use your favourite virtualisation solution for testing or use a bare metal setup. All you need is at least two running machines with Ubuntu 12.04 on them. Instead of vagrant up you would then use knife bootstrap with the single-controller and single-compute role.

So let's cook:

Checkout the Chef cookbooks and Vagrantfile

git clone https://www.github.com/cloudbau/openstack-chef-repo.git
librarian-chef update

Set up Chef server environment

Using hosted chef is the easiest way to get started, but you can of course use your own Chef server. The Vagrantfile uses 2GB per node at the moment. So be careful not to exceed your RAM if you increase the compute node count.

vi config.rb # Change the Chef server settings

Upload cookbooks to chef server

spiceweasel infrastructure.yml | sh

Deploy

Now deploy Openstack!

vagrant up

Get a coffee, tea or whatever you like while it's cooking…

Use it

Open a browser window at http://10.0.112.10 to log in to your OpenStack dashboard. The default username and password is "admin" and "secrete". If you want to change that, have a look at the attributes of the keystone cookbook.

What's next

The community chef cookbooks are still under development and the version used here is a slightly modified so that it works with OpenStack Folsom. These changes will be merged back (pull requests are pending) to the community cookbooks and the cookbooks will certainly evolve and also cover OpenStack services like cinder and quantum.

Status of the SmartOS OpenStack Port

2012-11-06T00:00:00+01:00

Since the first blog post about the SmartOS port and after my lightning talk at the Grizzly Summit, several people have asked me about the status of the port. So here's an update.

Current status

First announcement: It is not production ready. That's not because the code is bad or unstable. It is just incomplete at the moment. The whole network management code is still missing.

The whole thing is not a part of core OpenStack. I'm not sure if it ever will be, but that's not really a problem. The code should be easily pluggable to the current OpenStack code base. If the code is good enough and sufficiently complete I do not see why it should not be part of the core project. With my current understanding of the code organisation this would mean the code would be merged to the nova code tree. If - for some reason - it will not be accepted, it should be easily be distributable as a separate python egg.

Future plans

Several people asked me why I'm doing this. Who is behind this etc. Well, it's currently just a hobby project out of curiosity. I've been following the Solaris/OpenSolaris/Illumos/ development for some time because I think they have some pretty cool and different solutions to problems (ZFS, dtrace, etc.).

Besides my hobby projects I do have work to do, so the development progress on this project will be not predictable. If you are interested in the further development, just watch this blog or the wiki page mentioned below.

More information

Andy Edmonds one of the original creators of the Blueprint created a SmartOS wiki page at the OpenStack wiki. I added some info to it - including the current code base and information about how to set up a SmartOS/OpenStack installation.

The fear of OpenStack fragmentation and the holy grail of cloud

2012-10-25T00:00:00+02:00

Last week I was at the OpenStack Grizzly Summit and one thing was clear from the beginning: OpenStack generates a lot of interest in all kinds of businesses: Hosting, Enterprise IT, Development shops. All kinds of people want to use cloud infrastructure in one way or another to get the benefits of a more flexible infrastructure.

At the Folsom Summit in April a lot of talk was about "What is OpenStack?". A Distribution? A cloud operating system? A Framework. It seems to me that the consesus now is "its a toolkit to build cloud infrastructures".

I agree. This is fairly accurate and reflects the way most people use it. There are several cloud products out there that use OpenStack in different ways to solve their problems. It also shows that it's not an off the shelf product that you just buy right now.

When people say "OpenStack is not mature", it's like saying "The iPhone SDK is not mature". Well, it is not a product. So this assessment basically does not make sense. It is something that you use to create a product.

At this point it is unclear in what direction OpenStack will evolve. Certainly people use it to create products or services with it. And I can see that this creates a fear of fragmentation. A fear that was prominently put out by Gartner and rebutted by several OpenStack thought leaders.

Saying that this will not happen is not enough, though. It will happen if we do not actively fight it. That's life. This is how the world works. Things decay. Entropy.

Why is this important anyway? OpenStack is and will stay OpenSource it will be developed further, it will get better.

It might get better. Without a clear definition or goal of what OpenStack will provide and what it will not provide, it might end up being what it is now: a toolkit. Everybody picks their tool from it and creates a frankencloud.

Why is that bad? It solves your IT problem, right? Indeed it does for you, right now. And then, after the successful internal OpenStack deployment project your CIO proclaims "We're going hybrid!" And you realise that just by using OpenStack internally does not make it compatible with the public Cloud offerings that are out there that use OpenStack as well.

And this is not an API issue. Just because OpenStack has EC2 APIs does not make OpenStack compatible with EC2. There is more needed to compatibility than just an API. The API is necessary but not sufficient.

This is the holy grail of the cloud: Cloud interoperability.

I think it can be done. But it has to be actively developed. This kind of stuff is hard and it does not "happen". At the analyst panel at the grizzly summit Steve O'Grady of Red Monk shared this view. He compared it with Java/J2EE: You can write once and run anywhere with Java - up to a point.

"100% no modifications needed"-portability between cloud service providers internal/external clouds will not be possible in every case. But: It is possible up to a certain point and this point should be the goal.

How can we achieve this? OpenStack needs an executable Test Suite that everybody can run. Similar to the Java TCK just without the licensing issues. The OpenStack Foundation should make this kind of thing officially available. From a technical standpoint there already is code that could be a starting point (Tempest Project).

It should be clear that if your internal cloud deployment passes this test suite and your service provider does also pass it, you should have no problems to deploy your application on the public provider.

Rackspace already is starting to create something like this. While it might make business sense for them to do so, I think this approach is wrong. This kind of offering has to come from the OpenStack foundation! Nobody (except Rackspace) wants a "Rackspace compatible" cloud.

Troy Toman from Rackspace said: "We have a core that we know is the right thing. So how do we continue to innovate?". By putting the certification process into the OpenStack Foundation instead of Rackspace as a company.

For true cloud interop we need a vendor and service provider independent entity - the OpenStack Foundation - that defines what is needed for your cloud service or private cloud product to call it "OpenStack compatible".

Everybody goes ahead and creates their own little or large OpenStack cloud and solves problems. This is fine but if we really want to make the best use of the possibilities of cloud computing with OpenStack in the next years a lot of work has to be done to make cloud interop happen.

This work can either happen at every IT shop that wants to deploy cloud services in a hybrid model - or - at a central point that already manages OpenStack related things: the OpenStack Foundation.

The latter would be much more effective and would end up costing everybody less. It also would be much more in alignment with how the cloud model works than the "everybody does their own" model.

There was a comment at the summit that OpenStack can tell what Amazon has to do and at this point it was certainly arrogant to says so. With a cloud behaviour defining test suite and a strong statement from the OpenStack Foundation that proclaims that "This is how clouds behave" Amazon's cloud might just behave the same way. And then this is not an arrogant statement anymore but just reality.

Cloud interoperability is essential to the success of OpenStack in my opinion. This is certainly not the last post on that topic.

Nuking a big cluster with PXE boot, DBAN and Crowbar

2012-10-05T00:00:00+02:00

When you are deploying clusters on real hardware - again and again - to test the deployment - it is quite helpful to have the hardware in a clean state. Unfortunately there is no real "Factory reset" button on hard drives.

However, there is an simple solution to this problem: When you install your cluster using PXE boot (like when using Crowbar), you can easily wipe all the hard drives of the whole cluster using this config.

A word of warning at this point: The following steps describe how to DELETE data from your whole data centre (if you are not careful). So do backups, use with care, etc. You have been warned.

Step 1: Getting DBAN

DBAN is a small custom linux boot image that has only one purpose: Delete all disks.

Download the ISO and extract the DBAN.BZI file.

Step 2: Set up PXE boot

We are using this setup out of our Crowbar installation, so the PXE environment is already setup. If you use it stand alone, with Cobbler or something else, adjust the paths accordingly.

In the Crowbar case, create a file /tftpboot/discovery/pxelinux.cfg/nuke with the following content:

DEFAULT nuke
PROMPT 0
TIMEOUT 10
LABEL nuke
  KERNEL DBAN.BZI
  append nuke="dwipe --autonuke --method zero" silent vga=785
  IPAPPEND 2

Copy the DBAN.BZI file to /tftpboot/discovery/

Step 3: Prepare the nuke

Since we are going to nuke anything anyway and do not want chef to interfere with our evil plans, we stop chef-client on the admin node:

bluepill chef-client stop

Now we enable the self destruct button:

cd /tftpboot/discovery/pxelinux.cfg/
ln -fs nuke default

Step 4: Nuke the cluster

To nuke the entire cluster (excluding the admin), just delete all nodes from crowbar - either by clicking through the UI or by using the crowbar CLI tool.

Then reboot the nodes via IPMI and wait for the data destruction to commence.

Nuking the admin node can be done using the DBAN iso via some virtual IPMI drive.

Summary

You can easily reset a lot of machines using PXE boot and DBAN. When you are using Crowbar the setup shown here makes it easy to start completely fresh any time.

Crowbar "cleans" the hard disks itself when you trigger an install, but some times this does not suffice. Some LVM parts remain intact and the following installation does not work properly. Using the DBAN nuke we can ensure that we really do start with empty hard drives on every node.

Deploying OpenStack with Crowbar at scale

2012-09-26T00:00:00+02:00

I've been working with @ehaselwanter on an OpenStack deployment with Crowbar for a project we are currently doing for T-Labs.

"Scale" means different things to different people. So let us be specific here: We deployed 80 nova compute nodes and 18 Swift nodes, so that adds up to close to 100 nodes with Crowbar.

The installation from empty, humming bare metal to a running OpenStack cluster takes about 1 to 2 hours. Of course the development effort to make this work took a bit longer.

I want to share the things we learned because either nobody has done this at this scale before or if someone did they did not talk about it and/or did not share the code changes. And code changes are necessary to get it working.

The naive approach to this kind of deployment is like this: You say "What works with 5 VMs or 10 physical servers surely works with 100 servers as well". Of course, it does not.

We identified the following problems:

Coordination problems

Crowbar uses chef in a very specific way. The Chef run on the Crowbar admin node is basically part of the Crowbar application. It takes place in the crowbar internal installation state transitioning process. This means it creates DHCP and DNS configuration in order to serve the right PXE boot configs etc.

The problem that arises at about 50 nodes (in our case) is this: - A node is in state "discovered" and is triggered to be "allocated" - This triggers a chef-client run on the to-be-installed-node as well as the admin server - The chef-client run on admin takes longer than the client-run + reboot on the to-be-installed-node which then boots again into discovery mode.

This was the basic high-level problem that we identified. I guess internally all kinds of strange things happened. The symptoms where like this:

We had the following options:

a) Real solution: Coordinate state between admin node and installed node (e.g. reboot node when chef-client run on admin has succeeded and set up the installation correctly)

b) Hacky, time pressured, get-it-done solution: Make chef-client run on admin fast enough (which is the current behaviour case for smaller setups).

After we wiped our engineering tears out of our eyes when we realised there was no time for a) we went with b) and reduced the chef-client run time on the admin node from about 3-5 minutes to about 50 seconds. To find slow chef cookbooks, we used this simple script.

The fixes where quite simple: Remove unnecessary searches, make sure that e.g. DHCP, DNS and PXE boot configs only get changed, when actual changes happen. This was mostly done by enforcing order in hashes/arrays that were used to generate templates. The DNS part was a bit tricky because the "zone serial number" should only change when something else changes and it should not trigger changes itself.

Networking setup

The way the nova network config was set up, clearly was created with /24 networks in mind. With 256 hosts it does not matter if you change something for every IP in that range. With a /16 network and over 65k IPs it does matter.

The fixes here were straight forward and we actually were surprised that there were not more problems with the IP range and network configuration.

Bonding

… or how I stopped worrying about udev and love the crowbar approach. According to the documentation (as I understand it) udev is supposed to name the ethernet devices according to the MAC address. This should result in an ordered naming scheme. However, this is not what we observed.

With crowbar you can define how network cards will be addressed in the bc-network-template.json file. We use this approach to create two bonding interfaces (one for 1G and one for 10G networks). The ethX naming is not consistent across nodes, however the two 1G and two 10G interfaces are bonded and assigned to the right interface. The bonding interfaces have to be consistent across nodes (We changed crowbar to make it that way) because OpenStack refers to several network interfaces not only in the config files (which would be fine) but also in the database. With the database as central storage the naming has to be consistent so that OpenStack can find the right interfaces to do its work on.

The whole bonding setup was quite hard to get working. The reason for this is the way Ubuntu (and I guess Linux networking in general) uses the networking config files: As a way to provide command line options for several tools. This means that transitioning from one setup to another requires steps like this:

Tear down interfaces based on current settings
change config
create new networking config from files

There is already a lot of code in Crowbar to orchestrate transitions from one setup to another but we had to extend some parts - and we wanted to keep the networking management parts of the distribution.

Development workflow

We do not use the Crowbar dev-tool. It does not do a lot more (that we need) that git does not do already but has a lot of assumptions about how you have to manage our code. So we just use git with git flow.

We think that git submodules are a pain to use and should be avoided in the future. Different versions of dependencies (barclamps, chef cookbooks, packages etc.) should be managed with tools like bundler, berkshelf or librarian. We're working on a setup that uses these tools in the Crowbar context and we are glad to see that submodules already seem to have vanished from the current development branch.

For the initial setup it is fine to use the ISO installation approach. To change things on the fly (and or make Crowbar work with actual 100 nodes) we needed a way to change things quicker. Crowbar uses Chef and Chef can do that, so we just configure knife to point to the crowbar server and to include all cookbook directories from the barclamp source tree and we are good to go. A simple knife cookbook upload nova and the latest OpenStack config changes can be applied.

Screenshots

Everybody loves screenshots:

Next steps

We've just finished the code for internal use. It is not public yet. We will start to integrate changes to the Crowbar open source repositories in the next days.

We would love to hear or read other stories from deployments at this scale. You can reach us via Twitter @hvolkmer and @ehaselwanter

Reducing development-production parity for OpenStack development with SmartOS zones

2012-09-14T00:00:00+02:00

Reducing Dev/prod parity has several advantages that are listed and explained at the linked page. Without a useful reduction of the parity Continuous delivery is impossible or at least akin to madness.

In context of OpenStack development and deployment dev/prod parity seems to be not one of the most addresses problems right now. From what I hear the de facto standard development environment for OpenStack is devstack. Devstack is a perfect fit to get OpenStack running and to start developing but it is not the way people deploy OpenStack in production. This leads to all kinds of problems that could be avoided.

Reducing dev/prod parity is very important and that's why I think of it at this point even though a SmartOS based OpenStack production deployment seems to be a thing of the future right now.

Development setup

It turns out that SmartOS is fantastic to address this problem. Let's see how the current development setup looks like:

I basically start up nova-compute in the global zone to have access to vmadm and start all the other services in separate zones. At first this seems to be a hassle and to much for development. But it forces you to think of all the things that will go wrong in production. For example: nova-compute needs to be able to access the database (which will be fixed in the future). So I need to setup the mysql credentials in a way to support that.

Multi node setup

How would this model look like in a more production like environment?

You can see that the basic service separation is the same. From an service standpoint it does not matter if the zones were deployed on a development host or on several physical hosts in the multi node production case.

Being able to develop OpenStack in a production like setup will reduce the likelihood of surprises when it comes to deployment. SmartOS zones help a lot to achieve this goal.

Do you want to know more?

If you want to know more about SmartOS and OpenStack, vote for my talk proposal "Porting OpenStack to SmartOS" at the upcoming OpenStack San Diego Summit.

Why SmartOS as an OpenStack base operating system?

2012-09-07T00:00:00+02:00

When SmartOS was first announced about a year ago, I downloaded the ISO, booted it in VMware, logged in and then… nothing. What is this? It's small, it is not supposed to be installed on disk. What do I do with it? What is so special about it? It is just an Illumos distro - a small and strange one. I did not get it.

I am now porting OpenStack to SmartOS and I think it is a perfect fit for that purpose. It is truly a Cloud OS. What does that mean? Let's go through the features of SmartOS that make it a perfect fit to be run as a Cloud Base OS.

ZFS

The ZFS storage model is based on Copy-On-Write which means that snapshots and clones are essentially free. This is fantastic for a Cloud compute node. Say you have a small set of defined VM base images and then spin up 20 VMs of Ubuntu 12.04. The Ubuntu image takes up about 500 MB of disk space. How much do 20 VMs of Ubuntu 12.04 use? Simple math right? Well, with ZFS these 20 VMs take 500 and a few MB. Total. Of course as the instances diverge this ratio gets lower but it is still impressive and very useful. Also spinning up Instances means you only have to clone the base image in ZFS which will take a few seconds the most, compared to copying it on a traditional file system. Somewhere in the internets I can hear someone shouting: "My SAN does the same thing since the 80ies, smartass". Sure, but it costs quite a bit more and you also get a lot of complexity.

The local storage needs for the base operating system get even better when you use Zones (see below).

DTrace

While the open source SmartOS (to my knowledge) lacks the cool graphical reporting possibilities based on DTrace it still contains the real DTrace capabilities. Everyone who had to hunt a memory leak or strange process behaviour in production will know what it is worth to have a good tracing capability at hand. DTrace will enable you to find performance and other problems that in a stable and very low overhead way.

Zones

Zones let you securely partition your OS without the overhead of another hypervisor (OS level virtualisation). In SmartOS a zone is always the outer "hull" of virtualisation. Either inside is just another SmartOS, or a qemu process with KVM that runs any other operating system. In combination with ZFS and Crossbow this is fantastic basis for virtual environments.

As Zones have minimal overhead they provide an ideal foundation for PaaS services like databases, load balancers etc. In an OpenStack context this would allow us to just use a special VM image that translates to a zone for a DB. We could thus avoid the whole RedDwarf/RedDwarf lite-back-and-forth-disaster.

KVM

Using KVM allows us to run (almost) any x86 operating system using the Intel VT-D, VT-X extensions with very little overhead. SmartOS has integrated KVM in a way that is easy to use. Actually I'm pretty glad that they did not use libvirt as abstraction layer but created their own.

SMF - Service Management Facility

With SMF you can define services, dependencies and let SMF manage them for you. I do believe that dependencies are important and a good way to model things for services (in contrast to what the Upstart designers think). Managing services in this model has been proving to work well e.g. with the Erlang Supervision Tree. With SMF you can define dependencies and restart behaviours of services as well as alerts when stuff goes awry.

Crossbow

Illumos comes with its own network virtualisation layer (basically a virtual switch) called Crossbow. In SmartOS the network virtualisation is integrated in the vmadm tool and works seamlessly. The virtualisation is based on VLANs: Each tenant will get their own VLAN. Crossbow was way ahead of the competition when it was released. With OpenVSwitch the Linux world has caught up and maybe even surpassed Crossbow. It will be interesting to see how the development of these technologies will continue.

SmartOS VM tools

imgadm, vmadm are tools to manage images and VMs. They are clearly written in a way to be used as part of a cloud platform. No config files with strange syntax or super long command line options. Instead these tools work with short and clear command line commands that take JSON files as options to do the real work. This is fantastic when used as an API from a cloud orchestration layer like OpenStack. And it feels a lot more like an API than libvirt (I save the libvirt "API" rant for another time... XML snippets as arguments. Really?).

A true cloud OS

The - at first look strange - model of usb key or PXE booting the base OS is a fantastic fit for cloud environments and makes a lot of problems go away. OS updates? Just reboot. How to install the OS? Don't. Just use a Ramdisk. Why waste space on disk when you can have a Ramdisk of 250 MB with the whole OS in it?

As you can see I'm pretty excited about SmartOS in this context. In combination with OpenStack this might be the best open source cloud orchestration stack that you can get.

Porting OpenStack to SmartOS

2012-08-31T00:00:00+02:00

General idea

SmartOS is a modern operating system that was actually created to run cloud orchestration software. Joyent uses it in their commercial Smart Data centre software. So it makes perfect sense to port OpenStack to it.

In fact, this idea is so obvious that there is a blueprint describing this.

Thijs Metsch wrote a there part (1,2,3) blog posts series about this endeavour. But it ends where it gets interesting: The part where you actually would start VMs, copy images, set up networking etc.

I built on his work and started where he stopped.

The plan

Create a nova Hypervisor backend for SmartOS based on the current folsom (master) branch of Openstack Nova.
Integrate networking through Quantum
???
Profit

Current status

I can currently start and stop VMs (both SmartOS zones and KVM based VMs) through the OpenStack API. Glance integration is there (Images from glance will be put into ZFS the way "imgadm" wants them). On the SmartOS side I use the vmadm and imgadm tools as an integration API. Basic networking also works. Theres quite a lot of work to do to get security groups, floating IPs etc. working.

Here are two screenshots that show how it currently looks:

Expect more info to come in the next days and weeks.

Refactoring life. Mercilessly. >> OpenStack

There will be no reliable cloud (part 3)

How I stopped worrying and love the cloud

Green field cloud native apps

Other benefits of apps in the cloud that help with reliability

But I have a legacy system and want to put it into the cloud

Further reading:

Closing thoughts

There will be no reliable cloud (part 2)

Complexity

Failure domains

Some math - or not

Problems with failures in big complex systems

The importance of partial failure

You have to design for failure anyway

Business side of things

Now what?

There will be no reliable cloud (part 1)

High Avaiblity vs. Service resiliency

Non-distributed base system with HA on top

Distributed System and the right tradeoffs

Definition of availability / reliability?

We have all the ingredients, so let's build that reliable cloud!

Deploying a multi node setup of OpenStack Folsom on Ubuntu 12.04.1 LTS with one command

Checkout the Chef cookbooks and Vagrantfile

Set up Chef server environment

Upload cookbooks to chef server

Deploy

Use it

What's next

Status of the SmartOS OpenStack Port

Current status

Future plans

More information

The fear of OpenStack fragmentation and the holy grail of cloud

Nuking a big cluster with PXE boot, DBAN and Crowbar

Step 1: Getting DBAN

Step 2: Set up PXE boot

Step 3: Prepare the nuke

Step 4: Nuke the cluster

Summary

Deploying OpenStack with Crowbar at scale

Coordination problems

Networking setup

Bonding

Development workflow

Screenshots

Next steps

Reducing development-production parity for OpenStack development with SmartOS zones

Development setup

Multi node setup

Do you want to know more?

Why SmartOS as an OpenStack base operating system?

ZFS

DTrace

Zones

KVM

SMF - Service Management Facility

Crossbow

SmartOS VM tools

A true cloud OS

Porting OpenStack to SmartOS

General idea

The plan

Current status