Category Archives: development

Getting Image and Flavor information from the Rackspace API

At Gamevy we’re lucky enough to have been accepted onto the Rackspace UK Startup Programme. In combination with that, we’ve been using Ansible to deploy new servers, and perform application deployments.

When you create a new server at Rackspace using Ansible rax module there are two pieces of information you’ll need from Rackspace. One of those is the flavor, this defines the instance type (the configuration of the instance – e.g. how much RAM it is given). The second is the id of the image (operating system) you wish to use to create the server.

To date, we’ve not found an easy way to get this information. Our current method is to use cUrl, get an Access Token, and to then make further requests to the API to get the required information.

Given this is an infrequent task, I normally forget the necessary steps. I thought that by noting them down I might save myself, and others, some time in the future. This post assumes that you already have a Rackspace account, and that you have API access via a key which is associated with your account. If that’s not the case, Sign Up here, and then follow these steps to get your API key.

It’s worth noting that this post focuses on accessing functionality provided by Rackspace UK. If you’re accessing from the US, the steps are the same, but  you’ll need to change the URLs that are accessed. Full information about Rackspace’s API can be found at their documentation site.

Getting An Auth Token

First things first, let’s get an Auth Token. To do this, you’ll need a few pieces of information from your Rackspace account: your username, your API Key and your account number.

Login to your Cloud Control Panel, and in the top right hand corner of the screen you’ll see your username with an arrow pointing down next to it. Clicking on that will reveal a menu at the top of which will be your account number, note that down, you’ll need when you make calls to the API, later.

From the menu, select Account Settings. On the resulting screen, access your API secret and note that too.

Create a file, call it whatever you want, I called mine rackspace-creds.json on your machine (I put mine in my home directory, for ease of access). The file should contain the following:

{
    "auth":
    {
        "RAX-KSKEY:apiKeyCredentials":
        {
            "username": "YOUR_RACKSPACE_USERNAME",
            "apiKey": "YOUR_RACKSPACE_API_KEY"
        }
    }
}

To get your Auth Token, open up a terminal and issue the following command

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X 'POST' \
-d @rackspace-creds.json -H "Content-Type: application/json" | python -m json.tool

I’ve piped the output of the cUrl command through python’s json tool to prettify it. In the response you receive, you’ll need to grab your :

"token": {
            "RAX-AUTH:authenticatedBy": [
                "APIKEY"
            ],
            "expires": "",
            "id": "YOUR_AUTH_TOKEN",
            "tenant": {
                "id": "",
                "name": ""
            }
}

List The Flavors And Images Available To You

Now that you’ve got an Auth Token, you can go ahead and make calls to the API endpoints for Flavors and Images. I’ve pasted the corresponding examples below:

Flavors

curl https://lon.servers.api.rackspacecloud.com/v2/YOUR_ACCOUNT_ID/flavors \
> -H "X-Auth-Token: YOUR_AUTH_TOKEN" | python -m json.tool

Images

curl https://lon.servers.api.rackspacecloud.com/v2/YOUR_ACCOUNT_ID/flavors \
> -H "X-Auth-Token: YOUR_AUTH_TOKEN" | python -m json.tool

 

Tagged , , , , , , , ,

Iterating an ansible task using a dictionary object of variables

Recently we have been writing ansible scripts to deploy one of our applications. In doing so, we came across the need to configure 3 node processes as services. All of these node processes used a similar init script but required different values writing into the script.

We chose to use the template task thinking that with the 4 variables we needed to populate on each occasion we should be able to use the with_items method on the task to do what we wanted. We did plenty of Googling and couldn’t find a way forward and so were all set to revert to writing three separate implementations of the template task, and three associated templates.

Asking a question on the ansible IRC channel eventually yielded an answer and I thought I would share our implementation.

We start with a generic template, service_init_script.j2. The script itself is not important to this blog post, you should note that way in which the ansible ( / jinja2) variables refer to item.foo – this is important, and was our initial stumbling block. Each time the task is iterated upon the items within the with_items block are referred to as an item:

#!/bin/bash
### BEGIN INIT INFO
 # Provides: {{ item.service_name }}
 # Required-Start: $all
 # Required-Stop: $all
 # Default-Start: 2 3 4 5
 # Default-Stop: 0 1 6
 # Short-Description: {{ item.service_description }}
 ### END INIT INFO
# Taken loosely from https://gist.github.com/tilfin/5004848
 prgcmd={{ item.service_exec }} # What gets executed?
 prgname={{ item.service_name }} # What's the name (used to ensure only one instance is running)
 prguser={{ item.service_user }} # Which user should be used
 pidfile=/var/run/{{ item.service_name }}.pid # Where should the pid file be stored?
 logfile=/var/log/{{ item.service_name }}.log
export BJA_CONFIG={{ bja_config }}
start() {
 if [ -f $pidfile ]; then
 pid=`cat $pidfile`
 kill -0 $pid >& /dev/null
 if [ $? -eq 0 ]; then
 echo "{{ item.service_name }} has already been started."
 return 1
 fi
 fi
echo -n "Starting {{ item.service_name }}"
nohup start-stop-daemon -c $prguser -n $prgname -p $pidfile -m --exec /usr/bin/env --start $prgcmd >> $logfile 2>&1 &
if [ $? -eq 0 ]; then
 echo "."
 return 0
 else
 echo "Failed to start {{ item.service_name }}."
 return 1
 fi
 }
stop() {
if [ ! -f $pidfile ]; then
 echo "{{ item.service_name }} not started."
 return 1
 fi
echo -n "Stopping {{ item.service_name }}."
start-stop-daemon -p $pidfile --stop
if [ $? -ne 0 ]; then
 echo "Failed to stop {{ item.service_name }}."
 return 1
 fi
rm $pidfile
 echo "."
 }
status() {
if [ -f $pidfile ]; then
 pid=`cat $pidfile`
 kill -0 $pid >& /dev/null
 if [ $? -eq 0 ]; then
 echo "{{ item.service_name }} running. (PID: ${pid})"
 return 0
 else
 echo "{{ item.service_name }} might have crashed. (PID: ${pid} file remains)"
 return 1
 fi
 else
 echo "{{ item.service_name }} not started."
 return 0
 fi
 }
restart() {
 stop
 if [ $? -ne 0 ]; then
 return 1
 fi
sleep 2
start
 return $?
 }
case "$1" in
 start | stop | status | restart)
 $1
 ;;
 *)
 echo "Usage: $0 {start|stop|status|restart}"
 exit 2
 esac
exit $?

We then wrote our ansible script and call the template task thus:

- name: Create the service init scripts
template: src=service_init_script.j2 dest=/etc/init.d/{{ item.service_name }} owner=root group=root mode=755
with_items:
- { service_name: paymentHandler, service_description: 'Handles payments', service_user: blackjack_attack, service_exec: "{{ application_directory }}/apps/paymentHandler.js" }
- { service_name: tablePool, service_description: 'The pool of tables available', service_user: blackjack_attack, service_exec: "{{ application_directory }}/apps/tablePool.js" }
- { service_name: userManager, service_description: 'Manages users', service_user: blackjack_attack, service_exec: "{{ application_directory }}/apps/userManager.js" }
sudo: yes

So, we are calling the same template task, which in turn refers to the same jinja2 template, passing in a collection of dictionary objects.

The key point to understand here is that when the playbook is run ansible enumerates the items specified referring to each object as ‘item’. This is the reason why we had to prefix the variable names in the template with item.*

I’ve exposed both of these pieces of code as gists here, and here.

Tagged , , , , , , , ,

Do we need clearer roles and responsibilities?

Why are people are still preaching that better, more clearly articulated roles and responsibilities will improve productivity? Provide a transparent environment where people have a clear understanding of what they’re working toward, and its value, and you’ll get better results.

As I write this, I’m sitting on a train travelling home. I’ve just been through the blog posts in my RSS feed. Three offer very similar advice about how to improve an individual’s productivity in the workplace. Their answer to this perennial conundrum is that we need to provide clearer objectives and roles and responsibilities on a daily basis. Given that the posts originate from people promoting Agile as a more enlightened way of working, I shake my head in wonder. Really, I do.

The value of roles

There are times when clear roles and responsibilities are essential – this post is not to suggest that their use be completely dropped. Consider the outcome of the incident now known as the Miracle On The Hudson; having just taken off from New York City’s LaGuardia Airport, a US Airways aeroplane, flight 1549, struck a flock of geese. Both engines failed. Some 3 minutes later, the Captain and his crew, a First Officer and 3 Flight Attendants, had performed an emergency landing on the Hudson River. In an air crash, speed, accuracy of trained response and calm are paramount. Although the First Officer had been flying at the time, the Captain took the controls, leaving the officer to go through a 3-page emergency procedure to restart the engines. The three flight attendants showed passengers how to brace themselves for impact and took control of evacuating the plane, even as it filled with water. The captain walked up and down the sinking aircraft to be sure no-one had been left behind and was the last person to leave. Each of the crew members was helped by their training and their knowledge of their specific, unique roles and responsibilities.

The difference for IT

The problem, is that few business teams – IT or otherwise – face the same kind of challenge as existed for the air crew. In IT, problems tends to require a creative response, one that cannot be considered in advance, which no training will prepare us for, but which must be invented anew each time. Strict roles and responsibilities don’t help with this. Indeed, they may hinder it because instead of thinking freely, people are prescribed. Control does not sit well with creativity, originality or motivation. Attempting to control an individual in a group setting reinforces a silo-based mentality to work. That means that the attempt to control an individual through a strictly defined role and associated responsibilities, affects the whole system. In my opinion it compels people to work within the boundaries that are set for them – intentionally or otherwise.

Do rigid role boundaries assist in speed?

In the air crash example, knowing who is responsible for each task helped speed. Proponents of clear roles and responsibilities suggest that decisions are made faster and are accepted with less confusion. Yet in my experience one of the negative results is a reduction in collaboration. The classic case of this, which I am sure we have all experienced, is somebody justifying their inaction with: “well, it’s not my responsibility”. I’ve known many cases where people fought NOT to take responsibility for an unpleasant job or fought over who TOOK responsibility for a prestigious job and the result was always that the product slowed or sometimes stalled as collaboration evaporated.

Individuals can always change the rules

So far I’ve focused on the work. But software development is ultimately a people-orientated endeavour. I was thinking about this as I sat with my son – a four-and-a-half year old, to do his numeracy homework. For some reason he was writing a lot of his numbers back to front. I kept on trying to correct him and at one point said to him “just write me five nines”. His response was to write the number 5 and 9. Perfectly, I might add. Some might say it showed intelligence. I’m inclined to say that he was being subversive. Either way, it was a proud Dad moment. But that’s one of the problems with the advice being offered: people don’t like being told what to do. They either do what they think is best at the time, or in a small amount of cases intentionally subvert the system.

Demotivating boundaries

When we’re children recognition from our dads and teachers motivates us, as we grow older the influence of our peers is what counts. I tend to perform best when I know I am there to make a defined contribution. To this end, a degree of definition of my role and responsibilities helps, but a rigid boundary is demotivating, because it constrains my ability to contribute as much as I could or would like to. This demotivation is exacerbated when people have to discover where their responsibilities lie. I’ve heard people refer to this as being knocked back – they’ve tried to do something which is perhaps of interest to them, or where they perceive there to be an opportunity, only to be told that their contribution wasn’t wanted.

Doing things differently

So what instead? As with all pernicious problems the first step is to observe what is happening. You might find people in a stand up talking about the piece of work they’re doing, rather than the work the team needs to do. Perhaps you see executives or departments becoming more rigid in role demarcation. A lack of vision or sense of purpose might pervade the organisation. Worst of all, perhaps no-one thinks making things better is any part of their ‘responsibility’. Cause and effect is difficult to establish in a software development scenario, but you should be able to state your observations and assumptions. Establish what effect an improvement might have. Test your assumptions with others – what’s their opinion? Don’t fall in to the trap that it is your role to change people. Remember, the way in which people behave is far more likely to be driven by the environment around them than anything else. Once there is agreement in the group about the perceived problem, consider what you might change to bring about improvements. Test whether these improvements have had the desired effect. Respond to the feedback you receive. Rinse and repeat.

I know that those advocating clarity of roles and responsibilities see the same problems in demotivation and wasted efforts as I do. The difference is that they believe more control is the answer, while I believe that less direct control frees people to create their own solutions. Those 5 crew members on US Airways flight 1549 worked as a team, each within their specialisms, and brought off “the most successful ditching in aviation history”[1]. Software development teams need to be more cross-functional in nature, but I believe that they too can perform their own miracles – as long as they have an environment that supports them rather than dividing them.

Thanks go to the inimitable Joshua James Arnold, Lady H of Edmonton (but of no fixed Twitter abode), and the sartorially precise Mark Krishan (50 shades of) Gray for helping me to improve this blog post.

[1] New York Post, 2009. Quiet Air Hero Is Captain America. [online] Available at: <http://www.nypost.com/p/news/regional/item_Goem4fAiUd2hsctASfAjGJ>. [Accessed 28 November 2012].

Madmen In The Halls

I read Tom De Marco’s book Slack a few months ago, the first chapter titled “Madmen in the Halls” starts by suggesting that:

“The legacy of the nineties has been a dangerous corporate delusion: the idea that organizations are effective ony to the extent that all their workers are totally and eternally busy. Anyone who’s not overworked (sweating, staying late, racing from one task to the next, working Saturdays, unable to squeeze time for even the briefest meeting till two weeks after next) is looked on with suspicion. People with a little idle time on their hands may not even be safe. As [one of the authors friends] at Digital Equipment Corporation told [him] during the company’s darkest days, “There are madmen in the halls looking for someone to ax.” Of course, the ones they were looking to ax were the folks who weren’t all that busy.”

It’s sad to be able to draw parallels with some of the rash actions that were being taken then and the drastic ones people are being forced to take now.

Inevitably, as people look to make their respective companys’ more efficient and trim their bottom lines, Development teams are called upon to get the latest cost / time saving feature delivered as soon as possible.

It’s at times like this, when we’re being asked to go as fast as we can it becomes increasingly difficult to sell the fact that we need to deliver a quality product rather than one in which we’ve cut corners, as Jason suggests in his post, perhaps quality will suffer as part of the credit crunch. But now is precisely the time that we should be pushing harder to see the quality of our product improved.

I’m a believer in the Lean principle that you should decide as late as possible and that development teams should seek to defer decisions until the last responsible moment. Should we defer that decision to include unit tests, to tackle that particularly naughty piece of the code that is always causing us problems or to automate our build and deployment process. Definitely not.

Choosing not to pursue quality from the outset is almost like behaving like a student, leave doing it until the last minute and you will either hand in something that’s rushed, doesn’t really meet the expectations and will therefore need to be reworked or having to ask for an extension to the deadline.

Tagged

Let The Inmates Run The Asylum

One of my colleagues recently sent me a quote from this site which tickled me a bit.

“Those who do not have a clue are still debating about the process.
Those who know, just do it. (56)”

When reading through the rest of the quotes on the page I find the following one more interesting though:

“If you want to be a great leader, stop trying to control.
Let go of fixed plans and concepts and the team will govern itself.
The more prohibitions you have, the less disciplined the team will be.
The more coercion you exert, the less secure the team will be.
The more external help you call, the less self-reliant the team will be. (57)”

I must confess, I’m no Taoist, nor (other than having read The Tao of Pooh, which is another story) do I know that much about it, particularly in the context of Software Development, but in the respect of what a manager can do to establish a well functioning team or department, I agree with a lot of the points above.

I’ve witnessed my fair share of change in what is a relatively short career, I’ve seen it instigated by managers and I’ve also observed it come from the people on the ground. Each approach has it’s relative merits in any given situation, in my opinion though, the act of facilitating change will tend to success more than a command and control structure.

Self organising teams need to be nurtured, they need to feel as though they are empowered to make change, furthermore and perhaps most importantly they need a clear vision of their remit as well as some parameters to operate within.

I think that it is relatively easy to define the vision and parameters for a development team; they should seek to deliver value, aggressively tackle waste and to do both of those continually.

One of the things that is starting to interest me more and more though is business transformation and how some of the agile techniques such as self organisation can be applied. I believe that self organisation can be used as a mechanism to deliver enterprise change however, I think it is even more important to ensure success that a clear vision is communicated and that some definite parameters even if these may change later, to work within are stated.

Tagged

A little light reading

I’ve decided to switch the feed from my del.icio.us account to this blog off, if you still want to receive the feed you can subscribe to it here.

I’ve been following Karl Seguin’s foundations of programming series and have been impressed, it’s a good starting point I think and covers a lot of the technologies that we use here. He’s recently posted them all, compiled as a PDF, you can go get it here.

Tagged , , , , ,