Distributed Systems : Workshops

Basic information

Thu Nov 27

No program. Ossi will be available in C222 for questions.

Thu Nov 20

No program. Ossi will be available in C222 for questions.

Workshop on Thu Nov 6 in C222

Discussion about Paper 6

Workshop on Thu Oct 30 in C222

Discussion about the first two Home exercises

Workshop on Thu Oct 16 in C222

Q&A about the first two Home exercises

Workshop on Thu Oct 9 in C222

Q&A about the first two Home exercises

Workshop on Thu Oct 2 in computer class Exactum B221

Going through the exercise given for last week (read below)

There will be no workshop on September 25, instead do the following two exercises that should ready you for the second home exercise. We will go though them in the next workshop on October 2, but the idea is that everyone should try to have a working solution before that.

The Tale of Cat and Mouse, The Prologue

1) Make two bash scripts, one for the mouse and one for the cat.
The mouse listens a port through nc (netcat). If it receives a "MEOW" message it has 25 percentage change of running away and 75 % change of being caught. If the mouse is caught, print out its dramatic last words. If you grew up on a farmhouse, feel free to change the odds to better reflect your own experience.

The cat has to first find out whether to the mouse process is running or not. When the cat detects the mouse it will send its growling "MEOW" to the mouse using nc and tries to catch it.

The port number to use is 30000 + last three digits of the mouse's process id.
You can choose two ukko nodes or just use localhost to establish the communication.

2) Select 5-20 Ukko nodes. Modify the codes so that the mouse will now go to one of the selected nodes. The mouse it will choose the node where the most delicious cheese is available. The cat goes through the selected nodes to find the mouse, and when found, tries to catch it.

Tasks for Workshop on Thu Sep 18 in computer class Exactum B221

Bash tutoring

File redirection, signals
Netcat, ports
Look the exercises found on http://wiki.helsinki.fi/display/linuxfun. We can go through them in the workshop, if there is something that is interesting/useful

Tasks for Workshop on Thu Sep 11 in computer class Exactum B221

The aim is to finish the tasks from the last week's workshop so that everybody would have their ssh keys set up and so on. Please set up your keys even if you don't attend the workshop. You'll need to have them working later on the course.

If you have the keys working you might want to try these exercises.

1. "RPC" over SSH
Take a look at the Ukko cluster if you already haven’t done so:
http://www.cs.helsinki.fi/en/compfac/high-performance-cluster-ukko
http://www.cs.helsinki.fi/ukko/hpc-report.txt

Select 4 nodes from the HPC report that are not reserved for research groups. Use the parallel-ssh tool installed on the Ukko cluster nodes ('man
parallel-ssh') to connect to these 4 nodes and run the 'uptime' command in all of them.
Notice how the responses of the different cluster nodes are (probably) not in the same order as you put them in the hosts file. What could cause this, and
what do you expect would be the most likely explanation?

2. SSH RPC
Your task is to approximate RPC function calls using the SSH protocol. This task is also an intro to cluster- and cloud-level computation management.
Use nodes from the Ukko cluster for experimentation. Take care not to use nodes reserved for research projects.
Your task is to build a shell or Ruby/Perl/Python script that does the following:

1. Accept as input n node names in the form of a directory structure
./
./ukko001
[...]
./ukko240

You can select which nodes to use. The minimum requirement is five nodes.

2. Accept as command input n commands as text files
./ukko001/INPUT
[...]
./ukko150/INPUT
The specific node names will vary with the node names chosen in #1.
The input commands COULD be the same, but if you implement the task properly, this will not matter.

3. Run the given commands on each Ukko node concurrently (hint: call the SSH connection using &)

4. Record the output from each node into a corresponding text file
./ukko001/OUTPUT
[...]
./ukko150/OUTPUT
The output contents will almost certainly be different for each node, depending on which commands you chose to execute.

Tasks for Workshop on Thu Sep 4 in computer class Exactum B221

1. Your first shell script.

Create a new text file (say, helloworld.sh) and input the following content into it:

#!/bin/sh

x=5

echo Hello world!

Then save the file, make it executable ('chmod u+x helloworld.sh') and run it (./helloworld.sh).

Now open the file again, and find out how to make a 'for' loop (to echo 'hello world' x times) and an 'if' clause in it (to echo 'hello world' only if the variable x equals 4). Alternatively, you can do this part in Python or whatever scripting language you are familiar with.

Note that you can use #!/bin/bash instead of #!/bin/sh. This allows a different and some cases more flexible syntax, but all-in-all they are mostly the same.

If you find you have problems with this exercise, you should join the C Programming course as soon as possible to ensure you have the necessary skills to do the programming exercises on the course. The language used does not matter so much as the basic coding skills. If you want to deepend your skills, the course Linux Fundamentals may be of interest.

Helpful material:

http://www.ee.surrey.ac.uk/Teaching/Unix/
http://www.tldp.org/LDP/abs/html/index.html

2. SSH key management.

For the upcoming exercises involving the computational cluster, you will need to set up a key pair for ssh.

Take a look at the manual pages ('man ssh-keygen') and generate a private and public key pair for yourself. To use the key pair, the public key must be accessible in the Ukko cluster node you will use. Remember to protect your private key with a passphrase that is at least as strong as your department login's password, because this key is equivalent to your login and password. (You can use the same password as is your department login, IT support is ok with this; why?) The key passphrase is not actually stored on the key. Just do not mix your department password into any access to your home computer or other systems.

Install your generated key to your department (fs) home directory according to the instructions linked below.

Reading material as background and further instructions: OpenSSH key management, part 1: http://www.ibm.com/developerworks/library/l-keyc/index.html

Things to learn: How ssh keys work. The keys are needed to interact meaningfully with multiple nodes in the department network, such as in any exercises done on the cluster computer. While in normal department existence you can get along without this login aid, this is setup to save your time in later exercises.

3. A first glance at the Ukko cluster.

Read the instructions on how to use the department high performance cluster Ukko. http://www.cs.helsinki.fi/en/compfac/high-performance-cluster-ukko

Test your ssh key by connecting to an Ukko cluster node. First examine the report at http://www.cs.helsinki.fi/ukko/hpc-report.txt to find a node that is live and not in use by a research project.

Then use ssh-add to activate your key identity unless you already have done so.

Log on to the chosen Ukko cluster node, make sure your password was not asked, and take a look around before logging off:

'ls' - (directory listing) notice that you have access to your home directory here too. It's just like another interactive server setup!

'who' - are other interactive users logged in on the same node? For contrast, you can try this command on the actual department interactive servers.

'uptime' - how long has the node been up and running? Do they boot these nodes every week?

'top' - you can look for resource-intensive jobs running on the node, 'man top' for documentation. 'q' to quit.