Computational Cognitive Neuroscience : Information on Projects

Assistant responsible with the programming projects: Hande Celikkanat
Email: hande.celikkanat@helsinki.fi
Office: Exactum A340
Office hours: Friday 15:00-17:00. You are always welcome to have an appointment by email for any time outside of the office hours.
 

Quick Links
#Project 2 Topics
#Project 1 Topics
#Specifications
    #TL;DR
    #1. About the programming language selections
    #2. What to submit
    #3. How to submit
 
 
 
 
Project 2 Topics
For the second project, you can choose between:

Option 1: [Programming Project] PCA and ICA on natural images

Option 2: [Programming Project] The Olshausen-Field model

Option 3: [Essay] Bayesian inference in vision
Possible discussion points include, but are not limited to: Vision as Bayesian inference. Bayesian inference in early vision, eg., V1. Bayesian explanations of higher level vision - examples include: pattern recognition, object perception, scene classification, etc.

Option 4: [Essay] Frequency channels in early vision and their modelling
Possible discussion points include, but are not limited to: Types of cells which exhibit selectivity to frequencies. Experimental findings. Modeling of frequency channels. Gabor filters.

Option 5: [Article Summary] Brett Vintch, J. Anthony Movshon, and Eero P. Simoncelli, "A convolutional subunit model for neuronal responses in macaque V1", The Journal of Neuroscience, 2015. [pdf]

Option 6: [Article Summary] Jeremy Freeman, Corey. M Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon, "A functional and perceptual signature of the second visual area in primates", Nature Neuroscience, 2013. [pdf]

Option 7: [Article Summary] Haruo Hosoya and Aapo Hyvarinen, "A hierarchical statistical model of natural images explains tuning properties in V2", The Journal of Neuroscience, 2015. [pdf]

Option 8: [Article Summary] Mariano Sigman, Guillermo A. Cecchi, Charles D. Gilbert, and Marcelo O. Magnasco, "On a common circle: Natural scenes and Gestalt rules", Proceedings of the National Academy of Sciences, 2001. [pdf]

Option 9: [Article Summary] Odelia Schwartz and Eero P. Simoncelli, "Natural signal statistics and sensory gain control", Nature Neuroscience, 2011. [pdf]
 
 
Project 1 Topics
For your first project, you are welcome to choose between:

Option 1: [Programming Project] FeedForward-Backpropagation Neural Network, Autoencoders

Option 2: [Programming Project] Associative Memory: Linear Associative Memory, Hopfield Networks

Option 3: [Essay] Hebbian Learning
Possible discussion points include, but are not limited to: Investigating the different facets of Hebbian learning: biological, computational and cognitive. Associative memory (Sections 12.2 and 13.3 from Rojas). PCA using Hebbian algorithms. Predictive mirror neurons (paper by Keysers and Gazzola). Long-term potentiation.

Option 4: [Essay] Associative Learning
Possible discussion points include, but are not limited to: Linear associative memory (Rojas section 12.1.2). Hopfield networks (Rojas section 13). Associative memory (Sections 12.2 and 13.3 from Rojas). Similarities with human memory. Stability/plasticity dilemma. Convergence expectations.

Option 5: [Essay] Deep Learning
Possible discussion points include, but are not limited to: How is deep learning different from the classical neural networks approach? In which characteristics it is similar? Why has deep learning been regarded as a breakthrough in machine learning even though multi-layered architectures have been invented long ago? What are the strong vs. weak points of its current state-of-the-art? Why?

Option 6: [Essay] Restricted Boltzmann Machines
Possible discussion points include, but are not limited to: Inspirations for the Restricted Boltzmann machines. Their differences from the classical Boltzmann machines, and the consequences of these differences. The "stacking" of RBMs and their usage within deep networks.

Option 7: [Essay] Recurrent Neural Networks and Backpropagation Through Time
Possible discussion points include, but are not limited to: Computational implications of using recurrent connections. Possible applications. Sequence Learning. Advantages and disadvantages of using Backpropagation Through Time for training recurrent networks.

Option 8: [Essay] Recursive Neural Networks and Backpropagation Through Structure
Possible discussion points include, but are not limited to: Computational implications of using a recursive structure. Possible applications. Differences and similarities between recurrent neural networks and recursive neural networks.

Option 9: [Article Summary] Alex Krizhevsky and Geoffrey E. Hinton, "Using very deep autoencoders for content-based image retrieval", Proceedings of the European Symposium on Artificial Neural Networks, 2011. [pdf]

Option 10: [Article Summary] Carina Silberer and Mirella Lapata, "Learning grounded meaning representations with autoencoders", Proceedings of the Association for Computational Linguistics, 2014. [pdf]

Option 11: [Article Summary] Celia Abernathy and Lisa Bao, "Grounding Color and Shape Words in Visual Context", 2012. [pdf]

Option 12: [Article Summary] Thomas Riga, Angelo Cangelosi and Alberto Greco, "Symbol grounding transfer with hybrid self-organizing/supervised neural networks", Proceedings of the IEEE International Joint Conference on Neural Networks, 2004. [pdf]

Option 13: [Article Summary] Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng and Christopher D. Manning, "Parsing natural scenes and natural language with recursive neural networks", Proceedings of the International Conference on Machine Learning, 2011. [pdf]

Option 14: [Article Summary] Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng and Christopher Potts, "Recursive deep models for semantic compositionality over a sentiment treebank", Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013. [pdf]
 
 

Specifications

TL;DR:
For programming projects:
    - Using the provided templates is mandatory for any language choice.
    - Returning both the code and the report is mandatory.
    - Please kindly notify me in advance if you're planning to use another programming language than Python.

For all projects:
    - Late submission is allowed in exchange of a deduction per day.
 

Long Version:

1. About the programming language selection
- As noted in the class, we strongly suggest using python for your programming projects, due to a number of reasons. Python is free, extremely flexible, and gaining more popularity everyday, which means it is a good idea to get familiar with it. In case you decide to use Python, you will be needing the numpy, scipy, and matplotlib libraries. These can be imported for instance as following:

    import numpy as np
    import matplotlib as mpl
    import matplotlib.pyplot as plt
    from scipy import linalg,stats

- In case you are planning to use any other programming languages than this, please send an email to me first so that I have an idea. This is mostly for me to be aware of everyone's choices, and to confirm that I'm able to assist and partially-grade everyone's project if needed. (Most widely-used languages should be ok.)

- '''Importantly''' note that using the templates is mandatory even if you are using a different programming language than Python, in which case you are expected to utilize the same template function signatures in your language of choice.

2. What to submit
(This section applies only to the programming projects.)

- You are expected to submit BOTH your code and a report.

- The code is ultimately expected to run as completely as possible, though you will be awarded partial grades even for incomplete code and design.

- The report is expected to explain your design, the experiments you have conducted on your program, and the results you have observed. It is important to discuss your opinions about the strong and weak points of the method, why/how different settings produce different results, and how you tie these to the expectations you have acquired from the literature. Ultimately, what we wish is that you gain valuable insight by seeing real-life implications of the (explicit and implicit) assumptions inevitably made in every model.

- The report should additionally include how to run your code exactly. Indicate which files are included, as well as the directory structure, how to run the code (eg. Does your program run with command-line parameters? Is the user required to enter input from a GUI and if so, how? Does a tarball need to be extracted? Is a compilation via make necessary? etc.) It should also specifiy the dataset you have used clearly and provide the link if you have proposed your own dataset. In general, information in the report needs to be sufficient for running your code.

- Please kindly write your comments, function names, and variable names (if applicable) in English. This is not a requirement per se, but will make partial grading a lot easier if necessary.

3. How to submit
(This section applies to all projects.)

- Submissions are via email to hande.celikkanat@helsinki.fi before 23:59 on the announced deadline.
- You are allowed to make late submissions in return for a deduction in your grade. The late day penalty is additive:
    Overall grade = Net grade - 5 x (day)^2,
where (day) is the number of late days, each late again extending until 23:59 of the associated day.

Regarding any unclear points/questions/suggestions, you are more than welcome to contact me either by email or in person.

(Don't let the long text discourage you, and) Have fun :)

Hande

 

 
Programming Project 2.1 Details: PCA and ICA on natural images
For this project, you are going to try out PCA and ICA on natural images, using ready-made implementations.

Use PCA and ICA from the scikit-learn package, see e.g.:
http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html
http://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_iris.html

Do the following:
a) Generate artificial data from ICA model. In other words, create independent components by a random number generator, and mix them with some mixing matrix A.
b) Do PCA and ICA on that data.
c) Evaluate the results, i.e. do you recover the original components with PCA or ICA? (Note that ordering of the components is not defined.)
d) Take some image patch data (e.g. as in Exercise Session 5), and again do PCA and ICA on it.
e) Plot the results creating some nice visualization like in NIS book (see exercises for some plotting advice).
f) For both datasets (artificial and image patch), compute correlations between the estimated independent components. Compute also nonlinear correlations, e.g. correlations of the squares of the components. Compare results and discuss (Hint: read section 9.1 in the NIS book).

 
 
Programming Project 2.2 Details: Implementing the Olshausen-Field model.
Read the paper Olshausen and Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images", Nature, 1996.

Implement the algorithm described in the paper and apply it on image patch data (e.g. the one in Exercise session 5). Reproduce Figure 4.

Note the following notation used in the paper:
a) <...> means expectation (over the set of image patches)
b) dot over "a" is time derivative, Delta is change of parameter (from the viewpoint of programming an algorithm, the dot and the Delta mean the same thing: how the quantities should be changed at every step of the algorithm.)

 

 
 
Programming Project 1.1 Details: FeedForward-Backpropagation Neural Network, Autoencoders
Your main task is to implement (1) a feed-forward classifier network, (2) a one-layer autoencoder and (3) a backpropagation learning algorithm (based on Rojas 7.3.3) which can be used to train both of them.

Play with these and try to get a sense of the effects of certain settings. What happens when you change the learning rate? The initialization scheme? How does the number of layers effect performance? Do you observe strong improvement in performance from the transition to 0-hidden-layer case to 1-hidden layer case? Between 1-hidden-layer case and 2-hidden-layer case? Does the number of neurons in the hidden layer(s) effect the performance significantly? Is it better or worse to provide more hidden neurons than the number of neurons in the input layer? How do certain changes in the network structure reflect on the time performance, and is this compatible with your expectations given your knowledge of the structure of the networks?

Importantly, observe what happens when you train networks of 0 hidden layer, 1 hidden layers, 2 hidden layers, and more. How does the performance of a large number of hidden layers (i.e., more "deep" networks) compare with that of more shallow ones? Why do you think that is?

Your next task is related to this finding: You will combine the feed-forward network with the autoencoder and see if you can obtain better results this way. Try using your previously trained autoencoder as your first hidden layer, and train the further layers with standard supervised backpropagation algorithm. Do you observe a performance difference?

In addition, please design and make your own experiments to your heart's content.
Report on the effects. We encourage you to try to get a feeling of the strong and weak points of the respective approaches. Do you encounter any behavior that seems contrary to the literature? Why do you think that happens? You are welcome to report failures as well as successes, please do comment on the most possible reasons of failures.

Notice also the sensitivity of the network to the learning parameters.
Note that the sensitivity can be especially strong in the autoencoder case, we encourage you to try different parameters in order to observe the difference in performance.

Announcements
1. Possible datasets you can choose from (same with project 2):
   (These are provided for your convenience only. You are very welcome to propose a dataset of your own choice.)

There are also a large number of dataset options in these two links, have a look:

You can use these datasets anyway you like, feel free to downsample, convert to grayscale, or apply any preprocessing. The only requirement is that you are expected to divide your data into training and testing parts, and perform evaluations mainly based on the test data. Please indicate which dataset you are using clearly in your report, and add a full link to a download location in case it is not one of the above.

2. Template code for this project: project_1_template.py (Mandatory to use, please see above.)
3. A GUI template code that you can use to prompt input from the user: gui_template.py (optional to use)

 
Programming Project 1.2 Details: Associative Memory: Linear Associative Memory, Hopfield Networks
For this project, you are going to implement (1) a linear associative memory (Rojas section 12.1.2) and (2) a Hopfield network (Rojas section 13), and compare their performances.

Play with these and try to get a sense of the effects of certain settings. What happens when you change the learning rate? The initialization scheme? How does varying the connectivity structure effect the outcome? How does adding more neurons affect the training time? The test time? How do changes in the network connectivity structure reflect on the overall performance as well as the time performance, and is this compatible with your expectations given your knowledge of the structure of the networks?

Comment on the convergence of your network. How often do you obverse a convergence to a local minimum instead of a global minimum? Are those cases predictable?

Comment on the similarities of the Hopfield model with human memory. Do the basic principles make sense to you? Do you find them realistic? Why / why not?

In addition, please design and make your own experiments to your heart's content.
Report on the effects. We encourage you to try to get a feeling of the strong and weak points of the respective approaches. Do you encounter any behavior that seems contrary to the literature? Why do you think that happens? You are welcome to report failures as well as successes, please do comment on the most possible reasons of failures.

Announcements
1.Possible datasets you can choose from (same with project 1):
   (These are provided for your convenience only. You are very welcome to propose a dataset of your own choice.)

There are also a large number of dataset options in these two links, have a look:

You can use these datasets anyway you like, feel free to downsample, convert to grayscale, or apply any preprocessing. The only requirement is that you are expected to divide your data into training and testing parts, and perform evaluations mainly based on the test data. Please indicate which dataset you are using clearly in your report, and add a full link to a download location in case it is not one of the above.

2. Template code for this project: project_2_template.py (Mandatory to use, please see above.)
3. A GUI template code that you can use to prompt input from the user: gui_template.py (optional to use)

 
 

Tulostettava sivuTulostettava sivu