Welcome to **Reinforcement Learning and Dynamic Programming**

**Description:**

An intelligent system is expected to generate policies autonomously in order to achieve a goal, which is mostly to maximize a given reward function or minimize a given cost function. Reinforcement learning is a set of methods in machine learning that can produce such policies. In order to learn optimal actions in an environment that is not fully comprehensible to itself, an intelligent system can use reinforcement algorithms to leverage its experience to figure out optimal policies. Nowadays, reinforcement learning techniques are successfully applied in various engineering fields, including robotics (DeepMind’s walking robot) and computers playing games (AlphaGo and TD-Gammon). Developed independently from reinforcement learning, dynamic programming is a set of algorithms in optimal control theory that generate policies assuming that the environment is fully comprehensible to the intelligent system. Therefore, dynamic programming provides an essential base to learn reinforcement learning. The course aims at building a fundamental understanding of both methods based on their intimate relations to each other and on their applications to similar problems.

The course consists of the following topics:

• a short introduction to machine learning

• an introduction to Markov decision processes

• basics of dynamic programming

• basics of reinforcement learning

• approximate dynamic programming for high dimensional problems

• approximate reinforcement learning for high dimensional problems

• an introduction to quantum decision models

Throughout the course, the source book

*Reinforcement Learning and Dynamic Programming Using Function Approximators, **Lucian Busoniu, Robert Babuska, Bart De Schutter, Damien Ernst, CRC Press, 2010*

will be used for which MATLAB codes for numerical applications are available online on http://rlbook.busoniu.net/.

The goal of this course is to make doctoral students able to apply reinforcement learning and dynamic programming techniques to high dimensional machine learning problems.

** **

**Keywords: **Machine learning, reinforcement learning, dynamic programming, Markov decision processes, policy iteration, value iteration, policy search, approximate reinforcement, approximate dynamic programming, approximate policy iteration, approximate value iteration, approximate policy search, quantum decision theory.

**Prerequisite: **mathematics on the level of Master in science and engineering

**Organizer:** Rafael Wisniewski, Øzkan Karabacak, Zheng-Hua Tan**Lecturers:** **ECTS:** 2**Time: **September 30^{th} to October 4th, 2019

**Place:**

Zip code:

City: **Number of seats:****Deadline: **September 9th 2019

- Teacher: Ozkan Karabacak
- Teacher: Zheng-Hua Tan
- Teacher: Rafal Wisniewski

Welcome to Optimal Control

**Description:**

Optimal control is the problem of finding control strategies for a dynamic system such that a certain performance function is minimized (or maximized). The subject stems from the calculus of variations and was developed into an independent discipline during the early 1950's mainly due to two discoveries: the maximum principle by L.S. Pontryagin and dynamic programming by R. Bellman. Optimal control finds its application in a variety of areas including engineering, economics, biology and logistics.

The course has two main parts, which in headlines are: (1) Foundation of optimal control, and (2) Special topics. If time permits we will also discuss numerical implementation.

In the first part of the course, we will concentrate on the foundation of optimal control. We will discuss necessary and sufficient condition for optimality, and various types of constraints. We will address the question of existence of optimal strategies. We cover two main results in optimal control theory, the Hamilton-Jacobi-Bellman (HJB) equation and the (Pontryagin) maximum principle. We show how the dynamic programming principle works for an optimal control problem by using the HJB equation to solve linear quadratic control problems. Moreover, we apply the maximum principle to linear quadratic control problems. We end this part by introducing the notion of viscosity solution to the HJB equation.

In the second part of the course we will give an introduction to two areas of optimal control: singular optimal control where higher order conditions such as the generalized Legendre–Clebsch condition is used to obtain sufficient condition for local optimality, and optimal control of Markov processes where the state variables are not known with certainty (they are the outcome of stochastic differential equations). Finally, if time permits we will discuss software solutions for optimal control problems.

**Prerequisites: **A basic knowledge of mathematics as obtained through undergraduate engineering studies.

**Organizer:** Associate Professor John Leth, jjl@es.aau.dk

**Lecturers:** Associate Professor John Leth

**ECTS:** 5

**Time:** 12 - 16 August, 2019

**Place:** TBA ** **

**Zip code: **9220

**City: **Aalborg Øst** **

**Number of seats: **50

**Deadline: **22 July, 2019

Important information concerning PhD courses: We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 5,000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately three months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

- Teacher: John-Josef Leth

Description: The notion of Artificial Intelligence (AI) dates back approx. 70 years as a research field and even longer if one considers fiction writers. A number of different definitions of AI has been suggested over the years, but none seem to capture what AI is. This might be due to the fact that AI is about computer algorithms that behave intelligently. And since the capabilities of computer algorithms improve over time, no static definition is possible.

One aspect of AI is the ability to learn or adapt dynamically. This concept has inspired numerous Sci-fi books and movies with the underlying theme of man vs AI (often manifested in a robot). From this follows naturally ethical and regulatory considerations. But until recently, such considerations (see for example the three Robotic laws defined by the sci-fi writer I. Asimov) have been speculative since current AI algorithms (and their manifestation in mechanical devices) have performed poorly and hence never left university labs around the world. Recently, however, fast hardware and massive amount of data have allowed revisiting one particular AI algorithm invented in the 80s, namely Artificial Neural Networks (ANN), and increasing the size of the networks used in these models. This was exemplified via image processing for recognizing hand-written digits and resulted in amazing results. Inspired by this success ANN (now known as Deep Learning (DL)) was quickly picked up by other research fields where similar successes have been witnessed.

DL algorithms can now outperform humans on a number of tasks. Moreover, they can, to a certain degree, learn new tasks. An important point in this regard is that the algorithm is so complex that it is next to impossible to understand its inner workings. So, we seem to be facing a reality where AI, in a not too distant future, will be used to make decisions (simply because it is of better than humans). This raises a number of ethical and regulative questions such as, for instance, 1) how we ensure that AI systems are not discriminating against certain groups in the population, 2) how do we ensure transparency about the decisions made by AI systems, and relatedly 3) could and should individuals be given a substantial right to an explanation of decisions made by such systems and a substantial right not to be subjected to automated decision-making (GDPR). Since many of the currently developed AI systems operate on the basis of large amounts of data, the development and use of such systems also reinvigorate the ethical issues related to ‘Big data’. Finally, there are problems related to the efficacy and safety of AI systems. This raises questions not only of how appropriate monitoring of the development of these systems can be secured, but also and more importantly about the appropriate domains for use.

These questions and related questions are the core focus of the PhD course on ‘AI for the people’. The aim is to raise an awareness in the participants. To this end the course will be a combination of lectures and debates including the following topics:

• Introduction to AI

• Ethical issues in the development and use of AI

• Industry perspective on AI

Organisers: Professor Thomas Ploug, The Faculty of Humanities, Professor Thomas B. Moeslund, The Technical Faculty of IT and Design

Lecturers: Thomas Ploug, Thomas B. Moeslund

ECTS: 1

Time: February 27th (9-12) and March 28 (9-16)

Place:

February 27th: CREATE, room 5.127 Rendsburggade 14

March 28: CREATE, room 2.449 Rendsburggade 14

City: Aalborg

Number of seats: 30

Deadline: February 6th, 2019

Important information concerning PhD courses: We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 5,000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately three months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

- Teacher: Thomas Baltzer Moeslund
- Teacher: Thomas Ploug

Description: In many situations, a number of observations are made which contain some information about an underlying phenomenon we are interested in. Examples of this are

• the diagnosis of the Parkinson’s disease from a telephone recording,

• the assessment of bearing wear from vibrational data,

• the automatic transcription of music,

• order tracking analysis of rotating machines,

• automated analysis of the heart sound, and

• harmonic analysis in power systems.

To solve these and many other problems, a signal analysis toolbox is needed. This course focuses on developing, explaining, understanding, and using such tools. Specifically, the course covers important and general concepts such as:

Signal modelling: Which models exist, what are their applicability and limitations, and how do you compare different models?

Spectral analysis: Why is signal analysis often performed as a function of frequency

and how do you do it?

Inference and parameter estimation: How do you estimate model parameters accurately and quantify how well you do and how certain you are?

The course is primarily developed for doctoral students from medicine and various engineering and natural science disciplines who wish to not only apply, but also to understand signal and spectral analysis. Consequently, the course is rooted in a principled and systematic exposition of fundamental concepts and tools and in a scientific approach which promotes the creation of knowledge over improving state-of-the-art by ε percent. An important goal of the course is to make doctoral students able to solve a signal and spectral analysis task based on data from their own Ph.D.-project. This is integrated in the course via a mini project.

Keywords: Filtering, statistical signal processing, estimation theory, maximum likelihood, Bayesian statistics, separation, modelling, least squares, enhancement, nonnegative matrix factorizations, periodic signals, Fourier analysis.

Prerequisites: Basic probability theory, linear algebra, basic signal processing, and experience with MATLAB or Python programming.

Organizers Prof. Mads Græsbøll Christensen and Ass. Prof. Jesper Kjær Nielsen

Lecturers: Prof. Mads Græsbøll Christensen and Ass. Prof. Jesper Kjær Nielsen

ECTS: 3

Time: September 30th-October 4th, 2019, all days.

Place:

City:

Number of seats: 30

Deadline: September 9th 2019

Important information concerning PhD courses: We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 5,000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately three months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

- Teacher: Mads Græsbøll Christensen
- Teacher: Jesper Kjær Nielsen
- Teacher: Jesper Kjær Nielsen

Description: While Virtual Reality (VR) has been around since the 1960s, only recently it has been democratised thanks to the availability of low cost hardware technologies together with accessible development tools.

In this course we will introduce virtual, augmented and mixed reality technologies (VR, AR, XR) with a focus on:

-multisensory perception and embodied interaction

-hardware and software technologies for VR, AR and XR: head mounted displays, augmented reality displays, 3D user interfaces and tracking

-3D sound rendering and haptic technologies

-user experience evaluation: quantitative and qualitative methods.

-Presence research

-applications in art, entertainment, rehabilitation and learning

The class is a combination of theory and practical work in the multisensory experience lab at Aalborg University Copenhagen: https://melcph.create.aau.dk/

Organizers: Professor Stefania Serafin, sts@create.aau.dk

Lecturers: Professor Stefania Serafin, sts@create.aau.dk, Associate Professor Rolf Nordahl, rn@create.aau.dk, Postdoc Michele Geronazzo, mge@create.aau.dk, Assistant Professor Niels Christian Nilsson, ncn@create.aau.dk and Associate Professor Cumhur Erkut, cer@create.aau.dk

ECTS: 5

Time: 30 September to 4 October 2019, all day.

Place: Aalborg University Copenhagen

Number of seats: 30

Deadline: 9 September 2019

Important information concerning PhD courses: We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 5,000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately three months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

- Teacher: Cumhur Erkut
- Teacher: Michele Geronazzo
- Teacher: Niels Christian Nilsson
- Teacher: Rolf Bjørn Jørgensen Nordahl
- Teacher: Stefania Serafin

Description: Typical attacks on critical infrastructures: How hackers can abuse Internet of Things, and how attacks can be prevented and detected. Recently, methods from fault detection originated in control theory found its application in detection of cyber-attacks. In the second part of the lecture we will discuss public-key cryptosystems, which are widely used for secure data transmission. Public key cryptography aims at solving the problem of how two parties can communicate securely when they have not agreed on some secret common key, which is often the case in for example communication through the internet. It includes both public key encryption, which guarantees the secrecy of a message, and digital signatures, which provides authentication and integrity. In connection to this, we will also discuss key exchange. We will introduce some of the more classical, but still widely used, public key cryptosystems, such RSA and El Gamal/ Diffie-Hellman and we will discuss which security properties are usually required from them nowadays.

Subsequently, we will investigate the problem of secure multi-party computation, which studies how to "compute on encrypted data": how several mutually distrustful parties can collaborate to jointly perform computations involving private data without needing to actually reveal their private information to the others.

Prerequisite: mathematics on the level of Master in science and engineering

Organizers: Rafael Wisniewski, Ignacio Cascudo Pueyo, Jens Myrup Pedersen

Lecturers: Rafael Wisniewski, Ignacio Cascudo Pueyo, Jens Myrup Pedersen

ECTS: 3

Time: May 13th to 17th, 2019, all days

Place:

City:

Number of seats: 30

Deadline: April 22nd 2019

Important information concerning PhD courses: We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 5,000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately three months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

- Teacher: Jens Myrup Pedersen
- Teacher: Ignacio Cascudo Pueyo
- Teacher: Rafal Wisniewski