Electrical and Electronic Engineering (2023)

In applications within modeling, estimation, control, detection and forecasting, state space models are used. Normally the whole state can not be measured but only some function of the state is measured and perhaps also with substantial measurement noise. In at least the above applications it is necessary to estimate the whole state from the measurement. Exactly this is the topic for the course.

For linear discrete time stochastic systems there exist the Kalman filter solution which to some extend is included in the master curriculum. The topic for this course is the advanced methods for nonlinear and continuous time systems and also to include parameter estimation. The purpose of this PhD course is to give the participant a comprehensive knowledge on both basic and more advanced aspects of state and parameter estimation. The goal is to enable the students with knowledge and tools for stochastic modeling of physical systems.

The participant should be able to apply software for state and parameter estimation for the model structures. The software used are programs from MATLAB. Topics include among others Kalman filters (KF), Extended KF, Unscented KF, continuous discrete filtering, stochastic differential equation (SDE), parameter estimation by extending the state or maximum likelihood (ML), particle filter (PF). The course is evaluated as passed/not passed.

In order to pass the student must be actively participating and deliver written solutions to the exercises which must be accepted by the lecture. 

Organizer:          Torben Knudsen

Lecturers:           Torben Knudsen

ECTS:                   3

Time:                   6 - 10 November 2023

Place:

Zip code:

City:

Number of seats:      30

Deadline:             16 October 2023

Important information concerning PhD courses:

We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 3.000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately four months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.

An intelligent system is expected to generate policies autonomously in order to achieve a goal, which is mostly to maximize a given reward function or minimize a given cost function. 

Reinforcement learning is a set of methods in machine learning that can produce such policies. In order to learn optimal actions in an environment that is not fully comprehensible to itself, an intelligent system can use reinforcement algorithms to leverage its experience to figure out optimal policies. Nowadays, reinforcement learning techniques are successfully applied in various engineering fields, including robotics (DeepMind’s walking robot) and computers playing games (AlphaGo and TD-Gammon).

Developed independently from reinforcement learning, dynamic programming is a set of algorithms in optimal control theory that generate policies assuming that the environment is fully comprehensible to the intelligent system. Therefore, dynamic programming provides an essential base to learn reinforcement learning. The course aims at building a fundamental understanding of both methods based on their intimate relations to each other and on their applications to similar problems. 

The course consists of the following topics:   Markov decision processes, dynamic programming for infinite time and stopping time, reinforcement learning, and verification tools for reinforcement learning.

Optimal control is the problem of finding control strategies for a dynamic system such that a certain performance function is minimized (or maximized). The subject stems from the calculus of variations and was developed into an independent discipline during the early 1950's mainly due to two discoveries: the maximum principle by L.S. Pontryagin and dynamic programming by R. Bellman. Optimal control finds its application in a variety of areas including engineering, economics, biology and logistics.  

The course will be conducted as a (traditionally) lecture series with physical attendance. Course evaluation will be done through attendance and homework assignment. It has two main parts, which in headlines are: (1) Foundation of optimal control, and (2) Special topics, including a discussion on numerical implementation. 

In the first part of the course, we will concentrate on the foundation of optimal control. We will discuss necessary and sufficient condition for optimality, and various types of constraints. We will address the question of existence of optimal strategies. We cover two main results in optimal control theory, the Hamilton-Jacobi-Bellman (HJB) equation and the (Pontryagin) maximum principle. We show how the dynamic programming principle works for an optimal control problem by using the HJB equation to solve linear quadratic control problems. Moreover, we apply the maximum principle to linear quadratic control problems. We end this part with an introduction to the notion of viscosity solution to the HJB equation, and singular optimal control where higher order conditions such as the generalized Legendre–Clebsch condition is used to obtain sufficient condition for local optimality. If time permits we will introduce optimal control of Markov processes where the state variables are not known with certainty (they are the outcome of stochastic differential equations).  

In the second part of the course we will explore various software solutions for optimal control problems, and end by discussing numerical solutions of optimal control problems and their implementation.

Important information concerning PhD courses:

We have over some time experienced problems with no-show for both project and general courses. It has now reached a point where we are forced to take action. Therefore, the Doctoral School has decided to introduce a no-show fee of DKK 3.000 for each course where the student does not show up. Cancellations are accepted no later than 2 weeks before start of the course. Registered illness is of course an acceptable reason for not showing up on those days. Furthermore, all courses open for registration approximately four months before start. This can hopefully also provide new students a chance to register for courses during the year. We look forward to your registrations.