Emergent software systems take a reward signal, an environment signal, and a collection of possible behavioural compositions implementing the system logic in a variety of ways, to learn in real-time how to best assemble a system to maximise reward. This reduces the burden of complexity in systems building by making human programmers responsible only for developing potential building blocks while the system determines how best to use them in its deployment conditions – with no architectural models or training regimes.
This is a hands-on tutorial in which participants will be encouraged to build their own emergent system, with autonomously learned behaviour that is a consequence of the current deployment environment. We will structure the tutorial into two parts: (i) the Dana programming language, and (ii) our emergent systems framework. In the first part we will introduce our hyper-adaptive Dana programming language. Dana makes runtime adaptation safe, lightning fast, and pervasive to every element of a software system from TCP socket implementations up to graphical user widgets. In the second part of the tutorial we will introduce our emergent systems framework, built in Dana, which uses the language’s capability to abstract entire compositions of behaviour into simple action/reward tables. We will cover the three major parts of our framework in detail, including assembly, perception, and learning. As part of our discussion of the learning element, we will also introduce the concept of multi-armed bandits as one of the most prominent reinforcement learning techniques; we will particularly focus on the theory of UCB1 as a simple implementation of this learning technique.
Organizers
Roberto Rodrigues Filho (Lancaster University, UK) and Barry Porter (Lancaster University, UK)
Target audience
This tutorial is aimed at all researchers and practitioners in self-adaptive software systems, especially those interested in reinforcement learning techniques, with a particular focus on those wanting to gain new implementation experience with new and emerging systems-building technologies. The tutorial will be equally well suited both to those who are new to adaptive systems, and experienced researchers interested in developing new skills. Some experience in programming is helpful, and you should bring your own laptop to try out the hands-on parts of the tutorial. By the end of this tutorial, participants will have had the opportunity to experiment with adaptation, compositional search, and real-time machine learning to drive autonomous system assembly and re-assembly as a product of a perceived environment. We hope that this will provoke new ways of thinking about autonomous and adaptive systems, where complex behavioural decisions can be driven by apparently simple actions and rewards in a reinforcement learning algorithm.
Material provided
The tutorial will be divided into short lectures followed by hands-on sessions to try out the concepts; we will provide the lecture slides and worksheets for the practical elements.
Prerequisites
Because this is a practically-focused tutorial, some prior experience of programming will be very helpful in being able to fully engage with the activities.