Home > Research > Publications & Outputs > The Design Space of Emergent Scheduling for Dis...

Electronic data

  • conference_101719

    Rights statement: ©2022 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

    Accepted author manuscript, 1.81 MB, PDF document

Links

Text available via DOI:

View graph of relations

The Design Space of Emergent Scheduling for Distributed Execution Frameworks

Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSNConference contribution/Paperpeer-review

Published
Publication date29/06/2021
Host publicationSymposium on Software Engineering for Adaptive and Self-Managing Systems
PublisherIEEE
Pages186-195
Number of pages10
ISBN (electronic)9781665402897
ISBN (print)9781665402903
<mark>Original language</mark>English
EventSEAMS International Workshop on Software Engineering for Adaptive and Self-Managing Systems, ICSE - Madrid, Spain
Duration: 18/05/202118/05/2022
https://ieeexplore.ieee.org/xpl/conhome/9461924/proceeding

Symposium

SymposiumSEAMS International Workshop on Software Engineering for Adaptive and Self-Managing Systems, ICSE
Country/TerritorySpain
CityMadrid
Period18/05/2118/05/22
Internet address

Symposium

SymposiumSEAMS International Workshop on Software Engineering for Adaptive and Self-Managing Systems, ICSE
Country/TerritorySpain
CityMadrid
Period18/05/2118/05/22
Internet address

Abstract

Distributed Execution Frameworks (DEFs) such as Apache Spark have become ubiquitous as a solution for the execution of user-defined jobs to process terabytes of data across hundreds of nodes. One of the key costs of DEFs is scheduling of which parts of each job are placed on each host; better scheduling decisions provide lower overall execution time for each job, more efficient resource usage, and reduced energy consumption. Existing DEFs use a static approach to scheduling, either with a single generalised scheduler which aims to be a good fit for most workloads, or with a special-purpose scheduler which is tuned to optimise for a particular kind of workload. In both cases the scheduling implementation is fixed at design-time such that the DEF is unable to adjust to the actual characteristics of workloads that arrive at deployment time. In this paper we introduce an emergent scheduler for Distributed Execution Frameworks. This scheduler can be composed and re-composed at runtime from a set of different building blocks, allowing the system to dynamically provide the benefits of differing scheduling policies over time depending on the actual properties of incoming workloads - with improved performance and resource usage. In this paper we present the overall design of our emergent scheduler, we discuss the theoretical design space of different scheduling approaches, and we examine a specific research question to determine the correlation between workload properties and scheduling performance for different scheduler implementations. Our results are based on a real implementation of our emergent DEF running across multiple hosts in a real datacentre, and our implementation is made available as open-source software.

Bibliographic note

©2022 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.