Schedflow: An integrating system of scheduling policies and workflow engine in grid environments

Abstract

Workflow applications running on distributed environments are a promising solution for resource and computing intensive problems. However, the heterogeneity of resources in these environments complicates resource management and the scheduling of such applications. Sophisticated scheduling policies are being developed for workflows, but they have little impact in practice because their integration into existing workflow engines is complex due to the restrictions imposed by each workflow engine. In this paper, we introduce SchedFlow a system that allows users to easily incorporate scheduling policies in existing workflow engines. Our system provides a simple API with the essential mechanisms needed for scheduling policies to interact with workflow engines. Experiments have been conducted to show the functionality of SchedFlow with both static and dynamic scheduling policies. The workflow engine used in these experiments was Condor-DAGMan, the workflow application was Montage, and three scheduling policies were considered: Random and dynamic Min-Min. As a result we were able to implement once compare different workflow scheduling policies, and see makespan reduction of the scheduling policies with event management about static scheduling policies.