This project is read-only.

Framework Concepts

In order to understand how the framework operates, the concepts are explained below.

Framework version v1:

Runs, Tasks and Packages

The Framework executes sets of packages in a group called a Run. A Run executes individual packages as Tasks. Runs are defined in the Run Master table. The Tasks to be carried out by the Run are defined in the Task Master table, and are associated with a Run by setting the Run Id of the Task to the corresponding Run Id of the Run. Multiple Runs can be defined which execute independently. Each Run can consists of as many Tasks as required.

The location of each Package is stored in the Package Master table. This allows the package details to be decoupled from the Task so an individual package can be called multiple times and still uniquely identified.

Execution Tracking

On initiation a Run generates an entry in the Run Log table with a unique Run Execution Id. Similarly, log entries for each task within that run are generated in the Task Log table with a unique Task Execution Id for each execution of the package. The creation of the Task Log entries will reference the Extract Limits defined in the Extract Limits table to apply the Extract Limits that are defined at the start of the run. As the tasks execute, the log tables are updated to reflect the progress of the Run.


When a Run is initiated it doesn't only run the current Run Execution, it attempts to recover any previous incomplete components that are marked for recovery. An incomplete component is one that has not completed with a status of "Successful". The Run can be initiated in "Recovery Only" mode, which means the Run Execution will not attempt to run a new set of Tasks, and only recover incomplete components marked for recovery.

Whether a incomplete component is included as part of the recovery process depends on the Recovery Mode setting for that Run or Task. This can be set at Run or Task level. The available options are to either Recover or Ignore the Run or Task when looking for components to recover. NB: A Recovery Mode setting of Ignore at the Run level will override any settings at the Task Level.

Extract Limits

Extract Limits define a range for which Data will be processed. They are maintained centrally in the Extract Limits table, and the current values in the table are applied to the log entries of any Run or Task that references them. This ensures that during recovery, a consistent range of data will be extracted. The ranges are stored as nvarchar and need to be converted to the appropriate data type by a script in the Task that uses them.

NB: Extract Limits set at the Task level will override Extract limits set at the Run Level.

Last edited May 12, 2010 at 1:55 AM by BI_Monkey, version 2


No comments yet.