| java.lang.Object
 org.griphyn.cPlanner.parser.pdax.PDAX2MDAG
| PDAX2MDAG | | public class PDAX2MDAG implements Callback(Code) | | This callback ends up creating the megadag that contains the smaller dags
each corresponding to the one level as identified in the pdax file
generated by the partitioner.
author:
Karan Vahi
version:
$Revision: 464 $ |
| Method Summary | |
| public void | cbDocument(Map attributes)
Callback when the opening tag was parsed. | | public void | cbDone()
Callback when the parsing of the document is done. | | public void | cbParents(String child, List parents)
Callback for child and parent relationships from section 3. | | public void | cbPartition(Partition partition)
Callback for the partition . | | protected SubInfo | constructDAGJob(Partition partition, File directory, String dax)
Constructs a job that plans and submits the partitioned workflow,
referred to by a Partition. | | protected String | constructDAGManKnobs(PegasusProperties properties)
Constructs Any extra arguments that need to be passed to dagman, as determined
from the properties file. | | protected String | createSubmitDirectory(String label, String dir, String user, String vogroup, boolean timestampBased)
Creates the submit directory for the workflow. | | protected boolean | createSymlink(String source, File destDir)
Returns the number of partitions referred to in the PDAX file.
Parameters:
source - the source file that has to be symlinked.
Parameters:
destDir - the destination directory where the symlink has to beplaced. | | protected String | getAbsolutePath(Partition partition, String directory, String suffix)
Returns the absolute path to a dagman (usually) related file for a
particular partition in the submit directory that is passed as an input
parameter. | | protected String | getBaseName(Partition partition)
Returns the base name of the submit directory in which the submit files
for a particular partition reside.
Parameters:
partition - the partition for which the base directory is to beconstructed. | | protected String | getBasename(Partition partition, String suffix)
Returns the basename of a dagman (usually) related file for a particular
partition.
Parameters:
partition - the partition for which the dagman is responsible forexecution.
Parameters:
suffix - the suffix for the file basename. | | protected String | getBasenamePrefix(SubInfo job)
Returns the basename prefix of a dagman (usually) related file for a
a job that submits nested dagman.
Parameters:
job - the job that submits a nested dagman. | | protected String | getCacheFilePath(SubInfo job)
Returns the full path to a cache file that corresponds for one partition.
The cache file resides in the submit directory for the partition for which
the job is responsible for.
Parameters:
job - the job running on the submit host that submits the partition. | | public Object | getConstructedObject()
| | protected SubInfo | getJob(String id)
Returns the job that has been constructed for a particular partition.
Parameters:
id - the partition id. | | protected int | getPartitionCount(String pdax)
Returns the number of partitions referred to in the PDAX file.
Parameters:
pdax - the path to the pdax file. | | protected int | parseInt(String s)
Parses a string into an integer. | | protected static void | sanityCheck(File dir)
Checks the destination location for existence, if it can
be created, if it is writable etc. | | protected void | setPrescript(SubInfo job, String daxURL, String log)
Sets the prescript that ends up calling to the default wrapper that
introduces retry into Pegasus for a particular job. | | protected void | setPrescript(SubInfo job, String daxURL, String log, String namespace, String name, String version)
Sets the prescript that ends up calling to the default wrapper that
introduces retry into Pegasus for a particular job. | | protected String | writeOutBraindump(File directory, Partition partition, String dax, String dag)
Writes out the braindump.txt file for a partition in the partition submit
directory. | | protected String | writeOutProperties(String directory)
Writes out the properties to a temporary file in the directory passed.
Parameters:
directory - the directory in which the properties file needs tobe written to. |
| CODE_GENERATOR_CLASS | | final public static String CODE_GENERATOR_CLASS(Code) | | The SubmitWriter that has to be loaded for now.
|
| CONDOR_DAGMAN_LOGICAL_NAME | | final public static String CONDOR_DAGMAN_LOGICAL_NAME(Code) | | The logical name with which to query the transformation catalog for the
condor_dagman executable, that ends up running the mini dag as one
job.
|
| CONDOR_DAGMAN_NAMESPACE | | final public static String CONDOR_DAGMAN_NAMESPACE(Code) | | The namespace to use for condor dagman.
|
| CPLANNER_LOGICAL_NAME | | final public static String CPLANNER_LOGICAL_NAME(Code) | | The logical name with which to query the transformation catalog for
cPlanner executable.
|
| DAGMAN_KNOBS | | final public static String DAGMAN_KNOBS(Code) | | The dagman knobs controlled through property. They map the property name to
the corresponding dagman option.
|
| HEAD_INDEX | | final public static int HEAD_INDEX(Code) | | The index of the head job.
|
| NAMESPACE | | final public static String NAMESPACE(Code) | | The namespace to which the job in the MEGA DAG being created refer to.
|
| NUM_OF_EXPANDED_JOBS | | final public static int NUM_OF_EXPANDED_JOBS(Code) | | The number of jobs into which each job in the partition graph is
expanded to.
|
| RETRY_LOGICAL_NAME | | final public static String RETRY_LOGICAL_NAME(Code) | | The planner utility that needs to be called as a prescript.
|
| SUBMIT_DIRECTORY_PREFIX | | final public static String SUBMIT_DIRECTORY_PREFIX(Code) | | The prefix for the submit directory.
|
| TAIL_INDEX | | final public static int TAIL_INDEX(Code) | | The index of the tail job.
|
| mSeparator | | protected static char mSeparator(Code) | | The file Separator to be used on the submit host.
|
| PDAX2MDAG | | public PDAX2MDAG(String directory, PegasusProperties properties, PlannerOptions options)(Code) | | The overloaded constructor.
Parameters:
directory - the directory where the pdax and all the daxescorresponding to the partitions reside.
Parameters:
properties - the PegasusProperties to be used.
Parameters:
options - the options passed to the planner. |
| cbDocument | | public void cbDocument(Map attributes)(Code) | | Callback when the opening tag was parsed. This contains all
attributes and their raw values within a map. This callback can
also be used to initialize callback-specific resources.
Parameters:
attributes - is a map of attribute key to attribute value |
| cbDone | | public void cbDone()(Code) | | Callback when the parsing of the document is done. This ends up
triggering the writing of the condor submit files corresponding to the
mega dag.
|
| cbParents | | public void cbParents(String child, List parents)(Code) | | Callback for child and parent relationships from section 3. This ties
in the relations between the partitions to the relations between the jobs
that are responsible for partitions. In addition, appropriate cache
file arguments are generated.
Parameters:
child - is the IDREF of the child element.
Parameters:
parents - is a list of IDREFs of the included parents. |
| cbPartition | | public void cbPartition(Partition partition)(Code) | | Callback for the partition . These partitions are completely
assembled, but each is passed separately.
Parameters:
partition - is the PDAX-style partition. |
| constructDAGJob | | protected SubInfo constructDAGJob(Partition partition, File directory, String dax)(Code) | | Constructs a job that plans and submits the partitioned workflow,
referred to by a Partition. The main job itself is a condor dagman job
that submits the concrete workflow. The concrete workflow is generated by
running the planner in the prescript for the job.
Parameters:
partition - the partition corresponding to which the job has to beconstructed.
Parameters:
directory - the submit directory where the submit files for thepartition should reside.
Parameters:
dax - the absolute path to the partitioned dax file thatcorresponds to this partition. the constructed DAG job. |
| constructDAGManKnobs | | protected String constructDAGManKnobs(PegasusProperties properties)(Code) | | Constructs Any extra arguments that need to be passed to dagman, as determined
from the properties file.
Parameters:
properties - the PegasusProperties any arguments to be added, else empty string |
| createSubmitDirectory | | protected String createSubmitDirectory(String label, String dir, String user, String vogroup, boolean timestampBased) throws IOException(Code) | | Creates the submit directory for the workflow. This is not thread safe.
Parameters:
label - the label of the workflow being worked upon.
Parameters:
dir - the base directory specified by the user.
Parameters:
user - the username of the user.
Parameters:
vogroup - the vogroup to which the user belongs to.
Parameters:
timestampBased - boolean indicating whether to have a timestamp based dir or not the directory name created relative to the base directory passedas input.
throws:
IOException - in case of unable to create submit directory. |
| createSymlink | | protected boolean createSymlink(String source, File destDir)(Code) | | Returns the number of partitions referred to in the PDAX file.
Parameters:
source - the source file that has to be symlinked.
Parameters:
destDir - the destination directory where the symlink has to beplaced. the number of partitions in the pdax file. |
| getAbsolutePath | | protected String getAbsolutePath(Partition partition, String directory, String suffix)(Code) | | Returns the absolute path to a dagman (usually) related file for a
particular partition in the submit directory that is passed as an input
parameter. This does not create the file, just returns an absolute path
to it. Useful for constructing argument string for condor_dagman.
Parameters:
partition - the partition for which the dagman is responsible forexecution.
Parameters:
directory - the directory where the file should reside.
Parameters:
suffix - the suffix for the file basename. the absolute path to a file in the submit directory. |
| getBaseName | | protected String getBaseName(Partition partition)(Code) | | Returns the base name of the submit directory in which the submit files
for a particular partition reside.
Parameters:
partition - the partition for which the base directory is to beconstructed. the base name of the partition. |
| getBasename | | protected String getBasename(Partition partition, String suffix)(Code) | | Returns the basename of a dagman (usually) related file for a particular
partition.
Parameters:
partition - the partition for which the dagman is responsible forexecution.
Parameters:
suffix - the suffix for the file basename. the basename. |
| getBasenamePrefix | | protected String getBasenamePrefix(SubInfo job)(Code) | | Returns the basename prefix of a dagman (usually) related file for a
a job that submits nested dagman.
Parameters:
job - the job that submits a nested dagman. the basename. |
| getCacheFilePath | | protected String getCacheFilePath(SubInfo job)(Code) | | Returns the full path to a cache file that corresponds for one partition.
The cache file resides in the submit directory for the partition for which
the job is responsible for.
Parameters:
job - the job running on the submit host that submits the partition. the full path to the file. |
| getConstructedObject | | public Object getConstructedObject()(Code) | | Returns the MEGADAG that is generated
ADag object containing the mega daga |
| getJob | | protected SubInfo getJob(String id)(Code) | | Returns the job that has been constructed for a particular partition.
Parameters:
id - the partition id. the corresponding job, else null if not found. |
| getPartitionCount | | protected int getPartitionCount(String pdax)(Code) | | Returns the number of partitions referred to in the PDAX file.
Parameters:
pdax - the path to the pdax file. the number of partitions in the pdax file. |
| parseInt | | protected int parseInt(String s)(Code) | | Parses a string into an integer. Non valid values returned as -1
Parameters:
s - the String to be parsed as integer the int value if valid, else -1 |
| sanityCheck | | protected static void sanityCheck(File dir) throws IOException(Code) | | Checks the destination location for existence, if it can
be created, if it is writable etc.
Parameters:
dir - is the new base directory to optionally create.
throws:
IOException - in case of error while writing out files. |
| setPrescript | | protected void setPrescript(SubInfo job, String daxURL, String log)(Code) | | Sets the prescript that ends up calling to the default wrapper that
introduces retry into Pegasus for a particular job.
Parameters:
job - the job whose prescript needs to be set.
Parameters:
daxURL - the path to the dax file on the filesystem.
Parameters:
log - the file where the output of the prescript needs to beredirected to.
See Also: PDAX2MDAG.RETRY_LOGICAL_NAME |
| setPrescript | | protected void setPrescript(SubInfo job, String daxURL, String log, String namespace, String name, String version)(Code) | | Sets the prescript that ends up calling to the default wrapper that
introduces retry into Pegasus for a particular job.
Parameters:
job - the job whose prescript needs to be set.
Parameters:
daxURL - the path to the dax file on the filesystem.
Parameters:
log - the file where the output of the prescript needs to beredirected to.
Parameters:
namespace - the namespace of the replanner utility.
Parameters:
name - the logical name of the replanner.
Parameters:
version - the version of the replanner to be picked up. |
| writeOutBraindump | | protected String writeOutBraindump(File directory, Partition partition, String dax, String dag) throws IOException(Code) | | Writes out the braindump.txt file for a partition in the partition submit
directory. The braindump.txt file is used for passing to the tailstatd
daemon that monitors the state of execution of the workflow.
Parameters:
directory - the directory in which the braindump file needs tobe written to.
Parameters:
partition - the partition for which the braindump is to be written out.
Parameters:
dax - the dax file
Parameters:
dag - the dag file the absolute path to the braindump file.txt written in the directory.
throws:
IOException - in case of error while writing out file. |
| writeOutProperties | | protected String writeOutProperties(String directory) throws IOException(Code) | | Writes out the properties to a temporary file in the directory passed.
Parameters:
directory - the directory in which the properties file needs tobe written to. the absolute path to the properties file written in the directory.
throws:
IOException - in case of error while writing out file. |
|
|