| java.lang.Object
 org.griphyn.vdl.euryale.FlatFileFactory
org.griphyn.vdl.euryale.HashedFileFactory
All known Subclasses: org.griphyn.vdl.euryale.VirtualDecimalHashedFileFactory,
| HashedFileFactory | | public class HashedFileFactory extends FlatFileFactory (Code) | | This file factory generates a stream of submit files in a dynamically
determinable directory structure. By default, a 2-level subdirectory
structure is assumed, which should be able to accomodate about 500k
files.
mult=16, offset=30, fpd=254: nr=15 => l=1
mult=16, offset=30, fpd=254: nr=4047 => l=2
mult=16, offset=30, fpd=254: nr=1028222 => l=3
With the given multiplicator, offset and files per directory, nr is
smallest number of jobs at which a level change to l occurs.
author:
Kavitha Ranganathan
author:
Jens-S. Vöckler
author:
Yong Zhao
version:
$Revision: 289 $
See Also: DAX2DAG |
| Field Summary | |
| protected int | m_count
Counts the number of times the structured virtual constructor was
called. | | protected int | m_filesPerDirectory
Number of entries per level. | | protected int | m_levels
Contains the total number of directory levels. | | protected int | m_multiplicator
Multiplicative factor to estimate the number of result leaf
filenames for each virtual constructor invocation. | | protected int | m_offset
Offset of files expected to reside at the top level directory. | | protected StringBuffer | mh_buffer
Helping structure to avoid repeated memory requests. | | protected int | mh_digits
Helping structure to avoid repeated memory requests. | | protected int | mh_level
Helping structure to avoid repeated memory requests. |
| Constructor Summary | |
| public | HashedFileFactory(File baseDirectory)
Constructor: Creates the base directory and employs sanity checks. | | public | HashedFileFactory(String baseDirectory)
Constructor: Creates the directory and employs sanity checks. | | public | HashedFileFactory(File baseDirectory, int totalFiles)
Constructor: Creates the base directory and employs sanity checks. | | public | HashedFileFactory(String baseDirectory, int totalFiles)
Constructor: Creates the directory and employs sanity checks. |
| Method Summary | |
| public static int | calculateLevels(int totalFiles, int multiplicator, int filesPerDirectory, int offset)
Determines dynamically the number of directory levels required
to accomodate a certain number of files.
levels = |log ( tf * m + offset )|
fpd
Parameters:
totalFiles - is the total number of files estimated to generate
Parameters:
multiplicator - is a corrective factor to account for filesthat are created by the run-time system on the fly. | | protected File | createDirectory()
Creates a directory for the hashed file directory structure on the
submit host. | | public File | createFile(String basename)
Creates the next file with the given basename. | | public String | format(int number)
Converts the given integer into hexadecimal notation, using
the given number of digits, prefixing with zeros as necessary.
Parameters:
number - is the number to format. | | public int | getCount()
Returns the number of times the regular virtual constructor for
structured entries was called. | | public int | getFilesPerDirectory()
Accessor: Obtains the number of entries per directory. | | public int | getLevels()
Accessor: Obtains the total number of directory levels. | | public int | getMultiplicator()
Accessor: Obtains the multiplicative factor for an estimation
of total files from calls to the virtual constructor. | | public int | getOffset()
Accessor: Obtains the offset for an estimation of total files from
calls to the virtual constructor. | | public static void | main(String arg)
| | public void | reset()
Resets the helper structures after changing layout parameters. | | public void | setFilesPerDirectory(int entries)
Accessor: Sets the optimal maximum number of files per directory
excluding dot and dotdot. | | public void | setLevels(int levels)
Accessor: Sets the number of directory levels. | | public void | setLevelsFromTotals(int totalFiles)
Accessor: Sets the number of directory levels. | | public void | setMultiplicator(int multiplicator)
Accessor: Sets the multiplicative factor to account for files which
may be created without calling the virtual constructor. | | public void | setOffset(int offset)
Accessor: Sets the offset for files which may be created without
calling the virtual constructor. |
| m_filesPerDirectory | | protected int m_filesPerDirectory(Code) | | Number of entries per level. The number 254 is optimized for the
Linux VFS ext2, and consequently ext3, which works fastest, if the
number of entries per directory, including dot and dotdot, don't
exceed 256.
|
| m_levels | | protected int m_levels(Code) | | Contains the total number of directory levels. Defaults to a
reasonable level for hashing.
|
| m_multiplicator | | protected int m_multiplicator(Code) | | Multiplicative factor to estimate the number of result leaf
filenames for each virtual constructor invocation. We assume that
Euryale produces ~12 files per submit file. It is better to err
on the larger side than makeing the multiplicator too small.
|
| m_offset | | protected int m_offset(Code) | | Offset of files expected to reside at the top level directory.
This is counted in addition to the directories being created.
|
| HashedFileFactory | | public HashedFileFactory(File baseDirectory) throws IOException(Code) | | Constructor: Creates the base directory and employs sanity checks.
Parameters:
baseDirectory - is the place where the other dirs are created,and where the DAG file resides.
throws:
IOException - if the location is not a writable directory,or cannot be created as such. |
| HashedFileFactory | | public HashedFileFactory(String baseDirectory) throws IOException(Code) | | Constructor: Creates the directory and employs sanity checks.
Parameters:
baseDirectory - is the place where the other dirs are created,and where the DAG file resides.
throws:
IOException - if the location is not a writable directory,or cannot be created as such. |
| HashedFileFactory | | public HashedFileFactory(File baseDirectory, int totalFiles) throws IOException(Code) | | Constructor: Creates the base directory and employs sanity checks.
Parameters:
baseDirectory - is the place where the other dirs are created,and where the DAG file resides.
Parameters:
totalFiles - is the number of files to support, and the numberof times, the virtual constructor is expected to be called.
throws:
IOException - if the location is not a writable directory,or cannot be created as such. |
| HashedFileFactory | | public HashedFileFactory(String baseDirectory, int totalFiles) throws IOException(Code) | | Constructor: Creates the directory and employs sanity checks.
Parameters:
baseDirectory - is the place where the other dirs are created,and where the DAG file resides.
Parameters:
totalFiles - is the number of files to support, and the numberof times, the virtual constructor is expected to be called.
throws:
IOException - if the location is not a writable directory,or cannot be created as such. |
| calculateLevels | | public static int calculateLevels(int totalFiles, int multiplicator, int filesPerDirectory, int offset)(Code) | | Determines dynamically the number of directory levels required
to accomodate a certain number of files.
levels = |log ( tf * m + offset )|
fpd
Parameters:
totalFiles - is the total number of files estimated to generate
Parameters:
multiplicator - is a corrective factor to account for filesthat are created by the run-time system on the fly. For Euryale andPegasus it is safe to assume a multiplicator of at least 8.
Parameters:
filesPerDirectory - is the optimal maximum number of directoryentries in any directory. The value of 254 for Linux ext2, and thusext3, is a safe bet.
Parameters:
offset - is the number of (expected) files in the top level. the number of directory levels necessary to accomodate thegiven number of files. |
| createDirectory | | protected File createDirectory() throws IOException(Code) | | Creates a directory for the hashed file directory structure on the
submit host.
the File structure to the created directory
throws:
IOException - the exception. |
| createFile | | public File createFile(String basename) throws IOException(Code) | | Creates the next file with the given basename. This is the factory
standard virtual constructor. Once invoked, the directory structure
can not be changed any more.
Parameters:
basename - is the filename to create. Don't specify dirs here. a File structure which points to the new file. Nothing iscreated through this method, and creation may still fail.
See Also: HashedFileFactory.getCount() |
| format | | public String format(int number)(Code) | | Converts the given integer into hexadecimal notation, using
the given number of digits, prefixing with zeros as necessary.
Parameters:
number - is the number to format. a string of appropriate length, filled with leading zeros,representing the number hexadecimally. |
| getCount | | public int getCount()(Code) | | Returns the number of times the regular virtual constructor for
structured entries was called.
the count for createFile invocations.
See Also: HashedFileFactory.createFile(String) |
| getFilesPerDirectory | | public int getFilesPerDirectory()(Code) | | Accessor: Obtains the number of entries per directory.
the chosen number of entries per directory excluding thedot and dotdot files. |
| getLevels | | public int getLevels()(Code) | | Accessor: Obtains the total number of directory levels.
the total number of directory levels chosen. |
| getMultiplicator | | public int getMultiplicator()(Code) | | Accessor: Obtains the multiplicative factor for an estimation
of total files from calls to the virtual constructor.
the multiplicator.
See Also: HashedFileFactory.setMultiplicator(int) |
| getOffset | | public int getOffset()(Code) | | Accessor: Obtains the offset for an estimation of total files from
calls to the virtual constructor.
the offset
See Also: HashedFileFactory.setOffset(int) |
| reset | | public void reset()(Code) | | Resets the helper structures after changing layout parameters. You
will also need to call this function after you invoked the virtual
constructors, but want to change parameter pertaining to the
directory structure. The structured file count will also be reset!
|
| setFilesPerDirectory | | public void setFilesPerDirectory(int entries)(Code) | | Accessor: Sets the optimal maximum number of files per directory
excluding dot and dotdot. For a Linux ext2 and thus ext3 system,
the optimal maximum number is 254.
Parameters:
entries - is the number of optimal maximum entries per dir.
throws:
VTorInUseException - if the virtual constructor is already in use.
throws:
IllegalArgumentException - if the argument is less than one.
See Also: HashedFileFactory.getFilesPerDirectory() |
| setLevels | | public void setLevels(int levels)(Code) | | Accessor: Sets the number of directory levels. Note that this
modificator can only be called before the virtual constructor
is called the first time.
Parameters:
levels - is the number of directory levels to use
throws:
VTorInUseException - if the virtual constructor is already in use.
throws:
IllegalArgumentException - if the argument is less than zero.
See Also: HashedFileFactory.getLevels() |
| setLevelsFromTotals | | public void setLevelsFromTotals(int totalFiles)(Code) | | Accessor: Sets the number of directory levels. Note that this
modificator can only be called before the virtual constructor
is called the first time. It takes as argument the total number
of expected files instead of the level.
Parameters:
totalFiles - is the total number of files to accomodate.
throws:
VTorInUseException - if the virtual constructor is already in use.
throws:
IllegalArgumentException - if the argument is less than zero.
See Also: HashedFileFactory.getLevels() |
| setMultiplicator | | public void setMultiplicator(int multiplicator)(Code) | | Accessor: Sets the multiplicative factor to account for files which
may be created without calling the virtual constructor.
Parameters:
multiplicator - is the new multiplicator.
throws:
VTorInUseException - if the virtual constructor is already in use.
throws:
IllegalArgumentException - if the argument is less than one.
See Also: HashedFileFactory.getMultiplicator() |
|
|