Class HdfsBlobStore
- All Implemented Interfaces:
- AutoCloseable,- Shutdownable
We currently have NIMBUS_ADMINS and SUPERVISOR_ADMINS configuration. NIMBUS_ADMINS are given READ, WRITE and ADMIN access whereas the SUPERVISOR_ADMINS are given READ access in order to read and download the blobs form the nimbus.
The ACLs for the blob store are validated against whether the subject is a NIMBUS_ADMIN, SUPERVISOR_ADMIN or USER who has read, write or admin privileges in order to perform respective operations on the blob.
For hdfs blob store 1. The USER interacts with nimbus to upload and access blobs through NimbusBlobStore Client API. Here, unlike local blob store which stores the blobs locally, the nimbus talks to HDFS to upload the blobs. 2. The USER sets the ACLs, and the blob access is validated against these ACLs. 3. The SUPERVISOR interacts with nimbus through HdfsClientBlobStore to download the blobs. Here, unlike local blob store the supervisor interacts with HDFS directly to download the blobs. The call to HdfsBlobStore is made as a "null" subject. The blobstore gets the hadoop user and validates permissions for the supervisor.
- 
Nested Class SummaryNested classes/interfaces inherited from class org.apache.storm.blobstore.BlobStoreBlobStore.BlobStoreFileInputStream, BlobStore.BlobStoreFileOutputStream, BlobStore.KeyTranslationIterator
- 
Field SummaryFields inherited from class org.apache.storm.blobstore.BlobStoreBASE_BLOBS_DIR_NAME
- 
Constructor SummaryConstructors
- 
Method SummaryModifier and TypeMethodDescriptionbooleanblobExists(String key, Subject who) Checks if a blob exists.createBlob(String key, SettableBlobMeta meta, Subject who) Creates the blob.voiddeleteBlob(String key, Subject who) Deletes the blob data and metadata.voidfullCleanup(long age) Gets the InputStream to read the blob details.getBlobMeta(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.intgetBlobReplication(String key, Subject who) Gets the replication factor of the blob.longlistKeys()Returns an iterator with all the list of keys currently available on the blob store.voidprepare(Map<String, Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.protected voidprepareInternal(Map<String, Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf) Allow a Hadoop Configuration to be passed for testing.voidsetBlobMeta(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.voidsetLeaderElector(ILeaderElector leaderElector) Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.voidshutdown()updateBlob(String key, Subject who) Updates the blob data.intupdateBlobReplication(String key, int replication, Subject who) Modifies the replication factor of the blob.voidUpdates the last update time of existing blobs in the blobstore to the current time.voidValidates that the blob update time of the blobstore is up to date with the current existing blobs.voidwriteMetadata(String key, SettableBlobMeta meta) Methods inherited from class org.apache.storm.blobstore.BlobStoreclose, createBlob, createBlob, filterAndListKeys, readBlob, readBlobTo, startSyncBlobs, storedTopoIds, updateBlob, validateKey
- 
Constructor Details- 
HdfsBlobStorepublic HdfsBlobStore()
 
- 
- 
Method Details- 
preparepublic void prepare(Map<String, Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Description copied from class:BlobStoreAllows us to initialize the blob store.
- 
prepareInternalprotected void prepareInternal(Map<String, Object> conf, String overrideBase, org.apache.hadoop.conf.Configuration hadoopConf) Allow a Hadoop Configuration to be passed for testing. If it's null then the hadoop configs must be in your classpath.
- 
createBlobpublic AtomicOutputStream createBlob(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyAlreadyExistsException Description copied from class:BlobStoreCreates the blob.- Specified by:
- createBlobin class- BlobStore
- Parameters:
- key- Key for the blob
- meta- Metadata which contains the acls information
- who- Is the subject creating the blob
- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
- AuthorizationException
- KeyAlreadyExistsException
 
- 
updateBlobpublic AtomicOutputStream updateBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreUpdates the blob data.- Specified by:
- updateBlobin class- BlobStore
- Parameters:
- key- Key for the blob
- who- Is the subject having the write privilege for the blob
- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
getBlobMetapublic ReadableBlobMeta getBlobMeta(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreGets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.- Specified by:
- getBlobMetain class- BlobStore
- Parameters:
- key- Key for the blob
- who- Is the subject having the read privilege for the blob
- Returns:
- AtomicOutputStream returns a stream into which the data can be written
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
setLeaderElectorSets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.- Specified by:
- setLeaderElectorin class- BlobStore
- Parameters:
- leaderElector- the leader elector
 
- 
setBlobMetapublic void setBlobMeta(String key, SettableBlobMeta meta, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreSets the metadata with renewed acls for the blob.- Specified by:
- setBlobMetain class- BlobStore
- Parameters:
- key- Key for the blob
- meta- Metadata which contains the updated acls information
- who- Is the subject having the write privilege for the blob
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
deleteBlobDescription copied from class:BlobStoreDeletes the blob data and metadata.- Specified by:
- deleteBlobin class- BlobStore
- Parameters:
- key- Key for the blob
- who- Is the subject having write privilege for the blob
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
getBlobpublic InputStreamWithMeta getBlob(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreGets the InputStream to read the blob details.- Specified by:
- getBlobin class- BlobStore
- Parameters:
- key- Key for the blob
- who- Is the subject having the read privilege for the blob
- Returns:
- InputStreamWithMeta has the additional file length and version information
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
blobExistsChecks if a blob exists.- Parameters:
- key- blobstore key
- who- subject
- Throws:
- AuthorizationException- if authorization is failed
 
- 
listKeysDescription copied from class:BlobStoreReturns an iterator with all the list of keys currently available on the blob store.
- 
shutdownpublic void shutdown()
- 
getBlobReplicationpublic int getBlobReplication(String key, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreGets the replication factor of the blob.- Specified by:
- getBlobReplicationin class- BlobStore
- Parameters:
- key- Key for the blob
- who- Is the subject having the read privilege for the blob
- Returns:
- BlobReplication object containing the replication factor for the blob
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
updateBlobReplicationpublic int updateBlobReplication(String key, int replication, Subject who) throws AuthorizationException, KeyNotFoundException Description copied from class:BlobStoreModifies the replication factor of the blob.- Specified by:
- updateBlobReplicationin class- BlobStore
- Parameters:
- key- Key for the blob
- replication- The replication factor the blob has to be set
- who- Is the subject having the update privilege for the blob
- Returns:
- BlobReplication object containing the updated replication factor for the blob
- Throws:
- AuthorizationException
- KeyNotFoundException
 
- 
writeMetadatapublic void writeMetadata(String key, SettableBlobMeta meta) throws AuthorizationException, KeyNotFoundException 
- 
fullCleanup- Throws:
- IOException
 
- 
getLastBlobUpdateTime- Throws:
- IOException
 
- 
updateLastBlobUpdateTimeDescription copied from class:BlobStoreUpdates the last update time of existing blobs in the blobstore to the current time.- Overrides:
- updateLastBlobUpdateTimein class- BlobStore
- Throws:
- IOException- on any error
 
- 
validateBlobUpdateTimeDescription copied from class:BlobStoreValidates that the blob update time of the blobstore is up to date with the current existing blobs.- Overrides:
- validateBlobUpdateTimein class- BlobStore
- Throws:
- IOException- on any error
 
 
-