Package org.apache.storm.scheduler
Class Cluster
java.lang.Object
org.apache.storm.scheduler.Cluster
- All Implemented Interfaces:
- ISchedulingState
- Direct Known Subclasses:
- SingleTopologyCluster
The current state of the storm cluster.  Cluster is not currently thread safe.
- 
Constructor SummaryConstructorsConstructorDescriptionCopy constructor.Cluster(Cluster src, Topologies topologies) Testing Constructor that takes an existing cluster and replaces the topologies in it.Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) 
- 
Method SummaryModifier and TypeMethodDescriptionprotected voidassertValidTopologyForModification(String topologyId) Check if the given topology is allowed for modification right now.voidassign(SchedulerAssignment assignment, boolean ignoreSingleExceptions) Assign everything for the given topology.voidassign(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors) Assign the slot to the executors for this topology.voidblacklistHost(String host) voidfreeSlot(WorkerSlot slot) Free the specified slot.voidfreeSlots(Collection<WorkerSlot> slots) free the slots.Get all scheduled resources for node.getAssignablePorts(SupervisorDetails supervisor) Get the ports that are not blacklisted.Get all non-blacklisted slots in the cluster.getAssignableSlots(SupervisorDetails supervisor) Return all non-blacklisted slots on this supervisor.static doublegetAssignedMemoryForSlot(Map<String, Object> topConf) Get heap memory usage for a worker's main process and logwriter process.intgetAssignedNumWorkers(TopologyDetails topology) Get the number of workers assigned to a topology.getAssignmentById(String topologyId) get the current assignment for the topology.Get all the assignments.getAvailablePorts(SupervisorDetails supervisor) Return the available ports of this supervisor.Get the resources on the supervisor that are available to be scheduled.Get all the available worker slots in the cluster.getAvailableSlots(SupervisorDetails supervisor) Return all the available slots on this supervisor.Get all of the hosts that are blacklisted.doubleGet the total amount of CPU resources in cluster.Get the total amount of generic resources (excluding CPU and memory) in cluster.doubleGet the total amount of memory resources in cluster.getConf()Get the nimbus configuration.Map a supervisor to a given host.doubleGet the component name to executor list for executors that need to be scheduled.Get the executor to component name map for executors that need to be scheduled.Get the network topography (rackId -> nodes in the rack).getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds) Get all the available worker slots in the cluster, that are not blacklisted.getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds) Get the resources in the cluster that are available for scheduling.doublegetScheduledCpuForNode(String nodeId) Get the total cpu currently scheduled on a node.doublegetScheduledMemoryForNode(String nodeId) Get the total memory currently scheduled on a node.Get all topology scheduler statuses.getSupervisorById(String nodeId) Get a specific supervisor with thenodeId.Get all the supervisors.getSupervisorsByHost(String host) Get all the supervisors on the specifiedhost.Get the amount of used and free resources on a supervisor.Get all of the topologies.Get the amount of resources used by topologies.getUnassignedExecutors(TopologyDetails topology) get the unassigned executors of the topology.getUsedPorts(SupervisorDetails supervisor) Get all the used ports of this supervisor.Get all currently occupied slots.getUsedSlotsByTopologyId(String topologyId) get slots used by a topology.Get the resources for a given slot.Gets the reference to the full topology->worker resource map.booleanisBlackListed(String supervisorId) Check is a given supervisor is on a blacklisted host.booleanisBlacklistedHost(String host) Check if a given host is blacklisted.booleanisSlotOccupied(WorkerSlot slot) Check if a slot is occupied or not.booleanneedsScheduling(TopologyDetails topology) Does the topology need scheduling.booleanneedsSchedulingRas(TopologyDetails topology) LikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested.Get all of the topologies that need scheduling.voidsetAssignments(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster.voidsetBlacklistedHosts(Set<String> hosts) Set the list of hosts that are blacklisted.voidsetGreyListedSupervisors(Set<String> greyListedSupervisors) voidsetNetworkTopography(Map<String, List<String>> networkTopography) voidset scheduler status for a topology.voidsetStatus(TopologyDetails td, String statusMessage) set scheduler status for a topology.voidsetStatusIfAbsent(String topologyId, String statusMessage) voidsetStatusMap(Map<String, String> statusMap) set scheduler status map.voidUnassign everything for the given topology id.voidupdateFrom(Cluster other) Update the assignments and status from the other cluster.booleanwouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Would scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.Methods inherited from class java.lang.Objectclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface org.apache.storm.scheduler.ISchedulingStategetAssignedRacks, getHostToRack
- 
Constructor Details- 
Clusterpublic Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String, SupervisorDetails> supervisors, Map<String, ? extends SchedulerAssignment> assignments, Topologies topologies, Map<String, Object> conf) 
- 
ClusterCopy constructor.
- 
ClusterTesting Constructor that takes an existing cluster and replaces the topologies in it.- Parameters:
- src- the original cluster
- topologies- the new topolgoies to use
 
 
- 
- 
Method Details- 
getAssignedMemoryForSlotGet heap memory usage for a worker's main process and logwriter process.- Parameters:
- topConf- - the topology config
- Returns:
- the assigned memory (in MB)
 
- 
assertValidTopologyForModificationCheck if the given topology is allowed for modification right now. If not throw an IllegalArgumentException else go on.- Parameters:
- topologyId- the id of the topology to check
 
- 
getTopologiesDescription copied from interface:ISchedulingStateGet all of the topologies.- Specified by:
- getTopologiesin interface- ISchedulingState
- Returns:
- all of the topologies that are a part of the cluster.
 
- 
getBlacklistedHostsDescription copied from interface:ISchedulingStateGet all of the hosts that are blacklisted.- Specified by:
- getBlacklistedHostsin interface- ISchedulingState
- Returns:
- all of the hosts that are blacklisted
 
- 
setBlacklistedHostsSet the list of hosts that are blacklisted.- Parameters:
- hosts- the new hosts that are blacklisted.
 
- 
blacklistHost
- 
isBlackListedDescription copied from interface:ISchedulingStateCheck is a given supervisor is on a blacklisted host.- Specified by:
- isBlackListedin interface- ISchedulingState
- Parameters:
- supervisorId- the id of the supervisor
- Returns:
- true if it is else false
 
- 
isBlacklistedHostDescription copied from interface:ISchedulingStateCheck if a given host is blacklisted.- Specified by:
- isBlacklistedHostin interface- ISchedulingState
- Parameters:
- host- the name of the host
- Returns:
- true if it is else false.
 
- 
getHostDescription copied from interface:ISchedulingStateMap a supervisor to a given host.- Specified by:
- getHostin interface- ISchedulingState
- Parameters:
- supervisorId- the id of the supervisor
- Returns:
- the actual host name the supervisor is on
 
- 
needsSchedulingTopologiesDescription copied from interface:ISchedulingStateGet all of the topologies that need scheduling.- Specified by:
- needsSchedulingTopologiesin interface- ISchedulingState
- Returns:
- all of the topologies that are not fully scheduled.
 
- 
needsSchedulingDescription copied from interface:ISchedulingStateDoes the topology need scheduling.A topology needs scheduling if one of the following conditions holds: - Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
- There are unassigned executors in this topology
 - Specified by:
- needsSchedulingin interface- ISchedulingState
 
- 
needsSchedulingRasDescription copied from interface:ISchedulingStateLikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested. This is because the number of workers is ignored in RAS- Specified by:
- needsSchedulingRasin interface- ISchedulingState
- Parameters:
- topology- the topology to check
- Returns:
- true if the topology needs scheduling else false.
 
- 
getNeedsSchedulingExecutorToComponentsDescription copied from interface:ISchedulingStateGet the executor to component name map for executors that need to be scheduled.- Specified by:
- getNeedsSchedulingExecutorToComponentsin interface- ISchedulingState
- Parameters:
- topology- the topology this is for
- Returns:
- a executor -> component-id map which needs scheduling in this topology.
 
- 
getNeedsSchedulingComponentToExecutorspublic Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology) Description copied from interface:ISchedulingStateGet the component name to executor list for executors that need to be scheduled.- Specified by:
- getNeedsSchedulingComponentToExecutorsin interface- ISchedulingState
- Parameters:
- topology- the topology this is for
- Returns:
- a component-id -> executors map which needs scheduling in this topology.
 
- 
getUsedPortsDescription copied from interface:ISchedulingStateGet all the used ports of this supervisor.- Specified by:
- getUsedPortsin interface- ISchedulingState
 
- 
getAvailablePortsDescription copied from interface:ISchedulingStateReturn the available ports of this supervisor.- Specified by:
- getAvailablePortsin interface- ISchedulingState
 
- 
getAssignablePortsDescription copied from interface:ISchedulingStateGet the ports that are not blacklisted.- Specified by:
- getAssignablePortsin interface- ISchedulingState
- Parameters:
- supervisor- the supervisor
- Returns:
- the ports that are not blacklisted
 
- 
getNonBlacklistedAvailableSlotsDescription copied from interface:ISchedulingStateGet all the available worker slots in the cluster, that are not blacklisted.- Specified by:
- getNonBlacklistedAvailableSlotsin interface- ISchedulingState
- Parameters:
- blacklistedSupervisorIds- list of supervisor ids that should also be considered blacklisted.
 
- 
getAvailableSlotsDescription copied from interface:ISchedulingStateGet all the available worker slots in the cluster.- Specified by:
- getAvailableSlotsin interface- ISchedulingState
 
- 
getAvailableSlotsDescription copied from interface:ISchedulingStateReturn all the available slots on this supervisor.- Specified by:
- getAvailableSlotsin interface- ISchedulingState
 
- 
getAssignableSlotsDescription copied from interface:ISchedulingStateReturn all non-blacklisted slots on this supervisor.- Specified by:
- getAssignableSlotsin interface- ISchedulingState
- Parameters:
- supervisor- the supervisor
- Returns:
- the non-blacklisted slots
 
- 
getAssignableSlotsDescription copied from interface:ISchedulingStateGet all non-blacklisted slots in the cluster.- Specified by:
- getAssignableSlotsin interface- ISchedulingState
 
- 
getUnassignedExecutorsDescription copied from interface:ISchedulingStateget the unassigned executors of the topology.- Specified by:
- getUnassignedExecutorsin interface- ISchedulingState
- Parameters:
- topology- the topology to check
- Returns:
- the unassigned executors of the topology.
 
- 
getAssignedNumWorkersDescription copied from interface:ISchedulingStateGet the number of workers assigned to a topology.- Specified by:
- getAssignedNumWorkersin interface- ISchedulingState
- Parameters:
- topology- the topology this is for
- Returns:
- the number of workers assigned to this topology.
 
- 
getAvailableResourcesDescription copied from interface:ISchedulingStateGet the resources on the supervisor that are available to be scheduled.- Specified by:
- getAvailableResourcesin interface- ISchedulingState
- Parameters:
- sd- the supervisor.
- Returns:
- the resources available to be scheduled.
 
- 
wouldFitpublic boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap) Description copied from interface:ISchedulingStateWould scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.- Specified by:
- wouldFitin interface- ISchedulingState
- Parameters:
- ws- the slot to put it in
- exec- the executor to investigate
- td- the topology detains for this executor
- resourcesAvailable- all the available resources
- maxHeap- the maximum heap size for ws
- Returns:
- true it fits else false
 
- 
assignAssign the slot to the executors for this topology.- Throws:
- RuntimeException- if the specified slot is already occupied.
 
- 
assignAssign everything for the given topology.- Parameters:
- assignment- the new assignment to make
 
- 
freeSlotFree the specified slot.- Parameters:
- slot- the slot to free
 
- 
freeSlotsfree the slots.- Parameters:
- slots- multiple slots to free
 
- 
isSlotOccupiedDescription copied from interface:ISchedulingStateCheck if a slot is occupied or not.- Specified by:
- isSlotOccupiedin interface- ISchedulingState
- Parameters:
- slot- the slot be to checked.
- Returns:
- true if the specified slot is occupied.
 
- 
getAssignmentByIdDescription copied from interface:ISchedulingStateget the current assignment for the topology.- Specified by:
- getAssignmentByIdin interface- ISchedulingState
 
- 
getUsedSlotsByTopologyIdDescription copied from interface:ISchedulingStateget slots used by a topology.- Specified by:
- getUsedSlotsByTopologyIdin interface- ISchedulingState
 
- 
getSupervisorByIdDescription copied from interface:ISchedulingStateGet a specific supervisor with thenodeId.- Specified by:
- getSupervisorByIdin interface- ISchedulingState
 
- 
getUsedSlotsDescription copied from interface:ISchedulingStateGet all currently occupied slots.- Specified by:
- getUsedSlotsin interface- ISchedulingState
 
- 
getSupervisorsByHostDescription copied from interface:ISchedulingStateGet all the supervisors on the specifiedhost.- Specified by:
- getSupervisorsByHostin interface- ISchedulingState
- Parameters:
- host- hostname of the supervisor
- Returns:
- the SupervisorDetailsobject.
 
- 
getAssignmentsDescription copied from interface:ISchedulingStateGet all the assignments.- Specified by:
- getAssignmentsin interface- ISchedulingState
 
- 
setAssignmentspublic void setAssignments(Map<String, ? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions) Set assignments for cluster.
- 
getSupervisorsDescription copied from interface:ISchedulingStateGet all the supervisors.- Specified by:
- getSupervisorsin interface- ISchedulingState
 
- 
getNonBlacklistedClusterAvailableResourcespublic NormalizedResourceOffer getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds) Description copied from interface:ISchedulingStateGet the resources in the cluster that are available for scheduling.- Specified by:
- getNonBlacklistedClusterAvailableResourcesin interface- ISchedulingState
- Parameters:
- blacklistedSupervisorIds- other ids that are tentatively blacklisted.
 
- 
getClusterTotalCpuResourcepublic double getClusterTotalCpuResource()Description copied from interface:ISchedulingStateGet the total amount of CPU resources in cluster.- Specified by:
- getClusterTotalCpuResourcein interface- ISchedulingState
 
- 
getClusterTotalMemoryResourcepublic double getClusterTotalMemoryResource()Description copied from interface:ISchedulingStateGet the total amount of memory resources in cluster.- Specified by:
- getClusterTotalMemoryResourcein interface- ISchedulingState
 
- 
getClusterTotalGenericResourcesDescription copied from interface:ISchedulingStateGet the total amount of generic resources (excluding CPU and memory) in cluster.- Specified by:
- getClusterTotalGenericResourcesin interface- ISchedulingState
 
- 
getNetworkTopographyDescription copied from interface:ISchedulingStateGet the network topography (rackId -> nodes in the rack).- Specified by:
- getNetworkTopographyin interface- ISchedulingState
 
- 
setNetworkTopography
- 
setStatusset scheduler status for a topology.
- 
setStatusset scheduler status for a topology.
- 
setStatusIfAbsent
- 
getStatusMapDescription copied from interface:ISchedulingStateGet all topology scheduler statuses.- Specified by:
- getStatusMapin interface- ISchedulingState
 
- 
setStatusMapset scheduler status map.
- 
getStatus
- 
getTopologyResourcesMapDescription copied from interface:ISchedulingStateGet the amount of resources used by topologies. Used for displaying resource information on the UI.- Specified by:
- getTopologyResourcesMapin interface- ISchedulingState
- Returns:
- a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
 
- 
getSupervisorsResourcesMapDescription copied from interface:ISchedulingStateGet the amount of used and free resources on a supervisor. Used for displaying resource information on the UI- Specified by:
- getSupervisorsResourcesMapin interface- ISchedulingState
- Returns:
- a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}
 
- 
getWorkerResourcesMapDescription copied from interface:ISchedulingStateGets the reference to the full topology->worker resource map.- Specified by:
- getWorkerResourcesMapin interface- ISchedulingState
- Returns:
- map of topology -> map of worker slot ->resources for that worker
 
- 
getWorkerResourcesDescription copied from interface:ISchedulingStateGet the resources for a given slot.- Specified by:
- getWorkerResourcesin interface- ISchedulingState
- Parameters:
- ws- the slot
- Returns:
- the resources currently assigned
 
- 
getResourceMetrics
- 
getAllScheduledResourcesForNodeDescription copied from interface:ISchedulingStateGet all scheduled resources for node.- Specified by:
- getAllScheduledResourcesForNodein interface- ISchedulingState
 
- 
getScheduledMemoryForNodeDescription copied from interface:ISchedulingStateGet the total memory currently scheduled on a node.- Specified by:
- getScheduledMemoryForNodein interface- ISchedulingState
- Parameters:
- nodeId- the id of the node
- Returns:
- the total memory currently scheduled on the node
 
- 
getScheduledCpuForNodeDescription copied from interface:ISchedulingStateGet the total cpu currently scheduled on a node.- Specified by:
- getScheduledCpuForNodein interface- ISchedulingState
- Parameters:
- nodeId- the id of the node
- Returns:
- the total cpu currently scheduled on the node
 
- 
getINimbus
- 
getConfDescription copied from interface:ISchedulingStateGet the nimbus configuration.- Specified by:
- getConfin interface- ISchedulingState
 
- 
unassignUnassign everything for the given topology id.- Parameters:
- topoId- the is of the topology to unassign
 
- 
updateFromUpdate the assignments and status from the other cluster.- Parameters:
- other- the cluster to get the assignments and status from
 
- 
getMinWorkerCpupublic double getMinWorkerCpu()
- 
getGreyListedSupervisors
- 
setGreyListedSupervisors
 
-