Add managed capacity kill controls
This commit is contained in:
@@ -246,7 +246,7 @@ Active remote nodes show a small progress panel with a Modal, SSH terminal, or V
|
||||
|
||||
Remote planning and capacity acquisition count as active queue work even though they happen before the rewritten prompt enters ComfyUI's native execution queue. During Vast startup, the canvas receives readable updates for rental, worker-image download, SSH readiness, and runtime initialization. A provider-side terminal image-pull failure is reported immediately rather than appearing as an empty queue until the startup timeout expires. A rental that never produces an SSH-usable worker is marked unusable and destroyed after the bounded initialization wait so a later workflow cannot silently reuse it.
|
||||
|
||||
For workflows using a **Remote Execution Configurator**, that node is also the primary planning surface. As soon as scheduling finishes, the scheduler result is streamed to the panel before any Vast rental or SSH-readiness wait begins. The browser retains that plan across late node-panel mounts, a full page refresh, tab refocus, queue failure, and extension reloads, while removing any orphaned prior panel before remounting. The execution plan is grouped into one section per concrete target, with labeled components beneath it and the best-known machine type, GPU/HBM or VRAM capacity, system RAM, predicted completion time, and predicted cost. Modal system RAM is explicitly shown as provider-managed because it is assigned dynamically rather than inferred as a fixed machine capacity. Configuration names and selected targets therefore remain inspectable even when capacity never becomes ready. Connected storage backends are summarized beneath the execution plan rather than presented as scheduler targets; an R2 summary includes its bucket, current storage use and object count, jurisdiction, cache policy, and optional key prefix without exposing credentials. Bucket usage is read through the existing bucket-scoped S3 credential and cached for five minutes so repeated workflow submissions do not repeatedly enumerate the bucket. The storage area's **Reload** button bypasses that cache, retrieves fresh aggregate state, and updates the retained plan without queueing a workflow. Each selected environment has its own status and progress lane, so parallel Vast, Modal, and self-hosted setup or execution updates cannot overwrite each other; once a newer prompt owns the panel, delayed events from an older prompt remain isolated. Planned environments begin in a waiting state. Vast capacity then reports rental, worker-image download, initialization, and asset preparation before becoming ready. SSH rows distinguish completed asset synchronization from worker readiness and stream the later runtime check, image build, container start, and health wait; they report ready only after the worker socket answers with the expected fingerprint. A bounded controller pool prepares custom nodes and prompt assets concurrently across distinct environments, while preserving component order and request-cache reuse within each environment. A prompt-wide failure persistently terminates every environment lane and clears its in-progress meter, including when a later plan render or panel remount occurs. Live preparation and execution status use the same durable event stream. Queue-history reconciliation reads the prompt's actual terminal result, preserving a failure or interruption in the Configurator and replacing stale submission text when the prompt is no longer executing. Legacy workflows without a configurator continue to use the global status pill.
|
||||
For workflows using a **Remote Execution Configurator**, that node is also the primary planning surface. As soon as scheduling finishes, the scheduler result is streamed to the panel before any Vast rental or SSH-readiness wait begins. The browser retains that plan across late node-panel mounts, a full page refresh, tab refocus, queue failure, and extension reloads, while removing any orphaned prior panel before remounting. The execution plan is grouped into one section per concrete target, with labeled components beneath it and the best-known machine type, GPU/HBM or VRAM capacity, system RAM, predicted completion time, and predicted cost. Modal system RAM is explicitly shown as provider-managed because it is assigned dynamically rather than inferred as a fixed machine capacity. Configuration names and selected targets therefore remain inspectable even when capacity never becomes ready. The **Managed capacity** area independently lists every lease or container owned by this installation, including idle leftovers from earlier workflows, with provider, resource ID, lifecycle state, activity, GPU/RAM, and hourly price where available. Its **Reload** button refreshes the list, while each exact Vast lease or Modal container has a destructive **KILL** button with confirmation; ownership is revalidated by the server, and killing active capacity is explicitly allowed because the confirmation warns that its workflow will fail. Connected storage backends are summarized beneath the execution plan rather than presented as scheduler targets; an R2 summary includes its bucket, current storage use and object count, jurisdiction, cache policy, and optional key prefix without exposing credentials. Bucket usage is read through the existing bucket-scoped S3 credential and cached for five minutes so repeated workflow submissions do not repeatedly enumerate the bucket. The storage area's **Reload** button bypasses that cache, retrieves fresh aggregate state, and updates the retained plan without queueing a workflow. Each selected environment has its own status and progress lane, so parallel Vast, Modal, and self-hosted setup or execution updates cannot overwrite each other; once a newer prompt owns the panel, delayed events from an older prompt remain isolated. Planned environments begin in a waiting state. Vast capacity then reports rental, worker-image download, initialization, and asset preparation before becoming ready. SSH rows distinguish completed asset synchronization from worker readiness and stream the later runtime check, image build, container start, and health wait; they report ready only after the worker socket answers with the expected fingerprint. A bounded controller pool prepares custom nodes and prompt assets concurrently across distinct environments, while preserving component order and request-cache reuse within each environment. A prompt-wide failure persistently terminates every environment lane and clears its in-progress meter, including when a later plan render or panel remount occurs. Live preparation and execution status use the same durable event stream. Queue-history reconciliation reads the prompt's actual terminal result, preserving a failure or interruption in the Configurator and replacing stale submission text when the prompt is no longer executing. Legacy workflows without a configurator continue to use the global status pill.
|
||||
|
||||
Queue-time preparation appears in ComfyUI's queue through the server-side preparation bridge, including the native job metadata required by the queue panel, while the Configurator panel and global status pill show its detailed phase. The frontend acknowledges the prompt immediately so ComfyUI owns a visible, cancellable job while remote capacity is still starting; ComfyUI's standard Jobs cancellation also signals the prompt's preparation task. Preparation deliberately does not emit fake native node-execution events: doing so starts ComfyUI's execution watchdog, which can incorrectly mark a node as stalled or failed during a healthy multi-minute asset download. A preparation failure remains a queue/setup error with its original message rather than being recast as a node execution crash.
|
||||
|
||||
|
||||
+89
-21
@@ -91,7 +91,7 @@ from .ssh_runtime import SshRuntimeManager
|
||||
from .vast_config_node import extract_vast_profiles
|
||||
from .vast_service import VastProfileQuote, VastSearchRequirements, VastService
|
||||
from .vast_api import VastApiClient
|
||||
from .vast_leases import VastLeaseRegistry
|
||||
from .vast_leases import VastLeaseManager, VastLeaseRegistry
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
SetupStatusCallback = Callable[[str, int | None, int | None], None]
|
||||
@@ -9064,6 +9064,9 @@ def setup_modal_queue_route(
|
||||
container_status_route_path = _container_status_route_path(
|
||||
resolved_settings.route_path
|
||||
)
|
||||
container_stop_route_path = container_status_route_path.replace(
|
||||
"/container_status", "/container_stop"
|
||||
)
|
||||
delete_caches_route_path = _delete_modal_caches_route_path(
|
||||
resolved_settings.route_path
|
||||
)
|
||||
@@ -9183,17 +9186,27 @@ def setup_modal_queue_route(
|
||||
containers_task = asyncio.create_task(
|
||||
list_active_modal_containers(resolved_settings)
|
||||
)
|
||||
billing_task = asyncio.create_task(
|
||||
get_hourly_modal_app_billing(
|
||||
selected_settings.modal_gpu,
|
||||
resolved_settings,
|
||||
include_billing = request.query.get("include_billing", "true").casefold() not in {
|
||||
"0",
|
||||
"false",
|
||||
"no",
|
||||
}
|
||||
billing_task = (
|
||||
asyncio.create_task(
|
||||
get_hourly_modal_app_billing(
|
||||
selected_settings.modal_gpu,
|
||||
resolved_settings,
|
||||
)
|
||||
)
|
||||
if include_billing
|
||||
else None
|
||||
)
|
||||
try:
|
||||
containers = await containers_task
|
||||
except ModalContainerStatusError as exc:
|
||||
billing_task.cancel()
|
||||
await asyncio.gather(billing_task, return_exceptions=True)
|
||||
if billing_task is not None:
|
||||
billing_task.cancel()
|
||||
await asyncio.gather(billing_task, return_exceptions=True)
|
||||
logger.warning("Unable to refresh Modal container status: %s", exc)
|
||||
return web.json_response(
|
||||
{"containers": [], "error": str(exc), "polled_at": time.time()},
|
||||
@@ -9201,11 +9214,12 @@ def setup_modal_queue_route(
|
||||
)
|
||||
billing = None
|
||||
billing_error = None
|
||||
try:
|
||||
billing = await billing_task
|
||||
except ModalBillingStatusError as exc:
|
||||
billing_error = str(exc)
|
||||
logger.warning("Unable to refresh Modal hourly billing: %s", exc)
|
||||
if billing_task is not None:
|
||||
try:
|
||||
billing = await billing_task
|
||||
except ModalBillingStatusError as exc:
|
||||
billing_error = str(exc)
|
||||
logger.warning("Unable to refresh Modal hourly billing: %s", exc)
|
||||
return web.json_response(
|
||||
{
|
||||
"containers": [container.as_dict() for container in containers],
|
||||
@@ -9215,9 +9229,30 @@ def setup_modal_queue_route(
|
||||
}
|
||||
)
|
||||
|
||||
@prompt_server.routes.post(container_stop_route_path)
|
||||
async def modal_container_stop(request: web.Request) -> web.Response:
|
||||
"""Stop one exact active Modal container owned by this installation."""
|
||||
from .remote.modal_app import (
|
||||
ModalContainerStatusError,
|
||||
stop_managed_modal_container,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = await request.json()
|
||||
container_id = str(payload.get("container_id") or "").strip()
|
||||
stopped = await stop_managed_modal_container(
|
||||
container_id,
|
||||
resolved_settings,
|
||||
)
|
||||
except (ModalContainerStatusError, TypeError, ValueError) as exc:
|
||||
return web.json_response({"error": str(exc)}, status=502)
|
||||
return web.json_response(
|
||||
{"container_id": container_id, "stopped": stopped}
|
||||
)
|
||||
|
||||
@prompt_server.routes.get("/remote/vast/status")
|
||||
async def vast_status(request: web.Request) -> web.Response:
|
||||
"""Return credential-free managed lease state and controller readiness."""
|
||||
"""Return refreshed credential-free managed Vast lease inventory."""
|
||||
del request
|
||||
if vast_registry is None:
|
||||
return web.json_response(
|
||||
@@ -9225,17 +9260,34 @@ def setup_modal_queue_route(
|
||||
status=503,
|
||||
)
|
||||
try:
|
||||
state = await asyncio.to_thread(vast_registry.load)
|
||||
except ValueError as exc:
|
||||
if not os.getenv("VAST_API_KEY"):
|
||||
state = await asyncio.to_thread(vast_registry.load)
|
||||
leases = state.leases
|
||||
else:
|
||||
api_key = str(os.getenv("VAST_API_KEY") or "").strip()
|
||||
base_url = str(
|
||||
os.getenv("COMFY_MODAL_VAST_API_BASE_URL") or ""
|
||||
).strip()
|
||||
api_client = VastApiClient(
|
||||
api_key,
|
||||
**({"base_url": base_url} if base_url else {}),
|
||||
)
|
||||
manager = VastLeaseManager.for_inventory(
|
||||
api_client=api_client,
|
||||
registry=vast_registry,
|
||||
owner_id=resolved_settings.app_name,
|
||||
)
|
||||
leases = await manager.refresh_owned_leases()
|
||||
except (OSError, RuntimeError, ValueError) as exc:
|
||||
return web.json_response(
|
||||
{"configured": bool(os.getenv("VAST_API_KEY")), "leases": [], "error": str(exc)},
|
||||
status=500,
|
||||
status=502,
|
||||
)
|
||||
return web.json_response(
|
||||
{
|
||||
"configured": bool(os.getenv("VAST_API_KEY")),
|
||||
"image_configured": bool(os.getenv("COMFY_MODAL_VAST_IMAGE")),
|
||||
"leases": [lease.to_dict() for lease in state.leases],
|
||||
"leases": [lease.to_dict() for lease in leases],
|
||||
}
|
||||
)
|
||||
|
||||
@@ -9342,11 +9394,27 @@ def setup_modal_queue_route(
|
||||
try:
|
||||
payload = await request.json()
|
||||
instance_id = int(payload.get("instance_id"))
|
||||
service = VastService.from_environment(
|
||||
resolved_settings,
|
||||
repo_root=Path(__file__).resolve().parent,
|
||||
api_key = str(os.getenv("VAST_API_KEY") or "").strip()
|
||||
if not api_key:
|
||||
raise RuntimeError("Set VAST_API_KEY before destroying Vast capacity.")
|
||||
if vast_registry is None:
|
||||
raise RuntimeError("ComfyUI user directory is unavailable.")
|
||||
base_url = str(
|
||||
os.getenv("COMFY_MODAL_VAST_API_BASE_URL") or ""
|
||||
).strip()
|
||||
api_client = VastApiClient(
|
||||
api_key,
|
||||
**({"base_url": base_url} if base_url else {}),
|
||||
)
|
||||
manager = VastLeaseManager.for_inventory(
|
||||
api_client=api_client,
|
||||
registry=vast_registry,
|
||||
owner_id=resolved_settings.app_name,
|
||||
)
|
||||
destroyed = await manager.destroy_owned_lease(
|
||||
instance_id,
|
||||
allow_active_work=payload.get("force") is True,
|
||||
)
|
||||
destroyed = await service.lease_manager.destroy_owned_lease(instance_id)
|
||||
except (OSError, RuntimeError, TypeError, ValueError) as exc:
|
||||
return web.json_response({"error": str(exc)}, status=502)
|
||||
return web.json_response({"instance_id": instance_id, "destroyed": destroyed})
|
||||
|
||||
@@ -599,3 +599,5 @@
|
||||
- A persisted loading lease must consume only the remainder of its original readiness budget. Giving it a new full startup timeout after every controller restart can retain broken capacity indefinitely.
|
||||
- Vast can delete a contract between an ownership check and the destroy request. A not-found response from destruction confirms that capacity is already absent and should remove the stale registry record rather than turn successful cleanup into another failure.
|
||||
- Vast SSH and Jupyter runtypes replace the image entrypoint, inject access, and then run `onstart`. A custom base image can still leave `/root` or the injected key path incompatible with `sshd` StrictModes; normalize root ownership plus `.ssh` mode 0700 and `authorized_keys` mode 0600 in `onstart`. This repairs new rentals without rebuilding the heavyweight worker image.
|
||||
- Workflow plans are not a complete cloud-resource inventory: cancellation or a controller restart can leave valid idle leases that no longer appear in the current plan. Put provider-owned live capacity in a separately refreshable Configurator section, target destructive controls by immutable provider ID, and repeat the ownership check on the server. A deliberate kill path must be able to override an active-work guard only after an explicit warning; routine reaping and other callers should retain the guard.
|
||||
- Modal exposes active execution containers as task IDs, while its app-level stop operation can terminate unrelated warm capacity. Verify the requested task against the installation's managed app names, then call the exact SDK `ContainerStop` operation on Modal's synchronized event loop instead of stopping the whole deployed app.
|
||||
|
||||
@@ -582,6 +582,28 @@ def _list_modal_tasks_synchronously(
|
||||
return blocking_callable(client_module, api_pb2, environment_name)
|
||||
|
||||
|
||||
async def _stop_modal_task_on_sdk_loop(
|
||||
client_module: Any,
|
||||
api_pb2: Any,
|
||||
container_id: str,
|
||||
) -> Any:
|
||||
"""Stop one exact Modal task while running on Modal's managed event loop."""
|
||||
client = await client_module._Client.from_env()
|
||||
return await client.stub.ContainerStop(
|
||||
api_pb2.ContainerStopRequest(task_id=container_id)
|
||||
)
|
||||
|
||||
|
||||
def _stop_modal_task_synchronously(
|
||||
client_module: Any,
|
||||
api_pb2: Any,
|
||||
container_id: str,
|
||||
) -> Any:
|
||||
"""Run one Modal task-stop request through the synchronized SDK bridge."""
|
||||
blocking_callable = _synchronized_modal_callable(_stop_modal_task_on_sdk_loop)
|
||||
return blocking_callable(client_module, api_pb2, container_id)
|
||||
|
||||
|
||||
def _prune_modal_hourly_billing_cache() -> None:
|
||||
"""Bound retained hourly billing records across apps and GPU selections."""
|
||||
retained_keys = set(_MODAL_HOURLY_BILLING_CACHE) | set(
|
||||
@@ -866,6 +888,47 @@ async def list_active_modal_containers(
|
||||
return containers
|
||||
|
||||
|
||||
async def stop_managed_modal_container(
|
||||
container_id: str,
|
||||
settings: ModalSyncSettings | None = None,
|
||||
) -> bool:
|
||||
"""Stop one active container after verifying it belongs to this installation."""
|
||||
normalized_container_id = container_id.strip()
|
||||
if not normalized_container_id:
|
||||
raise ValueError("Modal container_id must not be empty.")
|
||||
|
||||
active_containers = await list_active_modal_containers(settings)
|
||||
if not any(
|
||||
container.container_id == normalized_container_id
|
||||
for container in active_containers
|
||||
):
|
||||
return False
|
||||
|
||||
try:
|
||||
client_module = importlib.import_module("modal.client")
|
||||
exception_module = importlib.import_module("modal.exception")
|
||||
api_pb2 = importlib.import_module("modal_proto.api_pb2")
|
||||
except ModuleNotFoundError as exc:
|
||||
raise ModalContainerStatusError(
|
||||
"The installed Modal SDK does not expose the container stop API."
|
||||
) from exc
|
||||
|
||||
modal_error_type = getattr(exception_module, "Error", RuntimeError)
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
_stop_modal_task_synchronously,
|
||||
client_module,
|
||||
api_pb2,
|
||||
normalized_container_id,
|
||||
)
|
||||
except (modal_error_type, OSError, AttributeError, RuntimeError) as exc:
|
||||
raise ModalContainerStatusError(
|
||||
f"Unable to stop Modal container {normalized_container_id}: {exc}"
|
||||
) from exc
|
||||
logger.warning("Stopped managed Modal container %s.", normalized_container_id)
|
||||
return True
|
||||
|
||||
|
||||
async def _stream_remote_container_logs_via_modal_sdk_async(
|
||||
task_id: str,
|
||||
stop_event: threading.Event,
|
||||
|
||||
@@ -69,6 +69,11 @@ class FakeElement {
|
||||
".remote-configurator-environments",
|
||||
".remote-configurator-empty",
|
||||
".remote-configurator-targets",
|
||||
".remote-configurator-capacity",
|
||||
".remote-configurator-capacity-list",
|
||||
".remote-configurator-capacity-empty",
|
||||
".remote-configurator-capacity-reload",
|
||||
".remote-configurator-capacity-refresh-status",
|
||||
".remote-configurator-storage",
|
||||
".remote-configurator-storage-list",
|
||||
".remote-configurator-storage-reload",
|
||||
@@ -145,6 +150,8 @@ const transformedSource = `${[
|
||||
" mountRemoteExecutionConfiguratorPanel,",
|
||||
" remoteConfiguratorStorageEntries,",
|
||||
" refreshRemoteConfiguratorStorage,",
|
||||
" remoteManagedCapacityEntries,",
|
||||
" refreshRemoteManagedCapacity,",
|
||||
" remoteStorageSizeLabel,",
|
||||
" remoteStorageUsageLabel,",
|
||||
" resolveComponentNodeIds,",
|
||||
@@ -217,6 +224,37 @@ assert.equal(
|
||||
"1.00 KiB · 1 object",
|
||||
);
|
||||
|
||||
const managedCapacityEntries = modalToggle.remoteManagedCapacityEntries(
|
||||
{
|
||||
leases: [{
|
||||
instance_id: 49047119,
|
||||
profile_name: "vast-small",
|
||||
actual_status: "running",
|
||||
active_invocations: 0,
|
||||
gpu_name: "RTX PRO 6000 WS",
|
||||
gpu_count: 1,
|
||||
gpu_ram_mb: 97894,
|
||||
cpu_ram_mb: 148275,
|
||||
hourly_cost_usd: 0.056,
|
||||
}],
|
||||
},
|
||||
{
|
||||
containers: [{
|
||||
container_id: "ta-managed",
|
||||
app_name: "comfy-modal-sync-B300",
|
||||
modal_gpu: "B300",
|
||||
state: "running",
|
||||
estimated_gpu_cost_per_second: 0.001972,
|
||||
}],
|
||||
},
|
||||
);
|
||||
assert.equal(managedCapacityEntries.length, 2);
|
||||
assert.equal(managedCapacityEntries[0].provider, "vast");
|
||||
assert.equal(managedCapacityEntries[0].resourceId, "49047119");
|
||||
assert.equal(managedCapacityEntries[0].details.at(-1), "$0.056/hr");
|
||||
assert.equal(managedCapacityEntries[1].provider, "modal");
|
||||
assert.equal(managedCapacityEntries[1].resourceId, "ta-managed");
|
||||
|
||||
assert.equal(
|
||||
modalToggle.queueErrorMessage({
|
||||
message: "Prompt validation failed",
|
||||
|
||||
@@ -398,6 +398,7 @@ def test_remote_environment_routes_save_and_probe_hosts(
|
||||
assert ("GET", "/remote/vast/status") in routes.handlers
|
||||
assert ("POST", "/remote/vast/reap") in routes.handlers
|
||||
assert ("POST", "/remote/vast/destroy") in routes.handlers
|
||||
assert ("POST", "/modal/container_stop") in routes.handlers
|
||||
assert ("POST", "/remote/storage/r2/oauth/start") in routes.handlers
|
||||
assert ("GET", "/remote/storage/r2/oauth/callback") in routes.handlers
|
||||
assert ("POST", "/remote/storage/r2/credentials") in routes.handlers
|
||||
|
||||
@@ -8452,6 +8452,7 @@ def test_list_active_modal_containers_filters_and_classifies_managed_apps(
|
||||
request_environment_names: list[str] = []
|
||||
request_thread_ids: list[int] = []
|
||||
request_event_loops: list[Any] = []
|
||||
stopped_container_ids: list[str] = []
|
||||
caller_thread_id = threading.get_ident()
|
||||
sdk_event_loop = asyncio.new_event_loop()
|
||||
|
||||
@@ -8489,6 +8490,13 @@ def test_list_active_modal_containers_filters_and_classifies_managed_apps(
|
||||
]
|
||||
)
|
||||
|
||||
async def ContainerStop(self, request: Any) -> Any:
|
||||
"""Capture one exact Modal task termination request."""
|
||||
stopped_container_ids.append(request.task_id)
|
||||
request_thread_ids.append(threading.get_ident())
|
||||
request_event_loops.append(asyncio.get_running_loop())
|
||||
return types.SimpleNamespace()
|
||||
|
||||
class FakeClient:
|
||||
"""Expose the synchronized Modal TaskList stub."""
|
||||
|
||||
@@ -8529,7 +8537,8 @@ def test_list_active_modal_containers_filters_and_classifies_managed_apps(
|
||||
return types.SimpleNamespace(Error=RuntimeError)
|
||||
if name == "modal_proto.api_pb2":
|
||||
return types.SimpleNamespace(
|
||||
TaskListRequest=lambda **kwargs: types.SimpleNamespace(**kwargs)
|
||||
TaskListRequest=lambda **kwargs: types.SimpleNamespace(**kwargs),
|
||||
ContainerStopRequest=lambda **kwargs: types.SimpleNamespace(**kwargs),
|
||||
)
|
||||
return original_import_module(name)
|
||||
|
||||
@@ -8540,6 +8549,9 @@ def test_list_active_modal_containers_filters_and_classifies_managed_apps(
|
||||
remote_modal_app_module.list_active_modal_containers(settings)
|
||||
)
|
||||
containers = asyncio.run(remote_modal_app_module.list_active_modal_containers(settings))
|
||||
stopped = asyncio.run(
|
||||
remote_modal_app_module.stop_managed_modal_container("ta-running", settings)
|
||||
)
|
||||
|
||||
assert first_containers == containers
|
||||
assert [container.container_id for container in containers] == ["ta-running", "ta-starting"]
|
||||
@@ -8547,13 +8559,84 @@ def test_list_active_modal_containers_filters_and_classifies_managed_apps(
|
||||
assert all(container.modal_gpu == "B300" for container in containers)
|
||||
assert containers[0].as_dict()["started_at"] == 101.0
|
||||
assert containers[0].as_dict()["estimated_gpu_cost_per_second"] == 0.001972
|
||||
assert request_environment_names == ["test-env", "test-env"]
|
||||
assert stopped
|
||||
assert stopped_container_ids == ["ta-running"]
|
||||
assert request_environment_names == ["test-env", "test-env", "test-env"]
|
||||
assert request_thread_ids
|
||||
assert all(thread_id != caller_thread_id for thread_id in request_thread_ids)
|
||||
assert request_event_loops == [sdk_event_loop, sdk_event_loop]
|
||||
assert request_event_loops == [sdk_event_loop] * 4
|
||||
sdk_event_loop.close()
|
||||
|
||||
|
||||
def test_stop_managed_modal_container_verifies_ownership_before_exact_stop(
|
||||
remote_modal_app_module: Any,
|
||||
monkeypatch: Any,
|
||||
) -> None:
|
||||
"""Container termination must target only a currently listed managed task."""
|
||||
stopped_container_ids: list[str] = []
|
||||
|
||||
async def fake_list_active(_settings: Any) -> list[Any]:
|
||||
"""Return one container already filtered to this ComfyUI installation."""
|
||||
return [
|
||||
remote_modal_app_module.ModalContainerStatus(
|
||||
container_id="ta-managed",
|
||||
app_id="ap-managed",
|
||||
app_name="comfy-modal-sync-B300",
|
||||
modal_gpu="B300",
|
||||
estimated_gpu_cost_per_second=0.001972,
|
||||
state="running",
|
||||
enqueued_at=100.0,
|
||||
started_at=101.0,
|
||||
)
|
||||
]
|
||||
|
||||
def fake_stop(
|
||||
_client_module: Any,
|
||||
_api_pb2: Any,
|
||||
container_id: str,
|
||||
) -> None:
|
||||
"""Record the exact task passed to the synchronized SDK bridge."""
|
||||
stopped_container_ids.append(container_id)
|
||||
|
||||
original_import_module = remote_modal_app_module.importlib.import_module
|
||||
|
||||
def fake_import_module(name: str) -> Any:
|
||||
"""Supply the minimal modules needed after ownership verification."""
|
||||
if name == "modal.client":
|
||||
return types.SimpleNamespace()
|
||||
if name == "modal.exception":
|
||||
return types.SimpleNamespace(Error=RuntimeError)
|
||||
if name == "modal_proto.api_pb2":
|
||||
return types.SimpleNamespace()
|
||||
return original_import_module(name)
|
||||
|
||||
monkeypatch.setattr(
|
||||
remote_modal_app_module,
|
||||
"list_active_modal_containers",
|
||||
fake_list_active,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
remote_modal_app_module,
|
||||
"_stop_modal_task_synchronously",
|
||||
fake_stop,
|
||||
)
|
||||
monkeypatch.setattr(remote_modal_app_module.importlib, "import_module", fake_import_module)
|
||||
|
||||
assert asyncio.run(
|
||||
remote_modal_app_module.stop_managed_modal_container(
|
||||
"ta-managed",
|
||||
remote_modal_app_module.get_settings(),
|
||||
)
|
||||
)
|
||||
assert not asyncio.run(
|
||||
remote_modal_app_module.stop_managed_modal_container(
|
||||
"ta-unrelated",
|
||||
remote_modal_app_module.get_settings(),
|
||||
)
|
||||
)
|
||||
assert stopped_container_ids == ["ta-managed"]
|
||||
|
||||
|
||||
def test_modal_gpu_estimated_rates_cover_supported_aliases(
|
||||
remote_modal_app_module: Any,
|
||||
) -> None:
|
||||
|
||||
@@ -517,6 +517,45 @@ def test_reconcile_after_restart_clears_stale_activity_count(
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_inventory_refresh_preserves_live_activity_count(
|
||||
tmp_path: Any,
|
||||
vast_api_module: Any,
|
||||
vast_leases_module: Any,
|
||||
vast_models_module: Any,
|
||||
vast_simulator_module: Any,
|
||||
) -> None:
|
||||
"""A UI inventory refresh must not make an active lease appear idle."""
|
||||
|
||||
async def scenario() -> None:
|
||||
"""Refresh provider state without applying restart reconciliation semantics."""
|
||||
state = vast_simulator_module.VastSimulatorState(polls_until_running=1)
|
||||
async with _running_simulator(
|
||||
vast_simulator_module.create_vast_simulator_app(state)
|
||||
) as base_url:
|
||||
manager = vast_leases_module.VastLeaseManager(
|
||||
api_client=vast_api_module.VastApiClient(
|
||||
state.api_key,
|
||||
base_url=base_url,
|
||||
),
|
||||
registry=vast_leases_module.VastLeaseRegistry.for_user_directory(
|
||||
tmp_path
|
||||
),
|
||||
owner_id="comfy-owner",
|
||||
runtime_fingerprint=_runtime_fingerprint(),
|
||||
launch_spec_factory=_launch_factory(vast_models_module),
|
||||
startup_timeout_seconds=2.0,
|
||||
)
|
||||
lease = await manager.ensure_lease(_profile(vast_models_module))
|
||||
manager.begin_activity(lease.instance_id)
|
||||
|
||||
refreshed = await manager.refresh_owned_leases()
|
||||
|
||||
assert refreshed[0].active_invocations == 1
|
||||
assert manager.registry.load().leases[0].active_invocations == 1
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_manual_destroy_requires_owned_matching_label(
|
||||
tmp_path: Any,
|
||||
vast_api_module: Any,
|
||||
@@ -606,6 +645,48 @@ def test_destroy_removes_registry_when_instance_vanishes_after_ownership_check(
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_explicit_force_destroy_can_terminate_active_owned_lease(
|
||||
tmp_path: Any,
|
||||
vast_api_module: Any,
|
||||
vast_leases_module: Any,
|
||||
vast_models_module: Any,
|
||||
vast_simulator_module: Any,
|
||||
) -> None:
|
||||
"""The destructive UI path may kill active work after exact ownership checks."""
|
||||
|
||||
async def scenario() -> None:
|
||||
"""Acquire, mark busy, and explicitly force-destroy one owned lease."""
|
||||
state = vast_simulator_module.VastSimulatorState(polls_until_running=1)
|
||||
async with _running_simulator(
|
||||
vast_simulator_module.create_vast_simulator_app(state)
|
||||
) as base_url:
|
||||
registry = vast_leases_module.VastLeaseRegistry.for_user_directory(tmp_path)
|
||||
manager = vast_leases_module.VastLeaseManager(
|
||||
api_client=vast_api_module.VastApiClient(
|
||||
state.api_key,
|
||||
base_url=base_url,
|
||||
),
|
||||
registry=registry,
|
||||
owner_id="comfy-owner",
|
||||
runtime_fingerprint=_runtime_fingerprint(),
|
||||
launch_spec_factory=_launch_factory(vast_models_module),
|
||||
startup_timeout_seconds=2.0,
|
||||
)
|
||||
lease = await manager.ensure_lease(_profile(vast_models_module))
|
||||
manager.begin_activity(lease.instance_id)
|
||||
|
||||
with pytest.raises(RuntimeError, match="active work"):
|
||||
await manager.destroy_owned_lease(lease.instance_id)
|
||||
assert await manager.destroy_owned_lease(
|
||||
lease.instance_id,
|
||||
allow_active_work=True,
|
||||
)
|
||||
assert registry.load().leases == ()
|
||||
assert lease.instance_id not in state.instances
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
def test_registry_is_versioned_atomic_and_credential_free(
|
||||
tmp_path: Any,
|
||||
vast_leases_module: Any,
|
||||
|
||||
@@ -392,6 +392,23 @@ def test_global_modal_status_badge_polls_and_renders_active_containers() -> None
|
||||
assert "?modal_gpu=${encodeURIComponent(requestedModalGpu)}" in source
|
||||
|
||||
|
||||
def test_configurator_lists_and_kills_exact_managed_capacity() -> None:
|
||||
"""The Configurator should expose guarded exact-instance lifecycle controls."""
|
||||
source = _modal_toggle_source()
|
||||
|
||||
assert 'const VAST_STATUS_ROUTE = "/remote/vast/status";' in source
|
||||
assert 'const VAST_DESTROY_ROUTE = "/remote/vast/destroy";' in source
|
||||
assert '"/container_stop"' in source
|
||||
assert "function remoteManagedCapacityEntries(" in source
|
||||
assert "function refreshRemoteManagedCapacity(" in source
|
||||
assert "function killRemoteManagedCapacity(" in source
|
||||
assert "Any workflow using it will fail. This cannot be undone." in source
|
||||
assert "?include_billing=false" in source
|
||||
assert 'class="remote-configurator-capacity-title">Managed capacity</div>' in source
|
||||
assert 'button.textContent = "KILL";' in source
|
||||
assert "{ instance_id: Number(entry.resourceId), force: true }" in source
|
||||
|
||||
|
||||
def test_global_modal_status_badge_estimates_prompt_gpu_cost() -> None:
|
||||
"""The global pill should integrate active GPU-container seconds per prompt."""
|
||||
source = _modal_toggle_source()
|
||||
|
||||
+55
-14
@@ -432,6 +432,32 @@ class VastLeaseManager:
|
||||
self._profile_locks_guard = threading.Lock()
|
||||
self._profile_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
@classmethod
|
||||
def for_inventory(
|
||||
cls,
|
||||
*,
|
||||
api_client: VastApiClient,
|
||||
registry: VastLeaseRegistry,
|
||||
owner_id: str,
|
||||
) -> "VastLeaseManager":
|
||||
"""Build a lifecycle-only manager without requiring a worker image."""
|
||||
|
||||
def unavailable_launch_spec(
|
||||
profile: VastResourceProfile,
|
||||
label: str,
|
||||
) -> VastInstanceLaunchSpec:
|
||||
"""Reject acquisition through an inventory-only manager."""
|
||||
del profile, label
|
||||
raise RuntimeError("Inventory-only Vast manager cannot acquire capacity.")
|
||||
|
||||
return cls(
|
||||
api_client=api_client,
|
||||
registry=registry,
|
||||
owner_id=owner_id,
|
||||
runtime_fingerprint="0" * 64,
|
||||
launch_spec_factory=unavailable_launch_spec,
|
||||
)
|
||||
|
||||
async def ensure_lease(
|
||||
self,
|
||||
profile: VastResourceProfile,
|
||||
@@ -839,7 +865,12 @@ class VastLeaseManager:
|
||||
)
|
||||
return tuple(destroyed)
|
||||
|
||||
async def destroy_owned_lease(self, instance_id: int) -> bool:
|
||||
async def destroy_owned_lease(
|
||||
self,
|
||||
instance_id: int,
|
||||
*,
|
||||
allow_active_work: bool = False,
|
||||
) -> bool:
|
||||
"""Destroy one exact registry-owned lease after checking its API label."""
|
||||
lease = next(
|
||||
(
|
||||
@@ -852,7 +883,7 @@ class VastLeaseManager:
|
||||
)
|
||||
if lease is None:
|
||||
return False
|
||||
if lease.active_invocations:
|
||||
if lease.active_invocations and not allow_active_work:
|
||||
raise RuntimeError(
|
||||
f"Vast lease {instance_id} has active work and cannot be destroyed."
|
||||
)
|
||||
@@ -876,8 +907,12 @@ class VastLeaseManager:
|
||||
self.registry.remove(instance_id)
|
||||
return True
|
||||
|
||||
async def reconcile(self) -> tuple[VastLeaseRecord, ...]:
|
||||
"""Refresh owned registry records and remove instances no longer visible."""
|
||||
async def refresh_owned_leases(
|
||||
self,
|
||||
*,
|
||||
clear_stale_activity: bool = False,
|
||||
) -> tuple[VastLeaseRecord, ...]:
|
||||
"""Refresh managed inventory without normally changing live activity counts."""
|
||||
visible_instances = {
|
||||
instance.instance_id: instance
|
||||
for instance in await self.api_client.list_instances()
|
||||
@@ -892,22 +927,28 @@ class VastLeaseManager:
|
||||
self.registry.remove(lease.instance_id)
|
||||
continue
|
||||
if instance.label != lease.label:
|
||||
self.registry.upsert(
|
||||
replace(
|
||||
lease,
|
||||
draining=True,
|
||||
last_error="Vast instance label no longer matches managed state.",
|
||||
)
|
||||
mismatched = replace(
|
||||
lease,
|
||||
draining=True,
|
||||
last_error="Vast instance label no longer matches managed state.",
|
||||
)
|
||||
self.registry.upsert(mismatched)
|
||||
reconciled.append(mismatched)
|
||||
continue
|
||||
recovered = replace(
|
||||
refreshed = replace(
|
||||
self._refresh_record(lease, instance),
|
||||
active_invocations=0,
|
||||
active_invocations=(
|
||||
0 if clear_stale_activity else lease.active_invocations
|
||||
),
|
||||
)
|
||||
self.registry.upsert(recovered)
|
||||
reconciled.append(recovered)
|
||||
self.registry.upsert(refreshed)
|
||||
reconciled.append(refreshed)
|
||||
return tuple(reconciled)
|
||||
|
||||
async def reconcile(self) -> tuple[VastLeaseRecord, ...]:
|
||||
"""Refresh leases after restart and clear activity owned by the dead process."""
|
||||
return await self.refresh_owned_leases(clear_stale_activity=True)
|
||||
|
||||
def incremental_retention_cost_usd(
|
||||
self,
|
||||
profile: VastResourceProfile,
|
||||
|
||||
+304
-6
@@ -9,6 +9,7 @@ const MODAL_ROUTE = "/modal/queue_prompt";
|
||||
const MODAL_ANALYZE_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/analyze_remote_nodes");
|
||||
const MODAL_PROGRESS_STATE_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/progress_state");
|
||||
const MODAL_CONTAINER_STATUS_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/container_status");
|
||||
const MODAL_CONTAINER_STOP_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/container_stop");
|
||||
const MODAL_DELETE_CACHES_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/delete_caches");
|
||||
const MODAL_DELETE_VOLUME_ROUTE = MODAL_ROUTE.replace(/\/queue_prompt$/, "/delete_volume");
|
||||
const MODAL_CANCEL_PREPARATION_ROUTE = MODAL_ROUTE.replace(
|
||||
@@ -16,6 +17,8 @@ const MODAL_CANCEL_PREPARATION_ROUTE = MODAL_ROUTE.replace(
|
||||
"/cancel_preparation",
|
||||
);
|
||||
const R2_STORAGE_USAGE_ROUTE = "/remote/storage/r2/usage";
|
||||
const VAST_STATUS_ROUTE = "/remote/vast/status";
|
||||
const VAST_DESTROY_ROUTE = "/remote/vast/destroy";
|
||||
const COMFY_QUEUE_ROUTE = "/queue";
|
||||
const COMFY_HISTORY_ROUTE = "/history";
|
||||
const INTERNAL_NODE_PREFIX = "ModalUniversalExecutor";
|
||||
@@ -3875,6 +3878,186 @@ function renderRemoteConfiguratorStorage(panel, configurations) {
|
||||
: 24 + entries.reduce((height, entry) => height + 34 + entry.details.length * 17, 0);
|
||||
}
|
||||
|
||||
/** Return safe, provider-neutral rows for managed Vast leases and Modal containers. */
|
||||
function remoteManagedCapacityEntries(vastPayload, modalPayload) {
|
||||
const vastEntries = (Array.isArray(vastPayload?.leases) ? vastPayload.leases : []).map(
|
||||
(lease) => ({
|
||||
provider: "vast",
|
||||
resourceId: String(lease?.instance_id ?? ""),
|
||||
title: String(lease?.profile_name ?? "Vast.ai lease"),
|
||||
subtitle: `Instance ${lease?.instance_id ?? "unknown"}`,
|
||||
state: String(lease?.actual_status ?? "unknown"),
|
||||
active: Math.max(0, Number(lease?.active_invocations ?? 0)),
|
||||
details: [
|
||||
`${Math.max(1, Number(lease?.gpu_count ?? 1))} × ${String(lease?.gpu_name ?? "GPU")}`,
|
||||
Number(lease?.gpu_ram_mb) > 0
|
||||
? `${remoteStorageSizeLabel(Number(lease.gpu_ram_mb) * 1024 ** 2)} VRAM`
|
||||
: null,
|
||||
Number(lease?.cpu_ram_mb) > 0
|
||||
? `${remoteStorageSizeLabel(Number(lease.cpu_ram_mb) * 1024 ** 2)} RAM`
|
||||
: null,
|
||||
Number.isFinite(Number(lease?.hourly_cost_usd))
|
||||
? `$${Number(lease.hourly_cost_usd).toFixed(3)}/hr`
|
||||
: null,
|
||||
].filter(Boolean),
|
||||
}),
|
||||
);
|
||||
const modalEntries = normalizedModalContainerStatuses(modalPayload).map((container) => ({
|
||||
provider: "modal",
|
||||
resourceId: container.containerId,
|
||||
title: container.appName || "Modal container",
|
||||
subtitle: `Container ${container.containerId}`,
|
||||
state: container.state,
|
||||
active: container.state === "running" ? 1 : 0,
|
||||
details: [
|
||||
container.modalGpu || "GPU unavailable",
|
||||
container.estimatedGpuCostPerSecond > 0
|
||||
? `$${(container.estimatedGpuCostPerSecond * 3600).toFixed(3)}/hr`
|
||||
: null,
|
||||
].filter(Boolean),
|
||||
}));
|
||||
return [...vastEntries, ...modalEntries].filter((entry) => entry.resourceId);
|
||||
}
|
||||
|
||||
/** Request JSON and convert an unsuccessful API response into a useful error. */
|
||||
async function requestRemoteCapacityJson(route, options = {}) {
|
||||
const response = await api.fetchApi(route, options);
|
||||
const payload = await response.json().catch(() => ({}));
|
||||
if (!response.ok) {
|
||||
throw new Error(payload.error || `${response.status} ${response.statusText}`);
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
|
||||
/** Kill one exact managed resource after explicit destructive confirmation. */
|
||||
async function killRemoteManagedCapacity(panel, entry, button) {
|
||||
const providerName = entry.provider === "vast" ? "Vast.ai lease" : "Modal container";
|
||||
const confirmed = globalThis.confirm?.(
|
||||
`KILL ${providerName} ${entry.resourceId}?\n\nAny workflow using it will fail. This cannot be undone.`,
|
||||
);
|
||||
if (!confirmed || panel.capacityKillInFlight) {
|
||||
return;
|
||||
}
|
||||
panel.capacityKillInFlight = `${entry.provider}:${entry.resourceId}`;
|
||||
button.disabled = true;
|
||||
button.textContent = "KILLING…";
|
||||
panel.capacityRefreshStatus.hidden = false;
|
||||
panel.capacityRefreshStatus.dataset.state = "loading";
|
||||
panel.capacityRefreshStatus.textContent = `Stopping ${entry.resourceId}`;
|
||||
panel.capacityRefreshStatus.title = "";
|
||||
try {
|
||||
const isVast = entry.provider === "vast";
|
||||
await requestRemoteCapacityJson(
|
||||
isVast ? VAST_DESTROY_ROUTE : MODAL_CONTAINER_STOP_ROUTE,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(
|
||||
isVast
|
||||
? { instance_id: Number(entry.resourceId), force: true }
|
||||
: { container_id: entry.resourceId },
|
||||
),
|
||||
},
|
||||
);
|
||||
panel.capacityRefreshStatus.dataset.state = "success";
|
||||
panel.capacityRefreshStatus.textContent = `${entry.resourceId} stopped`;
|
||||
await refreshRemoteManagedCapacity(panel, { preserveStatus: true });
|
||||
} catch (error) {
|
||||
panel.capacityRefreshStatus.dataset.state = "error";
|
||||
panel.capacityRefreshStatus.textContent = "Kill failed";
|
||||
panel.capacityRefreshStatus.title = String(error?.message ?? error);
|
||||
button.disabled = false;
|
||||
button.textContent = "KILL";
|
||||
} finally {
|
||||
panel.capacityKillInFlight = null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Render every currently managed cloud resource with an exact kill control. */
|
||||
function renderRemoteManagedCapacity(panel, entries) {
|
||||
panel.capacityList.replaceChildren();
|
||||
for (const entry of entries) {
|
||||
const card = document.createElement("div");
|
||||
card.className = "remote-configurator-capacity-card";
|
||||
const icon = document.createElement("img");
|
||||
icon.className = "remote-configurator-capacity-icon";
|
||||
icon.alt = entry.provider === "vast" ? "Vast.ai" : "Modal";
|
||||
icon.src = REMOTE_LOCATION_ICON_SOURCES[entry.provider];
|
||||
const identity = document.createElement("div");
|
||||
identity.className = "remote-configurator-capacity-identity";
|
||||
const name = document.createElement("span");
|
||||
name.className = "remote-configurator-capacity-name";
|
||||
name.textContent = entry.title;
|
||||
const subtitle = document.createElement("span");
|
||||
subtitle.className = "remote-configurator-capacity-subtitle";
|
||||
subtitle.textContent = entry.subtitle;
|
||||
identity.append(name, subtitle);
|
||||
const summary = document.createElement("span");
|
||||
summary.className = "remote-configurator-capacity-summary";
|
||||
const activity = entry.active > 0 ? ` · ${entry.active} active` : " · idle";
|
||||
summary.textContent = `${entry.state}${activity} · ${entry.details.join(" · ")}`;
|
||||
const button = document.createElement("button");
|
||||
button.className = "remote-configurator-capacity-kill";
|
||||
button.type = "button";
|
||||
button.textContent = "KILL";
|
||||
button.title = `Permanently stop ${entry.subtitle}`;
|
||||
button.addEventListener("click", () => {
|
||||
void killRemoteManagedCapacity(panel, entry, button);
|
||||
});
|
||||
card.append(icon, identity, summary, button);
|
||||
panel.capacityList.appendChild(card);
|
||||
}
|
||||
panel.capacityEmpty.hidden = entries.length > 0;
|
||||
panel.capacityHeight = 29 + Math.max(20, entries.length * 49);
|
||||
}
|
||||
|
||||
/** Reload all managed Vast leases and Modal containers shown by one panel. */
|
||||
async function refreshRemoteManagedCapacity(panel, { preserveStatus = false } = {}) {
|
||||
if (
|
||||
!panel ||
|
||||
panel.capacityRefreshInFlight ||
|
||||
typeof api.fetchApi !== "function"
|
||||
) {
|
||||
return;
|
||||
}
|
||||
panel.capacityRefreshInFlight = true;
|
||||
panel.capacityReload.disabled = true;
|
||||
panel.capacityReload.textContent = "Reloading…";
|
||||
if (!preserveStatus) {
|
||||
panel.capacityRefreshStatus.hidden = false;
|
||||
panel.capacityRefreshStatus.dataset.state = "loading";
|
||||
panel.capacityRefreshStatus.textContent = "Refreshing capacity";
|
||||
panel.capacityRefreshStatus.title = "";
|
||||
}
|
||||
const results = await Promise.allSettled([
|
||||
requestRemoteCapacityJson(VAST_STATUS_ROUTE),
|
||||
requestRemoteCapacityJson(`${MODAL_CONTAINER_STATUS_ROUTE}?include_billing=false`),
|
||||
]);
|
||||
const errors = results
|
||||
.filter((result) => result.status === "rejected")
|
||||
.map((result) => String(result.reason?.message ?? result.reason));
|
||||
const entries = remoteManagedCapacityEntries(
|
||||
results[0].status === "fulfilled" ? results[0].value : {},
|
||||
results[1].status === "fulfilled" ? results[1].value : {},
|
||||
);
|
||||
renderRemoteManagedCapacity(panel, entries);
|
||||
if (errors.length > 0) {
|
||||
panel.capacityRefreshStatus.hidden = false;
|
||||
panel.capacityRefreshStatus.dataset.state = "error";
|
||||
panel.capacityRefreshStatus.textContent = entries.length > 0 ? "Partially refreshed" : "Refresh failed";
|
||||
panel.capacityRefreshStatus.title = errors.join("\n");
|
||||
} else if (!preserveStatus) {
|
||||
panel.capacityRefreshStatus.hidden = false;
|
||||
panel.capacityRefreshStatus.dataset.state = "success";
|
||||
panel.capacityRefreshStatus.textContent = "Updated";
|
||||
panel.capacityRefreshStatus.title = "";
|
||||
}
|
||||
panel.capacityRefreshInFlight = false;
|
||||
panel.capacityReload.disabled = false;
|
||||
panel.capacityReload.textContent = "Reload";
|
||||
renderRemoteConfiguratorPlan(panel, panel.assignments, panel.configurations);
|
||||
}
|
||||
|
||||
/** Return compact GPU-memory and RAM capacity labels for one target. */
|
||||
function remoteConfiguratorHardwareLabels(hardware) {
|
||||
const gpuCount = Math.max(0, Number(hardware?.gpu_count ?? 0));
|
||||
@@ -4060,7 +4243,8 @@ function renderRemoteConfiguratorPlan(panel, assignments, configurations) {
|
||||
118 +
|
||||
Math.max(31, planHeight) +
|
||||
panel.environmentRows.size * 52 +
|
||||
panel.storageHeight;
|
||||
panel.storageHeight +
|
||||
panel.capacityHeight;
|
||||
const currentWidth = Number(panel.node.size?.[0]) || 0;
|
||||
const currentHeight = Number(panel.node.size?.[1]) || 0;
|
||||
panel.node.setSize?.([
|
||||
@@ -4314,6 +4498,17 @@ function mountRemoteExecutionConfiguratorPanel(node) {
|
||||
<div class="remote-configurator-plan-title">Execution plan</div>
|
||||
<div class="remote-configurator-empty">The selected environments will appear here after planning.</div>
|
||||
<div class="remote-configurator-targets" hidden></div>
|
||||
<div class="remote-configurator-capacity">
|
||||
<div class="remote-configurator-capacity-header">
|
||||
<div class="remote-configurator-capacity-title">Managed capacity</div>
|
||||
<div class="remote-configurator-capacity-refresh">
|
||||
<span class="remote-configurator-capacity-refresh-status" hidden></span>
|
||||
<button class="remote-configurator-capacity-reload" type="button">Reload</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="remote-configurator-capacity-empty">No managed Vast.ai leases or Modal containers.</div>
|
||||
<div class="remote-configurator-capacity-list"></div>
|
||||
</div>
|
||||
<div class="remote-configurator-storage" hidden>
|
||||
<div class="remote-configurator-storage-header">
|
||||
<div class="remote-configurator-storage-title">Storage backends</div>
|
||||
@@ -4328,9 +4523,12 @@ function mountRemoteExecutionConfiguratorPanel(node) {
|
||||
node,
|
||||
root,
|
||||
promptId: null,
|
||||
minHeight: 149,
|
||||
minHeight: 198,
|
||||
storageHeight: 0,
|
||||
capacityHeight: 49,
|
||||
storageRefreshInFlight: false,
|
||||
capacityRefreshInFlight: false,
|
||||
capacityKillInFlight: null,
|
||||
assignments: {},
|
||||
configurations: [],
|
||||
statusText: root.querySelector(".remote-configurator-status-text"),
|
||||
@@ -4341,6 +4539,13 @@ function mountRemoteExecutionConfiguratorPanel(node) {
|
||||
environmentRows: new Map(),
|
||||
emptyText: root.querySelector(".remote-configurator-empty"),
|
||||
targets: root.querySelector(".remote-configurator-targets"),
|
||||
capacity: root.querySelector(".remote-configurator-capacity"),
|
||||
capacityList: root.querySelector(".remote-configurator-capacity-list"),
|
||||
capacityEmpty: root.querySelector(".remote-configurator-capacity-empty"),
|
||||
capacityReload: root.querySelector(".remote-configurator-capacity-reload"),
|
||||
capacityRefreshStatus: root.querySelector(
|
||||
".remote-configurator-capacity-refresh-status",
|
||||
),
|
||||
storage: root.querySelector(".remote-configurator-storage"),
|
||||
storageList: root.querySelector(".remote-configurator-storage-list"),
|
||||
storageReload: root.querySelector(".remote-configurator-storage-reload"),
|
||||
@@ -4351,6 +4556,9 @@ function mountRemoteExecutionConfiguratorPanel(node) {
|
||||
panel.storageReload.addEventListener("click", () => {
|
||||
void refreshRemoteConfiguratorStorage(panel);
|
||||
});
|
||||
panel.capacityReload.addEventListener("click", () => {
|
||||
void refreshRemoteManagedCapacity(panel);
|
||||
});
|
||||
let widget;
|
||||
try {
|
||||
widget = node.addDOMWidget(
|
||||
@@ -4378,6 +4586,7 @@ function mountRemoteExecutionConfiguratorPanel(node) {
|
||||
remoteConfiguratorPanels.set(safeNodeId, panel);
|
||||
node.__remoteConfiguratorPanelMounted = true;
|
||||
restoreRemoteConfiguratorPanel(panel);
|
||||
void refreshRemoteManagedCapacity(panel);
|
||||
const originalOnRemoved = node.onRemoved;
|
||||
node.onRemoved = function onRemoved() {
|
||||
remoteConfiguratorPanels.delete(nodeId(this));
|
||||
@@ -7278,6 +7487,7 @@ function installGlobalStatusStyles() {
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity,
|
||||
.remote-configurator-storage {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
@@ -7286,10 +7496,7 @@ function installGlobalStatusStyles() {
|
||||
padding-top: 2px;
|
||||
}
|
||||
|
||||
.remote-configurator-storage[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-title,
|
||||
.remote-configurator-storage-title {
|
||||
color: #94a3b8;
|
||||
font-size: 9px;
|
||||
@@ -7298,6 +7505,8 @@ function installGlobalStatusStyles() {
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-header,
|
||||
.remote-configurator-capacity-refresh,
|
||||
.remote-configurator-storage-header,
|
||||
.remote-configurator-storage-refresh {
|
||||
display: flex;
|
||||
@@ -7306,10 +7515,12 @@ function installGlobalStatusStyles() {
|
||||
gap: 7px;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-header,
|
||||
.remote-configurator-storage-header {
|
||||
justify-content: space-between;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-refresh-status,
|
||||
.remote-configurator-storage-refresh-status {
|
||||
overflow: hidden;
|
||||
color: #64748b;
|
||||
@@ -7318,14 +7529,17 @@ function installGlobalStatusStyles() {
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-refresh-status[data-state="error"],
|
||||
.remote-configurator-storage-refresh-status[data-state="error"] {
|
||||
color: #f87171;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-refresh-status[data-state="success"],
|
||||
.remote-configurator-storage-refresh-status[data-state="success"] {
|
||||
color: #4ade80;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-reload,
|
||||
.remote-configurator-storage-reload {
|
||||
padding: 2px 7px;
|
||||
border: 1px solid rgba(56, 189, 248, 0.28);
|
||||
@@ -7336,16 +7550,100 @@ function installGlobalStatusStyles() {
|
||||
font: 600 9px/1.35 ui-sans-serif, system-ui, sans-serif;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-reload:hover:not(:disabled),
|
||||
.remote-configurator-storage-reload:hover:not(:disabled) {
|
||||
border-color: rgba(56, 189, 248, 0.55);
|
||||
background: rgba(14, 116, 144, 0.24);
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-reload:disabled,
|
||||
.remote-configurator-storage-reload:disabled {
|
||||
cursor: wait;
|
||||
opacity: 0.65;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-empty {
|
||||
color: #64748b;
|
||||
font-size: 9px;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-list {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
flex-direction: column;
|
||||
gap: 5px;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-card {
|
||||
display: grid;
|
||||
min-width: 0;
|
||||
grid-template-columns: 20px minmax(100px, 0.8fr) minmax(140px, 1.2fr) max-content;
|
||||
align-items: center;
|
||||
gap: 7px;
|
||||
padding: 6px 7px;
|
||||
border: 1px solid rgba(148, 163, 184, 0.2);
|
||||
border-radius: 7px;
|
||||
background: rgba(15, 23, 42, 0.54);
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-icon {
|
||||
width: 18px;
|
||||
height: 18px;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-identity {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
flex-direction: column;
|
||||
gap: 1px;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-name,
|
||||
.remote-configurator-capacity-subtitle,
|
||||
.remote-configurator-capacity-summary {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-name {
|
||||
color: #e2e8f0;
|
||||
font-size: 10px;
|
||||
font-weight: 650;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-subtitle,
|
||||
.remote-configurator-capacity-summary {
|
||||
color: #94a3b8;
|
||||
font: 8.5px/1.3 ui-monospace, SFMono-Regular, monospace;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-kill {
|
||||
padding: 3px 7px;
|
||||
border: 1px solid rgba(248, 113, 113, 0.55);
|
||||
border-radius: 5px;
|
||||
background: rgba(127, 29, 29, 0.26);
|
||||
color: #fca5a5;
|
||||
cursor: pointer;
|
||||
font: 750 9px/1.3 ui-monospace, SFMono-Regular, monospace;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-kill:hover:not(:disabled) {
|
||||
border-color: #ef4444;
|
||||
background: rgba(185, 28, 28, 0.42);
|
||||
color: #fee2e2;
|
||||
}
|
||||
|
||||
.remote-configurator-capacity-kill:disabled {
|
||||
cursor: wait;
|
||||
opacity: 0.65;
|
||||
}
|
||||
|
||||
.remote-configurator-storage[hidden] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.remote-configurator-storage-list {
|
||||
display: flex;
|
||||
min-width: 0;
|
||||
|
||||
Reference in New Issue
Block a user