Replace provisional Vast status lanes
This commit is contained in:
@@ -246,7 +246,7 @@ Active remote nodes show a small progress panel with a Modal, SSH terminal, or V
|
||||
|
||||
Remote planning and capacity acquisition count as active queue work even though they happen before the rewritten prompt enters ComfyUI's native execution queue. During Vast startup, the canvas receives readable updates for rental, worker-image download, SSH readiness, and runtime initialization. A provider-side terminal image-pull failure is reported immediately rather than appearing as an empty queue until the startup timeout expires. A rental that never produces an SSH-usable worker is marked unusable and destroyed after the bounded initialization wait so a later workflow cannot silently reuse it.
|
||||
|
||||
For workflows using a **Remote Execution Configurator**, that node is also the primary planning surface. As soon as scheduling finishes, the scheduler result is streamed to the panel before any Vast rental or SSH-readiness wait begins. The browser retains that plan across late node-panel mounts, a full page refresh, tab refocus, queue failure, and extension reloads, while removing any orphaned prior panel before remounting. The panel follows the node's current width as it is resized, including shrinking within narrow layouts instead of retaining its initial DOM-widget width. The execution plan is grouped into one section per concrete target, with labeled components beneath it and the best-known machine type, GPU/HBM or VRAM capacity, system RAM, predicted completion time, and predicted cost. Modal system RAM is explicitly shown as provider-managed because it is assigned dynamically rather than inferred as a fixed machine capacity. Configuration names and selected targets therefore remain inspectable even when capacity never becomes ready. The **Managed capacity** area independently lists every lease or container owned by this installation, including idle leftovers from earlier workflows, with provider, resource ID, lifecycle state, activity, GPU/RAM, and hourly price where available. Its **Reload** button refreshes the list, while each exact Vast lease or Modal container has a destructive **KILL** button with confirmation; ownership is revalidated by the server, and killing active capacity is explicitly allowed because the confirmation warns that its workflow will fail. These controls opt into pointer handling inside ComfyUI's otherwise click-through canvas overlay, provide immediate pressed/loading feedback, and isolate their pointer gestures from node dragging. Connected storage backends are summarized beneath the execution plan rather than presented as scheduler targets; an R2 summary includes its bucket, current storage use and object count, jurisdiction, cache policy, and optional key prefix without exposing credentials. Bucket usage is read through the existing bucket-scoped S3 credential and cached for five minutes so repeated workflow submissions do not repeatedly enumerate the bucket. The storage area's **Reload** button bypasses that cache, retrieves fresh aggregate state, and updates the retained plan without queueing a workflow. Each selected environment has its own status and progress lane, so parallel Vast, Modal, and self-hosted setup or execution updates cannot overwrite each other; once a newer prompt owns the panel, delayed events from an older prompt remain isolated. Planned environments begin in a waiting state. Vast capacity then reports rental, worker-image download, initialization, and asset preparation before becoming ready. SSH rows distinguish completed asset synchronization from worker readiness and stream the later runtime check, image build, container start, and health wait; they report ready only after the worker socket answers with the expected fingerprint. A bounded controller pool prepares custom nodes and prompt assets concurrently across distinct environments, while preserving component order and request-cache reuse within each environment. A prompt-wide failure persistently terminates every environment lane and clears its in-progress meter, including when a later plan render or panel remount occurs. Live preparation and execution status use the same durable event stream. Queue-history reconciliation reads the prompt's actual terminal result, preserving a failure or interruption in the Configurator and replacing stale submission text when the prompt is no longer executing. Legacy workflows without a configurator continue to use the global status pill.
|
||||
For workflows using a **Remote Execution Configurator**, that node is also the primary planning surface. As soon as scheduling finishes, the scheduler result is streamed to the panel before any Vast rental or SSH-readiness wait begins. The browser retains that plan across late node-panel mounts, a full page refresh, tab refocus, queue failure, and extension reloads, while removing any orphaned prior panel before remounting. The panel follows the node's current width as it is resized, including shrinking within narrow layouts instead of retaining its initial DOM-widget width. The execution plan is grouped into one section per concrete target, with labeled components beneath it and the best-known machine type, GPU/HBM or VRAM capacity, system RAM, predicted completion time, and predicted cost. Modal system RAM is explicitly shown as provider-managed because it is assigned dynamically rather than inferred as a fixed machine capacity. Configuration names and selected targets therefore remain inspectable even when capacity never becomes ready. The **Managed capacity** area independently lists every lease or container owned by this installation, including idle leftovers from earlier workflows, with provider, resource ID, lifecycle state, activity, GPU/RAM, and hourly price where available. Its **Reload** button refreshes the list, while each exact Vast lease or Modal container has a destructive **KILL** button with confirmation; ownership is revalidated by the server, and killing active capacity is explicitly allowed because the confirmation warns that its workflow will fail. These controls opt into pointer handling inside ComfyUI's otherwise click-through canvas overlay, provide immediate pressed/loading feedback, and isolate their pointer gestures from node dragging. Connected storage backends are summarized beneath the execution plan rather than presented as scheduler targets; an R2 summary includes its bucket, current storage use and object count, jurisdiction, cache policy, and optional key prefix without exposing credentials. Bucket usage is read through the existing bucket-scoped S3 credential and cached for five minutes so repeated workflow submissions do not repeatedly enumerate the bucket. The storage area's **Reload** button bypasses that cache, retrieves fresh aggregate state, and updates the retained plan without queueing a workflow. Each selected environment has its own status and progress lane, so parallel Vast, Modal, and self-hosted setup or execution updates cannot overwrite each other; once a newer prompt owns the panel, delayed events from an older prompt remain isolated. Planned environments begin in a waiting state. When a planned Vast configuration resolves to a concrete lease, its existing lane is promoted in place so the configuration and instance identities do not briefly appear as duplicate capacity. Vast capacity then reports rental, worker-image download, initialization, and asset preparation before becoming ready. SSH rows distinguish completed asset synchronization from worker readiness and stream the later runtime check, image build, container start, and health wait; they report ready only after the worker socket answers with the expected fingerprint. A bounded controller pool prepares custom nodes and prompt assets concurrently across distinct environments, while preserving component order and request-cache reuse within each environment. A prompt-wide failure persistently terminates every environment lane and clears its in-progress meter, including when a later plan render or panel remount occurs. Live preparation and execution status use the same durable event stream. Queue-history reconciliation reads the prompt's actual terminal result, preserving a failure or interruption in the Configurator and replacing stale submission text when the prompt is no longer executing. Legacy workflows without a configurator continue to use the global status pill.
|
||||
|
||||
Queue-time preparation appears in ComfyUI's queue through the server-side preparation bridge, including the native job metadata required by the queue panel, while the Configurator panel and global status pill show its detailed phase. The frontend acknowledges the prompt immediately so ComfyUI owns a visible, cancellable job while remote capacity is still starting; ComfyUI's standard Jobs cancellation also signals the prompt's preparation task. Preparation deliberately does not emit fake native node-execution events: doing so starts ComfyUI's execution watchdog, which can incorrectly mark a node as stalled or failed during a healthy multi-minute asset download. A preparation failure remains a queue/setup error with its original message rather than being recast as a node execution crash.
|
||||
|
||||
|
||||
@@ -606,3 +606,4 @@
|
||||
- Workflow plans are not a complete cloud-resource inventory: cancellation or a controller restart can leave valid idle leases that no longer appear in the current plan. Put provider-owned live capacity in a separately refreshable Configurator section, target destructive controls by immutable provider ID, and repeat the ownership check on the server. A deliberate kill path must be able to override an active-work guard only after an explicit warning; routine reaping and other callers should retain the guard.
|
||||
- Modal exposes active execution containers as task IDs, while its app-level stop operation can terminate unrelated warm capacity. Verify the requested task against the installation's managed app names, then call the exact SDK `ContainerStop` operation on Modal's synchronized event loop instead of stopping the whole deployed app.
|
||||
- A click-through DOM widget cannot become interactive merely by attaching button listeners. When the panel root uses `pointer-events: none` so the canvas remains draggable, opt each intended control back in with `pointer-events: auto`, stop its pointer and wheel events at the panel boundary, and provide pressed/loading state so canvas interaction cannot swallow a destructive or refresh action invisibly.
|
||||
- Vast setup changes identity when a planned configuration such as `vast:380` becomes a concrete lease such as `vast:380:49052528`. Because readiness telemetry can arrive before the post-acquisition plan, promote the provisional environment lane as soon as that concrete ID appears and transfer its retained state; treating the concrete ID as unrelated briefly renders the same capacity twice.
|
||||
|
||||
@@ -1649,7 +1649,7 @@ modalToggle.registerRemoteConfiguratorPlan(
|
||||
{
|
||||
"14": {
|
||||
provider: "vast",
|
||||
environment_id: "vast:profile:slot-0",
|
||||
environment_id: "vast:vast-big",
|
||||
configuration_id: "vast-big",
|
||||
node_ids: ["14", "15"],
|
||||
predicted_cost_usd: 0.031,
|
||||
@@ -1882,7 +1882,17 @@ modalToggle.handleModalStatus({
|
||||
prompt_id: "plan-before-capacity",
|
||||
node_ids: ["14", "15"],
|
||||
configurator_node_id: "381",
|
||||
execution_environment_id: "vast:profile:slot-0",
|
||||
execution_environment_id: "vast:vast-big",
|
||||
status_message: "Vast.ai worker is ready",
|
||||
},
|
||||
});
|
||||
modalToggle.handleModalStatus({
|
||||
detail: {
|
||||
phase: "setup",
|
||||
prompt_id: "plan-before-capacity",
|
||||
node_ids: ["14", "15"],
|
||||
configurator_node_id: "381",
|
||||
execution_environment_id: "vast:vast-big:49052528",
|
||||
status_message: "Uploading Vast assets",
|
||||
status_current: 3,
|
||||
status_total: 10,
|
||||
@@ -1900,8 +1910,24 @@ modalToggle.handleModalStatus({
|
||||
status_total: 4,
|
||||
},
|
||||
});
|
||||
const vastEnvironmentRow = retainedPanel.environmentRows.get("vast:profile:slot-0");
|
||||
const vastEnvironmentRow = retainedPanel.environmentRows.get("vast:vast-big:49052528");
|
||||
const modalEnvironmentRow = retainedPanel.environmentRows.get("modal:lambda");
|
||||
assert.equal(retainedPanel.environmentRows.size, 2);
|
||||
assert.equal(retainedPanel.environmentRows.has("vast:vast-big"), false);
|
||||
assert.equal(
|
||||
vastEnvironmentRow.label.textContent,
|
||||
"Vast Big · Vast.ai vast:vast-big:49052528",
|
||||
);
|
||||
assert.equal(
|
||||
retainedPanel.assignments["14"].environment_id,
|
||||
"vast:vast-big:49052528",
|
||||
);
|
||||
assert.equal(
|
||||
modalToggle.modalPromptStates
|
||||
.get("plan-before-capacity")
|
||||
.remoteEnvironmentStatuses.has("vast:vast-big"),
|
||||
false,
|
||||
);
|
||||
assert.equal(vastEnvironmentRow.statusText.textContent, "Uploading Vast assets");
|
||||
assert.equal(vastEnvironmentRow.progressValue.textContent, "3/10");
|
||||
assert.equal(vastEnvironmentRow.progressFill.style.width, "30%");
|
||||
@@ -1920,7 +1946,7 @@ modalToggle.handleModalStatus({
|
||||
});
|
||||
const failedPromptState = modalToggle.modalPromptStates.get("plan-before-capacity");
|
||||
assert.equal(
|
||||
failedPromptState.remoteEnvironmentStatuses.get("vast:profile:slot-0")?.phase,
|
||||
failedPromptState.remoteEnvironmentStatuses.get("vast:vast-big:49052528")?.phase,
|
||||
"error",
|
||||
);
|
||||
assert.equal(
|
||||
@@ -1935,7 +1961,7 @@ modalToggle.registerRemoteConfiguratorPlan(
|
||||
failedPromptState.remoteExecutionConfigurations,
|
||||
);
|
||||
assert.equal(
|
||||
retainedPanel.environmentRows.get("vast:profile:slot-0").root.dataset.phase,
|
||||
retainedPanel.environmentRows.get("vast:vast-big:49052528").root.dataset.phase,
|
||||
"error",
|
||||
);
|
||||
assert.equal(
|
||||
|
||||
@@ -92,6 +92,7 @@ def test_configurator_renders_out_of_band_plan_and_status() -> None:
|
||||
assert "function registerRemoteConfiguratorPlan(promptId, assignments, configurations)" in source
|
||||
assert "function renderRemoteConfiguratorStatus(panel, state)" in source
|
||||
assert "function renderRemoteConfiguratorEnvironments(panel, assignments, configurations)" in source
|
||||
assert "function promoteProvisionalVastEnvironment(" in source
|
||||
assert "function remoteConfiguratorStorageEntries(configurations)" in source
|
||||
assert "function renderRemoteConfiguratorStorage(panel, configurations)" in source
|
||||
assert "function updateRemoteConfiguratorEnvironmentStatus(promptId, detail)" in source
|
||||
|
||||
+50
-1
@@ -4320,6 +4320,52 @@ function registerRemoteConfiguratorPlan(promptId, assignments, configurations) {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace one provisional Vast configuration lane with its concrete lease lane.
|
||||
* @param {Record<string, any>} panel
|
||||
* @param {Record<string, any>} promptState
|
||||
* @param {string} concreteEnvironmentId
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function promoteProvisionalVastEnvironment(
|
||||
panel,
|
||||
promptState,
|
||||
concreteEnvironmentId,
|
||||
) {
|
||||
const provisionalEnvironmentIds = Object.values(panel?.assignments ?? {})
|
||||
.filter((assignment) => assignment?.provider === "vast")
|
||||
.map((assignment) => `vast:${String(assignment?.configuration_id ?? "")}`)
|
||||
.filter(
|
||||
(environmentId) =>
|
||||
panel.environmentRows.has(environmentId) &&
|
||||
/^\d+$/.test(concreteEnvironmentId.slice(environmentId.length + 1)) &&
|
||||
concreteEnvironmentId.startsWith(`${environmentId}:`),
|
||||
)
|
||||
.sort((left, right) => right.length - left.length);
|
||||
const provisionalEnvironmentId = provisionalEnvironmentIds[0];
|
||||
if (!provisionalEnvironmentId) {
|
||||
return false;
|
||||
}
|
||||
const assignments = Object.fromEntries(
|
||||
Object.entries(promptState.remoteExecutionPlanAssignments).map(([componentId, assignment]) => [
|
||||
componentId,
|
||||
assignment?.environment_id === provisionalEnvironmentId
|
||||
? { ...assignment, environment_id: concreteEnvironmentId }
|
||||
: assignment,
|
||||
]),
|
||||
);
|
||||
const provisionalState = promptState.remoteEnvironmentStatuses.get(
|
||||
provisionalEnvironmentId,
|
||||
);
|
||||
promptState.remoteEnvironmentStatuses.delete(provisionalEnvironmentId);
|
||||
if (provisionalState && !promptState.remoteEnvironmentStatuses.has(concreteEnvironmentId)) {
|
||||
promptState.remoteEnvironmentStatuses.set(concreteEnvironmentId, provisionalState);
|
||||
}
|
||||
promptState.remoteExecutionPlanAssignments = assignments;
|
||||
renderRemoteConfiguratorPlan(panel, assignments, promptState.remoteExecutionConfigurations);
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Store and render one environment's latest status independently of its peers.
|
||||
* @param {string} promptId
|
||||
@@ -4331,6 +4377,10 @@ function updateRemoteConfiguratorEnvironmentStatus(promptId, detail) {
|
||||
if (!environmentId || !promptState) {
|
||||
return;
|
||||
}
|
||||
const panel = remoteConfiguratorPanel(promptState.configuratorNodeId);
|
||||
if (panel?.promptId === promptId && !panel.environmentRows.has(environmentId)) {
|
||||
promoteProvisionalVastEnvironment(panel, promptState, environmentId);
|
||||
}
|
||||
const existing = promptState.remoteEnvironmentStatuses.get(environmentId) ?? {};
|
||||
const state = {
|
||||
phase: String(detail?.phase ?? existing.phase ?? STATE_SETUP),
|
||||
@@ -4342,7 +4392,6 @@ function updateRemoteConfiguratorEnvironmentStatus(promptId, detail) {
|
||||
updatedAt: nowMs(),
|
||||
};
|
||||
promptState.remoteEnvironmentStatuses.set(environmentId, state);
|
||||
const panel = remoteConfiguratorPanel(promptState.configuratorNodeId);
|
||||
if (!panel || panel.promptId !== promptId) {
|
||||
return;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user