An encoding clusterfrom the PCs you already own
NyXGridEncode can spread jobs across several Windows PCs on the same LAN. There is no dedicated server or web console to set up. Install the software on each PC and select an operating mode to build the cluster.
No dedicated infrastructure required
In distributed encoding, one PC acts as the controller and holds the queue, while the agent PCs share the conversion work. Controller and agents alike are ordinary Windows PCs running the same software. Each PC taking part in distributed encoding is called a node. The controller and agents communicate within the same LAN using a shared auth token; the traffic is not encrypted.
What you need
- The software on each PCNo separate agent program
- External encoding toolsRequired on every PC that encodes, and on the controller
- A connection to the same LANController address configured on each agent
- A shared auth token16 to 64 characters, using ASCII letters, digits and symbols
- A license key on the controllerAn edition that includes distributed encoding (Pro or ProPlus). The number of agents depends on the edition
Even when the PCs differ in performance, jobs are assigned based on the processing capacity (performance ratio) set on each agent, so that the whole queue finishes as early as possible.
If a node stops responding for a set period, its jobs return to the queue and are processed by the other nodes.
Files added by folder watch are assigned to agents just like any other job in the queue.
Distributed encoding is available with a Pro or ProPlus license.Activation
With a trial license, you can use Pro features, including distributed encoding, free of charge for 7 days. No payment details are required.
Four operating modes
Each PC's role is chosen in Settings > Mode. A single PC cannot act as a controller and an agent at the same time, and the mode cannot be changed while encoding is in progress.
Scroll horizontally to see the rest of the table
| Mode | Role | Encodes | Accepts agents | Required external tools | |
|---|---|---|---|---|---|
| Encoding | Media info | ||||
| Standalone | Encodes on this PC only | Required | Required | ||
| Controller + Encoder | Assigns jobs and encodes | Required | Required | ||
| Controller only | Assigns jobs only | Not required | Required * | ||
| Agent | Encodes assigned jobs | Required | Not required | ||
Encoding uses either a GPU tool (NVEncC, QSVEncC or VCEEncC) or FFmpeg; media information is always read with FFmpeg. * If a preset uses automatic black bar detection, the controller's FFmpeg is used for that as well.
Even in "Controller only" mode, the controller reads each input file's media information with FFmpeg when preparing the jobs it assigns.
Agents cannot add to the queue, start encoding, choose presets, use Adjust, set the output folder, use folder watch or send completion notifications. These controls are grayed out, with the reason shown.
Received jobs appear in the queue list with progress, FPS, elapsed time, output size and status. The list is display-only and is not saved. When it is empty, the agent shows "Waiting for the controller to assign jobs".
Job assignment that accounts for performance differences
Each agent requests its next job from the controller as soon as it has capacity. On every request, the controller recalculates the assignment so that all waiting jobs finish as early as possible.
Example: 9 jobs assigned to 3 PCs with performance ratios of 100, 50 and 40, simplified to one parallel job per PC.
When a node has capacity but no job has been assigned to it, the smallest job is moved to that node. The move is limited to the smallest job because a slow node that takes a large job delays the whole queue.
Near the end of the queue, a job can finish sooner by waiting for a fast node to free up than by going to a slow node. In that case, the assignment is held.
Conditions for releasing a hold- The node the hold relies on has not responded for a set period
- The expected wait has been exceeded by a set margin
- A node disconnects or stops
- The situation changes and the reason for the hold no longer applies
The performance ratio is configured on each PC that runs as an agent. The agent sends this value automatically with every job request, and the controller uses the value it receives without adjustment.
The performance ratio describes how fast a single job runs, while the number of parallel jobs is how many run at once. The two settings are independent. An agent's parallel job count and GPU selection are taken from its own Settings > Encoding page.
"Efficiency first" (the default) assigns the largest jobs first, each to the node that can finish it soonest. "List order" assigns jobs in queue order. Standalone mode always uses "List order".
The assignment plan is never stored. It is recalculated in full every time a job is assigned, so disconnections, reconnections, new nodes and changes to parallel jobs take effect from the next assignment, and no update can be missed.
* Assignment uses an approximation that takes the longest jobs first and picks the node that finishes soonest. It does not guarantee a mathematically optimal schedule.
* Jobs are assigned per file. A single video cannot be split across multiple PCs.
Preparing the controller and adding agents
First prepare the PC that acts as the controller, then set up each PC that joins as an agent. There is no separate agent program or screen: both run the same software.
Preparing the controller
- Install the software, start it and register a Pro or ProPlus license key.
- In Settings > Mode, select "Controller + Encoder" or "Controller only".
- Set the port and the auth token. Select "Generate" to create a hard-to-guess token automatically.
- Shared paths are optional. If you add locations that agents can read and write directly as shared paths, files in them are not transferred. For a location the controller uses through a drive letter, also set "Location on This PC". For a UNC shared path, you can also set the user name and password used to access it.
- Note down this PC's host name or IP address, the port and the auth token. You will enter them on each agent.
- Set the output folder in the main window, or an output folder for a watched folder. The controller does not accept agents until an output folder is set. Jobs are assigned only while encoding is running on the controller.
Steps on each agent
Do the following on each PC that joins.
-
1
Installing the software
Run the installer, or extract the ZIP file into a folder. On first launch, NyXGridEncode guides you through downloading the external tools it uses for encoding.
-
2
Launching the software
Keep NyXGridEncode running while the PC works as an agent, or register it as a Windows service. There is no separate background program. By default, the PC is kept awake while it encodes.
-
3
Setting the operating mode
In Settings > Mode, select "Agent", enter the controller's host name or IP address, port and auth token that you noted down, then save. The PC then joins the controller. The display name is optional. "Test Connection" checks whether this PC can read and write the controller's paths directly.
For the second and later agents, use "Export" at the bottom left of Settings and then "Import" on the next PC to copy the Mode settings, including the address, port and auth token. The exported file contains the auth token and the shared path passwords, so do not put it where others can read it.
Direct mode or transfer mode, chosen automatically
Input files reach the agents by one of two routes, chosen automatically for each job. The controller requires a transfer when the input file or the output folder is outside the shared paths, or when a "Location on This PC" cannot be verified. Otherwise, the agent checks whether it can read the input file it was given and write to the output folder, and uses the transfer route if it cannot. If the agent can only read a share, the transfer route is used. No setting is needed to choose the route.
When you set shared paths, files in them that an agent can read and write directly are not transferred. The output folder and watched folders are added to the shared paths automatically. For a location the controller uses through a drive letter, such as E:\Video, set it as "Location on This PC" for the shared path, and agents receive the shared path instead. A drive-letter path that is not replaced is opened as the same path on the agent PC. Files outside the shared paths, and files under a shared path that cannot be verified to point to the same location as its "Location on This PC", are transferred.
Even while jobs encode in parallel, files are transferred one at a time, since concurrent transfers saturate the link and slow everything down. On a 1 Gbps LAN, transferring 10 GB takes about 100 seconds. The "File Transfer Benchmark" in Settings > Benchmark measures the actual transfer speed to connected agents.
Transferred files go to a temporary folder on the agent and are deleted afterwards, whether or not the job succeeded. The agent needs free space for two files: the input and the output.
Jobs that went through a transfer show "node name - Transfer" in the queue's Node column, so you can see later why a job took longer. Before encoding, "Test Connection" in the agent's settings shows for each path whether the agent reads and writes it directly, and why.
Output files received by the controller are written under a temporary name ending in .part. If the declared length does not match the number of bytes actually written, the file is deleted and only that job is retried. A received output gets its final name only when the controller accepts the agent's result. Input files received by an agent are also deleted and not used if their length does not match.
Transfer requests carry only a job number and a type, never a file location. The controller opens only the location it gave to that job. Requests for unknown job numbers are refused, so a request cannot be used to read arbitrary files.
Recovering jobs and preventing duplicate work
If an agent stops responding for a set period, the controller treats it as disconnected and returns its running jobs to the queue. Progress updates from agents arrive at short intervals, and the set period includes an ample margin so that a brief network delay does not trigger a recovery.
Three safeguards against duplicate work
Progress and results received from an agent are checked against the node the job is assigned to. If that agent no longer owns the job, it is told to stop.
A result that arrives from an agent after recovery is accepted only if the job is still waiting and the conversion succeeded. It is discarded if another node is already processing the job, if the job has already completed or failed, or if the conversion failed.
Output written to a shared path gets a .part suffix and receives its final name only after the result arrives from the agent.
Current limitation
Until communication is restored, an agent cannot tell that its job has been recovered. If the recovered job is assigned to another node, the same file may be processed on two PCs for a while. Once communication is restored, the reply to the agent's next progress update instructs it to stop. A successful result that arrives while the job is still waiting after recovery is accepted, so work is wasted only when the disconnection is prolonged.
For a setup that prepares for a controller PC failure, see Using the second PC.
Choosing a target and canceling jobs
The controller normally decides where each job runs, but you can also choose the node for a specific job. Jobs already assigned to agents can be canceled from the controller.
Choosing a target
- Right-click a job in the queue and select a target from the "Encode" submenu. The choices are "Auto assign" and each connected agent, plus "Local" in "Controller + Encoder" mode.
- A chosen target is treated as an explicit instruction. The job is never moved to another node automatically because of a slow response or because an estimate was exceeded.
- If the chosen node stops participating, for example because it disconnects, a grace period applies. If the node has not rejoined by the end of that period, the choice is cleared and a warning appears in the log and the status bar. The job is never moved to another node without notice.
- If the node is seen joining at any point during the grace period, the countdown starts over.
The wait is not indefinite because, in unattended operation, waiting for a node that never returns would waste all the processing time spent waiting.
Canceling
- Agents request jobs from the controller, so the controller has no way to contact them directly. Instead, each time an agent sends its progress, the controller's reply tells it whether to continue.
- Jobs that are encoding stop within moments, since the check happens each time progress arrives. Jobs that are transferring or waiting stop at the next heartbeat.
- Canceling one job does not stop the other work on that node.
- Canceled jobs return to the queue rather than being marked as failed, so they can be run again later.
Starting and stopping agent PCs
The controller starts disconnected agent PCs with Wake-on-LAN, and agents shut down automatically after the queue finishes. When you use both, agent PCs stay on only while they process jobs. Wake-on-LAN is available on a controller with the Pro edition or higher. Auto shutdown works when the controller that the agent connects to has the Pro edition or higher. Agent PCs do not need a license.
An agent PC stays on from the time a wake request starts it until it stops after the queue finishes.
Starting PCs with Wake-on-LAN
- The controller sends a wake request to a disconnected agent PC. Requests go to agents whose MAC address is known.
- Requests are sent when encoding starts, again at intervals while encoding is running, and when you select "Wake" in Node Info.
- To have an agent connect to the controller without anyone signing in, register NyXGridEncode as a Windows service on the agent PC.
Auto shutdown
- If the queue remains finished for the wait time, the agent PC performs the selected action: shut down, force shut down, sleep or hibernate.
- Select when it runs: "Only when started by Wake-on-LAN" or "Always".
- A warning appears before the action, and you can cancel the action while the warning is shown. If the keyboard or mouse is used after the queue finishes, the action waits until a set time has passed since the last input.
- The controller and PCs in Standalone mode can also shut down automatically after the queue finishes (Standard edition and higher).
Requirements and limitations
- Enable Wake-on-LAN in the BIOS/UEFI and NIC settings of each agent PC. For the steps, see the documentation from the manufacturer of the PC, motherboard or NIC. NyXGridEncode support does not cover setting up Wake-on-LAN or troubleshooting PCs on which it cannot be enabled.
- Wake requests are broadcast, so they reach only PCs on the same subnet.
- With "Force shut down", unsaved work is lost.
- Whether Wake-on-LAN can start a PC depends on the PC model, the motherboard and its BIOS/UEFI version and settings, the NIC and its driver, the Windows power settings (such as Fast Startup), how the PC was stopped (shutdown, sleep, hibernation or unplugging and reconnecting the power cable), whether the PC uses Wi-Fi or a wired LAN, and the switches, routers and other network equipment and their configuration. Some PCs also do not support sleep or hibernation for auto shutdown. Operation is not guaranteed in every environment. Before you start using these functions, test starting and stopping each agent PC.
For details on the settings and operating conditions, see User manual: Wake-on-LAN and auto shutdown.
Points to note when running a cluster
Use the same version of NyXGridEncode on the controller and every agent. A controller and an agent whose distributed encoding protocol versions differ cannot connect. If an agent runs a different version from the controller, Node Info shows ▲ in its Connection column, and the status bar shows the number of such agents. You can check the version of each PC in the node details.
Setting parallel jobs with the benchmark
The number of parallel jobs for an agent is set on the agent PC itself and sent to the controller automatically. To base that number on measurement, benchmark with the presets that will actually be used.
- 1Downloading the presets
In "Edit Presets", select "Get from Controller" to download the controller's presets.
- 2Measurement
In Settings > Benchmark, select the downloaded preset and run the measurement.
- 3Applying the result to parallel jobs
Apply the calculated recommended value to the number of parallel jobs in Settings > Encoding.
Before downloading presets
"Get from Controller" replaces all presets on that PC with the controller's presets once you confirm with OK in "Edit Presets", and the original presets are lost. Export any presets you want to keep beforehand. Downloaded presets are used only for measurement. Encoding always uses the settings specified by the controller.
