01
The control problem
A camera can make robot control feel direct while hiding the exact failures that matter: occlusion, stale frames, depth ambiguity, mode confusion, link delay, and unsafe restart.
The former prototype converted image movement into values labelled as metres, treated handedness probability as tracking confidence, and held the last target after dropout. It had no clutch, independent stop path, command acknowledgement, robot feedback, force sensing, collision planner, or replayable safety decision.
The replacement estimates hand landmarks. Calibration converts their positions into dimensionless X, Y, depth, roll and grip commands. A supervisor checks those commands before use. A robot-specific gateway would need its own measured calibration and hazard analysis.
Safety premise
The browser must be allowed to fail without creating hazardous motion.
Browser crash, model failure, camera loss, and network loss are normal design inputs. A real stop circuit and final motion limits cannot depend on the same software, machine, or link.
02
Keep safety controls near the robot
The bench separates hand tracking from the controls a deployed robot would need. The browser implements the first two stages. Transport, the robot gateway and an independent stop path remain design requirements.
01
Vision input
A local MediaPipe model estimates 21 hand landmarks. Calibration converts their positions into relative movement commands, without measuring distance in metres.
Untrusted
02
Command supervisor
Each camera sample is checked for calibration, hand visibility, geometry and age. The supervisor also checks hold-to-run, reacquisition, speed changes and workspace limits.
Tested in browser
03
Command transport
A deployed connection must authenticate the session and reject expired, duplicated, out-of-order or out-of-range commands.
Specified / not connected
04
Robot-side gateway
A controller beside the robot owns kinematics, collision, force, speed, tool, and task limits. That controller is not implemented by this website.
Production target
05
Independent stop path
Protective stop and emergency stop must not share the camera, browser, network, or application failure path.
Required / outside browser
The vision model, browser and network provide commands. Safety functions must be implemented separately.
03
The rebuilt control bench
The bench runs camera inference locally. Its model and WASM runtime load from the same origin, inference runs in a worker, and the application does not upload or retain frames. Accepted commands update a limited Cartesian target before the simulator calculates robot motion and feedback.
- Landmark runtime
- MediaPipe Tasks Vision 0.10.35 / local assets
- Model
- Hand Landmarker full float16 v1 / pinned hash
- Inference
- Dedicated browser worker / camera stays on device
- Command units
- Normalized rates and grip request / no metric claim
- Supervisor tests
- 8 deterministic tests / passing 10 July 2026
- Physical robot link
- Not implemented
Three profile presets change simulator watchdog, reacquisition, command slew, workspace, injected latency, and packet loss. They are bench inputs, not standard limits. A deployment must replace them with task-specific budgets and verified stop behavior.
Hand tracking
The current model does not cover the workwear and contact cases.
The MediaPipe model card puts gloves and occluded hands outside intended use and does not position the model for life-critical decisions. Space, subsea, and nuclear tasks commonly introduce gloves, tools, self-occlusion, poor views, and costly contact. The model must remain replaceable and cannot carry the safety case.
04
The command is an expiring request
Position targets can preserve dangerous intent after the operator or link disappears. The rebuilt bench emits a normalized rate request with an explicit validity decision. A failed gate makes every motion rate zero and marks the command invalid. The downstream controller still chooses the task-specific safe action.
{
"schema": "li.teleop.intent/1",
"sessionId": "local-sim-…",
"sequence": 1842,
"capturedAtMonotonicMs": 91342.5,
"expiresAfterMs": 140,
"source": "camera",
"intent": {
"linearRate": [0.18, -0.04, 0.00],
"rollRate": 0.07,
"grip": 0.61
},
"gates": {
"calibrated": true,
"handVisible": true,
"geometryValid": true,
"clutchHeld": true
}
}Deployment addition
A real transport must also bind operator identity, authorization, configuration/model hash, clock basis, signature, acknowledgement, and replay protection. These fields are requirements for a real connection.
05
Supervisor states and recovery
Tracking a hand, arming motion and moving the robot are separate states. The operator must hold the control while all automatic checks pass.
| State | Invariant | Exit condition |
|---|---|---|
| Standby | Output is invalid and all motion rates are zero. | Valid source, calibration, live link, then deliberate two-step arm. |
| Reacquiring | A returning hand cannot immediately move the simulated plant. | Tracking remains valid for the selected profile's dwell time. |
| Armed / hold | Motion stays at zero until the operator holds the control. | The operator holds the control and all sample checks pass. |
| Active / limited | Only fresh commands within the rate and speed-change limits are accepted. | A failed check returns to hold and discards the previous command. |
| Emergency stop | The simulated stop is latched and output remains invalid. | A deliberate reset returns to standby; a new two-step arm is required. |
“Hold” is only the simulator’s zero-rate behavior. It is not a universal physical safe state. A collision in orbit, a retained subsea load, and a tool inside a radiation field can require different responses: controlled stop, maintain-grasp, retreat, park, release, or torque-off.
06
One input method, three qualification paths
There is no certificate called “space, subsea, and nuclear compliant.” The complete robot, task, operating environment, communications path, operator, and safety case determine which standards and evidence apply.
| Domain | First constraint | Architecture response | Qualification route |
|---|---|---|---|
| Orbital servicing | Delay changes the control problem. Direct continuous rate control stops making sense when command age exceeds the task budget. | Keep safety and contact response local; move delayed operations toward supervised goals, preview, and robot-side autonomy. | Tailor NASA-STD-3001, NASA software assurance, or ECSS human-factors, software, safety, verification, and security requirements to the mission. |
| Subsea intervention | Visibility, tether/link degradation, vehicle motion, tool loads, and remote-operations-centre handoff change what a safe hold means. | Keep vehicle and manipulator limits subsea or topside-local, record command acceptance, and prove the complete ROV interface and operator workflow. | Use IMCA R004/R006/R025, the applicable NORSOK, API/ISO subsea interfaces, vessel/class rules, and customer acceptance basis. |
| Nuclear remote handling | Contact force, retained loads, contamination, radiation, recovery access, and human-system mode errors can dominate camera tracking quality. | Keep the ML/browser path non-safety-classified behind independent controls where possible. Add qualified force feedback when the manipulator class requires it. | ISO 17874 is the direct remote-manipulator family. IEC nuclear I&C, human-factors, cyber, QA, and environmental standards apply only when the system scope invokes them. |
Nuclear remote handling
Vision-only control is not an electrical master-slave replacement.
ISO 17874-3 treats force reflection as an essential feature of the covered electrical master-slave manipulators. This bench has no force-feedback device and cannot claim that equivalence or conformance.
07
What the current tests prove
The automated tests exercise individual supervisor decisions. Browser timing, camera accuracy and stopping a physical robot require separate tests.
| Evidence | Result | Scope and limit |
|---|---|---|
| Hand mapping fixtures | Pass | Neutral calibration, normalized saturation, invalid landmark geometry. No accuracy or glove/lighting claim. |
| Supervisor fixtures | Pass | Stale-frame inhibit, latched stop priority, reacquisition dwell, command slew, and workspace guard. |
| Browser interaction | In verification | Camera permission, worker/model load, simulation, keyboard clutch, stop/reset, telemetry export, responsive layout. |
| Hardware in the loop | Not run | No robot, ROS 2/vendor adapter, force/torque sensor, physical stop circuit, or measured end-to-end stop time. |
| Field qualification | Not started | No mission, subsea spread, nuclear cell, customer asset, operator study, environmental test, or certification assessment. |
Test baseline: Node 23.9.0 / eight deterministic tests / 10 July 2026. Browser and production-build results are added only after the route is exercised.
08
The route from tested bench to deployed system
A deployment needs a specified robot and task. The command format, safety design, hazard log and tests can then be adapted to that application.
| Gate | Evidence required before acceptance |
|---|---|
| Task and hazard basis | Named robot, tool, load, environment, communications topology, safe-state analysis, foreseeable misuse, and accepted residual risk. |
| Robot-side controls | Validated kinematic, collision, speed, force/torque, workspace, grasp, stop, restart, and authorization behavior independent of the browser. |
| Trace capture and fault injection | Versioned traces for occlusion, gloves, extra hands, frozen frames, clock skew, delay, jitter, loss, reordering, spoofing, restart, and takeover failure. |
| Hardware in the loop | Measured motion-to-photon, pose-to-command, command age, jitter, stop time, safe-state entry, command acknowledgement, and recovery on the selected hardware. |
| Human factors | Representative operators complete defined tasks without hidden mode changes, unsafe workload, ambiguous authority, or unobserved command rejection. |
| Domain acceptance | Mission/customer-tailored standards crosswalk, cyber assessment, environmental/EMC evidence, operating procedures, training, maintenance, and signed acceptance. |
Status
The bench is parked.
The retained bench demonstrates local hand tracking, supervisor checks and simulated faults. A deployment would need the evidence listed above.
09
Primary references
- Google AI Edge: Hand Landmarker for Web
Local model setup, video inference, output semantics, configuration, and the main-thread blocking warning addressed by the worker architecture.
- MediaPipe: Hand Tracking model card
Intended use, Apache-2.0 license, training/evaluation limits, glove and occlusion exclusions, and the life-critical decision boundary.
- ISO 12100:2010
Machinery risk assessment and risk reduction for the complete system.
- ISO 10218-2:2025
Industrial robot application and cell integration. Applicability depends on the selected robot and deployment.
- IEC 62443-4-1:2018
Secure product-development lifecycle for an industrial control component.
- NASA-STD-3001 Volume 2 Revision E (2025)
Human-system integration requirements including visible mode, health, recording, override, safe takeover, stop, and authorized restart.
- ECSS-E-ST-40C Revision 1 (2025)
Software engineering requirements for a tailored European space project.
- IMCA R004 Revision 6 (2024)
ROV system code of practice, including remote-operations-centre guidance.
- ISO 17874-3:2011
Electrical master-slave manipulators for nuclear remote handling, including the force-reflection boundary relevant to this concept.
- IEC 61513:2026
Nuclear power plant I&C systems important to safety; conditional on the function and plant classification.
Standards review completed 10 July 2026. The standards register distinguishes design inputs from conditional, contractual, and certification requirements.