> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rafftechnologies.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Recover a locked-out VM
> Get back into a VM when SSH, RDP, or the OS itself has stopped letting you in
Updated May 8, 2026
You'll lock yourself out eventually — a misedited `sshd_config`, a too-aggressive UFW rule, a broken `/etc/fstab`, a forgotten password. Raff gives you five recovery paths, ordered from least to most destructive. **Try them in order.**
## 1. Power off, then power on — try this first
Before doing anything else, fully power-cycle the VM. This is the cheapest, fastest fix and clears every kind of *transient* in-memory state:
* Hung processes holding ports the SSH daemon needs
* A `iptables` / `nftables` ruleset loaded into memory that wasn't persisted to disk (and won't come back after boot)
* An OOM event that left the system in a degraded state
* Stuck routing or DNS state
* A frozen kernel or hypervisor-side stall
A full power cycle is more thorough than a reboot — the VM truly stops between, so any volatile state in the running OS is gone.
**How to do it:**
1. Open the VM detail page.
2. Use the **Running** toggle (top right) — flip it off, wait until status is `passive`, flip it back on. Or use the **Power** tab → **Turn Off VM**, then **Turn On VM**.
3. Wait for `active`. Try SSH or RDP again.
Power cycling fixes problems that **didn't survive a reboot in the first place** — transient runtime state. It does **not** fix on-disk misconfigurations like a broken `sshd_config`, a persisted firewall rule, or a bad `/etc/fstab`. If those are the cause, the VM will return to the same locked-out state after boot — move on to the next step.
If the VM was previously responsive and is now flaky for no obvious reason, this step has the highest hit rate and lowest cost. Always try it first.
## 2. VNC console — the universal recovery path
The in-app **VNC console** connects through the hypervisor instead of the network. Open it from the VM detail page → **Console** button (or row Actions menu → **Open Terminal**). When SSH isn't reachable, the dashboard automatically opens VNC instead.
This is your way in for *any* in-OS misconfiguration that didn't break the boot itself.
### Common scenarios fixed via VNC
1. Open the [VNC console](/products/build/virtual-machines/quickstart-guides/connect-via-ssh#3-vnc-console-graphical-fallback). Log in as `root` (use the password from the **Credentials** card → **Send to Email**, or [reset](/products/build/virtual-machines/quickstart-guides/reset-password) it).
2. Edit `/etc/ssh/sshd_config` — fix the offending line.
3. Validate the syntax: `sshd -t` (silence = OK).
4. Restart SSH: `systemctl restart ssh` (Ubuntu / Debian) or `systemctl restart sshd` (RHEL family).
5. Test from your local terminal.
1. VNC in, log in.
2. **Ubuntu / Debian:** `ufw status`, then `ufw allow OpenSSH` or `ufw disable` while you fix it.
3. **RHEL family:** `firewall-cmd --add-service=ssh --permanent && firewall-cmd --reload`, or `systemctl stop firewalld` while debugging.
4. Test SSH from outside.
1. VNC in.
2. Free space: `du -h --max-depth=1 / 2>/dev/null | sort -rh | head -20` to find big paths.
3. Quick wins: `apt clean`, `journalctl --vacuum-size=200M`, `rm /tmp/*`, `docker system prune -a`.
4. If you still can't free enough, [resize the disk](/products/build/virtual-machines/quickstart-guides/resize) up (one-way grow), then expand the filesystem inside the VM.
Use [Reset password](/products/build/virtual-machines/quickstart-guides/reset-password) from the dashboard. The new password is emailed to your account address; use the VNC console to log in once it arrives.
## 3. Single-user mode via GRUB — when the VM won't even boot to login
If the boot fails before you can log in (broken `/etc/fstab`, missing kernel module, runaway init), boot into single-user mode by editing the GRUB cmdline.
### Steps (Linux)
1. Open the [VNC console](/products/build/virtual-machines/quickstart-guides/connect-via-ssh#3-vnc-console-graphical-fallback). [Hard reboot](/products/build/virtual-machines/quickstart-guides/manage-power) the VM if it's hung.
2. The GRUB menu appears for \~3 seconds. Hit any key to pause it.
3. Highlight the default kernel entry and press `e` to edit.
4. Find the line beginning `linux …` (it's long). Move to the end with `End` or `Ctrl-E`.
5. Append one of:
* **Ubuntu / Debian:** ` single` — boots normally up to single-user mode; networking and most services off, root shell after entering the password
* **Most distros — emergency root shell with init:** ` init=/bin/bash` — skips most of init; minimal environment but works when systemd itself is broken
* **Hard fallback when even init=/bin/bash fails:** ` init=/bin/sh` — the most minimal recovery shell; useful when `/bin/bash` is missing or symlinked to a broken binary, or when libc itself is in a bad state. Works on virtually any broken system as long as the kernel can find `/bin/sh`
6. Press `Ctrl-X` (or `F10`) to boot.
You land at a root shell with the root filesystem mounted **read-only**. To make changes:
```bash theme={null}
mount -o remount,rw /
```
Now fix the underlying problem:
| Problem | Fix |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Bad `/etc/fstab` line | `vi /etc/fstab` — fix or comment out the bad line |
| Missing kernel module on boot | Reinstall the module package, or pin the previous kernel via GRUB |
| Bad systemd unit | `systemctl mask ` or remove the unit file |
| Forgotten root password | `passwd root`, then reboot (works only on distros that allow this — modern systemd may require additional steps) |
When done:
```bash theme={null}
sync
reboot -f
```
The VM should boot normally.
## 4. Windows recovery — Safe Mode, WinRE, and Ctrl-Alt-Del
The Linux paths above use GRUB's interactive menu — Windows has its own equivalents. **All of them go through the [browser console](/products/build/virtual-machines/quickstart-guides/open-browser-console)** because the VM's network is the thing you're trying to recover from.
Open the console (VM detail page → **Console** button) and reboot the VM with one of:
* **Power tab** → **Restart** in the dashboard
* Inside the console: **Special-keys menu → Ctrl-Alt-Delete** (sends the chord through NoVNC since your browser intercepts the literal combo)
Then you have three Windows-specific recovery paths:
### Safe Mode
Boot the VM into Windows with only essential drivers and services — the standard repair environment for "Windows boots but is unstable" (recent driver, recent update, login loop).
1. **Reboot the VM** from the dashboard or via Ctrl-Alt-Del in the console
2. Watch the console — as soon as the BIOS POST clears and you see the Windows logo, **immediately tap F8** repeatedly via the console keyboard. Some Windows versions need **Shift+F8** or **F11**; if F8 doesn't bring up the boot menu after a couple of tries, reboot and try Shift+F8 next time
3. The **Advanced Boot Options** menu appears — pick **Safe Mode** (network-disabled) or **Safe Mode with Networking** (if you want SSH/agent access from inside Safe Mode for tooling)
4. Sign in with the local Administrator account
From Safe Mode you can uninstall a problem driver, roll back a Windows Update, disable a service that's preventing login, or run `sfc /scannow` to repair system files. Reboot normally when done.
### WinRE — Windows Recovery Environment
WinRE is the more powerful recovery layer — Startup Repair, System Restore, Command Prompt, Reset This PC. Three ways to reach it:
| Path | When to use |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Three-failed-boot trigger** | Force-power-cycle the VM 3 times in a row (interrupt the boot before Windows finishes loading). On the 4th boot, Windows enters WinRE automatically. From the console: hold the dashboard's **Restart** action through 3 partial boots, then let it boot — WinRE shows up |
| **F11 at boot** | On most Windows Server versions, **tap F11** during the Windows logo screen via the console keyboard. The Advanced Startup options appear |
| **From inside Windows** | Settings → System → Recovery → **Advanced startup → Restart now**. Useful when Windows still boots but is misbehaving |
WinRE's **Command Prompt** is where most repair work happens — `bootrec /fixmbr`, `bootrec /rebuildbcd`, `chkdsk /f`, `dism` to repair Windows components. **Startup Repair** does these automatically for the common cases.
### Reset password
For "I can sign in but I forgot the Administrator password" or "RDP works but auth keeps failing":
The dashboard's **[Reset password](/products/build/virtual-machines/quickstart-guides/reset-password)** action handles this without any console / WinRE work — it sets a new Administrator password through the hypervisor's CONTEXT mechanism on the next reboot. Far simpler than booting into WinRE to manually edit the SAM hive.
### Console session limit during recovery
The browser console has a **1-hour session timeout**. Long Windows recoveries (DISM repair, full chkdsk, restoring from a system image) can run past that. If your session expires mid-recovery:
* The recovery operation **continues running** inside the VM — your VNC viewer disconnected, but the OS isn't affected
* **Close the console tab and re-open from the VM detail page** to get a fresh 1-hour session
* Reconnect picks up wherever Windows is — same screen, same keyboard
Plan for this: trigger the recovery operation, note the time, and reconnect every \~50 minutes to monitor progress.
## 5. Factory reset — when you can't fix the OS in place
[Factory reset](/products/build/virtual-machines/quickstart-guides/factory-reset) wipes the OS disk and reinstalls the **same** OS template fresh. Crucially, **factory reset preserves**:
* Attached volumes and their data
* Backups and volume snapshots
* Network configuration and IP addresses
* VM specs (CPU/RAM/disk)
If your important data lives on attached volumes — not the OS disk — factory reset is fast and clean. If your data is on the OS disk, **don't** factory reset until you've tried path 4.
## 6. Restore from backup via support
If the OS disk is unrecoverable and your data was on it:
1. Check whether you have a [backup](/products/build/virtual-machines/quickstart-guides/enable-backups) of the VM. **Compute Resources → Backups** lists them.
2. If yes — email [support@rafftechnologies.com](mailto:support@rafftechnologies.com) with the VM ID and the backup name. Support can spin up a new VM from the backup with the disk contents and installed software intact.
3. If no — your only options are factory reset (loses OS disk) or destroy and recreate. The OS-disk contents are gone.
This is why production VMs should always run a backup schedule. See [Enable backups](/products/build/virtual-machines/quickstart-guides/enable-backups).
## What Raff doesn't have
* **No bootable rescue ISO** — recovery happens through GRUB single-user mode or factory reset, not by booting a separate rescue environment.
* **No standalone "rescue mode" toggle** in the dashboard — the VNC console is the rescue path; combined with GRUB editing, it covers the same cases.
## Prevention checklist
To make recovery rare in the first place:
* [Initial server setup](/products/build/virtual-machines/quickstart-guides/initial-server-setup) — non-root user with sudo, key-only SSH, but always **keep a working password as fallback**
* Test SSH config changes with `sshd -t` before restarting the daemon
* Don't enable a firewall without first allowing 22 (or your SSH port)
* Run an [enable backups](/products/build/virtual-machines/quickstart-guides/enable-backups) schedule on any VM with data you can't lose
* Take a [snapshot](/products/build/virtual-machines/quickstart-guides/create-snapshot) before risky changes; convert to image if you'd want to keep it across a delete
## Next steps
Get a fresh password emailed.
SSH terminal, direct SSH, and VNC console — all the access methods.
Clean OS reinstall while keeping volumes and backups.
The single best protection against unrecoverable VMs.