Everything that broke running Omarchy in VMware Workstation

I run Omarchy in a VMware Workstation VM on a Windows host, and once it is set up properly it is a good way to work. Getting there took a couple of days, most of which I spent chasing what I was convinced were three separate problems.

The browser would stop launching, Discord along with it, and at some point the clipboard would stop syncing to Windows without any indication that it had. Logging out and back in brought everything back for a few hours.

All of it came from one dead process, and none of the error messages pointed anywhere near it. What follows is the chain, the fixes, and the handful of diagnostics that sent me the wrong way for hours at a stretch.

Everything here is Arch (Omarchy), Hyprland 0.56, open-vm-tools 13.1, on VMware Workstation with 3D acceleration turned on.

Xwayland dies on vmwgfx

Xwayland dies and Hyprland does not bring it back.

unknown object (36), message attach(?oii)
XWAYLAND: wl_display#1: error 1: invalid arguments for wl_surface#18.attach
(EE) failed to dispatch Wayland events: Invalid argument

I assumed this was some kind of race for a while, but it is not. Start a scratch Xwayland by hand with nothing connected to it and it dies at startup every single time. Glamor is what is doing it: the accelerated path allocates its buffers through GBM on vmwgfx, hands the compositor a wl_buffer, and the compositor refuses it. Since that counts as a protocol violation rather than a rendering glitch, the connection gets killed outright.

One variable in ~/.config/uwsm/env sorts it:

export XWAYLAND_NO_GLAMOR=1

Xwayland 24.1 honours it, and you can check your own build with:

strings /usr/bin/Xwayland | grep XWAYLAND_

You lose nothing by setting it. glxinfo reports the same llvmpipe GL 4.6 either way, since this guest was never getting accelerated GLX in the first place.

What cost me time here was assuming LIBGL_ALWAYS_SOFTWARE=1 already covered it. That variable moves glamor’s rendering onto llvmpipe while the buffers still come from GBM on vmwgfx, so Xwayland carries on dying with it set. It looks like it should be operating on the same layer, which is why I skipped past it for so long.

Apps that hang instead of failing

This is the same bug again, and it is the reason one dead process ends up looking like several unrelated ones.

When Xwayland dies, Hyprland carries on exporting DISPLAY=:0 and leaves /tmp/.X11-unix/X0 sitting in place, so X11 clients never get an error. They connect to a socket that nobody is accepting on and block forever in unix_wait_for_peer.

Hanging like that is a lot harder to debug than a clean failure, because there is nothing to go and read. Both omarchy-launch-browser and omarchy-launch-webapp start with xdg-settings get default-web-browser, whose desktop detection shells out to xprop -root _DT_SAVE_MODE. That call blocks, the launcher never reaches the line that starts the app, and every keypress leaves another stuck process behind. There were ten of them by the time I went looking.

One command settles it:

cat /proc/<pid>/task/*/wchan     # unix_wait_for_peer = dead Xwayland, live DISPLAY

Discord goes the same way, since on Omarchy it is a Chromium web app running through that same launcher, which is how two apps with nothing to do with each other end up broken by one blocked probe.

If you are going to build something here, build a dead-man switch that notices Xwayland is gone and unlinks the stale socket so X11 connections fail immediately. It does not fix anything, but it turns “my desktop is broken” into “X11 apps are unavailable”, and the launcher probe drops from an infinite hang to 0.12 seconds.

The clipboard reports itself healthy while carrying nothing

Copy and paste between Windows and the guest has more places to break than anything else here, and every one of them breaks silently.

There are three problems stacked on top of each other, roughly in the order you will meet them.

The plugin never loads. libdndcp.so links gtkmm3, which Arch ships as an optional dependency of open-vm-tools, so a stock install does not have it:

Opening plugin 'libdndcp.so' failed: libgtkmm-3.0.so.1: cannot open shared object file

Install gtkmm3 and it loads.

The plugin is X11-only. It calls gdk_x11_* and XOpenDisplay, while Omarchy exports GDK_BACKEND="wayland,x11,*", so GTK initialises on Wayland and every X11 call inside the plugin fails silently. A systemd user drop-in forcing GDK_BACKEND=x11 puts it back on Xwayland where it expects to be.

Hyprland does not sync the X11 and Wayland clipboards. VMware’s data lands on the X11 CLIPBOARD and stops there, invisible to every native app. Closing that gap needs a daemon that is an X11 client and a Wayland client simultaneously, copying whichever side changed most recently.

Do not trust systemctl is-active while you are working on this. When Xwayland dies the clipboard units exit, systemd restarts them straight back into the same dead display, and they will happily report active while nothing at all is moving.

The image support everyone says is missing

Common wisdom says open-vm-tools cannot carry images, and the test everyone reaches for agrees:

strings /usr/lib/open-vm-tools/plugins/vmusr/libdndcp.so | grep -cE '^image/'   # 0

I believed that for most of a day. The zero is real, but GTK registers image/png, image/bmp and the rest at runtime out of GdkPixbuf’s format list, so they are never in the binary as literal strings for grep to find. The symbol table tells a different story:

nm -D --undefined-only /usr/lib/open-vm-tools/plugins/vmusr/libdndcp.so | grep -i image

Back come Gtk::Clipboard::set_image(), wait_for_image(), Gdk::PixbufLoader and Pixbuf::save_to_buffer(), which is the entire image clipboard API in both directions. open-vm-tools 13 negotiates CopyPaste v4 with the host, and that version carries PNG. Images had been arriving on the guest’s X11 clipboard the whole time I was convinced they could not, and the only thing missing was something to carry them the last hop to Wayland.

It is a bad test in general, not just here. Any MIME type a toolkit assembles at runtime is invisible to grep.

Two things that will shape the bridge if you write one:

• Do not read the clipboard image on every poll just to spot changes, because each read makes vmtoolsd drag the whole payload across the backdoor from the host. Hash the first 64 KB and only do a full read when that hash moves.
• Re-encoding a PNG is not byte-stable. After a guest to host push, the bytes X11 hands back do not match the ones you supplied, so the image looks new and starts bouncing between the two clipboards indefinitely. Keep hashes of both forms.

GPU acceleration is blocked upstream

Your VM’s 3D acceleration is probably fine. Mine reports the full feature set:

vmwgfx: Capabilities: ... 3D, ... gbobject, dx
vmwgfx: Available shader model: SM_5_1X
EGL on GBM: SVGA3D, OpenGL 4.3

Everything still renders on the CPU, and the workaround everybody recommends is what is causing it. LIBGL_ALWAYS_SOFTWARE=1 does more than pick a software rasteriser, it hides the DRM node from EGL completely:

LIBGL_ALWAYS_SOFTWARE	EGL devices	DRM node
unset	2	/dev/dri/card0 + renderD128, VMware
=1	1	none

Aquamarine matches EGL devices against /dev/dri/card0, so with that variable set it finds nothing at all and Hyprland comes up without a GL renderer:

CDRMRenderer(drm): Can't create renderer, no matching devices found
drm: Failed to initialize renderer state for /dev/dri/card0

A compositor with no renderer cannot import a GPU buffer from anything, so it rejects every GPU client, which happens to be the exact symptom people set the variable to cure. It also retries the renderer indefinitely, and my hyprland.log got to 181 MB in 14 hours inside a 790 MB tmpfs, which is to say in RAM.

Removing it gets the compositor onto the GPU cleanly, but clients still die, including a plain GTK4 window, so it is not a Qt or Quickshell problem. A WAYLAND_DEBUG capture shows where it ends:

-> zwp_linux_buffer_params_v1#50.add(fd 15, 0, 0, 7680, 0, 0)
-> zwp_linux_buffer_params_v1#50.create_immed(new id wl_buffer#51, 1920, 1080, XR24, 0)
   discarded [unknown]#50.[event 1]          <- buffer_params.failed
   wl_display#1.error("invalid arguments for wl_surface#31.attach")

Hyprland advertises 116 format and modifier pairs, the client picks one off that list, and then the compositor refuses to import it. Nothing gets logged, even with AQ_TRACE=1 and HYPRLAND_TRACE=1 set.

To rule the driver out I wrote four small GBM/EGL programs that allocate a buffer and import it straight back through eglCreateImageKHR, and every combination works: implicit and explicit LINEAR modifiers, with and without modifier attributes, 1920x1080 at the exact stride the client asks for, across separate device fds, and into a card0 display. They all import cleanly, which puts the fault somewhere above Mesa.

It is filed as aquamarine#360. Until that lands, a Hyprland guest on VMware renders in software. Both hyprland and aquamarine come out of Arch’s extra repo, so whenever it is fixed it will arrive through an ordinary pacman -Syu without waiting on an Omarchy release.

There is one more trap in that error text. I spent hours reading unknown object (46), message attach(?oii) as the compositor rejecting a malformed buffer. What it is telling you is that the buffer does not exist: libwayland resolves the target object before it prints the method name and signature, so the object it cannot find is the wl_buffer argument rather than the surface. The compositor had already refused to create it, and once I read it that way there was enough to file.

Screen recording, and a simpler blocker

Omarchy binds ALT + PRINT to gpu-screen-recorder, which refuses to start on a VMware guest in two different ways depending on your config:

with LIBGL_ALWAYS_SOFTWARE=1
  "It's using llvmpipe (software rendering)... failed to load opengl"

without it
  "unknown gpu vendor: VMware, Inc."

The second one is a hard vendor whitelist covering AMD, NVIDIA and Intel, so it would refuse on this hardware even with acceleration working properly, and -fallback-cpu-encoding yes does not get you past it, because that flag only covers the encoder rather than the check itself.

wf-recorder does the job instead. It captures through wlr-screencopy, which is the same path grim already uses successfully here, and encodes x264 on the CPU. Full screen holds a steady 30.1 fps with no dropped frames at roughly 25% of one core, which is plenty for demos and bug reports.

Playing the result back crashed mpv, for the same underlying reason. It comes up on Wayland, builds itself a working GL 4.6 context, notices the renderer is llvmpipe and throws that context away as too slow, then falls through its probe list to x11egl and aborts:

[vo/gpu-next/wayland] GL_RENDERER='llvmpipe (LLVM 22.1.8, 256 bits)'
[vo/gpu-next/opengl]  Suspected software renderer or indirect context.
[vo/gpu-next/opengl]  Initializing GPU context 'x11egl'
mpv: video/out/x11_common.c:679: vo_x11_init: Assertion `!vo->x11' failed.

Telling it to accept the software renderer keeps it on the context that was already working. In ~/.config/mpv/mpv.conf:

gpu-sw=yes

That pattern turned up more than once. Anything that refuses software rendering on your behalf goes looking for a better path, and on this guest there is not one, so it finds a crash instead.

If you are setting this up

Check that Xwayland is actually alive before you diagnose anything else, because one dead process here shows up as several unrelated bugs and you can burn a lot of time treating them separately.

Most of the difficulty in this stack comes from calls that block forever instead of returning an error, so if something is unresponsive rather than broken, go looking for a hang before you go looking for a bug.

systemctl is-active will tell you a unit is healthy when it has restarted cleanly into a dead world, so do not let a green status talk you out of investigating.

And re-test whatever you wrote down earlier. Three notes I had been working from (images cannot cross, LIBGL fixes Xwayland, screenshots do not work here) turned out to be wrong when I finally checked them properly, and each one had sent me somewhere worse than the original bug.

Where it ended up: Xwayland stays up, text and images move both ways across the clipboard, and screenshots and recording both work. GPU acceleration is the one thing still outstanding, and it is sitting with upstream now rather than with me.