implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

sodiboo · 2024-08-26T20:56:40Z

My motivation for this is the use of KVM switching software, in particular i noticed that lan-mouse requires them (hence my branch name).

Two things need to be done:

Please someone check that what i am doing with pointer axis frames is sane. This was painful. It works. Smithay docs refer to a nonexistent function so i had to check 2-year-old git blames to find out what replaced it. We just ignore the axis source if it's done before an arbitrary other thing?? Help.
Need to implement a keyboard-shortcuts-inhibit release bind. The current implementation already doesn't inhibit the ChangeVt binds ever.

You can try the functionality of both of these protocols using lan-mouse --test-emulation (will cause the pointer to move in a lemniscate) or lan-mouse --test-capture (WARNING: YOUR FOCUS WILL BE UNRECOVERABLE FROM WITHIN NIRI. use Ctrl+Alt+F2 and then pkill -f test-capture to recover. the Ctrl+Alt+{F1..F12} binds are the ChangeVt binds i've mentioned already)

Another thing i've noticed is that virtual-keyboard inputs aren't ever handled by the compositor (i.e. Super+Left is passed to whatever client, instead of focusing the column to the left)? This is irrelevant to this PR, but something i noticed while using lan-mouse. This is probably a Smithay issue? Since it's uhh mostly implemented in Smithay.

YaLTeR

Smithay docs refer to a nonexistent function so i had to check 2-year-old git blames to find out what replaced it. We just ignore the axis source if it's done before an arbitrary other thing??

I haven't looked in detail but if it involves axis v120 events then yeah it's like this pretty much because of legacy, v120 events replaced the old ones: https://wayland.freedesktop.org/libinput/doc/latest/wheel-api.html

YaLTeR · 2024-08-27T13:42:57Z

src/input/mod.rs

@@ -2221,6 +2247,18 @@ fn should_intercept_key(
        }
    }

+    if is_inhibiting_shortcuts


I have a feeling that this should interact in a more complex manner with the match below because starting to inhibit while some modifier is held (and in suppressed_keys) then releasing the modifier will currently leave it in suppressed_keys, which does not seem right.

This behavior should also be tested in the unit tests below.

I've since updated the handling in a way that i think works with supressed_keys properly (and i've commented the behaviour), but to be honest, thinking about the order of events kinda makes my head hurt and i'm dreading writing the requested tests. I'm also not entirely confident that my intuition of how it should behave exactly aligns with what you think it should do? Do you wanna take a shot at writing tests for this? 🥺👉👈

Seems fine to me. For the tests, just add a few cases with the shortcuts inhibited = true. For pressing and releasing allow-inhibit, !allow-inhibit, which also check suppress keys. Basically check that a shortcut that would return Intercept() returns instead Forward if inhibited = true. And maybe also something which first does a press with inhibited = true, then release with inhibited = false in case that logic would change later.

src/input/mod.rs

sodiboo · 2024-09-07T20:56:56Z

I've implemented the allow-inhibiting property for binds, and the toggle-keyboard-shortcuts-inhibit action, which weren't present in the initial draft of this PR. This is basically the only thing that was missing from making this PR "complete". keyboard-shortcuts-inhibit is now fully implemented (should probably have some more tests).

I've also made sure you can't bind toggle-keyboard-shortcuts-inhibit but allow that bind to be inhibited. the allow-inhibiting property is ignored for this action

wlr-virtual-pointer worked flawlessly right off the bat for my use case, and you wouldn't be able to tell from using, but the underlying code is definitely janky!!! I'd like someone to verify that this is how the Axis* events are suppposed to work. Basically if we get more than one timestamp in a single frame, all but the first are ignored. And... AxisDiscrete overrides the values set by Axis? It feels very dirty, but judging by the AxisFrame struct in Smithay, it feels like the protocol should not really be allowing any of these requests to be sent twice in one frame. It feels very jank, and i know little about how axis events actually work, i just plugged together the parts of Smithay that "looked" about right to me.

The uhh... AxisSource request is optional (i think?) but PointerAxisEvent::source() does not allow a None value, so this is definitely wrong. The AxisFrame struct cannot be constructed without a timestamp, so for the AxisSource event, which doesn't include a timestamp, it's just dropped because no AxisFrame can be started; a different Axis* event must be sent first for the AxisSource to amend it.

YaLTeR

I'd like someone to verify that this is how the Axis* events are suppposed to work. Basically if we get more than one timestamp in a single frame, all but the first are ignored. And... AxisDiscrete overrides the values set by Axis? It feels very dirty, but judging by the AxisFrame struct in Smithay, it feels like the protocol should not really be allowing any of these requests to be sent twice in one frame. It feels very jank, and i know little about how axis events actually work, i just plugged together the parts of Smithay that "looked" about right to me.

The uhh... AxisSource request is optional (i think?) but PointerAxisEvent::source() does not allow a None value, so this is definitely wrong. The AxisFrame struct cannot be constructed without a timestamp, so for the AxisSource event, which doesn't include a timestamp, it's just dropped because no AxisFrame can be started; a different Axis* event must be sent first for the AxisSource to amend it.

Yeah idk tbh, I would just test with several wlr-virtual-pointer clients to see that they work. And if there's something weird, check in WAYLAND_DEBUG=1 what events they send.

(btw, what wlr-virtual-pointer clients are there apart from lan-mouse? I'll need a few to test this I guess)

YaLTeR · 2024-10-06T07:14:03Z

resources/default-config.kdl

+    //
+    // The allow-inhibiting=false property can be applied to other binds as well,
+    // which ensures niri always processes them, even when an inhibitor is active.
+    Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }


I think we should bind this to Mod+Escape by default because Super+Escape is the default in GNOME: https://gitlab.gnome.org/GNOME/mutter/-/blob/d7d92c68bd5c76525752feab65c44b3157deb7f1/data/org.gnome.mutter.wayland.gschema.xml.in#L58-61

The reasoning for not doing this is so:

The primary release mechanism should be implemented in the client. It will know best what keys are in use and easily accessible. In many cases, we'd expect the client to release on Super+Escape (because inhibiting shortcuts means you're free to use Super for whatever binds, and Super+Escape is unlikely to mean anything else), but if it does mean something else then the client knows best what is an appropriate release shortcut.

The reason why i chose Mod+Shift+Escape is because that's the shortcut to open Task Manager on Windows; so it already has the connotation of "An app is misbehaving!! Help me, operating system!" in some users' minds (including my own).

The difference is, however, ultimately arbitrary. So if you disagree with the above, i'll change it to Mod+Escape.

In many cases, we'd expect the client to release on Super+Escape (because inhibiting shortcuts means you're free to use Super for whatever binds, and Super+Escape is unlikely to mean anything else), but if it does mean something else then the client knows best what is an appropriate release shortcut.

Well, my thinking is, if Super+Escape is the default on GNOME, then client devs might decide not to do anything on Super+Escape client-side and generally expect it to already work, and definitely not bind it to something other important (considering GNOME is by and large the "default desktop", and any client testing likely occurs on GNOME).

The reason why i chose Mod+Shift+Escape is because that's the shortcut to open Task Manager on Windows

Oh? I always used Ctrl+Shift+Escape (and Ctrl-Alt-Delete before that started bringing up the menu).

Oh? I always used Ctrl+Shift+Escape

actually that sounds about right. i may have misremembered and just Made Up This Shortcut (by changing the first modifier in the sequence)

then yeah, let's make it parity with GNOME. i'm still happy with that 'cause i can just use the one i prefer in my config :3

Suggested change

Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

so yeah

src/handlers/mod.rs

src/input/mod.rs

YaLTeR · 2024-10-06T07:27:29Z

niri-config/src/lib.rs

                Mod+T allow-when-locked=true { spawn "alacritty"; }
                Mod+Q { close-window; }
                Mod+Shift+H { focus-monitor-left; }
                Mod+Ctrl+Shift+L { move-window-to-monitor-right; }
                Mod+Comma { consume-window-into-column; }
                Mod+1 { focus-workspace 1; }
                Mod+Shift+1 { focus-workspace "workspace-1"; }
-                Mod+Shift+E { quit skip-confirmation=true; }
+                Mod+Shift+E allow-inhibiting=false { quit skip-confirmation=true; }


I wonder if allow-inhibiting=false is a bit hard to wrap one's head around? Maybe something along prevent-inhibiting=true would be better, human config file reading and writing wise?

i don't feel like that change is an improvement, because prevent-inhibiting=true is a double negative. but also, i agree with your concern, and i think it's slightly confusing because allow-inhibiting=false is also a double negative, because the value it will be set to is false.

is there a way to name it that isn't a double negative? always-available=true, but more descriptive? borrow an idiom from CSS and call it important=true? maybe with the exclamation point to drive home the correlation? !important=true. though it reads as "not".

Hmmm, difficult question indeed. I like the word "inhibit" being there because it's the technical term for this (unless it isn't always? do some apps re-phrase it?).

If we drop inhibit, maybe something like "always-accessible" is better than "always-available"? That's kind of still not very descriptive, while it does control a very specific thing, so it would be good if it were descriptive...

src/input/mod.rs

src/protocols/virtual_pointer.rs

src/handlers/mod.rs

YaLTeR · 2024-10-06T07:51:08Z

src/input/mod.rs

@@ -2221,6 +2247,18 @@ fn should_intercept_key(
        }
    }

+    if is_inhibiting_shortcuts


Seems fine to me. For the tests, just add a few cases with the shortcuts inhibited = true. For pressing and releasing allow-inhibit, !allow-inhibit, which also check suppress keys. Basically check that a shortcut that would return Intercept() returns instead Forward if inhibited = true. And maybe also something which first does a press with inhibited = true, then release with inhibited = false in case that logic would change later.

sodiboo · 2024-10-06T12:34:26Z

(btw, what wlr-virtual-pointer clients are there apart from lan-mouse? I'll need a few to test this I guess)

just about any KVM or RDP software. Something like deskflow or input leap should do the same thing as lan-mouse. I also noticed that WayVNC works just fine with niri as of this PR, as it has several capture protocols including wlr-screencopy, so you can connect to it from wlvncc.

Most of these should probably also be using keyboard shortcuts inhibit? Unsure. I know lan-mouse does, at least.

ConsularParadi · 2024-10-19T17:38:22Z

Hi, I was trying to setup wayvnc and having some issues with wlr-virtual-pointer. Has this issue been resolved in niri-0.1.9-1. If yes, how can I set it up?

YaLTeR · 2024-10-19T18:39:45Z

wlr-virtual-pointer is part of this PR. It is not yet merged

sodiboo · 2024-10-19T20:46:42Z

I was trying to setup wayvnc [...]

Note that wayvnc still won't work smoothly once this is merged, at the very least because Smithay doesn't handdle ext-virtual-keyboard properly; all keyboard inputs are sent to the focused applications, and the compositor can't currently intercept them for purposes such as window management (e.g. you can only change focus by clicking with the mouse; over a VNC connection it will behave as if you have zero binds configured)

Currently, i'm fairly busy with a bunch of exams and other assignments at school, which is why i haven't been very actively working on this PR (and others) for some time. I'll get to it sometime, and also improve the virtual keyboard situation such that WayVNC will work properly, as i do actually also wish to use WayVNC eventually. I have no ETA on when all of this will be finished; it's a fairly broad scope of work requiring changes in Smithay's input backend and not just niri.

If you don't find the lack of keyboard shortcuts to be problematic, and still wish to use WayVNC and want to do it right now, you can just run this PR as-is, before it's merged. Its not necessarily up-to-date with the most recent version of niri, so you'd be "downgrading" slightly (not by much). I am running a branch based on this PR live on my machines and it works fine.

sodiboo · 2024-11-08T08:13:41Z

by the way:

closes Implement keyboard-shortcuts-inhibit-unstable-v1 #198

i didn't realize this was an issue that existed but it should definitely be linked to this PR.

current status: i have a lot of stuff in my backlog, including a lot of schoolwork. i'll get back to this (and my other PRs) eventually but right now it's not my top priority

the keyboard shortcuts inhibition part is actually stable and worked well last i recall, so like, if you want it right now just go merge it manually. i guess i could open another PR to separate these, because i think that one requires less work to "finish" and actually be good to merge.

sodiboo · 2024-11-23T20:30:04Z

Okay, so respecting the output parameter was actually surprisingly less work than expected. It's fairly trivial to define an extension trait for the input backend, such that we can easily expose the VirtualPointer::output() to the event handlers.

I've implemented this in the commit titled "add InputBackend extensions; use Device::output() for absolute pos events". This is the latest commit.

Virtual Pointer absolute motion when provided with an output should now work, and actually give output-relative coordinates.

I've also hooked up the touch and tablet positions to this function, falling back to the previous behaviour when it returns None, which is actually always for these event types at this time. But, this makes it easier to implement the ei protocol and also the potential to expand the config to allow per-tablet configuration of which output it maps to. Just change the impls in backend_ext.rs to make sure they return the correct Output, and everything else should just work as is. (and, maybe need to change its signature to take additional params, but besides that the callsites already work).

salman-farooq-sh · 2024-12-08T20:09:45Z

Thanks @sodiboo and @YaLTeR. What work is left in this now?

salman-farooq-sh · 2024-12-25T19:55:18Z

I am very excited for this.

YaLTeR · 2025-01-02T18:22:00Z

Remind me, what is left to do here? (Apart from the whole "virtual keyboard doesn't do compositor shortcuts" which is not really related to these code changes.) Just review and merge?

sodiboo · 2025-01-03T10:40:09Z

uihhh yeah, just review i guess. everything works i think.

YaLTeR

Some minor things left, otherwise looks good.

YaLTeR · 2025-01-05T07:10:50Z

resources/default-config.kdl

+    //
+    // The allow-inhibiting=false property can be applied to other binds as well,
+    // which ensures niri always processes them, even when an inhibitor is active.
+    Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }


Suggested change

Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

so yeah

src/input/backend_ext.rs

src/input/mod.rs

src/protocols/virtual_pointer.rs

…ents

YaLTeR · 2025-01-18T07:08:27Z

I added a few tests for inhibited shortcuts. Manually tested inhibiting with SDL3 test clients, seems to work fine.

However, I noticed that pointer input in winit is vertically inverted. What's up with that, does it work on your machine?

sodiboo · 2025-01-18T08:47:21Z

However, I noticed that pointer input in winit is vertically inverted. What's up with that, does it work on your machine?

(resolved on Matrix, up to my push just now; but summarizing here for future reference)

No, this didn't work on my machine. I don't know how i missed that.

It's because returning the Output from the winit backend made the pointer event code actually respect the Transform::Flipped180 that is unilaterally applied to the winit output.
This wasn't an issue before, because previously it was handled as a global coordinate space (where the output transform is not considered).
The correct way to solve the root cause of this is to apply a buffer transform at the end of rendering (e.g. in my Wayland backend, i tell the parent compositor to flip the buffer), but since that's in Smithay (and actually even winit, not just Smithay), we cannot easily do that right now,
so the current resolution is to just revert to the old behaviour. it's indistinguishable anyway, so it's whatever. buh. it works i guess. i just don't like that it's essentially two bugs cancelling each other out

…e intent

YaLTeR · 2025-01-18T14:27:19Z

Let's assume this works fine now, and i'll also pretend i didn't see how you added that new State parameter. Still, please re-test with lan-mouse (ideally with transformed outputs too)

YaLTeR · 2025-01-18T14:27:28Z

Thanks!

sodiboo · 2025-01-18T15:12:11Z

and i'll also pretend i didn't see how you added that new State parameter

hey there was already a comment about that in the previous commit, that we'd need it to implement per-device config (e.g. multiple touch screens, or multiple tablets mapped to different outputs). this is just preparing for that inevitability, at least

sodiboo · 2025-01-18T18:16:59Z

I've tested it now with lan-mouse, and it seems to still be fully functional! both wlr-virtual-pointer and keyboard-shortcuts-inhibit work fine!

i did notice, however, that disabling the keyboard shortcuts inhibitor is not very helpful against lan-mouse because, you see, lan-mouse still has keyboard focus, and if i open a fuzzel, i can't type into it; those keypresses go to my other machine. so best i can do is like, exit niri or switch VTs (though the latter doesn't require the inhibitor anyways). I guess that is basically a lan-mouse bug, because it should release its focus whenever the inhibitor is disabled methinks (it ought to run the same function as if a peer tells it that "hey you, you have the focus now, you're the main machine to control", but it doesn't)

of course, virtual keyboard still can't do compositor binds, but that's like, super not our fault, and not related to this PR. but this is the main thing limiting the usability of lan-mouse and similar such software now

YaLTeR · 2025-01-18T18:20:54Z

Alright, thanks for testing

This was referenced Aug 26, 2024

idle inhibit on unfocused computers feschber/lan-mouse#181

Open

binds are not processed for virtual-keyboard events #631

Closed

YaLTeR reviewed Aug 27, 2024

View reviewed changes

sodiboo changed the title ~~implement wlr-keyboard-shortcuts-inhibit and wlr-virtual-pointer~~ implement keyboard-shortcuts-inhibit and wlr-virtual-pointer Aug 27, 2024

YaLTeR force-pushed the main branch from 62f2af1 to a2c8553 Compare September 4, 2024 09:39

sodiboo force-pushed the support-lan-mouse branch 3 times, most recently from bfd9d78 to 62982c4 Compare September 7, 2024 19:53

sodiboo marked this pull request as ready for review September 7, 2024 21:08

YaLTeR reviewed Oct 6, 2024

View reviewed changes

sodiboo force-pushed the support-lan-mouse branch from 62982c4 to 234b6b5 Compare October 6, 2024 13:25

sodiboo force-pushed the support-lan-mouse branch 2 times, most recently from 33cdeac to e5298b7 Compare October 20, 2024 21:16

sodiboo force-pushed the support-lan-mouse branch 8 times, most recently from d19f77f to 2febcdc Compare November 23, 2024 20:22

YaLTeR linked an issue Nov 23, 2024 that may be closed by this pull request

Implement keyboard-shortcuts-inhibit-unstable-v1 #198

Closed

sodiboo force-pushed the support-lan-mouse branch from 4b29f6d to 9e8a5de Compare November 23, 2024 21:36

sodiboo force-pushed the support-lan-mouse branch from 9e8a5de to f337a27 Compare December 25, 2024 18:35

YaLTeR requested changes Jan 5, 2025

View reviewed changes

sodiboo force-pushed the support-lan-mouse branch 2 times, most recently from 4189442 to 5440002 Compare January 12, 2025 19:07

sodiboo added 6 commits January 18, 2025 09:26

stub keyboard-shortcuts-inhibit and virtual-pointer impls

5deb1d6

implement keyboard-shortcuts-inhibit

3ff900e

implement virtual-pointer

79f2205

deal with supressed key release edge-case; add allow-inhibiting property

a7fc24b

add toggle-keyboard-shortcuts-inhibit bind

ef8d527

add InputBackend extensions; use Device::output() for absolute pos ev…

8f4f644

…ents

YaLTeR force-pushed the support-lan-mouse branch from 5440002 to d8876d4 Compare January 18, 2025 06:46

sodiboo force-pushed the support-lan-mouse branch from d8876d4 to 9fa7423 Compare January 18, 2025 08:38

sodiboo and others added 2 commits January 18, 2025 09:51

add a State parameter to the backend exts and better document futur…

57c7a06

…e intent

Add some tests for is_inhibiting_shortcuts

a8d273d

sodiboo force-pushed the support-lan-mouse branch from 9fa7423 to a8d273d Compare January 18, 2025 08:51

YaLTeR merged commit 0584dd2 into YaLTeR:main Jan 18, 2025
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

sodiboo commented Aug 26, 2024

YaLTeR left a comment

YaLTeR Aug 27, 2024

sodiboo Sep 7, 2024

YaLTeR Oct 6, 2024

sodiboo commented Sep 7, 2024

YaLTeR left a comment

YaLTeR Oct 6, 2024

sodiboo Nov 23, 2024

YaLTeR Nov 28, 2024

sodiboo Nov 28, 2024

YaLTeR Jan 5, 2025

YaLTeR Oct 6, 2024

sodiboo Nov 23, 2024

YaLTeR Nov 28, 2024

YaLTeR Oct 6, 2024

sodiboo commented Oct 6, 2024

ConsularParadi commented Oct 19, 2024

YaLTeR commented Oct 19, 2024

sodiboo commented Oct 19, 2024

sodiboo commented Nov 8, 2024

sodiboo commented Nov 23, 2024

salman-farooq-sh commented Dec 8, 2024

salman-farooq-sh commented Dec 25, 2024

YaLTeR commented Jan 2, 2025

sodiboo commented Jan 3, 2025

YaLTeR left a comment

YaLTeR Jan 5, 2025

YaLTeR commented Jan 18, 2025

sodiboo commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

sodiboo commented Jan 18, 2025

sodiboo commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

	Mod+Shift+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }
	Mod+Escape allow-inhibiting=false { toggle-keyboard-shortcuts-inhibit; }

implement keyboard-shortcuts-inhibit and wlr-virtual-pointer #630

implement keyboard-shortcuts-inhibit and wlr-virtual-pointer #630

Conversation

sodiboo commented Aug 26, 2024

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sodiboo commented Sep 7, 2024

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sodiboo commented Oct 6, 2024

ConsularParadi commented Oct 19, 2024

YaLTeR commented Oct 19, 2024

sodiboo commented Oct 19, 2024

sodiboo commented Nov 8, 2024

sodiboo commented Nov 23, 2024

salman-farooq-sh commented Dec 8, 2024

salman-farooq-sh commented Dec 25, 2024

YaLTeR commented Jan 2, 2025

sodiboo commented Jan 3, 2025

YaLTeR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YaLTeR commented Jan 18, 2025

sodiboo commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

sodiboo commented Jan 18, 2025

sodiboo commented Jan 18, 2025

YaLTeR commented Jan 18, 2025

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630

implement `keyboard-shortcuts-inhibit` and `wlr-virtual-pointer` #630