Chromium

Plumbing for proximate character bounds

To enable gesture support for StylusHandwritingWin, the browser needs
to be aware of the most up-to-date character bounds from the renderer.
However, it may be expensive both to compute and cache all of the
character bounding boxes for very large editable text fields, for
example Google Docs or Microsoft Word.

As a compromise, only the most relevant character bounding boxes will
be collected, called "proximate character bounds". These bounds will
only be needed when stylus inputs result in handwriting, and they'll
only be needed for a brief window of time. Additionally, the bounds do
not need to be updated after collection, since StylusHandwritingWin
will only query a burst of character bounds once after the browser
responds with `ITfFocusHandwritingTargetArgs::SetResponse` when TSF
focus was successfully established.

This CL includes the logical plumbing for this mechanism, adding method
stubs that will be filled out in separate follow-up CLs.

 Does not contain any logical changes that require new tests.

Low-Coverage-Reason: TRIVIAL_CHANGE This CL only contains method stubs
Bug: 355578906
Change-Id: Idfb316360fd590627c9647c2d507fbb3db8c840d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5850661
Reviewed-by: danakj <danakj@chromium.org>
Commit-Queue: Adam Ettenberger <Adam.Ettenberger@microsoft.com>
Reviewed-by: Peter Boström <pbos@chromium.org>
Reviewed-by: Dave Tapuska <dtapuska@chromium.org>
Reviewed-by: Yaroslav Shalivskyy <yshalivskyy@microsoft.com>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1369588}

Files:

• M components/input/render_widget_host_view_input.h
• M components/stylus_handwriting/win/features.cc
• M content/browser/renderer_host/render_widget_host_impl.cc
• M content/browser/renderer_host/render_widget_host_impl.h
• M content/browser/renderer_host/render_widget_host_view_android.cc
• M content/browser/renderer_host/render_widget_host_view_android.h
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/browser/renderer_host/render_widget_host_view_aura.h
• M content/browser/renderer_host/text_input_manager.cc
• M content/browser/renderer_host/text_input_manager.h
• M third_party/blink/public/mojom/page/widget.mojom
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.cc
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.h
• M ui/base/ime/dummy_text_input_client.cc
• M ui/base/ime/dummy_text_input_client.h
• M ui/base/ime/fake_text_input_client.cc
• M ui/base/ime/fake_text_input_client.h
• M ui/base/ime/text_input_client.h
• M ui/base/ime/win/BUILD.gn
• A ui/base/ime/win/DEPS
• M ui/base/ime/win/tsf_text_store.cc
• M ui/base/ime/win/tsf_text_store_unittest.cc
• M ui/views/controls/prefix_selector.cc
• M ui/views/controls/prefix_selector.h
• M ui/views/controls/textfield/textfield.cc
• M ui/views/controls/textfield/textfield.h

Hash: 23b345bb4bb5b8a91e97f7fced61631fa3657021
Date:  Wed Oct 16 20:03:39 2024

Implement Shell Handwriting API controller

New class StylusHandwritingControllerWin. Per-thread singleton.
The class encapsulates the Shell Handwriting API. Receives signals from
the application to notify the Shell Handwriting API of relevant
interactions, and forwards information back to the renderer process to
perform hit testing and focus events.

The controller will be lazily initialized only on the pen mouse move
event to reduce the performance cost during the startup.

Added initialization of stylus_handwriting_enabled web contents pref
based on the feature flag value. API availability checks will guard
against processing handwriting-related touch events in case the
controller hasn't been able to initialize successfully. Covered
existing methods and member variables
(WebContentsImpl::SetStylusHandwritingEnabled() and
stylus_handwriting_enabled_) with android only buildflag as we don't
intend to use them on Windows.

More tests will be added in follow-up CLs as more functionality is
implemented.

Bug: 355578906
Change-Id: I1c9a849c28e5386265b5e6fa0a9fde3f428bb942
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5805258
Reviewed-by: Robert Flack <flackr@chromium.org>
Commit-Queue: Yaroslav Shalivskyy <yshalivskyy@microsoft.com>
Reviewed-by: Dave Tapuska <dtapuska@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1371604}

Files:

• M content/browser/BUILD.gn
• M content/browser/DEPS
• A content/browser/renderer_host/input/stylus_handwriting_controller_win.cc
• A content/browser/renderer_host/input/stylus_handwriting_controller_win.h
• A content/browser/renderer_host/input/stylus_handwriting_controller_win_browsertest.cc
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/browser/renderer_host/render_widget_host_view_aura.h
• M content/browser/web_contents/web_contents_impl.cc
• M content/browser/web_contents/web_contents_impl.h
• M content/test/BUILD.gn

Hash: d76a81f4157061cecc8bd12c6a67fa188fbff4f2
Date:  Mon Oct 21 20:08:20 2024

Add stylus handwriting event properties to ui::TouchEvent 
 
The CL adds stylus handwriting event properties, 
handwriting_pointer_id and handwriting_stroke_id, to 
ui::TouchEvent. Both properties are essential for 
communication with ShellHandwriting API to request 
handwriting for a pointer and initiate the experience. 
 
The handwriting pointer id is an unmapped pointer id 
supplied by the OS. The handwriting stroke id is supplied by 
the OS based on the pointer id. The browser calls 
`GetHandwritingStrokeIdForPointer()` from 
shellhandwriting.h to retrieve the id for the given pointer. 
 
Both fields are intended to stay within the browser process 
and not to be shared with other processes or used outside 
stylus handwriting feature context. 
 
Bug: 355578906 
Change-Id: I7768586f9b5d6d4f2ee80923e64c5ccd18005da8 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5912139 
Reviewed-by: Allen Bauer <kylixrd@chromium.org> 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org> 
Commit-Queue: Yaroslav Shalivskyy <yshalivskyy@microsoft.com> 
Cr-Commit-Position: refs/heads/main@{#1372776}

Files:

• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/browser/renderer_host/render_widget_host_view_aura.h
• M ui/events/BUILD.gn
• M ui/events/event.cc
• M ui/events/event.h
• M ui/events/event_utils.h
• A ui/events/win/stylus_handwriting_properties_win.cc
• A ui/events/win/stylus_handwriting_properties_win.h
• A ui/events/win/stylus_handwriting_properties_win_unittest.cc
• M ui/views/BUILD.gn
• M ui/views/win/DEPS
• M ui/views/win/pen_event_processor.cc
• M ui/views/win/pen_event_processor.h
• M ui/views/win/pen_event_processor_unittest.cc

Hash: 652a3b6bece5dadfb98591fe79a8e99e56e9965b
Date:  Wed Oct 23 16:45:55 2024

Implement ComputeProximateCharacterBounds 
 
+ WebFrameWidgetImpl::ComputeProximateCharacterBounds 
  Implementing this to collect `ProximateBoundsCollectionHalfLimit()` 
  character bounds in each direction of the `pivot_position`. 
 
+ stylus_handwriting::win::ProximateBoundsCollectionHalfLimit() 
  This will be used to remotely tune how many characters are collected 
  and cached for gesture support by ShellHandwritingWin. 
  This affects performance of CPU and memory usage, as well as the 
  reliability of handwriting gestures on Windows. 
  Bounds will be consumed by the following external API callers, 
  `ITextStoreACP::GetTextExt` and `ITextStoreACP::GetACPFromPoint`, 
  in a separate CL. 
 
+ RenderWidgetHostViewAura::OnFocusHandwritingTarget 
  This method is the `ITfHandwritingSink::FocusHandwritingTarget` 
  callback and will be connected with a separate CL that implements the 
  API boundary. For now, illustrates how Windows Aura will call 
  `RenderInputRouterClient::UpdateElementFocusForStylusWriting`. 
 
+ ::mojom::blink::FrameWidget::OnStartStylusWriting 
  Updating the signature to accept an optional `focus_rect_in_widget`. 
  This is used by Windows implementation to: 
  1. Specify an area in widget coordinates to be focused. When a valid 
     stylus writing target cannot be determined with this rect, will 
     fallback to `CurrentTouchDownElement()`. 
  2. Used to derive a "pivot" position for "proximate bounds" 
     collection, independent of caret placement. 
  3. (TBD) If [1] results in a focus change, the pivot position will be 
     used to update the caret position. If the target element already 
     has focus, the caret position will be unchanged. 
 
+ WebFrameWidgetImpl::OnStartStylusWriting 
  Implementing FrameWidget::OnStartStylusWriting points [1] and [2] 
  from the notes above. 
 
Bug: 355578906 
Change-Id: Ia10cd48cda97c12b91b876786855581e5ea6fe63 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5857350 
Reviewed-by: Dominic Farolino <dom@chromium.org> 
Reviewed-by: Kentaro Hara <haraken@chromium.org> 
Commit-Queue: Adam Ettenberger <Adam.Ettenberger@microsoft.com> 
Reviewed-by: Antonio Sartori <antoniosartori@chromium.org> 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Cr-Commit-Position: refs/heads/main@{#1375609}

Files:

• M components/input/render_input_router_client.h
• M components/stylus_handwriting/win/features.cc
• M components/stylus_handwriting/win/features.h
• M content/browser/renderer_host/render_widget_host_impl.cc
• M content/browser/renderer_host/render_widget_host_impl.h
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/public/test/fake_frame_widget.h
• M third_party/blink/public/mojom/page/widget.mojom
• M third_party/blink/renderer/core/BUILD.gn
• M third_party/blink/renderer/core/frame/DEPS
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.cc
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.h
• M third_party/blink/renderer/core/frame/web_frame_widget_test.cc
• M third_party/blink/tools/blinkpy/presubmit/audit_non_blink_usage.py

Hash: 72d7972b10365068a2771d6a3d79f720d563b3c2
Date:  Wed Oct 30 00:47:40 2024

Improve StylusHandwritingControllerWin browser tests infrastructure 
 
The CL implements mocks for some Text Framework's (ITf) 
interfaces to replace the StylusHandwritingControllerWin mock 
class. This increases the test coverage and improves the reusability 
of the testing infrastructure across multiple test files. 
 
Important changes: 
 
1. Instead of creating a mock for the whole controller, we now mock 
the thread manager. The `g_instance_for_testing` static pointer is no 
longer needed and has been replaced with 
`g_thread_manager_instance_for_testing`. 
 
2. A scoped closed runner was added to reset the mock thread 
manager and the static controller instances after each test. This 
prevents state and memory leakages across the tests, hides the 
implementation details from the caller, and decreases the number 
of the controller's friend classes. 
 
Bug: 355578906 
Change-Id: I1064f3ef0d604fb26a9652307c8ed33111bc6647 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5991877 
Reviewed-by: Adam Ettenberger <Adam.Ettenberger@microsoft.com> 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Commit-Queue: Yaroslav Shalivskyy <yshalivskyy@microsoft.com> 
Cr-Commit-Position: refs/heads/main@{#1380678}

Files:

• A content/browser/renderer_host/input/mock_tfhandwriting.cc
• A content/browser/renderer_host/input/mock_tfhandwriting.h
• M content/browser/renderer_host/input/stylus_handwriting_controller_win.cc
• M content/browser/renderer_host/input/stylus_handwriting_controller_win.h
• M content/browser/renderer_host/input/stylus_handwriting_controller_win_browsertest.cc
• M content/test/BUILD.gn

Hash: a390c22d018da5c6567fd18b02ea06c60439d583
Date:  Fri Nov 08 23:07:02 2024

Implement GetProximateCharacter *Bounds and *IndexFromPoint 
 
Implementing the RenderWidgetHostViewAura version of 
GetProximateCharacterBounds and GetProximateCharacterIndexFromPoint 
when pulling from the "proximate" character bounds cache. 
 
GetProximateCharacterBounds is meant to satisfy requirements 
outlined for ITextStoreACP::GetTextExt. 
 
If all character bounding boxes in the range [`acp_start`, `acp_end`) 
are available, combines them into one bounding box that encapsulates 
the entire range, otherwise returns `std::nullopt` which the caller 
handles by throwing `TS_E_NOLAYOUT` since it's unclear whether the 
provided range went out-of-bounds for the actual text range or only 
the cached "proximate" range. 
 
Chromium's `GetTextExt` implementation currently doesn't support the 
OUT parameter `clipped`, and so far doesn't seem to be necessary for 
StylusHandwritingWin. This can be added in a separate CL if any issues 
come up related to that parameter. 
 
GetProximateCharacterIndexFromPoint is meant to satisfy requirements 
outlined for ITextStoreACP::GetACPFromPoint. 
 
Depending on which `ui::IndexFromPointFlags` is provided, finds an 
appropriate character offset from the "proximate" character bounds 
cache, or returns `std::nullopt` if one cannot be determined which the 
caller handles by throwing `TS_E_INVALIDPOINT`. 
 
Bug: 355578906 
Change-Id: Ia4c81dcd0be7676c5639da2a8d39c1c113cc7b38 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5871295 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Reviewed-by: Yaroslav Shalivskyy <yshalivskyy@microsoft.com> 
Reviewed-by: Nico Weber <thakis@chromium.org> 
Commit-Queue: Adam Ettenberger <Adam.Ettenberger@microsoft.com> 
Cr-Commit-Position: refs/heads/main@{#1387028}

Files:

• M content/browser/renderer_host/DEPS
• M content/browser/renderer_host/input/stylus_handwriting_controller_win.cc
• M content/browser/renderer_host/input/stylus_handwriting_controller_win_browsertest.cc
• A content/browser/renderer_host/input/stylus_handwriting_win_test_helper.cc
• A content/browser/renderer_host/input/stylus_handwriting_win_test_helper.h
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/browser/renderer_host/render_widget_host_view_aura_unittest.cc
• M content/test/BUILD.gn
• M ui/base/ime/text_input_client.cc
• M ui/base/ime/text_input_client.h
• M ui/base/ime/win/tsf_text_store.cc

Hash: 043fffd131f7254b1dd0b9eca26951ac8c998c37
Date:  Fri Nov 22 20:19:47 2024

Implement StylusHandwritingCallbackSinkWin 
 
New class StylusHandwritingCallbackSinkWin implements 
ITfHandwritingSink interface from shellhandwriting.h. 
 
Shell Handwriting API will call ::FocusHandwritingTarget() requesting to 
set focus to the most likely text control. To handle 
ITfHandwritingSink::FocusHandwritingTarget, the browser will request 
that focus be updated in the renderer based on the RECT from 
ITfFocusHandwritingTargetArgs::GetPointerTargetInfo, which will be 
handled via mojom::blink::FrameWidget::OnStartStylusWriting. Focus will 
only be set on content eligible for handwriting. If focus cannot be set 
on content eligible for handwriting with the RECT provided by 
GetPointerTargetInfo, then focus will fallback to the eligible element 
that was initially tapped. 
 
::OnFocusedHandled()/::OnFocusFailed() notifies the Shell Handwriting 
API whether the focus was set or not. If the browser fails to set the 
focus, the handwriting will be abandoned and Shell Handwriting will 
stop processing incoming pointer input events. 
 
Also, the change improves handwriting controller browser tests by 
utilizing new mocks of handwriting-related interfaces. This enables 
testing the controller without the mocking the controller instance. 
 
Bug: 355578906 
Change-Id: I86e637ced9fbc679f11e65c66053ba7d54321577 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5882132 
Commit-Queue: Yaroslav Shalivskyy <yshalivskyy@microsoft.com> 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Cr-Commit-Position: refs/heads/main@{#1390517}

Files:

• M content/browser/BUILD.gn
• M content/browser/DEPS
• M content/browser/renderer_host/input/mock_tfhandwriting.cc
• M content/browser/renderer_host/input/mock_tfhandwriting.h
• A content/browser/renderer_host/input/stylus_handwriting_callback_sink_win.cc
• A content/browser/renderer_host/input/stylus_handwriting_callback_sink_win.h
• M content/browser/renderer_host/input/stylus_handwriting_controller_win.cc
• M content/browser/renderer_host/input/stylus_handwriting_controller_win.h
• M content/browser/renderer_host/input/stylus_handwriting_controller_win_browsertest.cc
• M content/browser/renderer_host/input/stylus_handwriting_win_test_helper.cc
• M content/browser/renderer_host/input/stylus_handwriting_win_test_helper.h
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/test/content_test_bundle_data.filelist
• A content/test/data/stylus_handwriting_test.html
• M ui/events/test/event_generator.cc
• M ui/events/test/event_generator.h
• M ui/events/win/stylus_handwriting_properties_win.cc
• M ui/events/win/stylus_handwriting_properties_win.h

Hash: a46bd534d1d18c5c27648028780a6c489dc14506
Date:  Mon Dec 02 19:42:53 2024

StylusHandwritingWin fix coordinate space bug (Widget DIP vs Physical) 
 
- WebFrameWidgetImpl::OnStartStylusWriting 
  Add a missing coordinate-space transformation from DIP widget space 
  to Blink Space. This fixes a hit testing issue when using different 
  device display scaling. 
 
- TSFTextStore::GetACPFromPoint 
  RenderWidgetHostViewAura::GetProximateCharacterIndexFromPoint 
  Instead of converting each proximate bounding box from a (cached) 
  DIP widget space rect to a physical screen rect, converting the 
  single API point from physical screen space to DIP widget space. 
 
Renaming a bunch of stylus handwriting specific function arguments to 
reflect which coordinate space they expect for point and rect inputs. 
 
Bug: 355578906 
Change-Id: I719bbcfa1057187391abbbb24429d0b118204c01 
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6128720 
Reviewed-by: Dana Fried <dfried@chromium.org> 
Commit-Queue: Adam Ettenberger <Adam.Ettenberger@microsoft.com> 
Reviewed-by: Mustafa Emre Acer <meacer@chromium.org> 
Reviewed-by: Dave Tapuska <dtapuska@chromium.org> 
Cr-Commit-Position: refs/heads/main@{#1405629}

Files:

• M components/input/render_input_router_client.h
• M content/browser/renderer_host/input/stylus_handwriting_callback_sink_win.cc
• M content/browser/renderer_host/render_widget_host_impl.cc
• M content/browser/renderer_host/render_widget_host_impl.h
• M content/browser/renderer_host/render_widget_host_view_aura.cc
• M content/browser/renderer_host/render_widget_host_view_aura.h
• M content/public/test/fake_frame_widget.h
• M third_party/blink/public/mojom/page/widget.mojom
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.cc
• M third_party/blink/renderer/core/frame/web_frame_widget_impl.h
• M third_party/blink/renderer/core/frame/web_frame_widget_test.cc
• M ui/base/ime/dummy_text_input_client.cc
• M ui/base/ime/dummy_text_input_client.h
• M ui/base/ime/fake_text_input_client.cc
• M ui/base/ime/fake_text_input_client.h
• M ui/base/ime/text_input_client.h
• M ui/base/ime/win/tsf_text_store.cc
• M ui/views/controls/prefix_selector.cc
• M ui/views/controls/prefix_selector.h
• M ui/views/controls/textfield/textfield.cc
• M ui/views/controls/textfield/textfield.h

Hash: 8b4b8bda97555498afdf48544e8354f3d6600f43
Date:  Mon Jan 13 11:31:22 2025