WAI

Bug

Status Update

No update yet.

Description

ch...@google.com

created issue on behalf of ko...@appmospheres.com #1

Aug 28, 2024 10:59AM

Steps to reproduce the problem

run the benchmark of https://github.com/palapala-app/palapala_pdf (in examples) against chrome 127.x vs latest 128.0.6613.85. it performs pdf generation from html content using the cdp protocol (ws based). it's a performance regression issue.

Problem Description

the performance of pdf generation over websocket based CDP has degraded seriously with the last upgrade, from an order of 500 docs/s to about 30 docs/s on a mac m3. this was confirmed by doing the same on a mac m1 with the same order of difference, we still had an older version of chrome on that machine to be able to compare.

Additional Comments

you can reach me also at koen.handekyn@gmail.com

Summary

serious performance degradation of pdf generation over CDP (a factor 20x slower)

Additional Data

Category: Developer Tools
Chrome Channel: Not sure
Regression: Yes

Comments

ko...@appmospheres.com <ko...@appmospheres.com> #2Aug 28, 2024 03:36PM

some good news

when using the latest greatest chrome headless mac_arm-130.0.6682.2 somehow it's fast again, so not sure if the issue was already fixed or that the issue does not occur with the headless builds

/chrome-headless-shell/mac_arm-130.0.6682.2/chrome-headless-shell-mac-arm64/chrome-headless-shell --remote-debugging-port=9222

ruby examples/performance_benchmark.rb
warmup
c:1, n:10 : Throughput = 107.46 docs/sec, Total time = 0.0931 seconds
benchmarking 20 docs: 1x20, 2x10, 4x5, 5x4, 20x1
c:1, n:20 : Throughput = 225.33 docs/sec, Total time = 0.0888 seconds
c:2, n:10 : Throughput = 209.71 docs/sec, Total time = 0.0954 seconds
c:4, n:5 : Throughput = 148.0 docs/sec, Total time = 0.1351 seconds
benchmarking 320 docs
c:1, n:320 : Throughput = 539.61 docs/sec, Total time = 0.593 seconds
c:2, n:160 : Throughput = 690.69 docs/sec, Total time = 0.4633 seconds
c:4, n:80 : Throughput = 785.5 docs/sec, Total time = 0.4074 seconds
c:8, n:40 : Throughput = 651.49 docs/sec, Total time = 0.4912 seconds

ko...@appmospheres.com <ko...@appmospheres.com> #3Aug 28, 2024 03:51PM

also this version is fine ./chrome-headless-shell/mac_arm-128.0.6613.84/chrome-headless-shell-mac-arm64/chrome-headless-shell

similar througput

ruby examples/performance_benchmark.rb
warmup
c:1, n:10 : Throughput = 146.97 docs/sec, Total time = 0.068 seconds
benchmarking 20 docs: 1x20, 2x10, 4x5, 5x4, 20x1
c:1, n:20 : Throughput = 232.83 docs/sec, Total time = 0.0859 seconds
c:2, n:10 : Throughput = 207.67 docs/sec, Total time = 0.0963 seconds
c:4, n:5 : Throughput = 145.54 docs/sec, Total time = 0.1374 seconds
benchmarking 320 docs
c:1, n:320 : Throughput = 539.5 docs/sec, Total time = 0.5931 seconds
c:2, n:160 : Throughput = 692.36 docs/sec, Total time = 0.4622 seconds
c:4, n:80 : Throughput = 786.01 docs/sec, Total time = 0.4071 seconds
c:8, n:40 : Throughput = 683.85 docs/sec, Total time = 0.4679 seconds

so the issue seems very specific for the mentioned version in the ticket

bu...@google.com <bu...@google.com> #4Aug 30, 2024 06:09AM

@Reporter, Thanks for filing the Issue.

Could you please elaborate on the same by providing Screen Cast for actual/expected behaviour with clear steps, it will helpful for further triage.

Thanks.!!

ko...@appmospheres.com <ko...@appmospheres.com> #5Aug 30, 2024 10:53AM

hey

a movie won't do a lot as it's performance related and the video just will show the 'timings' of the above

but let me explain again in detail my findings

i have now test bench that runs very slow as from chrome 128.0.6613.84/85 while any older version runs fast, e.g. chrome@127.0.6533.119 is still fast
we are talking about more then 10x slower !!! so it's not small difference ...

strangely, the chrome-headless-shell variants don't seem to suffer from it, e.g. chrome-headless-shell/mac_arm-128.0.6613.84 and newer
but also the normal chrome is started in headless mode

the test focusses purely on PDF generation, using toPDF over CDP in a loop basically

my conclusion is that there is a regression issue that impacts performance heavily but that only appears in the normal chrome builds even if run in headless mode while the pure headless shell build doesn't suffer from it for some reason

further: i've also noted that the headless=new mode slows down pdf generation

regards
koen

ko...@appmospheres.com <ko...@appmospheres.com> #6Aug 30, 2024 10:54AM

the test is written in ruby that drives the CDP, i can share the test bench / github repo if that would be helpfull or we can do a screenshare

pe...@google.com <pe...@google.com> #7Aug 30, 2024 10:57AM

Thank you for providing more feedback. Adding the requester to the CC list.

bu...@chromium.org <bu...@chromium.org> #8Sep 2, 2024 08:13AM

As per

comment #1

&#5, Unable to Triage the Issue from TE End. Hence, adding "TE-NeedsTriageHelp" hotlist. Someone from Dev Team could you please take a look into this issue.

Thanks!

bm...@google.com <bm...@google.com> #9Sep 9, 2024 07:54AM

Assigned to al...@google.com.

Alex, can you take a look please?

pe...@google.com <pe...@google.com> #10Sep 9, 2024 07:57AM

All Chrome DevTools regression bugs must be P0 or P1. Changing this bug to P1 because it is marked as a regression. See go/chrome-devtools/regressions for details about the process.

al...@google.com <al...@google.com> #11Sep 16, 2024 08:21AM

Could it be caused by the new headless (same as headful) being the default mode now? chrome-headless-shell and chrome --headless=old should be the same.

al...@google.com <al...@google.com> #12Sep 16, 2024 08:25AM

Status: Won't Fix (Intended Behavior)

Note that the default headless mode is new from https://chromium-review.googlesource.com/c/chromium/src/+/5226762 (since 128.0.6543.0)

The performance for PDF generation in the new headless mode should be the same as in the headful mode. If you need to keep using the old headless mode, the recommended way is to use the chrome-headless-shell binary.