Chromium

From the Android system logs:
Line 18473: 07-31 10:39:48.706  7781  7895 E chromium: [ERROR:video_stream_encoder.cc(1307)] Failed to initialize the encoder associated with codec type: H264 (4)
        Line 18474: 07-31 10:39:48.706  7781  7895 E chromium: [ERROR:video_stream_encoder.cc(1336)] Failed to configure encoder.
        Line 18475: 07-31 10:39:48.706  7781  7895 W chromium: [WARNING:video_stream_encoder.cc(1433)] Failed to initialize H264 encoder.switch_encoder_on_init_failures: 1
        Line 18476: 07-31 10:39:48.706  7781  8323 W chromium: [WARNING:webrtc_video_engine.cc(937)] Failed to switch encoder to: Codec name: VP8, parameters: { }. Is default fallback allowed: 1
        Line 18477: 07-31 10:39:48.706  7781  8323 W chromium: [WARNING:webrtc_video_engine.cc(893)] Encoder failed but no fallback codec is available

All our Microsoft group calling experience on web is affected. Skype Web, ACS calling SDK users are not able to send video.

Looks like this bug  fix broke the H264 encoder
https://bugs.chromium.org/p/chromium/issues/detail?id=1448816

Confused - does Android not have H264 SW support?

H264 SW may require additional build flags than that of open source chromium builds, but IIUC this is a problem on official builds as well?

We could exclude Android from the 360p SW fallback path if that helps since mobile and non-mobile may have different codec support

The actual interesting line is the first here:
07-31 10:39:36.971  7781  7895 W chromium: [WARNING:rtc_video_encoder.cc(1729)] Fallback to SW due to low resolution being less than 360p (180x320)
07-31 10:39:36.971  1318 26571 I OMX-VDEC-1080P: omx_vdec::component_init() success : fd=13
07-31 10:39:36.971  7781  7895 E chromium: [ERROR:video_stream_encoder.cc(1307)] Failed to initialize the encoder associated with codec type: H264 (4)
07-31 10:39:36.971  7781  7895 E chromium: [ERROR:video_stream_encoder.cc(1336)] Failed to configure encoder.
07-31 10:39:36.971  7781  7895 W chromium: [WARNING:video_stream_encoder.cc(1433)] Failed to initialize H264 encoder.switch_encoder_on_init_failures: 1
07-31 10:39:36.971  7841 12177 I ACodec  : [OMX.qcom.video.decoder.avc] Now Loaded
07-31 10:39:36.971  7781  8323 W chromium: [WARNING:webrtc_video_engine.cc(937)] Failed to switch encoder to: Codec name: VP8, parameters: { }. Is default fallback allowed: 1
07-31 10:39:36.971  7781  8323 W chromium: [WARNING:webrtc_video_engine.cc(893)] Encoder failed but no fallback codec is available

Both amazed and terrified that fallback to a different codec works... :)

E.g. #if !BUILDFLAG(IS_ANDROID) around this block:
https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/platform/peerconnection/rtc_video_encoder.cc;l=1697;drc=c837f06db70d3942f82989eca70ed51f7ddfc8d2

[Monorail components: -Blink>WebRTC Blink>WebRTC>Video]

Did it work on 114?

this is an official build we are testing :)

it did work on 114

hbos: can you trigger the killswitch for Android in M115 and M116 and exclude Android in M117 where the killswitch is removed?

[Empty comment from Monorail migration]

This issue is extremely serious and impacting many customers including in critical situations like healthcare. Can we please get confirmation whether this is being rolled back, and an approximate timeframe to communicate to impacted customers?

We've discovered that the sample linked in the description never worked.

To repro one needs to have H264 + resolution <360p

Edge has deployed a config to our Android clients to turn off ForceSoftwareForLowResolutions. We've confirmed it's addressed the customer-observed issue.

As noted above, the original repro does not appear to actually map to the underlying issue and should be ignored. I will defer to other folks at Microsoft to get you more actionable repro context if needed.

Sounds quite serious, raising to P0 and looping in Android release TPM (govind).

I was at first wondering why we were only hearing about this now, two weeks after 115 hitting stable on Android. But now I remember there was a delay in ramping M115 up on Android due to some other issues. I see we hit the >50% traffic point only on Saturday July 29, so for work-focused scenarios like teams video chat, it makes sense that reports would have first started coming in on Monday.

Flipping the killswitch for Android specifically makes sense to me. It's past EOD for the WebRTC team now but I'll see if I can find anyone on the west coast who can help.

Thank you for adding me.
Yeah, it is very late for WebRTC team, would be great if we can find someone in west coast.

Is this issue specific to Chrome on Android + M115 ONLY? And No impact on Android Webview?

Adding "RBS" for tracking.

+Candrada@ for test repro on M115.

[Empty comment from Monorail migration]

Good question about WebView, I'd expect it to be similarly impacted. I've tried but failed to reproduce this. I can create a teams video chat in Chrome Macos m116 which seems to work fine, but when I try to open the videochat link on my phone it just directs me to the play store to download the app. Installing the app (maybe the concern is strictly WebView only?) I just get stuck in an auth loop trying to sign into my microsoft account over and over again. Perhaps others will have better luck with a repro. Probably need to wait for WebRTC team to wake up tomorrow to proceed further.

Thank you Rick.

Requested repro for Webview test team as well.

[Empty comment from Monorail migration]

[Empty comment from Monorail migration]

I was able to successfully start a video chat in the teams Android app (with Webview M116) and share video with teams running in Chrome MacOS M116. So I don't think this is about the native app. Maybe there's some scenarios where teams uses mobile web instead of the app? Any chance this error is covered by UMA metrics so we can get a count of it's occurence?

I also tried using a desktop UA string on Chrome Android to get the desktop web app. That seemed to work fine. Maybe only an issue if the connection is slow enough to downgrade to less than 360p?

We should disable the fallback path on all of Android to be safe and backmerge this. We should not disable it on other platforms where this isn't a problem - it only occurs in situations where the codec is supported in HW but not in SW, it seems, which is a very rare situation (e.g. not the case on desktop).

I'll make a CL as per https://crbug.com/chromium/1469318#c7

> Good question about WebView, I'd expect it to be similarly impacted.
Yes, WebView is impacted as well

> so for work-focused scenarios like teams video chat, it makes sense that reports would have first started coming in on Monday.
Yes, that's what we see in telemetry too, the first bigger impact (aka users actually used M115 for their Teams/ACS workloads) for work scenarios came on Monday

> I was able to successfully start a video chat in the teams Android app (with Webview M116) and share video with teams running in Chrome MacOS M116. So I don't think this is about the native app.
Broken scenario involves running Teams in a mobile browser. This scenario can be triggered by scheduling a meeting and using Virtual Appointment as a template. This would generate a join link/invite that will not prompt to install a mobile app.

> We should disable the fallback path on all of Android to be safe and backmerge this.
@hbos, can we land a killswitch for ForceSoftwareForLowResolutions in the meanwhile, to mitigate the impact?

I will both backmerge the fix and do a GCL config for triggering the killswitch. If the killswitch can land before or has to land after the backmerges depends on the Finch reviewers, but I'll get the ball rolling today.

[Empty comment from Monorail migration]

[Empty comment from Monorail migration]

[Empty comment from Monorail migration]

Backing down off the P0. That's a "trigger the pagers" level.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src/+/3770ccf1732eab5ea8009c9ce8abcd298ec5bab3

commit 3770ccf1732eab5ea8009c9ce8abcd298ec5bab3
Author: Henrik Boström <hbos@chromium.org>
Date: Wed Aug 02 13:08:30 2023

Exclude Android from SW fallback path.

It was discovered that on Android, SW fallback can change codec, because
H264 is only available in HW. This is a P-0 regression, so we disable
the SW fallback path on this platform.

Bug: chromium:1469318
Change-Id: I4eda1c3ec8b9457515756f2ebf4ae86095503010
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4738001
Commit-Queue: Henrik Boström <hbos@chromium.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1178383}

[modify] https://crrev.com/3770ccf1732eab5ea8009c9ce8abcd298ec5bab3/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder.cc
[modify] https://crrev.com/3770ccf1732eab5ea8009c9ce8abcd298ec5bab3/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder_test.cc

Requesting to backmerge https://crbug.com/chromium/1469318#c35 (https://chromium.googlesource.com/chromium/src/+/3770ccf1732eab5ea8009c9ce8abcd298ec5bab3) to M116 and M115 on Android/AndroidWebView.

This CL addresses a serious WebRTC regression affecting Microsoft products, https://crbug.com/chromium/1469318#c2: "All our Microsoft group calling experience on web is affected. Skype Web, ACS calling SDK users are not able to send video."

This is not a problem on non-Android platforms where there exists SW fallback codecs for all HW codecs, so the problem only happened on Android.

Closing as Fixed, though https://crbug.com/chromium/1469318#c15 has already verified that disabling this feature fixes the issue, which my CL does using #if !BUILDFLAGs.

This CL is very safe and I will look into triggering the killswitch flag for this feature that was present in the codebase until recently.

[Empty comment from Monorail migration]

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src/+/641124d8c9011e7e8fa43e32110b1a85b83f6ebd

commit 641124d8c9011e7e8fa43e32110b1a85b83f6ebd
Author: Henrik Boström <hbos@chromium.org>
Date: Wed Aug 02 15:11:59 2023

[M117 Canary Merge] Exclude Android from SW fallback path.

It was discovered that on Android, SW fallback can change codec, because
H264 is only available in HW. This is a P-0 regression, so we disable
the SW fallback path on this platform.

(cherry picked from commit 3770ccf1732eab5ea8009c9ce8abcd298ec5bab3)

Bug: chromium:1469318
Change-Id: I4eda1c3ec8b9457515756f2ebf4ae86095503010
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4738001
Commit-Queue: Henrik Boström <hbos@chromium.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@chromium.org>
Cr-Original-Commit-Position: refs/heads/main@{#1178383}
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4742466
Reviewed-by: Krishna Govind <govind@chromium.org>
Owners-Override: Krishna Govind <govind@chromium.org>
Cr-Commit-Position: refs/branch-heads/5926@{#3}
Cr-Branched-From: c49b4ebf359f5a78ea15adc0c25aa20ba7819ad2-refs/heads/main@{#1178349}

[modify] https://crrev.com/641124d8c9011e7e8fa43e32110b1a85b83f6ebd/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder.cc
[modify] https://crrev.com/641124d8c9011e7e8fa43e32110b1a85b83f6ebd/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder_test.cc

Able to repro the issue on M115(115.0.5790.166) . will verify the fix once its landed in tomorrows Canary.

Please verify the fix on Android Canary #117.0.5926.2+ once available later in the afrernoon today.  Thank you.

Requesting a Postmortem for this as it required emergency finch push .

Tracking irm - irm/i_48B9nvJSHG6fZQea8WNi

We have confirmation from Microsoft, they've verified the  finch killswitch config and confirmed Chrome 115 on Android - h264 video works fine.

+Erhu, I  will do M116 merge review for this after canary verification

I am one of the affected customers who raised with Microsoft. I have restarted my phone and validated it now works on chrome 115 android.

Thanks All!

Re #44,  great, thank you  rogers.man@.

Canary #117.0.5926.2 with client side fix (https://chromium-review.googlesource.com/c/chromium/src/+/4742466) went live in play ~8:45 PM PT.

Issue still repro's on latest Canary(117.0.5927.0) with the steps mentioned in C#0.

Please find the video @ http://go/chrome-androidlogs1/14/1469318 

С#0 repro is unfortunately invalid and never worked in the first place (and likely needs to be extracted into a separate issue). We need a simple sample that sends 180p h264 stream for a faster validation.

Merge review required: M116 is already shipping to beta.

Please answer the following questions so that we can safely process your merge request:
1. Why does your merge fit within the merge criteria for these milestones?
- Chrome Browser: https://chromiumdash.appspot.com/branches
- Chrome OS: https://goto.google.com/cros-release-branch-merge-guidelines
2. What changes specifically would you like to merge? Please link to Gerrit.
3. Have the changes been released and tested on canary?
4. Is this a new feature? If yes, is it behind a Finch flag and are experiments active in any release channels?
5. [Chrome OS only]: Was the change reviewed and approved by the Eng Prod Representative? https://goto.google.com/cros-engprodcomponents
6. If this merge addresses a major issue in the stable channel, does it require manual verification by the test team? If so, please describe required testing.

Please contact the milestone owner if you have questions.
Owners: eakpobaro (Android), eakpobaro (iOS), obenedict (ChromeOS), danielyip (Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Merge review required: M115 is already shipping to stable.

Please answer the following questions so that we can safely process your merge request:
1. Why does your merge fit within the merge criteria for these milestones?
- Chrome Browser: https://chromiumdash.appspot.com/branches
- Chrome OS: https://goto.google.com/cros-release-branch-merge-guidelines
2. What changes specifically would you like to merge? Please link to Gerrit.
3. Have the changes been released and tested on canary?
4. Is this a new feature? If yes, is it behind a Finch flag and are experiments active in any release channels?
5. [Chrome OS only]: Was the change reviewed and approved by the Eng Prod Representative? https://goto.google.com/cros-engprodcomponents
6. If this merge addresses a major issue in the stable channel, does it require manual verification by the test team? If so, please describe required testing.

Please contact the milestone owner if you have questions.
Owners: govind (Android), govind (iOS), dgagnon (ChromeOS), pbommana (Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Can someone from Microsoft please help verify on latest Canary (117.0.5927.0 or later)?

Re #51, please use canary version Canary #117.0.5926.2+ for verification as updated at #46. Thank you.

I can not repro the problem in canary  117.0.5926.2 with either my reported problem or the repro steps listed in C#28

Thank you for canary verification.

M116 branch 5845 merge approved. Please merge.

M115 merge rejected.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src/+/cc49e333a1d1468834adb1961e6cefb6283f4b4d

commit cc49e333a1d1468834adb1961e6cefb6283f4b4d
Author: Henrik Boström <hbos@chromium.org>
Date: Fri Aug 04 08:16:47 2023

[Merge-M116] Exclude Android from SW fallback path.

It was discovered that on Android, SW fallback can change codec, because
H264 is only available in HW. This is a P-0 regression, so we disable
the SW fallback path on this platform.

(cherry picked from commit 3770ccf1732eab5ea8009c9ce8abcd298ec5bab3)

Bug: chromium:1469318
Change-Id: I25a8583fb30b15ce408131aa7dd1c4fba985197c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4738001
Commit-Queue: Henrik Boström <hbos@chromium.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@chromium.org>
Cr-Original-Commit-Position: refs/heads/main@{#1178383}
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4742403
Cr-Commit-Position: refs/branch-heads/5845@{#1116}
Cr-Branched-From: 5a5dff63a4a4c63b9b18589819bebb2566c85443-refs/heads/main@{#1160321}

[modify] https://crrev.com/cc49e333a1d1468834adb1961e6cefb6283f4b4d/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder.cc
[modify] https://crrev.com/cc49e333a1d1468834adb1961e6cefb6283f4b4d/third_party/blink/renderer/platform/peerconnection/rtc_video_encoder_test.cc

Re #51, we validated Canary 117.0.5927.0, video in Microsoft Teams/ACS scenarios works just fine. Thanks everyone for a quick turnaround!

Fixed, verified and backmerged to M116. M115 did not get the backmerge for a binary respin, but we did successfully trigger the killswitch on M115 using Finch, so even there it is fixed for endpoints with Finch. https://chromiumdash.appspot.com/schedule says M116 Stable release is on Tue, Aug 15.

Cheers everyone!

[Empty comment from Monorail migration]

This issue was migrated from crbug.com/chromium/1469318?no_tracker_redirect=1

[Auto-CCs applied]
[Monorail blocking: crbug.com/chromium/1469562]
[Monorail components added to Component Tags custom field.]