【トラブルシューティング】管理者がメッセージセンターの通知を大量に受信する SI（サービスインシデント）が発生【MO301479】

Read Time:12 Minute, 40 Second

Microsoft が提供する Microsoft 365 のサービス正常性（MO301479）にて、管理者がメッセージセンター内の通知を大量に受信する SI（サービスインシデント）が発生していたことを発表しています。

Admins may be receiving multiple notifications regarding a Message Center post - MO301479 — Admins may be receiving multiple notifications regarding a Message Center post – MO301479

この通知では、”MC301473″ の “Microsoft OneDrive & SharePoint Library dropdown menu.” に関する通知であった模様です。

なお、本事象は Microsoft 365 Admin アプリケーション内の通知を無効にすることで大量の通知の受信を回避できるとのことです。

また、事象が発生していた管理者ユーザーでは 2021 年 12 月 3 日（金）9：00（12：00 AM UTC）～ 2021 年 12 月 3 日（金）1：00（1：00 AM UTC）の間に “サービス正常性” や “メッセージセンター” の表示またはアクセスが行えなかった可能性もあるとのことです。

現時点では本事象が改善されていることを確認しているものの事象が発生していた要因の調査のため通知を一時的に無効にしているとのことです。

なお、Microsoft 365 Admin アプリケーション経由での “サービス正常性” および “メッセージセンター” の両方で通知の受信が行えなくなる可能性あるもののメールでの通知は引き続き受信が行えるとのことです。

※ この問題は、Microsoft 365 Admin アプリケーションを利用している管理者ユーザーにて通知を有効にしている場合に発生する問題になります。

Microsoft 365 suite 関連記事一覧

・【トラブルシューティング】Microsoft 365 管理センター内にて Microsoft Teams、E メールのストレージ、グループアクティビティレポート等が更新されない問題が発生【MO289071】

Microsoft 365 suite のメッセージセンター関連情報一覧

Microsoft 365 suite のメッセージセンター関連記事一覧

デルオンラインストアでチェックするなら、このボタンをタップ

Microsoft 365 suite 関連記事一覧

Microsoft 365 suite サービス正常性関連記事一覧

Admins may be receiving multiple notifications regarding a Message Center post – MO301479

サービス：Microsoft 365 suite

状態：Service restored

ユーザーへの影響：Admins may be receiving a large volume of notifications for a Message Center post.

Microsoft 365 のサービス正常性（MO301479）にて、管理者がメッセージセンター内の通知を大量に受信する SI（サービスインシデント）が発生していたことを発表しています。

この通知では、”MC301473″ の “Microsoft OneDrive & SharePoint Library dropdown menu.” に関する大量の通知が受信されていました。

なお、本事象は Microsoft 365 Admin アプリケーション内の通知を無効にすることで大量の通知の受信を回避することができました。

さらに、事象が発生していた管理者ユーザーでは 2021 年 12 月 3 日（金）9：00（12：00 AM UTC）～ 2021 年 12 月 3 日（金）10：00（1：00 AM UTC）の間に “サービス正常性” や “メッセージセンター” の表示またはアクセスが行えなくなっていた可能性もあるとのことです。

本事象の予防措置として原因を究明する間は、一時的に通知を無効にしています。

この間は、SHD（サービス正常性）とメッセージセンターの両方についてモバイルアプリケーション内に通知が届かない可能性もありますが、メールアドレス向けによる通知は受信されていたと想定されています。

現在は、影響を受けるすべての環境への修正プログラムの展開が完了し、広範囲におよぶ内部テストの後に、本事象が改善されていることを確認しており、モバイルデバイス向けに SHD（サービス正常性）とメッセージセンターの通知を再度有効にしました。

なお、5 営業日以内に PIR（Post – Incident Report）が公開されます。

※ この問題は、Microsoft 365 Admin アプリケーションを利用している管理者ユーザーにて通知を有効にしている場合に発生する問題になります。

Post Incident Repot（PIR）

Incident Information

Incident ID

MO301479

Incident Title

Admins may be receiving multiple notifications regarding a Message Center post

Service(s) Impacted

Message Center

User Impact

Admins may have received a large volume of notifications for a Message Center post.

・This notice was posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
・Admins could have disabled direct notifications within the Microsoft 365 admin center app to prevent further notifications while we were focused on mitigating impact.
・Additionally, admins may have been unable to view or access the Service Health Dashboard or Message Center between Friday, December 3, 2021, at 12:00 AM UTC and 1:00 AM UTC.
・As a precautionary measure, we temporarily disabled notifications while we determined the cause of the problem.
During this time, admins may have not received the expected mobile app notifications for both the SHD and Message Center; however, email notifications would have continued to work as expected.

Scope of Impact

This issue impacted admins that have notifications enabled within the Microsoft 365 admin center mobile app.

Incident Start Date and Time

Friday, December 3, 2021, at 12:00 AM UTC

Incident End Date and Time

Monday, December 6, 2021, at 5:00 AM UTC

Root Cause

A message creation optimization update exposed a code regression where an Application Programming Interface (API) was not implemented correctly in the Service Health Dashboard (SHD) notification service.

Instead of sending one queue item, an entire list was sent instead.

This sent out duplicate notifications when the admin notification service received multiple items, as the list was split into manageable queue items.

Actions Taken (All times UTC)

Thursday, December 2

・8:00 PM – We deployed an update which contained an optimization for our message creation and expansion logic.

Friday, December 3

・1:17 AM – We received reports of a problem where users received multiple notifications about Message Center posts to their mobile devices.
・1:47 AM – We confirmed that the message creation optimization update exposed a code regression where an API was not implemented correctly in the Service Health Dashboard (SHD) notification service. Instead of sending one queue item, an entire list was sent instead.
This sent out duplicate notifications when the admin notification service received multiple items, as the list was split into manageable queue items.
We confirmed that the impact started once the first global message was posted after the update had been deployed.
・2:30 AM – We disabled notifications to provide short-term relief whilst we investigated a long-term solution. We started to develop a fix.
・6:00 AM – We tested and deployed the fix.

Saturday, December 4

・1:03 AM – We confirmed that the deployment was progressing as expected.

Monday, December 6

・4:00 AM – We turned notifications back on.

Tuesday, December 7

・12:06 AM – We confirmed that the fix had completed deployment successfully.
We declared the incident resolved as of Monday, December 6, 2021, at 5:00 AM UTC, which is when notifications were turned back on.

Next Steps

Findings

Action

Completion Date

A deployment exposed a code regression, causing multiple notifications.

We’re reviewing our code updates within notifications to identify and prevent similar issues in the future.

January 2022

Additional testing steps have been added to pick up the scenario in future service updates.

January 2022

Additional telemetry has been set up to track messages by how we target messages (customer-level, regional etc.)

January 2022

December 14, 2021 4:14 AM

A post-incident report has been published.

December 9, 2021 8:25 AM

A post-incident report has been published.

December 7, 2021 9:42 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may have been receiving a large volume of notifications for a Message Center post.
・More info : This notice was posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins could have also disabled direct notifications within the Microsoft 365 admin center app to prevent further notifications while we were focused on mitigating impact.
Additionally, admins may have been unable to view or access the Service Health Dashboard or Message Center between Friday, December 3, 2021, 9:00 AM (12:00 AM UTC) and 1:00 AM UTC.
As a precautionary measure, we temporarily disabled notifications while we determined the cause of the problem.
During this time, admins may have not received the expected mobile app notifications for both the SHD and Message Center; however, email notifications would have continued to work as expected.
・Final status : We’ve completed deploying the fix to all affected environments.
After a period of extensive monitoring and internal testing we’ve confirmed the issue has been remediated.
We’ve re-enabled notifications for SHD and Message Center in mobile devices.
・Scope of impact : This issue impacted admins that have notifications enabled within the Microsoft 365 admin center mobile app.
・Start time : Friday, December 3, 2021, 9:00 AM (12:00 AM UTC)
・End time : Monday, December 6, 2021, 2:00 PM (5:00 AM UTC)
・Root cause : A message creation optimization update exposed a code regression where an Application Programming Interface (API) was not implemented correctly in the Service Health Dashboard (SHD) notification service.
Instead of sending one queue item, an entire list was sent instead.
This sent out duplicate notifications when the admin notification service received multiple items, as the list was split into manageable queue items.
・Next steps : – For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document.Next update by : Tuesday, December 7, 2021, 12:00 PM (3:00 AM UTC)
We’ll publish a post-incident report within five business days.

December 4, 2021 10:00 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This notice is posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins could have also disabled direct notifications within the Microsoft 365 admin center app to prevent further notifications while we were focused on mitigating impact.
As a precautionary measure, we’ve temporarily disabled notifications while we determine the cause of the problem.
During this time, you may not receive the expected mobile app notifications for both the SHD and Message Center; however, email notifications will continue to work as expected
・Current statu s : We’ve isolated the cause of the issue and we’ve developed a new fix that will properly address the problem.
We’re reviewing our notification queue to ensure there is no residual impact prior to re-enabling the notification service.
We expect our validation and re-enablement to be completed by our next update.
・Scope of impact : This issue impacts admins that have notifications enabled within the Microsoft 365 admin center mobile app.
・Start time : Friday, December 3, 2021, 9:00 AM (12:00 AM UTC)
・Next update by : Tuesday, December 7, 2021, 12:00 PM (3:00 AM UTC)

December 3, 2021 11:45 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This notice is posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins could have also disabled direct notifications within the Microsoft 365 admin center app to prevent further notifications while we were focused on mitigating impact.
Additionally, admins may have been unable to view or access the Service Health Dashboard or Message Center between Friday, December 3, 2021, 9:00 AM (12:00 AM UTC) and 1:00 AM UTC.
・Current status : We’ve confirmed that the issue has been resolved, though as a precautionary measure while we determine the cause of the problem, we’ve temporarily disabled notifications.
During this time you may not receive the expected mobile app notifications for both the SHD and Message center, however email notifications will continue to work as expected.
We will provide an update as soon as we reenable notifications.
・Scope of impact : This issue impacts admins that have notifications enabled within the Microsoft 365 admin center mobile app.
・Start time : Friday, December 3, 2021, 9:00 AM (12:00 AM UTC)
・Next update by : Saturday, December 4, 2021, 12:00 PM (3:00 AM UTC)

December 3, 2021 11:24 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This notice is posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins may also disable direct notifications within the Microsoft 365 Admin Center app to prevent further notifications while we are focused on mitigating impact.
Additionally, admins may have been unable to view or access the Service Health Dashboard or Message Center between Friday, December 3, 2021, 9:00 AM (12:00 AM UTC) and 1:00 AM UTC.
・Current status : While we were preparing the impacted environment for reversion of the fix, our telemetry indicated the issue has been resolved.
We’re monitoring the service, working with impacted admins, and investigating the underlying cause of the issue.
・Scope of impact : This issue impacts admins that have notifications enabled within the Microsoft 365 admin center mobile app.
・Start time : Friday, December 3, 2021, 9:00 AM (12:00 AM UTC)
・Next update by : Friday, December 3, 2021, 12:00 PM (3:00 AM UTC)

December 3, 2021 11:17 AM · クイック更新

・Current status : During our process to revert the recent changes, our telemetry indicates that the issue has self-resolved.
We’re investigating the underlying cause of the problem.
This quick update is designed to give the latest information on this issue.

December 3, 2021 11:10 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This issue is posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins may also disable direct notifications within the Microsoft 365 Admin Center app to prevent further notifications while we are focused on mitigating impact.
Additionally, admins may have been unable to view or access the Service Health Dashboard or Message Center between Friday, December 3, 2021, 9:00 AM (12:00 AM UTC) and 1:00 AM UTC.
・Current status : We’re continuing our process to revert the change that appears to be causing impact to the issue.
・Scope of impact : This issue impacts admins that have notifications enabled within the Microsoft 365 admin center mobile app.
・Start time : Friday, December 3, 2021, 9:00 AM (12:00 AM UTC)
・Next update by : Friday, December 3, 2021, 12:00 PM (3:00 AM UTC)

December 3, 2021 10:51 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This issue is posted under MC301473: “Microsoft OneDrive & SharePoint Library dropdown menu.”
Admins may also disable direct notifications within the Microsoft 365 Admin Center app to prevent further notifications while we are focused on mitigating impact.
・Current status : Our initial investigation indicates that a recent change may be contributing to the issue.
We are reverting our change as a preliminary troubleshooting issue while continuing to analyze the gathered data.
・Scope of impact : We’re currently investigating the scope of impact, but this issue may impact any admins who have enabled direct notifications for the Microsoft 365 admin center mobile app.
・Next update by : Friday, December 3, 2021, 12:00 PM (3:00 AM UTC)

December 3, 2021 10:46 AM

・Title : Admins may be receiving multiple notifications regarding a Message Center post
・User Impact : Admins may be receiving a large volume of notifications for a Message Center post.
・More info : This issue is posted under MC301473: “Microsoft OneDrive & SharePoint
・Current status : We’ve received reports of an issue in which admins are receiving a large volume of notifications for a Message Center post.
We’re gathering diagnostic data to determine the cause of the issue.
・Scope of impact : We’re currently investigating the scope of impact, but this issue may impact any admins who have enabled direct notifications for the Microsoft 365 admin center mobile app.
・Next update by : Friday, December 3, 2021, 12:00 PM (3:00 AM UTC)

Microsoft Store をチェックするなら、このボタンをタップ

Microsoft 365 suite 関連記事一覧

Microsoft 365 suite のメッセージセンター関連情報一覧

Microsoft 365 suite サービス正常性関連記事一覧