Teams Bot Stuck 'Incoming': Media Ports & Windows Server 2019

by Felix Dubois 62 views

Have you ever encountered a situation where your Microsoft Teams Policy Recording Bot gets stuck in the dreaded "Incoming" state, leaving you scratching your head? You're not alone, guys! This article dives deep into a perplexing issue encountered on Windows Server 2019, where calls initiated by a compliance recording bot remain indefinitely in the "Incoming" state. We'll explore the technical details, potential causes, attempted solutions, and critical questions surrounding this issue, providing a comprehensive guide for troubleshooting and understanding.

The Mystery of the Stuck Calls

The core problem? Media ports never bind during call setup, effectively preventing proper media negotiation. This leads to two distinct failure patterns: calls either fail immediately with 408/8544 errors or remain perpetually stuck in the "Incoming" state. Let's break down the environment and the observed behaviors to get a clearer picture.

Environment Configuration

Our setup involves a standalone Windows Service deployed on Windows Server 2019 Standard (Build 17763). The service, migrated from an Azure Cloud Service, boasts a direct internet connection with a public IP (Nat 1:1 through Firewall) and a valid SSL certificate. We've experimented with both stable and beta SDK versions to isolate the issue:

  • Stable SDK Versions (V1.0):
    • Microsoft.Graph.Communications.Calls.Media: v1.2.0.3742
    • Microsoft.Graph.Communications.Calls: v1.2.0.3742
    • Microsoft.Graph.Communications.Client: v1.2.0.3742
    • Microsoft.Graph.Communications.Common: v1.2.0.7270
    • Microsoft.Graph.Communications.Core: v1.2.0.3742
    • Microsoft.Skype.Bots.Media: v1.20.0.348-alpha (V1.0 version)
  • Beta SDK Versions:
    • Microsoft.Graph.Communications.Calls\1.2.0-beta.14141
    • Microsoft.Graph.Communications.Calls.Media\1.2.0-beta.9995
    • Microsoft.Graph.Communications.Client\1.2.0-beta.14141
    • Microsoft.Graph.Communications.Common\1.2.0-beta.14141
    • Microsoft.Graph.Communications.Core\1.2.0-beta.14141
    • Microsoft.Skype.Bots.Media\1.27.0.2-alpha
  • Framework: .NET Framework 4.7.2

The expected behavior is straightforward: a call notification is received, AnswerAsync() is executed, the call transitions to "Established", media ports bind, and the bot appears in the Teams meeting as a recording participant. However, the reality is far from this ideal scenario.

The Tale of Two Failure Patterns

We've observed two distinct failure patterns, each with its unique characteristics. Let's dissect them:

Pattern 1: The Immediate Failure (More Common with Stable SDK)

This pattern is characterized by a swift and decisive termination of the call.

  1. ✅ Call notification processing: Working flawlessly.
  2. ✅ Authentication (JWT validation): No issues here.
  3. ✅ AnswerAsync() initiated: Starts with promise.
  4. ❌ Immediate termination: The call transitions from "Incoming" to "Terminated" in the blink of an eye (around 1 second).
  5. ❌ Error 408/8544: The dreaded "Callee could not be reached" error rears its ugly head.
  6. ❌ AnswerAsync fails: A "Call not found" exception adds insult to injury.

This failure pattern, more prevalent with the stable SDK, suggests a fundamental problem in establishing the call connection. The quick termination and the 408/8544 error point towards a failure in reaching the callee, despite the successful initial processing and authentication. The "Call not found" exception further complicates the matter, indicating a potential issue with call state management within the SDK.

Pattern 2: The Perennial "Incoming" State

The second pattern is a slow burn, where calls get stuck in the "Incoming" state, seemingly for eternity.

  1. ✅ Call processing and AnswerAsync(): Working as expected.
  2. ❌ Call remains stuck: The call stubbornly stays in the "Incoming" state for an indefinite period (2+ minutes, in our observations).
  3. ❌ No state transition: The call never reaches the coveted "Established" state.
  4. ❌ No media ports binding: The crux of the issue – media ports remain unbound, hindering media negotiation.

This pattern is particularly frustrating as it indicates a partial success. The initial processing and AnswerAsync() execution suggest that the bot is receiving and acknowledging the call invitation. However, the failure to transition to "Established" and the absence of media port binding point to a roadblock in the media connection setup. The call's indefinite stay in the "Incoming" state suggests a deadlock or a missing trigger in the call establishment process.

Unearthing the Technical Evidence

To diagnose the problem, we've meticulously examined various components of the infrastructure. Here's what we've found:

Successful Infrastructure Components

Our investigation reveals that several components are functioning as expected, providing a solid foundation for the bot's operation.

  • ✅ HTTPS (443): Microsoft connects successfully, ensuring secure communication.
  • ✅ Media Control (8445): Microsoft also connects successfully, with TimeWait connections observed, indicating proper control channel establishment.
  • ✅ Call Control (10100): Another successful connection point for Microsoft, further solidifying the control plane.
  • ✅ Authentication: JWT validation works seamlessly, eliminating authentication as a potential culprit.
  • ✅ Service initialization: All components start correctly, ruling out any initial setup issues.

The Achilles' Heel: Failed Media Port Binding

Despite the successful infrastructure components, a critical piece is missing: media port binding.

  • ❌ NO media ports binding during calls: This is the core issue. Netstat shows no ports listening in the expected ranges during call attempts.
  • ❌ NO Microsoft connection attempts to media ports: Microsoft never tries to connect to the media ports, confirming the lack of binding.
  • ❌ Calls either fail immediately (408/8544) or remain stuck in "Incoming": The observed failure patterns directly correlate with the missing media port binding.
  • ❌ Two distinct failure patterns with stable SDK vs beta SDK: The difference in behavior between the SDK versions hints at potential SDK-specific issues.

The absence of media port binding is the smoking gun. Without bound media ports, media negotiation cannot occur, leading to call failures or the perpetual "Incoming" state. The lack of connection attempts from Microsoft to these ports further reinforces the notion that the ports are simply not available for communication.

MediaPlatformSettings Configuration

Our MediaPlatformSettings configuration is as follows:

MediaPlatformSettings = new MediaPlatformSettings
{
    ApplicationId = this.AadAppId,
    MediaPlatformInstanceSettings = new MediaPlatformInstanceSettings
    {
        CertificateThumbprint = certificate.Thumbprint,
        InstanceInternalPort = 8445,
        InstancePublicPort = 8445,
        InstancePublicIPAddress = IPAddress.Any,
        ServiceFqdn =