Dedicated servers no longer receiving SESSION_MEMBER_CHANGED messages via the hub

As of sometime last night (9/23) and today (9/24) our dedicated servers are no longer receiving the SESSION_MEMBER_CHANGED messages and are therefore unable to proceed after being claimed.

Hi @njupshot,

Could you please provide the session ID so that our internal team can investigate further?

Additionally, we tried to reproduce your issue using the Byte Wars sample, but the SESSION_MEMBER_CHANGED event is still being received on the DS.

We repro’d this morning.

Session ID: 400b17a090694fe6a02ef77341116f26
Pod ID: ds_01922859-f549-78fb-a60a-09979bf51ad7

Upon joining the session, the client realizes that the dedi is not ready yet and so it awaits the DS status changed notification. Upon receiving it, it then connects to the dedicated server. However, the dedicated server receives no further messages after the initial serverClaimed message.

The issue seems to only happen in the case that the dedi takes some additional time to provision which is indicative by the fact that it’s not present in the response when the client joins the session. In that case, the dedi will not receive any further messages from the hub after serverClaimed and therefore the dedi has no way of knowing which players are supposed to be in the session and therefore can’t permit said players to connect. In other words, if the server is provisioned after the client joins the session then the server won’t be notified of the session’s participants.

Hi @njupshot,

Thank you for the explanation. The SESSION_MEMBER_CHANGED event not being received by the DS is expected only if the user has already joined the session.

We recommend obtaining session information after the server is claimed so the DS can get the initial players who joined the session. If a new player joins the session later, the DS will receive the SESSION_MEMBER_CHANGED event.

Hope this answer your question.

Hi @Damar I did as you suggested and fetched the current state of the session at the time we are claimed, but we’re still running into some kind of race condition. At the time I fetch the session, the user has been invited but hasn’t joined yet. In that case, I’d expect to get a message once the user’s status moves from INVITED to JOINED but that isn’t happening. Here are some relevant log lines from our game server:

[2024-10-10T21:04:27.529Z][INFO] LogHub: received topic: serverClaimed
[2024-10-10T21:04:27.529Z][INFO] LogHub: claimed by session 
29d1d6eba09b4511875a724c88d04036, fetching...
[2024-10-10T21:04:27.615Z][INFO] LogHub: processing gameSessionResponse:
{
    "matchPool": "dev-battle-solo-ds-ams",
    "backfillTicketID": "",
    "code": "NWHGY3",
    "ticketIDs": null,
    "isActive": true,
    "isFull": false,
    "version": 1,
    "id": "29d1d6eba09b4511875a724c88d04036",
    "namespace": "__REDACTED__",
    "createdAt": "2024-10-10T21:04:26.903Z",
    "createdBy": "client-__REDACTED__",
    "updatedAt": "2024-10-10T21:04:26.931Z",
    "leaderID": "",
    "configuration": {
        "persistent": false,
        "textChat": false,
        "autoJoin": false,
        "tieTeamsSessionLifetime": false,
        "minPlayers": 0,
        "maxPlayers": 2,
        "inviteTimeout": 60,
        "inactiveTimeout": 60,
        "name": "battle-ds-ams",
        "type": "DS",
        "joinability": "INVITE_ONLY",
        "deployment": "",
        "clientVersion": "v0.0.0",
        "requestedRegions": [
            "us-east-2"
        ],
        "dsSource": "AMS",
        "preferredClaimKeys": [],
        "fallbackClaimKeys": [],
        "nativeSessionSetting": {
            "XboxSessionTemplateName": "",
            "XboxServiceConfigID": "",
            "XboxTitleID": "",
            "XboxSandboxID": "",
            "XboxAllowCrossPlatform": false,
            "PSNServiceLabel": 0,
            "SessionTitle": "",
            "ShouldSync": false,
            "PSNSupportedPlatforms": [
                "ps5"
            ],
            "localizedSessionName": {
                "localizedText": {}
            }
        },
        "PSNBaseURL": "https://s2s.sp-int.playstation.net",
        "maxActiveSession": -1,
        "disableResendInvite": false,
        "attributes": {
            "client_version": "v0.0.0",
            "game_mode": "__REDACTED__"
        },
        "leaderElectionGracePeriod": 0
    },
    "members": [
        {
            "id": "__REDACTED__",
            "status": "INVITED",
            "statusV2": "INVITED",
            "updatedAt": "2024-10-10T21:04:26.903Z",
            "platformID": "",
            "platformUserID": ""
        }
    ],
    "attributes": {
        "client_version": "__REDACTED__",
        "game_mode": "__REDACTED__"
    },
    "DSInformation": {
        "CreatedAt": "2024-10-10T21:04:26.926Z",
        "RequestedAt": "2024-10-10T21:04:26.926Z",
        "Status": "REQUESTED",
        "StatusV2": "REQUESTED"
    },
    "teams": [
        {
            "userIDs": [
                "__REDACTED__"
            ],
            "parties": [
                {
                    "partyID": "",
                    "userIDs": [
                        "__REDACTED__"
                    ]
                }
            ]
        }
    ]
}

Hi @njupshot,

The SESSION_MEMBER_CHANGED event should be triggered every time there is a status change. You mentioned that a user was invited—have you called JoinSession from the user’s perspective?

Here’s the flow that covers this:

  1. Users go through matchmaking and find a match.
  2. The match service creates a session and invites all matched players to join.
  3. The session requests a dedicated server (DS) from AMS.
  4. The DS receives a claim notification and queries the current game session.
  5. Players receive the invite notification and should call JoinSession.
  6. The DS will receive SESSION_MEMBER_CHANGED for each player who joins the session.

To ensure your data is up-to-date and to prevent race conditions when fetching session info in relation to SESSION_MEMBER_CHANGED, there is a version field you can use to compare which session data is the most recent.

Yes the client joined the session, here’s the relevant client log line:

[10_10_2024_21_04_27_421][INFO][MatchmakingHandler] session 29d1d6eba09b4511875a724c88d04036 joined successfully:
{
    "dsInformation": {
        "server": null,
        "status": "REQUESTED",
        "statusV2": "REQUESTED",
        "requestedAt": "2024-10-10T23:04:26.926+02:00"
    },
    "attributes": {
        "client_version": "v0.0.0",
        "game_mode": "__REDACTED__"
    },
    "backfillTicketId": "",
    "code": "NWHGY3",
    "configuration": {
        "name": "battle-ds-ams",
        "type": "DS",
        "joinability": "INVITE_ONLY",
        "minPlayers": 0,
        "maxPlayers": 2,
        "inviteTimeout": 60,
        "inactiveTimeout": 60,
        "deployment": "",
        "requestedRegions": [
            "us-east-2"
        ],
        "clientVersion": "v0.0.0",
        "persistent": false
    },
    "createdAt": "2024-10-10T21:04:26.903Z",
    "createdBy": "client-__REDACTED__",
    "id": "29d1d6eba09b4511875a724c88d04036",
    "isActive": true,
    "leaderId": "__REDACTED__",
    "matchPool": "dev-battle-solo-ds-ams",
    "members": [
        {
            "id": "__REDACTED__",
            "platformId": "",
            "platformUserId": "",
            "status": "JOINED",
            "statusV2": "JOINED",
            "updatedAt": "2024-10-10T21:04:27.524Z"
        }
    ],
    ■■■■■■■■■■■■■■■■■■■■■■■■■■■,
    "teams": [
        {
            "userIds": [
                "__REDACTED__"
            ]
        }
    ],
    "ticketIds": null,
    "updatedAt": "2024-10-10T21:04:27.524Z",
    "version": 2,
    "storage": null
}

Given the above, I’m fairly certain the expectation is that the server should have gotten a SESSION_MEMBER_CHANGED. Do you agree? If so, given the details in the logs provided is your team able to track down and verify what happened?

(Also just to clarify, by race condition I meant the potential for a race condition in the AB logic that decides whether or not to send a SESSION_MEMBER_CHANGED. For example, what’s the expectation if the server is claimed at the same time the client joins the session?)

Thanks

hi @njupshot, yes, the session service will only send the SESSION_MEMBER_CHANGED when the server information is available. We have checked some of the latest sessions in your namespace, and there is indeed a potential race condition between serverClaimed and player joined the session. You can take a look at session 1cd52f1b42d24d2bb215f734d9077a12.

Do you want to restrict only joined player can travel to the session?
The other way that we can try is to fetch the session data when the player travels to the DS.
Alternatively, you can also try implementing the player validation using session secret here: Integrate game session player validation | AccelByte Documentation
So that the DS doesn’t have to wait for SESSION_MEMBER_CHANGED to validate the player.