GBX Data Fetcher module for Challenges, Maps, Replays, Packs

[Xymph]

I received some requests to support Replays in my GBX Data Fetcher module (PHP), so I expanded that: the GBXDataFetcher class has been renamed to GBXChallengeFetcher, and there is a new GBXReplayFetcher class. Like for Challenges, it extracts all available data fields from a Replay file's header block and, optionally, also from the XML block. All known Replay types are supported.

Download the GBX Data Fetcher module here.

[TStarGermany]

ah nicey nicey, maybe i can make use of that ^^

[Xymph]

Programmers using my GBXChallengeFetcher, GBXReplayFetcher and ReplayParser classes to process track and replay files may be interested to know that these APIs have been updated several times over the past month or two. The same goes for the Extract GBX Data script that lists track and replay info for easy manual examination of these files. Improvements include the handling of XML parser errors, escaping of '&' characters, several enhancements and fixes in processing replays, and the odd bug fix.

Compare the version numbers of your copies with the latest ones, and download them from http://www.gamers.org/tmf/#scripts .

[nicomoore0]

Thanks

[w1lla]

don't want to bump this but i think you have missed some data in replays.

For example the trackname can't be found in the replayfetcher only the trackuid.

thats my poke and seek that i found in a replay.

[Xymph]

don't want to bump this but i think you have missed some data in replays.

For example the trackname can't be found in the replayfetcher only the trackuid.

Right, because it isn't included in the header/XML sections of replay files.

thats my poke and seek that i found in a replay.

If you want that to make sense, you'll have to be way more specific.

[Electron]

The scripts of Xymph are based on the old descriptions of the GBX file format in the TM-Wiki. Some of these old descriptions were not interpreted correctly. Hence, these scripts do not work with GBX files from Virtual Skipper and may not work with future versions of TrackMania 2 and ShootMania.

The following pseudo code is based on the new description of the GBX format, can be simply maintained and should still function after extending or remodelling the format by Nadeo.

I provide this only as technical information. Hence, the pseudo code uses neither any error handling, memory allocation nor is shown how the read data are utilized.

Sample pseudo code to read the uncompressed contents from a Challenge file:

Code: Select all

// Sample pseudo code to read the uncompressed contents from a challenge file

// Jump to header size
SetFilePointer(hFile, 13, FILE_BEGIN);

// Read header size (not used)
ReadFile(hFile, &ulHeaderSize, 4);

// Number of chunks in header
ReadFile(hFile, &ulNumHeaderChunks, 4);

// Determine chunk sizes and offsets
ulChunkOffset = 21 + (ulNumHeaderChunks * 8);
ulInfoChunkSize = 0;
ulStringChunkSize = 0;
ulVersionChunkSize = 0;
ulXmlChunkSize = 0;
ulThumbnailChunkSize = 0;
ulAuthorChunkSize = 0;
ulInfoChunkOffset = 0;
ulStringChunkOffset = 0;
ulVersionChunkOffset = 0;
ulXmlChunkOffset = 0;
ulThumbnailChunkOffset = 0;
ulAuthorChunkOffset = 0;

for (ulCouter = 1; ulCouter <= ulNumHeaderChunks; ulCouter++)
{
    ReadFile(hFile, &ulChunkId, 4);
    ReadFile(hFile, &ulChunkSize, 4);

    // Remove msb
    ulChunkSize &= 0x7FFFFFFF;

    // Not required blocks can be commented out
    switch (ulChunkId)
    {
    case 0x03043002:
    case 0x24003002:
        ulInfoChunkSize = ulChunkSize;
        ulInfoChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03043003:
    case 0x24003003:
        ulStringChunkSize = ulChunkSize;
        ulStringChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03043004:
    case 0x24003004:
        ulVersionChunkSize = ulChunkSize;
        ulVersionChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03043005:
    case 0x24003005:
        ulXmlChunkSize = ulChunkSize;
        ulXmlChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03043007:
    case 0x24003007:
        ulThumbnailChunkSize = ulChunkSize;
        ulThumbnailChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03043008:
        ulAuthorChunkSize = ulChunkSize;
        ulAuthorChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    default: // Remaining chunks
        ulChunkOffset += ulChunkSize;
        break;
    }
}

// "Version" chunk
if (ulVersionChunkSize > 0)
{
    // Jump to version chunk
    SetFilePointer(hFile, ulVersionChunkOffset, FILE_BEGIN);

    // Read challenge version
    ReadFile(hFile, &ulVersion, 4);
}

// "Author" chunk
if (ulAuthorChunkSize > 0)
{
    // Jump to author chunk
    SetFilePointer(hFile, ulAuthorChunkOffset, FILE_BEGIN);

    // Read version
    ReadFile(hFile, &ulVersion, 4);

    // Read author info version
    ReadFile(hFile, &ulAuthorVer, 4);

    // Read login size
    ReadFile(hFile, &ulLen, 4);

    // Read login
    ReadFile(hFile, pLogin, ulLen);

    // Read nickname size
    ReadFile(hFile, &ulLen, 4);

    // Read nickname
    ReadFile(hFile, pNickname, ulLen);

    // Read author zone size
    ReadFile(hFile, &ulLen, 4);

    // Read author zone
    ReadFile(hFile, pZone, ulLen);

    // Read extra info size
    ReadFile(hFile, &ulLen, 4);

    // Read extra info
    ReadFile(hFile, pExtraInfo, ulLen);
}

// "String" chunk
if (ulStringChunkSize > 0)
{
    // Jump to string chunk
    SetFilePointer(hFile, ulStringChunkOffset, FILE_BEGIN);

    // Read version
    ReadFile(hFile, &bStrVer, 1);

    // Jump to UID index
    SetFilePointer(hFile, 4, FILE_CURRENT);

    // Read UID index
    ReadFile(hFile, &ulIndex, 4);

    if ((ulIndex & 0x3FFFFFFF) == 0)
    {
        // Read UID size
        ReadFile(hFile, &ulLen, 4);

        // Read UID
        ReadFile(hFile, pUID, ulLen);
    }

    // Read environment index
    ReadFile(hFile, &ulIndex, 4);

    if ((ulIndex & 0xC0000000) == 0)
    {
        switch (ulIndex)
        {
            case 11: // TMValley
                strcpy(pEnvir, "Valley");
                break;
            case 12: // TMCanyon
                strcpy(pEnvir, "Canyon");
                break;
            case 13: // TMLagoon
                strcpy(pEnvir, "Lagoon");
                break;
            case 202: // SMStorm
                strcpy(pEnvir, "Storm");
                break;
        }
    }
    else if ((ulIndex & 0x3FFFFFFF) == 0)
    {
        // Read environment size
        ReadFile(hFile, &ulLen, 4);

        // Read Environment
        ReadFile(hFile, pEnvir, ulLen);
    }

    // Read author name index
    ReadFile(hFile, &ulIndex, 4);

    if ((ulIndex & 0x3FFFFFFF) == 0)
    {
        // Read author name size
        ReadFile(hFile, &ulLen, 4);

        // Read author name
        ReadFile(hFile, pAuthor, ulLen);
    }

    // Read track name size
    ReadFile(hFile, &ulLen, 4);

    // Read track name
    ReadFile(hFile, pTrackname, ulLen);

    // Read game mode
    ReadFile(hFile, &bGameMode, 1);

    if (bStrVer >= 1)
    {
        // Jump to password size
        SetFilePointer(hFile, 4, FILE_CURRENT);

        // Read password size
        ReadFile(hFile, &ulLen, 4);

        if (bStrVer >= 2)
        {
            // Jump to mood index
            SetFilePointer(hFile, ulLen, FILE_CURRENT);

            // Read mood index
            ReadFile(hFile, &ulIndex, 4);

            if ((ulIndex & 0x3FFFFFFF) == 0)
            {
                // Read mood size
                ReadFile(hFile, &ulLen, 4);

                // Read mood name
                ReadFile(hFile, pMood, ulLen);
            }
            else if ((ulIndex & 0x3FFFFFFF) == 2)
            {
                // 1: UID, 2: Environment, 3: Track author
                memcpy(pMood, pEnvir);
            }

            // Read environment index
            ReadFile(hFile, &ulIndex, 4);

            if ((ulIndex & 0xC0000000) == 0)
            {
                switch (ulIndex)
                {
                    case 11: // TMValley
                        strcpy(pBackground, "Valley");
                        break;
                    case 12: // TMCanyon
                        strcpy(pBackground, "Canyon");
                        break;
                    case 13: // TMLagoon
                        strcpy(pBackground, "Lagoon");
                        break;
                    case 202: // SMStorm
                        strcpy(pBackground, "Storm");
                        break;
                }
            }
            else if ((ulIndex & 0x3FFFFFFF) == 0)
            {
                // Read collection size
                ReadFile(hFile, &ulLen, 4);

                // Read Collection
                ReadFile(hFile, pBackground, ulLen);
            }
            else if ((ulIndex & 0x3FFFFFFF) == 2)
            {
                // 1: UID, 2: Collection, 3: Track author
                memcpy(pBackground, pEnvir);
            }

            // Read envir author index
            ReadFile(hFile, &ulIndex, 4);

            if ((ulIndex & 0x3FFFFFFF) == 0)
            {
                // Read envir author size
                ReadFile(hFile, &ulLen, 4);

                // Read envir author
                ReadFile(hFile, pEnvirAuthor, ulLen);
            }
            else if ((ulIndex & 0x3FFFFFFF) == 3)
            {
                // 1: UID, 2: Environment, 3: Developer
                memcpy(pEnvirAuthor, pAuthor);
            }

            if (bStrVer >= 9)
            {
                // Jump to map type size
                SetFilePointer(hFile, 32, FILE_CURRENT);

                // Read map type size
                ReadFile(hFile, &ulLen, 4);

                // Read map type
                ReadFile(hFile, pMapType, ulLen);

                // Read map style size
                ReadFile(hFile, &ulLen, 4);

                // Read map style
                ReadFile(hFile, pMapStyle, ulLen);

                // Jump to lightmap version
                SetFilePointer(hFile, 8, FILE_CURRENT);

                // Read lightmap version
                ReadFile(hFile, &bLightmap, 1);

                if (bStrVer >= 11)
                {
                    // Read title id index
                    ReadFile(hFile, &ulIndex, 4);

                    if ((ulIndex & 0x3FFFFFFF) == 0)
                    {
                         // Read title id size
                         ReadFile(hFile, &ulLen, 4);

                         // Read title id
                         ReadFile(hFile, pTitleId, ulLen);
                    }
                }
            }
        }
    }
}

// "Info" chunk
if (ulInfoChunkSize > 0)
{
    // Jump to info chunk
    SetFilePointer(hFile, ulInfoChunkOffset, FILE_BEGIN);

    // Read version
    ReadFile(hFile, &bInfoVer, 1);

    if (bInfoVer >= 3)
    {
        // Jump to bronze
        SetFilePointer(hFile, 4, FILE_CURRENT);

        // Bronze
        ReadFile(hFile, &lBronze, 4);

        // Silver
        ReadFile(hFile, &lSilver, 4);

        // Gold
        ReadFile(hFile, &lGold, 4);

        // Authortime
        ReadFile(hFile, &ulAuthortime, 4);

        if (bInfoVer >= 4)
        {
            // Coppers
            ReadFile(hFile, &ulPrice, 4);

            if (bInfoVer >= 6)
            {
                // Multilap
                ReadFile(hFile, &ulNbLaps, 4);

                // Track type
                ReadFile(hFile, &ulTrackType, 4);

                if (bInfoVer >= 10)
                {
                    SetFilePointer(hFile, 4, FILE_CURRENT);

                    // Author score
                    ReadFile(hFile, &ulAuthorscore, 4);

                    if (bInfoVer >= 11)
                    {
                        // Editor flags
                        ReadFile(hFile, &ulEditorFlags, 4);

                       if (bInfoVer >= 13)
                       {
                          SetFilePointer(hFile, 4, FILE_CURRENT);

                          // Checkpoints
                          ReadFile(hFile, &ulCheckpoints, 4);

                          // Number of laps
                          ReadFile(hFile, &ulNbLaps, 4);
                       }
                    }
                }
            }
        }
    }
}

// "XML" chunk
if (ulXmlChunkSize > 0)
{
    // Jump to XML chunk
    SetFilePointer(hFile, ulXmlChunkOffset, FILE_BEGIN);

    // Read XML size
    ReadFile(hFile, &ulLen, 4);
    ulLen &= 0x7FFFFFFF;

    // Read XML
    ReadFile(hFile, pXml, ulLen);
}

// "Thumbnail/Comments" chunk
if (ulThumbnailChunkSize > 0)
{
    // Jump to thumbnail chunk
    SetFilePointer(hFile, ulThumbnailChunkOffset, FILE_BEGIN);

    // Thumbnail version
    ReadFile(hFile, &ulVersion, 4);

    if (ulVersion == 1)
    {
        // Read thumbnail size
        ReadFile(hFile, &ulLen, 4);

        // Jump to thumbnail data
        SetFilePointer(hFile, 15, FILE_CURRENT);

        if (ulLen > 0)
        {
            // Read thumbnail
            ReadFile(hFile, pThumbnail, ulLen);
        }

        // Jump to comments size
        SetFilePointer(hFile, 26, FILE_CURRENT);

        // Read comments size
        ReadFile(hFile, &ulLen, 4);

        if (ulLen > 0)
        {
            // Read comments
            ReadFile(hFile, pComments, ulLen);
        }
    }
}

Sample pseudo code to read the uncompressed contents from a Replay file:

Code: Select all

// Sample pseudo code to read the uncompressed contents from a replay file

// Jump to header size
SetFilePointer(hFile, 13, FILE_BEGIN);

// Read header size (not used)
ReadFile(hFile, &ulHeaderSize, 4);

// Number of chunks in header
ReadFile(hFile, &ulNumHeaderChunks, 4);

// Determine chunk sizes and offsets
ulChunkOffset = 21 + (ulNumHeaderChunks * 8);
ulStringChunkSize = 0;
ulXmlChunkSize = 0;
ulAuthorChunkSize = 0;
ulStringChunkOffset = 0;
ulXmlChunkOffset = 0;
ulAuthorChunkOffset = 0;

for (ulCouter = 1; ulCouter <= ulNumHeaderChunks; ulCouter++)
{
    ReadFile(hFile, &ulChunkId, 4);
    ReadFile(hFile, &ulChunkSize, 4);

    // Remove msb
    ulChunkSize &= 0x7FFFFFFF;

    // Not required blocks can be commented out
    switch (ulChunkId)
    {
    case 0x03093000:
    case 0x2403F000:
        ulStringChunkSize = ulChunkSize;
        ulStringChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03093001:
    case 0x2403F001:
        ulXmlChunkSize = ulChunkSize;
        ulXmlChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    case 0x03093002:
        ulAuthorChunkSize = ulChunkSize;
        ulAuthorChunkOffset = ulChunkOffset;
        ulChunkOffset += ulChunkSize;
        break;
    default: // Remaining chunks
        ulChunkOffset += ulChunkSize;
        break;
    }
}

// "String" chunk
if (ulStringChunkSize > 4)
{
    // Jump to string chunk
    SetFilePointer(hFile, ulStringChunkOffset, FILE_BEGIN);

    // Read replay version
    ReadFile(hFile, &ulVersion, 4);

    if (ulVersion >= 2)
    {
        // Jump to UID index
        SetFilePointer(hFile, 4, FILE_CURRENT);

        // Read UID index
        ReadFile(hFile, &ulIndex, 4);

        if ((ulIndex & 0x3FFFFFFF) == 0)
        {
            // Read UID size
            ReadFile(hFile, &ulLen, 4);

            // Read UID
            ReadFile(hFile, pUID, ulLen);
        }

        // Read environment index
        ReadFile(hFile, &ulIndex, 4);

        if ((ulIndex & 0xC0000000) == 0)
        {
            switch (ulIndex)
            {
                case 11: // TMValley
                    strcpy(pEnvir, "Valley");
                    break;
                case 12: // TMCanyon
                    strcpy(pEnvir, "Canyon");
                    break;
                case 13: // TMLagoon
                    strcpy(pEnvir, "Lagoon");
                    break;
                case 202: // SMStorm
                    strcpy(pEnvir, "Storm");
                    break;
            }
        }
        else if ((ulIndex & 0x3FFFFFFF) == 0)
        {
            // Read environment size
            ReadFile(hFile, &ulLen, 4);

            // Read Environment
            ReadFile(hFile, pEnvir, ulLen);
        }

        // Read author name index
        ReadFile(hFile, &ulIndex, 4);

        if ((ulIndex & 0x3FFFFFFF) == 0)
        {
            // Read author name size
            ReadFile(hFile, &ulLen, 4);

            // Read author name
            ReadFile(hFile, pAuthor, ulLen);
        }

        // Read replay time
        ReadFile(hFile, &ulTime, 4);

        // Read nickname size
        ReadFile(hFile, &ulLen, 4);

        // Read nickname
        ReadFile(hFile, pName, ulLen);

        if (ulVersion >= 6)
        {
            // Read login size
            ReadFile(hFile, &ulLen, 4);

            // Read login
            ReadFile(hFile, pLogin, ulLen);

            if ((ulVersion >= 8)
            {
                // Read unknown byte
                ReadFile(hFile, &bUnknown, 1);

                // Read title id index
                ReadFile(hFile, &ulIndex, 4);

                if ((ulIndex & 0x3FFFFFFF) == 0)
                {
                     // Read title id size
                     ReadFile(hFile, &ulLen, 4);

                     // Read title id
                     ReadFile(hFile, pTitleId, ulLen);
                }
            }
        }
    }
}

// "Author" chunk
if (ulAuthorChunkSize > 0)
{
    // Jump to author chunk
    SetFilePointer(hFile, ulAuthorChunkOffset, FILE_BEGIN);

    // Read version
    ReadFile(hFile, &ulVersion, 4);

    // Read author info version
    ReadFile(hFile, &ulAuthorVer, 4);

    // Read login size
    ReadFile(hFile, &ulLen, 4);

    // Read login
    ReadFile(hFile, pLogin, ulLen);

    // Read nickname size
    ReadFile(hFile, &ulLen, 4);

    // Read nickname
    ReadFile(hFile, pNickname, ulLen);

    // Read author zone size
    ReadFile(hFile, &ulLen, 4);

    // Read author zone
    ReadFile(hFile, pZone, ulLen);

    // Read extra info size
    ReadFile(hFile, &ulLen, 4);

    // Read extra info
    ReadFile(hFile, pExtraInfo, ulLen);
}

// "XML" chunk
if (ulXmlChunkSize > 0)
{
    // Jump to XML chunk
    SetFilePointer(hFile, ulXmlChunkOffset, FILE_BEGIN);

    // Read XML size
    ReadFile(hFile, &ulLen, 4);
    ulLen &= 0x7FFFFFFF;

    // Read XML
    ReadFile(hFile, pXml, ulLen);
}

Electron

[Xymph]

The scripts of Xymph are based on the old descriptions of the GBX file format in the TM-Wiki. Some of these descriptions were not interpreted correctly. Hence, these scripts do not work with GBX files from Virtual Skipper and may not work with future versions of TrackMania 2 and ShootMania.

True, but then again, those scripts were written long before the new description became available, and are as "correct" as possible at the time. They were also never intended for the VSK series (I own nor care for them ). And I fully expect them to fail for the next generation of TM/etc games , but would have attempted to address that after gaining access to them.

The following pseudo code is based on the new description of the GBX format, can be simply maintained and should still function after extending or remodelling the format by Nadeo.

I provide this only as technical information. Hence, the pseudo code uses neither any error handling, memory allocation nor is shown how the read data are utilized.

Very useful, thank for sharing. With this it would probably be better to rewrite my Fetcher/Parser classes from scratch, rather than trying to kludge in TM2/etc support once that game arrives. I just don't know when I'll have time myself...

[Xymph]

Blast from the past, but this module has been repeatedly updated in recent years (which I forgot to note here), and now support for TrackMania² Canyon has been added to both the Challenge/Map and Replay fetcher classes.

Several related scripts/classes received the same treatment: the command-line 'Extract GBX data' script, the Replay Parser class, and the Tally GBX versions script (w/ challenges).

Descriptions and download links can be found on the XAseco site.

[Xymph]

Another necro-post (sorry ), but this module has been completely rewritten (based on Electron's pseudo-code above) and now parses ManiaPlanet (beta) maps for TrackMania² Canyon and ShootMania Storm the correct way. The new GBXChallMapFetcher and GBXReplayFetcher classes are not compatible with the old GBXChallengeFetcher and GBXReplayFetcher ones, but a new GBXChallengeFetcher wrapper class is included for backwards compatibility with third-party plugins.

The Tally GBX versions zip has also been updated with all known challenge/map versions, while the Replay Parser class is obsolete and has been removed. The 'Extract GBX data' script still needs to be rewritten.

As usual, descriptions and download links can be found on the XAseco site.

Edit: I merged the related "Updated GBX challenge & replay scripts" topic with this one - I must have inadvertedly created that one after having forgotten about this one.

[Xymph]

The GBX Data Fetcher module has now been extended with a new GBXPackFetcher class to process ManiaPlanet .Pack.Gbx and .pak files.

Also, as promised, the 'Extract GBX data' script has been completely rewritten to extract and pretty-print relevant information on all the file types that the GBX Data Fetcher module supports.

As always, descriptions and download links can be found on the XAseco site.

[Xymph]

The GBX Data Fetcher module has been improved, adding UTF-8 decoding of parsed XML chunk's elements and stripping UTF-8 BOMs from various string fields. It can be downloaded from the usual place.

[Xymph]

The GBX Data Fetcher module has been improved again, incorporating the latest insights from the Wiki about version-dependent processing of Challenge/Map and Pack files, and adding more lookback strings. As usual it can be obtained from my Scripts & Tools page.

[Xymph]

A few minor adjustments to the GBX Data Fetcher module make it handle the newly released TrackMania² Valley better. Get it from the usual place.

[Xymph]

A new release of the GBX Data Fetcher module is now online, improving the PHP classes that process Challenge/Map, Replay and Pack files. This update extends GBXPackFetcher with Included Packs info, adds a few more fields to all main classes, and generally improves GBX processing. The related 'Extract GBX data' script has been enhanced to output the new fields and Included Packs list too. As always both are available for download from the Scripts & Tools page of XAseco.org.

Edit: the module has been updated to v2.8 with a minor bug fix.