The big Dedimania discussion

This is the place where you can find everything related to the dedicated server, control scripts and community tools.

Moderators: Pit Crew, TM-Patrol

Post Reply
Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

The big Dedimania discussion

Post by Xymph » 07 Apr 2009 19:18

This post outlines a number of different topics regarding Dedimania, the global records database, that should be of interest to players, server admins, and tool authors.

Intro

As everyone can imagine, a global records database is only useful if it is reliable (rarely down) and trustworthy (no bogus entries), and with limited resources this is no easy task, as demonstrated by the demise of the other public database.

Over the past few years Slig has built something that generally works well, but also has its own share of problems. The two big ones are timeouts due to overload of the central Dedimania server, and cluttered record lists due to erroneous and cheated entries. A few statistics help put this into perspective:

- over 3600 game servers are known to Dedimania, distributed among XAseco (60.7 %), FAST (28.8 %), RCP (2.4 %), ADEDIMANIA (2.9 %), TRUCK (2.5 %) and the rest (2.7 %).
- the database contains more than 167 thousand challenges, half a million players, and 4.5 million records!
- the main Dedimania system is currently hosted on a dual core server with 3 GB RAM, and will need to be moved to a bigger server (quad core, 4 GB) soon


Here's what each of you should know and can do to help, and what we're doing and planning to do in order to improve the system.

Players

If you hold a number of world records, enjoy competing with other drivers on a world-wide level, or simply appreciate the Dedimania service, then make a donation. Renting the server for the central system costs real money, so your donation makes a real difference in sustaining and improving operations.

If you think you're funny or cool for cheating your way to the top of the record lists, you're wrong, and we ask you to stop. Just quit doing it. Over the past few months we have already deleted hundreds of erroneous and cheated entries manually, and are now moving towards systematically scanning the database for various inconsistencies, and deleting the bogus entries. The next pass will delete over 2100 cheated and broken entries, and further tests will be developed to weed out more junk.

Players found to be cheating on a large scale will have all their records deleted, including the (possibly) valid ones. This already happened to logins '0101010101010101001010101' and '_daniel-an_'.

Server admins

Besides making a donation when your server uses the Dedimania system, admins can also help by using the latest version of your controller or plugin of choice. Older versions typically contain bugs which can cause erroneous records, thus polluting the database, and do not meet all the requirements of the Dedimania API.

In the future the API requirements will be more strictly enforced, meaning that potential records submitted by obsolete tools will be ignored, so by not upgrading you're letting your players down.

Also, Rounds servers with multi-lap tracks should not alter the RoundForcedLaps value (<rounds_forcedlaps> in the match settings) unless you use the most recent version of a controller/plugin that handles this correctly. Currently FAST 3.2.x and XAseco v1.06 are known to be correct and, if not modified, those versions will simply not submit finishes to the central Dedimania system. Servers where the RoundForcedLaps test is missing or removed will eventually find records with too few laps to be ignored or deleted anyway.

In Rounds mode multi-lap tracks need to be raced with their default number of laps for fair comparison between players and servers. Some cleanup of inappropriate records has already been done, and systematic scans of the database for these situations will be performed soon.

Tool authors

Tool/plugin authors are already doing their part to support Dedimania by making it more widely available, but of course their donation is still welcome. More important however is that they insure their tool/plugin adheres to all API requirements.

Currently it is not mandatory to submit checkpoints data with each record, this was done for backward compatibility with old Onyx versions. But recent tools like CAR and plugins like some ADEDIMANIA variants (for Aseco/Rasp 2) also don't submit checkpoints. At some (yet to be announced) point in the future, such record submissions will no longer be accepted. Existing records without checkpoints (currently about 164000) may also be deleted partly or wholesale to enforce consistency after that point.

Also, to really combat cheating it will become necessary to submit validation replays with records, so authors will have to update their tool/plugin once Slig has worked out the API changes.

Outro

Thanks for reading and for your support of Dedimania. Please report bogus records in the Dedimania forum or in a PM to Xymph, do not clutter this thread with them. We do appreciate feedback on the above or any other Dedimania related topics.

Lastly, please point all your friends and all admins of your favorite servers to this thread, so that as many people in the Dedimania/TM community as possible become aware of the above discussion. Oh, and did I mention your donation is most welcome? :-) It will help renting that even bigger server.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
hal|Sascha
Pit Crew
Pit Crew
Posts: 671
Joined: 12 Aug 2005 16:22
Owned TM-games: TMU, TMN, TMS, TMO
Location: Germany Munich
Contact:

Re: The big Dedimania discussion

Post by hal|Sascha » 07 Apr 2009 21:20

Very important thread :)
I'll donate some euro's soon, hope it helps :) (btw. would be nice if the donation page would be in english^^, its french atm)
Just a question to the checkpoint thing. So the struct {'Login': string, 'Best': int, 'Checks': array of int} as documented should better be {'Login': string, 'Best': int}?

Ah yeah and I'm happy about the RCP 2.4% :mrgreen: (more then nothing :thumbsup: )
CPU: Intel Core 2 Duo E6600
Mainboard: Asus P5W DH Deluxe
RAM: 2 GB
Graphics: ATI Radeon X1950XTX
Audio: Soundblaster Audigy 4
Internet: ADSL 6Mbit
OS: Windows Vista Bussiness

Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: The big Dedimania discussion

Post by Xymph » 07 Apr 2009 21:30

hal|Sascha wrote:I'll donate some euro's soon, hope it helps :) (btw. would be nice if the donation page would be in english^^, its french atm)
I see it in English, but I asked Slig about that.
hal|Sascha wrote:Just a question to the checkpoint thing. So the struct {'Login': string, 'Best': int, 'Checks': array of int} as documented should better be {'Login': string, 'Best': int}?
No, the former is correct, the Checks field should be included, and will become required soon (breaking old tools that don't submit them).
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

nocturne
solid chaser
solid chaser
Posts: 1390
Joined: 08 Jun 2007 18:48
Owned TM-games: all
Contact:

Re: The big Dedimania discussion

Post by nocturne » 07 Apr 2009 21:49

I have an idea that can save dedimania... flush the db. It's already too corrupted to even be worthwhile anymore. If there are new requirements, the only records in the database should be the ones that pertain to them. All records without cps, either just delete them or call them 'unverified', and don't include in the standard lists.

User avatar
Slig
Pit Crew
Pit Crew
Posts: 2124
Joined: 05 Sep 2005 17:51
Owned TM-games: ALL
Location: TraxicoLand (Fr)
Contact:

Re: The big Dedimania discussion

Post by Slig » 07 Apr 2009 23:04

nocturne wrote:I have an idea that can save dedimania... flush the db. It's already too corrupted to even be worthwhile anymore. If there are new requirements, the only records in the database should be the ones that pertain to them. All records without cps, either just delete them or call them 'unverified', and don't include in the standard lists.
It will be done. Xymph is working with me to see what would be the best ting to do about it...
At least for tmuf/tmnf records those without cp will be removed, and also those with wrong number of cp. There will probably be some errors for maps where the number of cp itself is wrong, btw 4000000 records can't be handled manually.

I'll try to see how to separate old games/scripts records from forever ones : i don't want to break compatibility, but i'd like to stop having them pollute the main database.
Xymph wrote:
hal|Sascha wrote:I'll donate some euro's soon, hope it helps :) (btw. would be nice if the donation page would be in english^^, its french atm)
I see it in English, but I asked Slig about that.
should be fixed now (btw i don't know why some saw it in english and some others in french...)
Xymph wrote:
hal|Sascha wrote:Just a question to the checkpoint thing. So the struct {'Login': string, 'Best': int, 'Checks': array of int} as documented should better be {'Login': string, 'Best': int}?
No, the former is correct, the Checks field should be included, and will become required soon (breaking old tools that don't submit them).
It will become mandatory in 1 or 2 weeks, sure before the end of april. Records without cp times will then be rejected.
Note that for 'Checks', instead of an array of int you can send a string with comma separated times (ie something like: "1245,5695,34567") : it is supported and is far smaller when converted to xmlrpc.


I will try to extend the API to include validation replays before end of april too.

To try to help to avoid wrong and cheated records, all scripts maintainers should start to think about :
- verify that checkpoints indexes of a player run are positive and incremental.
- verify that checkpoints times are positive and incremental.
- verify that the last checkpoint has the same time as Finish time.
- for new player best, get its validation replay, then in it :
* that the map uid is same
* that the player login is same
* that the time is same
else don't send that time.
- the validation replay of the best sent time will have to be sent
* on dedimania side, all received records will be rejected if the validation replay differ from indicated uid and best player time.
- also, be sure to not send records in Rounds mode if RoundsForcedLaps is used !

Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: The big Dedimania discussion

Post by Xymph » 08 Apr 2009 08:43

nocturne wrote:I have an idea that can save dedimania... flush the db. It's already too corrupted to even be worthwhile anymore. If there are new requirements, the only records in the database should be the ones that pertain to them.
That's too drastic. Yes there are thousands of bogus entries, maybe even half a million or more, but that still means there are over 3.5 million valid records. Many thousands of players did their best to race them, so deleting them all would be a sure way to lose their trust. This thread aims for the opposite: to show we're actively working to improve the system and thus increase their trust in it.
nocturne wrote:All records without cps, either just delete them or call them 'unverified', and don't include in the standard lists.
Slig answered that already.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
hal|Sascha
Pit Crew
Pit Crew
Posts: 671
Joined: 12 Aug 2005 16:22
Owned TM-games: TMU, TMN, TMS, TMO
Location: Germany Munich
Contact:

Re: The big Dedimania discussion

Post by hal|Sascha » 08 Apr 2009 12:09

- Note that for 'Checks', instead of an array of int you can send a string with comma separated times (ie something like: "1245,5695,34567") : it is supported and is far smaller when converted to xmlrpc.
- verify that checkpoints indexes of a player run are positive and incremental.
- verify that checkpoints times are positive and incremental.
- verify that the last checkpoint has the same time as Finish time.
Added to rcp 4.0.3.0 :)
- for new player best, get its validation replay, then in it :
* that the map uid is same
* that the player login is same
* that the time is same
else don't send that time.
- the validation replay of the best sent time will have to be sent
* on dedimania side, all received records will be rejected if the validation replay differ from indicated uid and best player time.
Never tried that validation replay thing until now.
Some questions:
- can I get the validation replay just on "PlayerFinish" or later too?
- the documentation of GetValidationReplay isn't clear to me, which type of string this methods awaits? The player login?
- how can I check the data inside the replay? Does somebody may have a code snippets?
- how to send the base64 of the toprecord? any api changes?

- also, be sure to not send records in Rounds mode if RoundsForcedLaps is used !
Added to rcp 4.0.3.0 :)
CPU: Intel Core 2 Duo E6600
Mainboard: Asus P5W DH Deluxe
RAM: 2 GB
Graphics: ATI Radeon X1950XTX
Audio: Soundblaster Audigy 4
Internet: ADSL 6Mbit
OS: Windows Vista Bussiness

Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: The big Dedimania discussion

Post by Xymph » 08 Apr 2009 13:41

hal|Sascha wrote:Never tried that validation replay thing until now.
I think the technical details of implementing this feature belong in a separate thread, so I made that with my reply.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
xai
happy cruiser
happy cruiser
Posts: 162
Joined: 13 Nov 2008 15:46
Owned TM-games: TMU(F),TMN(F)
Manialink(s): xai

Re: The big Dedimania discussion

Post by xai » 10 Apr 2009 08:55

Xymph, you can remove the CAR part, as you probably noticed I made it into a Xaseco plugin ^^
Weeeeeeeeeeee

User avatar
jimos94
happy cruiser
happy cruiser
Posts: 122
Joined: 06 Jan 2007 22:52
Owned TM-games: TMUF
Manialink(s): fun
Contact:

Re: The big Dedimania discussion

Post by jimos94 » 10 Apr 2009 09:07

Good job to those who developed, are developing and probably or maybe ^^ will be developing Dedimania system .I hope there will be improvements about cheated records...
http://www.tm-planet.net/ Server hosting, Interviews, Events and more :)

Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: The big Dedimania discussion

Post by Xymph » 10 Apr 2009 09:10

xai wrote:Xymph, you can remove the CAR part, as you probably noticed I made it into a Xaseco plugin ^^
That's irrelevant in this context. There are still servers running CAR, thus records without checkpoints can still be submitted to the central database.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

donaldl43
smooth traffic navigator
smooth traffic navigator
Posts: 194
Joined: 18 Jan 2009 21:07
Owned TM-games: TMU
Contact:

Re: The big Dedimania discussion

Post by donaldl43 » 10 Apr 2009 17:36

I like the Dedimania concept but in my humble opinion it's a little too complicated to use as a programmer. You currently need a lot of programming to submit and get records.

I've mentioned in another thread that I would like to write a Python version of a Dedimania interface but it was pointed out just how involved that would be.

The documentation is also a bit lacking. Even if my PHP skills were top notch it would take quite a while to understand exactly what needs to be done to use it.

That said, I would be happy to donate and probably will since you're asking but I'm wondering if anything can be done to address my comments.

User avatar
w1lla
TM-Patrol
TM-Patrol
Posts: 1466
Joined: 23 May 2007 07:20
Owned TM-games: TMU, TMN, TMF
Manialink(s): intr
Location: Venray

Re: The big Dedimania discussion

Post by w1lla » 10 Apr 2009 20:40

Dedimania for Aseco will be continued next week when im free from my social work for school.

So please hold your hopes high.

Thnx...

Code: Select all

tmnforever is nations and united makes it special. tmnforever has united. I need united!

Xymph
Pit Crew
Pit Crew
Posts: 5653
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: The big Dedimania discussion

Post by Xymph » 11 Apr 2009 13:45

On my scans of the database I noticed one tool has particular trouble with submitting correct records with associated checkpoints. TM-DeReCo frequently submits entries with checkpoint lists like these:

Code: Select all

14630,31970,14750,32050,47740

10410,23950,30450,10410,10410,24260,30760,36200,10410,24060,30620,10410,23920,30320,10410,10410,24090,30570,36020,47680,50250,52150,10410,24260,30600,36100,47950,50330,51990,55920

7590,7530,23520,37490,45030,7550,23540,37710,44890,7540,23500,7520,23430,7520,7520,7540,23480,37510,44700,7560,23560,12740,29930,36120,48230,12830,30070,36430,12630,30180,36550,49250,12620,29590,35590,47690,12590,29620,35660,12720,10610,17590,28280,49030,51490,10470,17350,28420,40950,43380,48040,10440,17540,28130,10470,17390,27960,40840,43370,48400,10500,17440,10520,17530,28250,40870,43360,10420,24760,31520,36940,49170,51670,53460,57420

12360,17510,25870,32990,12360,17280,25370,32400,46180,58070
This looks to me as if the tool doesn't reset its internal CP list for the player when [s]he retires and can't make the finish in the first attempt. Thus all CPs ever crossed during the track are in the list.

Obviously this is unacceptable, and all such broken entries will be deleted from the database. We (Slig and I) urge the makers of TM-DeReCo to fix their tool immediately. Fortunately it isn't publicly available yet, and is in use on just a few servers, so only about 1500 records suffered from this problem, but fixing it is necessary to prevent it becoming much worse upon a public release.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

Speedy64
pedestrian
pedestrian
Posts: 7
Joined: 10 Feb 2009 10:56
Owned TM-games: TMUF, TMS

Re: The big Dedimania discussion

Post by Speedy64 » 11 Apr 2009 16:01

Oh you are right - thanks for the info.

I hope its already fixed in the current version (have worked on the checkpoint-thing this days) but i will look at it and also add one more check before sending to you.

if u have more/other problems feel free to contact me here or ingame under "speedy64" or in our small forum on trackmania.speedy64.de. nobody is perfect - but we are working on it ;)

(sry for my english - i'm german ;) )

Post Reply