Aseco's/GbxRemote.inc.php's 64Bit problem

This is the place where you can find everything related to the dedicated server, control scripts and community tools.

Moderators: Pit Crew, TM-Patrol

User avatar
Slig
Pit Crew
Pit Crew
Posts: 2124
Joined: 05 Sep 2005 17:51
Owned TM-games: ALL
Location: TraxicoLand (Fr)
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Slig » 24 Apr 2008 14:14

Moriddin wrote:But a utf 8 character can differ from size 1 bytes to 4 bytes (according to the unicode value) you can't use strlen for the datasize
explained here http://nl.php.net/manual/en/function.strlen.php#72274.

strlen is a string manipulation function so this will return the number of characters. UTF8 is an storage/tranport encoding and you have to calculate the size in bytes of the utf8 string (also explained there).
It's just false, at least up to php5. Only iconv and mb_string functions take care of charset, so what you say is true for mb_strlen() with utf8 for example.

With php6 it will really be a problem anyway it seems, i just don't know how a byte based string/buffer will have to be handled, btw it's not for tomorrow.

Moriddin
road tourist
road tourist
Posts: 77
Joined: 21 Mar 2008 13:29
Owned TM-games: TMN, TMU, TMUF

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Moriddin » 24 Apr 2008 14:55

Hmm, i'm not a php expert. But i did read some more on the matter.

Apparently if youre systemlocale supports utf-8 then php <= 5 supports string handling in this format too (source).

Since in the past all php based tm programs worked ok with all the unicode symbols I have to assume that this is handled in utf8 format (pure unicode double byte characters are not supported until php version 6).

So it should indeed be a wrong move to encode them (again).

But still the datapackage size keeps bugging me then.

For example:

Code: Select all

  Stringvalue : TËST
  Byte data (latin-1, 4 bytes, 4 characters) : 0x54 0xCB 0x53 0x54
  Byte Data (UTF-8, 5 bytes, 4 characters) : 0x54 0xC3 0x8B 0x53 0x54
If you use strlen in the pack method then the server will only read 4 bytes instead of 5 out of the socket.

Maybe i'm wrong with my assumption what type of string is used during php execution (I just can't find how its even posible that all those pretty names of tm players can be correctly used in a non unicode system like php) but on low level transport this should be correct.

User avatar
Phhere
sunday driver
sunday driver
Posts: 65
Joined: 16 Oct 2006 21:01
Owned TM-games: TMS, TMU
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Phhere » 24 Apr 2008 15:06

why not use mb_strlen($str) ?

Code: Select all

function strlen_mb($str){
if(function_exists("mb_strlen")) return mb_strlen($str);
else return strlen($str);
}
and change line 524 - 528 to

Code: Select all

		if ($this->protocol == 1) {
			$request = pack('Va*', strlen_mb($xml), $xml);
		} else {
			$request = pack('VVa*', strlen_mb($xml), $this->reqhandle, $xml);
		}
This code is untested but i think it should work.
*Sorry for my english*
webSPELL Development
Image

User avatar
Slig
Pit Crew
Pit Crew
Posts: 2124
Joined: 05 Sep 2005 17:51
Owned TM-games: ALL
Location: TraxicoLand (Fr)
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Slig » 24 Apr 2008 15:25

Moriddin wrote: (source).
your source indicate clearly that all function like strlen do byte count/index and not character count/index
Since in the past all php based tm programs worked ok with all the unicode symbols I have to assume that this is handled in utf8 format
exactly, if strlen in the line you speak about was not counting bytes, you would not have any chat or manialink at all working as soon there is some utf-8 char in it ;)
(pure unicode double byte characters are not supported until php version 6).
"pure unicode" means nothing, all utf8, utf16, ucs2 etc. are unicode. You mean ucs2, used by windows and java. And it seems that php6 will use it too, which will make for our scripts many difficulties, so they will probably stay php5 only long time...

Code: Select all

  Stringvalue : TËST
  Byte data (latin-1, 4 bytes, 4 characters) : 0x54 0xCB 0x53 0x54
  Byte Data (UTF-8, 5 bytes, 4 characters) : 0x54 0xC3 0x8B 0x53 0x54
If you use strlen in the pack method then the server will only read 4 bytes instead of 5 out of the socket.
did you really tested yourself php5 strlen() in both cases ?

Moriddin
road tourist
road tourist
Posts: 77
Joined: 21 Mar 2008 13:29
Owned TM-games: TMN, TMU, TMUF

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Moriddin » 24 Apr 2008 15:34

did you really tested yourself php5 strlen() in both cases ?
Tested it and : omg, i'm indeed a php noob I see :oops:

in all other languages I know str_len() returns the number of charaters (and that topic on php.net said the same).
Better test it next time instead of reading it from php.net :roflol:

User avatar
Slig
Pit Crew
Pit Crew
Posts: 2124
Joined: 05 Sep 2005 17:51
Owned TM-games: ALL
Location: TraxicoLand (Fr)
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Slig » 24 Apr 2008 15:48

Moriddin wrote:in all other languages I know str_len() returns the number of charaters (and that topic on php.net said the same).
Using utf8 (and not ucs2!) ? in which language ?

Moriddin
road tourist
road tourist
Posts: 77
Joined: 21 Mar 2008 13:29
Owned TM-games: TMN, TMU, TMUF

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Moriddin » 24 Apr 2008 16:04

I'm used to work with ansi and widestrings instead of encoded strings, this was my error.

Normally if I need unicode then I use a widestring internal in the program. Only if you store (or send it over a socket) you convert it to utf-8. When manipulating strings (like parsing) its much faster to always know in which byte position a certian char is located. Following this line it never occurred to me that php could/would manipulate encoded strings.

btw, i'm used to pascal and c(++), and when porting the transport layer to pascal I just thought that this was very odd in the php file.

Xymph
Pit Crew
Pit Crew
Posts: 5726
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Xymph » 20 May 2008 10:45

I applied a few changes to the GbxRemote module, after analyzing an XAseco problem that partly resulted from two minor issues in this file. My original resetError method was wrong as it caused the isError method to always return true, and the query method could produce unpack warnings when there was no data due to a dedicated crash.

Here are the change notes:
Release 2008-05-20 - Xymph:
Prevented unpack() warnings in query method when the connection dies
Changed resetError method to assign 'false' for correct isError method
Tweaked some 'transport error' messages
The regular and big-endian versions of the modules can be downloaded from my page.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
hal|Sascha
Pit Crew
Pit Crew
Posts: 671
Joined: 12 Aug 2005 16:22
Owned TM-games: TMU, TMN, TMS, TMO
Location: Germany Munich
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by hal|Sascha » 26 May 2008 07:57

Nice, had that unpack problem reported from some rcp users, will try it :mrgreen:
CPU: Intel Core 2 Duo E6600
Mainboard: Asus P5W DH Deluxe
RAM: 2 GB
Graphics: ATI Radeon X1950XTX
Audio: Soundblaster Audigy 4
Internet: ADSL 6Mbit
OS: Windows Vista Bussiness

User avatar
hal|Sascha
Pit Crew
Pit Crew
Posts: 671
Joined: 12 Aug 2005 16:22
Owned TM-games: TMU, TMN, TMS, TMO
Location: Germany Munich
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by hal|Sascha » 29 May 2008 12:50

Just a little more feedback :mrgreen:

Tested it and works fine, great work! THX! :)
CPU: Intel Core 2 Duo E6600
Mainboard: Asus P5W DH Deluxe
RAM: 2 GB
Graphics: ATI Radeon X1950XTX
Audio: Soundblaster Audigy 4
Internet: ADSL 6Mbit
OS: Windows Vista Bussiness

Gekko
speedy pilot
speedy pilot
Posts: 550
Joined: 03 Sep 2006 18:20
Owned TM-games: ALL
Location: Naples - Italy
Contact:

Re: Aseco's/GbxRemote.inc.php's 64Bit problem

Post by Gekko » 03 Jun 2008 19:51

I'm tring it too.

Post Reply