Sybase NNTP forums - End Of Life (EOL)

The NNTP forums from Sybase - forums.sybase.com - are now closed.

All new questions should be directed to the appropriate forum at the SAP Community Network (SCN).

Individual products have links to the respective forums on SCN, or you can go to SCN and search for your product in the search box (upper right corner) to find your specific developer center.

False duplicate key error

8 posts in General Discussion Last posting was on 2009-07-29 07:42:19.0Z
boekhold@gmail.com Posted on 2009-07-25 07:39:55.0Z
From: "boekhold@gmail.com" <boekhold@gmail.com>
Newsgroups: sybase.public.ase.general
Subject: False duplicate key error
Date: Sat, 25 Jul 2009 00:39:55 -0700 (PDT)
Organization: http://groups.google.com
Lines: 40
Message-ID: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com>
NNTP-Posting-Host: 195.149.65.71
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1256
Content-Transfer-Encoding: quoted-printable
X-Trace: posting.google.com 1248507595 16833 127.0.0.1 (25 Jul 2009 07:39:55 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Sat, 25 Jul 2009 07:39:55 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: w41g2000yqb.googlegroups.com; posting-host=195.149.65.71; posting-account=W4ZeygoAAACoZ0L7EZ5SeHMMvEmtKh_I
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.33 Safari/530.5,gzip(gfe),gzip(gfe)
Path: forums-1-dub!forums-master!newssvr.sybase.com!news-sj-1.sprintlink.net!news-peer1.sprintlink.net!nntp1.phx1.gblx.net!nntp.gblx.net!nntp.gblx.net!border2.nntp.dca.giganews.com!nntp.giganews.com!postnews.google.com!w41g2000yqb.googlegroups.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28048
Article PK: 77295

Hi all,

I have a database that started of with a default DB encoding of iso
(ISO8859-1). I used sqlloc to convert this DB into UTF-8, using
"Binary ordering, for the ISO 10646-1, UTF-8 multibyte encoding" for
the default sort order. One of the tables has a "name varchar(255)"
column with a unique index on it.

I'm trying to insert a specific row into that table with "name" set to
(in arabic, in case this gets screwed up by the newsgroup) "صندوق
التكافل الاجتماعي لمنسوبي وحدات الخدمات الطبية للقوات المسلحة بالخرج".
This fails because there already is a row with that same value of
"name" (Attempt to insert duplicate key row in object 'Cpty' with
unique index 'CptyIdx3'). So in my program I modify the data so that
I'm trying to insert

صندوق التكافل الاجتماعي لمنسوبي وحدات الخدمات الطبية للقوات المسلحة
بالخرج - 71043114000002

(this mixing of left-to-right and right-to-left writing is messing up
my browser a bit, but basically that string should read "concat
(orig_name, ' - 71043114000002')").

Strange thing is, that second insert /also/ gives me a duplicate key
error on the same index. This is happening with only 3 rows out of my
total of 89931 rows of test data. In all 3 cases, the value of "name"
that I'm trying to insert is very long, over 128 characters.

Is there some limit in Sybase varchar indices where it only indexes
the first 128 characters? Or is there some character set encoding
magic going on that causes this? I can imagine that the UTF-8
characters get expanded to 2 bytes for example (although in my UTF-8
source file these names are all well below 255 bytes, in the example
above the original arabic name is 139 bytes long).

Kind regards, Maarten


Neal Stack [Sybase] Posted on 2009-07-27 16:09:44.0Z
Message-ID: <4A6DD147.2000305@nospam.com>
From: "Neal Stack [Sybase]" <nstack@nospam.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
Newsgroups: sybase.public.ase.general
To: "boekhold@gmail.com" <boekhold@gmail.com>
Subject: Re: False duplicate key error
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com>
In-Reply-To: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
NNTP-Posting-Host: vip152.sybase.com
X-Original-NNTP-Posting-Host: vip152.sybase.com
Date: 27 Jul 2009 09:09:44 -0700
X-Trace: forums-3-dub.sybase.com 1248710984 10.22.241.152 (27 Jul 2009 09:09:44 -0700)
X-Original-Trace: 27 Jul 2009 09:09:44 -0700, vip152.sybase.com
Lines: 74
Path: forums-1-dub!forums-master!forums-3-dub.sybase.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28053
Article PK: 77298

Hello,

I suspect your data might be getting truncated. The default behavior for ASE
is to silently truncate character data.

Try doing a "set string_rtruncation on" before you attempt your insert.

You might also make use of the char_length() and datalength() built in functions:
http://infocenter.sybase.com/help/topic/com.sybase.help.ase_15.0.blocks/html/blocks/blocks42.htm

For example, I insert one character here but it takes two bytes of storage on my UTF8 server:

1> create table t4 (c1 char(2))
2> go
1> insert into t4 values ('á') /* 0xE1 */
2> go
(1 row affected)
1> select char_length(c1), datalength(c1) from t4
2> go

----------- -----------
1 2

(1 row affected)
1> select convert(binary, c1) from t4
2> go

--------------------------------------------------------------
0xc3a100000000000000000000000000000000000000000000000000000000

(1 row affected)


Regards,
Neal

boekhold@gmail.com wrote:
> Hi all,
>
> I have a database that started of with a default DB encoding of iso
> (ISO8859-1). I used sqlloc to convert this DB into UTF-8, using
> "Binary ordering, for the ISO 10646-1, UTF-8 multibyte encoding" for
> the default sort order. One of the tables has a "name varchar(255)"
> column with a unique index on it.
>
> I'm trying to insert a specific row into that table with "name" set to
> (in arabic, in case this gets screwed up by the newsgroup) "صندوق
> التكافل الاجتماعي لمنسوبي وحدات الخدمات الطبية للقوات المسلحة بالخرج".
> This fails because there already is a row with that same value of
> "name" (Attempt to insert duplicate key row in object 'Cpty' with
> unique index 'CptyIdx3'). So in my program I modify the data so that
> I'm trying to insert
>
> صندوق التكافل الاجتماعي لمنسوبي وحدات الخدمات الطبية للقوات المسلحة
> بالخرج - 71043114000002
>
> (this mixing of left-to-right and right-to-left writing is messing up
> my browser a bit, but basically that string should read "concat
> (orig_name, ' - 71043114000002')").
>
> Strange thing is, that second insert /also/ gives me a duplicate key
> error on the same index. This is happening with only 3 rows out of my
> total of 89931 rows of test data. In all 3 cases, the value of "name"
> that I'm trying to insert is very long, over 128 characters.
>
> Is there some limit in Sybase varchar indices where it only indexes
> the first 128 characters? Or is there some character set encoding
> magic going on that causes this? I can imagine that the UTF-8
> characters get expanded to 2 bytes for example (although in my UTF-8
> source file these names are all well below 255 bytes, in the example
> above the original arabic name is 139 bytes long).
>
> Kind regards, Maarten


boekhold@gmail.com Posted on 2009-07-28 06:43:21.0Z
From: "boekhold@gmail.com" <boekhold@gmail.com>
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
Date: Mon, 27 Jul 2009 23:43:21 -0700 (PDT)
Organization: http://groups.google.com
Lines: 37
Message-ID: <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com>
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com>
NNTP-Posting-Host: 195.149.65.71
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64
X-Trace: posting.google.com 1248763401 13087 127.0.0.1 (28 Jul 2009 06:43:21 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Tue, 28 Jul 2009 06:43:21 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: d4g2000yqa.googlegroups.com; posting-host=195.149.65.71; posting-account=W4ZeygoAAACoZ0L7EZ5SeHMMvEmtKh_I
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.33 Safari/530.5,gzip(gfe),gzip(gfe)
Path: forums-1-dub!forums-master!newssvr.sybase.com!news-sj-1.sprintlink.net!news-peer1.sprintlink.net!newsfeed.yul.equant.net!novia!news-out.readnews.com!transit4.readnews.com!postnews.google.com!d4g2000yqa.googlegroups.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28059
Article PK: 77308

Hi Neal,

On Jul 27, 8:09 pm, "Neal Stack [Sybase]" <nst...@nospam.com> wrote:
> Hello,
>
> I suspect your data might be getting truncated.  The default behavior for ASE
> is to silently truncate character data.
>
> Try doing a "set string_rtruncation on" before you attempt your insert.
>
> You might also make use of the char_length() and datalength() built in functions:
>    http://infocenter.sybase.com/help/topic/com.sybase.help.ase_15.0.bloc...

Yes, you are right:

1> select datalength('مكتبة الارشاد فرع شركة سالم عبدالله بالعمش
لاصحابها سعيد عبدالله وعبدالله محمد بالعمش وشركاهم')
2> go

-----------
333

(1 row affected)
1> select char_length('مكتبة الارشاد فرع شركة سالم عبدالله بالعمش
لاصحابها سعيد عبدالله وعبدالله محمد بالعمش وشركاهم')
2> go

-----------
173

(1 row affected)

So in this case, this name is 173 characters long, but Sybase will use
333 bytes of space to store this. The 'col' column is defined as
"varchar(255)", which means there are 255 bytes of storage available.
Sybase will in fact silently truncate the data in order to fit it into
that column.

If you disable that "silent truncation", you will see the following
(correct) error message:

1> set string_rtruncation on
2> go
1> insert into mbtest (col) values ('مكتبة الارشاد فرع شركة سالم
عبدالله بالعمش لاصحابها سعيد عبدالله وعبدالله محمد بالعمش وشركاهم')
2> go
Msg 9502, Level 16, State 19:
Server 'KGRPROD', Line 1:
Data exception - string data right truncated
1>

The strange thing is that for this customer, Excel insists that the
name is only 93 characters long, and in my source file, it 'only'
takes 139 bytes. And that file is in UTF8 as well, so I would expect
Sybase to need 93chars/139bytes as well. So how is it getting to 173
chars and 333 bytes?!?!?!

Could client encoding have something to do with this?

Rgds, Maarten


boekhold@gmail.com Posted on 2009-07-28 07:15:10.0Z
From: "boekhold@gmail.com" <boekhold@gmail.com>
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
Date: Tue, 28 Jul 2009 00:15:10 -0700 (PDT)
Organization: http://groups.google.com
Lines: 9
Message-ID: <f9f24417-d3ce-4f8a-8f64-b7178230a1c8@w6g2000yqw.googlegroups.com>
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com> <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com>
NNTP-Posting-Host: 195.149.65.71
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1248765310 7958 127.0.0.1 (28 Jul 2009 07:15:10 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Tue, 28 Jul 2009 07:15:10 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: w6g2000yqw.googlegroups.com; posting-host=195.149.65.71; posting-account=W4ZeygoAAACoZ0L7EZ5SeHMMvEmtKh_I
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.33 Safari/530.5,gzip(gfe),gzip(gfe)
Path: forums-1-dub!forums-master!newssvr.sybase.com!news-sj-1.sprintlink.net!news-peer1.sprintlink.net!newsfeed.yul.equant.net!nntp1.roc.gblx.net!nntp.gblx.net!nntp.gblx.net!nlpi057.nbdc.sbc.com!prodigy.net!border1.nntp.dca.giganews.com!nntp.giganews.com!postnews.google.com!w6g2000yqw.googlegroups.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28060
Article PK: 77304

Correction:

> The strange thing is that for this customer, Excel insists that the
> name is only 93 characters long, and in my source file, it 'only'
> takes 139 bytes. And that file is in UTF8 as well, so I would expect

It's actually 173 bytes...

Rgds


Neal Stack [Sybase] Posted on 2009-07-28 14:41:24.0Z
Message-ID: <4A6F0E12.1020004@nospam.com>
From: "Neal Stack [Sybase]" <nstack@nospam.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com> <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com>
In-Reply-To: <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
NNTP-Posting-Host: vip152.sybase.com
X-Original-NNTP-Posting-Host: vip152.sybase.com
Date: 28 Jul 2009 07:41:24 -0700
X-Trace: forums-3-dub.sybase.com 1248792084 10.22.241.152 (28 Jul 2009 07:41:24 -0700)
X-Original-Trace: 28 Jul 2009 07:41:24 -0700, vip152.sybase.com
Lines: 51
Path: forums-1-dub!forums-master!forums-3-dub.sybase.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28062
Article PK: 77310

Hello,

I can think of two reasons why you might see expansion:
1) Bcp gets confused about where one column ends and the next begins.
2) Your source data file is not actually utf8 but maybe cp1256.

If your source file is truly utf8 than I would not expect much difference in size
between what you see in ASE and what is in the file itself.

Are you using bcp to copy the file in? You have to be careful about what you specify
as row & column terminators because if you have embedded control characters in your
text (like tab), bcp can combine or split the rows unexpectedly.

I just used a single row of data that I copied from your email.
I pasted it into Excel and did a "save as" to a Unicode text file (Arabic_utf8.txt).
Notice that I had to use -r "\r\0\n\0" as the row terminator.

So in this first example, my Excel utf8 data file is 140 characters (136 data and 4 for the "\r\0\n\0").

07/28/2009 07:37 AM 140 Arabic_utf8.txt
1 File(s) 140 bytes
0 Dir(s) 29,877,370,880 bytes free

C:\Temp>od -x Arabic_utf8.txt
0000000000 FEFF 0627 0644 062A 0643 0627 0641 0644
0000000020 0020 0627 0644 0627 062C 062A 0645 0627
0000000040 0639 064A 0020 0644 0645 0646 0633 0648
0000000060 0628 064A 0020 0648 062D 062F 0627 062A
0000000100 0020 0627 0644 062F 0645 0627 062A 0020
0000000120 0627 0644 0637 0628 064A 0629 0020 0644
0000000140 0644 0642 0648 0627 062A 0020 0627 0644
0000000160 0645 0633 0644 062D 0629 0020 0628 0627
0000000200 0644 062E 0631 062C 000D 000A

C:\Temp>bcp tempdb..arabic in Arabic_utf8.txt -Usa -P -J utf8 -c -r "\r\0\n\0"

Starting copy...

1 rows copied.
Clock Time (ms.): total = 31 Avg = 31 (32.26 rows per sec.)

C:\Temp>isql -Usa -P
1> use tempdb
2> go
1> select datalength(c1), char_length(c1) from arabic
2> go

----------- -----------
136 136

(1 row affected)


boekhold@gmail.com Posted on 2009-07-28 16:22:21.0Z
From: "boekhold@gmail.com" <boekhold@gmail.com>
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
Date: Tue, 28 Jul 2009 09:22:21 -0700 (PDT)
Organization: http://groups.google.com
Lines: 20
Message-ID: <22e52ecd-1c6b-478c-8725-f936c7c591fb@k19g2000yqn.googlegroups.com>
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com> <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com> <4A6F0E12.1020004@nospam.com>
NNTP-Posting-Host: 212.71.32.94
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1248798141 26873 127.0.0.1 (28 Jul 2009 16:22:21 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Tue, 28 Jul 2009 16:22:21 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: k19g2000yqn.googlegroups.com; posting-host=212.71.32.94; posting-account=W4ZeygoAAACoZ0L7EZ5SeHMMvEmtKh_I
User-Agent: G2/1.0
X-HTTP-Via: 1.1 NetCache10 (NetCache NetApp/6.0.5)
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.33 Safari/530.5,gzip(gfe),gzip(gfe)
Path: forums-1-dub!forums-master!newssvr.sybase.com!news-sj-1.sprintlink.net!news-peer1.sprintlink.net!nntp1.phx1.gblx.net!nntp.gblx.net!nntp.gblx.net!newsfeed.news2me.com!news.glorb.com!news2.glorb.com!postnews.google.com!k19g2000yqn.googlegroups.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28063
Article PK: 77309

Hello,

> I can think of two reasons why you might see expansion:
> 1) Bcp gets confused about where one column ends and the next begins.
> 2) Your source data file is not actually utf8 but maybe cp1256.

Hmmm, my test as described in my previous post was actually just CTRL-
C in Excel, then paste into a putty terminal (set to UTF-8) directly
into ISQL. All done on a Windows PC. Perhaps Windows is doing some
conversion to cp1256 here.

Having said that, the source file /is/ UTF-8, 100%. I don't use BCP to
get the data into the table. In fact, my application is not actually
the one doing the insert. I have an import application that sends
messages to a server application which does the actual insert.

I think I need to do a little test with my source file, a dummy table
and BCP...

Maarten


Neal Stack [Sybase] Posted on 2009-07-28 16:56:33.0Z
Message-ID: <4A6F2DC1.8020104@nospam.com>
From: "Neal Stack [Sybase]" <nstack@nospam.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com> <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com> <4A6F0E12.1020004@nospam.com> <22e52ecd-1c6b-478c-8725-f936c7c591fb@k19g2000yqn.googlegroups.com>
In-Reply-To: <22e52ecd-1c6b-478c-8725-f936c7c591fb@k19g2000yqn.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
NNTP-Posting-Host: vip152.sybase.com
X-Original-NNTP-Posting-Host: vip152.sybase.com
Date: 28 Jul 2009 09:56:33 -0700
X-Trace: forums-3-dub.sybase.com 1248800193 10.22.241.152 (28 Jul 2009 09:56:33 -0700)
X-Original-Trace: 28 Jul 2009 09:56:33 -0700, vip152.sybase.com
Lines: 35
Path: forums-1-dub!forums-master!forums-3-dub.sybase.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28065
Article PK: 77311

I don't have Putty installed but with MS-DOS you have to change
the code page and use a font that works with cutting and pasting:

C:\>chcp
Active code page: 437

C:\>chcp 1256
Active code page: 1256


Regards,
Neal

boekhold@gmail.com wrote:
> Hello,
>
>> I can think of two reasons why you might see expansion:
>> 1) Bcp gets confused about where one column ends and the next begins.
>> 2) Your source data file is not actually utf8 but maybe cp1256.
>
> Hmmm, my test as described in my previous post was actually just CTRL-
> C in Excel, then paste into a putty terminal (set to UTF-8) directly
> into ISQL. All done on a Windows PC. Perhaps Windows is doing some
> conversion to cp1256 here.
>
> Having said that, the source file /is/ UTF-8, 100%. I don't use BCP to
> get the data into the table. In fact, my application is not actually
> the one doing the insert. I have an import application that sends
> messages to a server application which does the actual insert.
>
> I think I need to do a little test with my source file, a dummy table
> and BCP...
>
> Maarten


boekhold@gmail.com Posted on 2009-07-29 07:42:19.0Z
From: "boekhold@gmail.com" <boekhold@gmail.com>
Newsgroups: sybase.public.ase.general
Subject: Re: False duplicate key error
Date: Wed, 29 Jul 2009 00:42:19 -0700 (PDT)
Organization: http://groups.google.com
Lines: 20
Message-ID: <19f2070c-573f-4ef2-bf41-8e77e996b0ce@k30g2000yqf.googlegroups.com>
References: <c322d458-6532-44ad-bad5-509a4676bce2@w41g2000yqb.googlegroups.com> <4A6DD147.2000305@nospam.com> <4acb6938-3bee-4a8b-b977-4332d0599718@d4g2000yqa.googlegroups.com> <4A6F0E12.1020004@nospam.com> <22e52ecd-1c6b-478c-8725-f936c7c591fb@k19g2000yqn.googlegroups.com>
NNTP-Posting-Host: 195.149.65.71
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1248853340 13305 127.0.0.1 (29 Jul 2009 07:42:20 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Wed, 29 Jul 2009 07:42:20 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: k30g2000yqf.googlegroups.com; posting-host=195.149.65.71; posting-account=W4ZeygoAAACoZ0L7EZ5SeHMMvEmtKh_I
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.33 Safari/530.5,gzip(gfe),gzip(gfe)
Path: forums-1-dub!forums-master!newssvr.sybase.com!news-sj-1.sprintlink.net!news-peer1.sprintlink.net!nntp1.phx1.gblx.net!nntp.gblx.net!nntp.gblx.net!newsfeed.news2me.com!nx02.iad01.newshosting.com!newshosting.com!69.16.185.11.MISMATCH!npeer01.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!postnews.google.com!k30g2000yqf.googlegroups.com!not-for-mail
Xref: forums-1-dub sybase.public.ase.general:28066
Article PK: 77312

Hello,

> I think I need to do a little test with my source file, a dummy table
> and BCP...

OK, that pointed me to the right direction. After I imported my source
file directly into a temp table with bcp -J utf8, the Arabic text not
only displayed correctly in isql (well, sqsh actually), the datalength
() and char_length() were as expected as well.

Turns out my server application didn't automatically pick up the
database encoding, and inserted the data in whatever kind of charset
it felt like (still wonder why the front-end application did display
the arabic correctly though!). After telling the server application
explicitly to use utf8 everywhere the data arrives in the DB
correctly.

Thanks for your pointers Neal.

Regards, Maarten