Importing UTF-8 encoded CSV - Cobalt 8

Total posts: 18
16 Апр 2015 22:35

Hi,

don't know if only I had the problem to import UTF-8 encoded csv files. I was not able to import, if unicode characters are used. So I change the in controllers/import.php the function _load_csv as:

private function _load_csv($filename = '', $delimiter = ',')
{
    $header = NULL;
    if(($handle = fopen($filename, 'r')) !== FALSE)
    {
        while(($row = fgetcsv($handle, 2048, $delimiter)) !== FALSE)
        {
            if(!$header)
            {
                $header = $row;
            }
            else
            {
                $data = array_combine($header, $row);
                $data = array_map("utf8_encode", $data);
                $this->_row($data);
            }
        }
        fclose($handle);
    }
}

Just added the line $data = array_map("utf8_encode", $data);

Best, Dirk

Последние изменения: 29 Июнь 2016

Цитата

klox7

Total posts: 914
17 Апр 2015 09:06

Great, thanks for sharing.

Цитата

Sergey

Total posts: 13,748
22 Апр 2015 12:52

Updated source.

Цитата

pepperstreet VIP

Total posts: 3,837
09 Янв 2016 13:07

banana7777 Just added the line $data = array_map("utf8_encode", $data);

Funnily, I have no luck with this extra line!?
I have "norwegian" content with typical characters.
As far as I can tell, the CSV file is already in UTF-8 format. I tried different editors. The characters show-up correctly.
But after import, the characters are "converted" and unreadable. For instance å -> Ã¥

As soon as I remove/comment the line, everything works fine.
Any clues or thoughts?

It seems my file does not need a conversion. At least it is converted/encoded again.
I am far from understanding the utf8_encode documentation, but it seems it might need a check if the file is already in UTF-8 or another format...
(just thinking loud)

Цитата

Sergey

Total posts: 13,748
12 Янв 2016 09:44

pepperstreet but it seems it might need a check if the file is already in UTF-8

There is not such a PHP function. There is mb_* functions but those are additional extension and not installed with every PHP.

Цитата

pepperstreet VIP

Total posts: 3,837
12 Янв 2016 11:47

Sergey There is not such a PHP function. There is mb_* functions but those are additional extension and not installed with every PHP.

Ah, interesting. But are you sure? Looks like a standard function? The PHP Manual lists it in Reference:

mb_detect_encoding (PHP 4 >= 4.0.6, PHP 5, PHP 7)

Цитата

Sergey

Total posts: 13,748
15 Янв 2016 10:38

Php manual lists a lot of different extension functions that are not installed by default. And although MultiByte extension is on most of the installations, but not on all of them. Even iconv was a problem.

Anyway I cannot use it. My code is working taking in account that imported text is saved in UTF8.

Цитата

pepperstreet VIP

Total posts: 3,837
17 Янв 2016 01:12

Sergey My code is working taking in account that imported text is saved in UTF8.

o.k. but why does it "convert" the norwegian characters?

pepperstreet As far as I can tell, the CSV file is already in UTF-8 format. I tried different editors. The characters show-up correctly. But after import, the characters are "converted" and unreadable. For instance å -> Ã¥

Цитата

Sergey

Total posts: 13,748
18 Янв 2016 10:30

pepperstreet o.k. but why does it "convert" the norwegian characters?

Because that conversion should not be there. Add this conversion was suggested by only one person. Most probably it is not needed.

Цитата

pepperstreet VIP

Total posts: 3,837
18 Янв 2016 14:08

Sergey Because that conversion should not be there. And this conversion was suggested by only one person. Most probably it is not needed.

But you have implemented it ;) If it works for some but not all imports, what about adding a menuItem Parameter? So, one could choose to use this "conversion" or not.

Цитата

Sergey

Total posts: 13,748
25 Янв 2016 14:28

I think I have to delete it. I am not sure. I have to run some tests.

Цитата

banana7777 VIP