Import feature is ready - need testers :)

ID field in step 2 should not be mandatory ...

lets say i just want to import a batch of records and dont need the update feature for them later anymore, i still have to define a unique field

Step 3 import statistics are wrong ..

that was the first import into a new cobalt installation -- it created 3 new records , not updated them

There is no possibility to select child settings so that every imported article binded to different parent.

I have a very clear idea how it should work.. i will try to explain later in detail with exact example

See child import :

mandatory requirement is to import parent CSV first

For every import a json file will be left in the tmp folder .... i have already more than 100 files there :D

Do yo mean files like 1360141764.json? Those ar used for progress bar. I cannot delete them because i do not know with wat delay JS will access it. SO just clear you TMP folder. This is in fact why it is saved in tmp. You can easily clean this folder and delete everything without thinking you delete something important.

yes, i did it already ... i just thought the cleaning should be handled by Cobalt to make it more "maintenance free" .. but a cron job can take care of this also ...

Hi

I've been testing the import facility and it works pretty well.

But I notice some possible issues I will ask about:

1) I get a warning for each line in the csv

Warning: Failed to move file!

(as per screenshot)

I wonder is this because of the space in incremented filename?

Or perhaps a better question would be: Is the space in the incremented filename by design or error?

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected?

2) About tmp files: As Sackgesicht points out the csv files arent removed - obviously I can remove them, but is it intentional/permanent that they are left in tmp dir?

3) About Import Settings: I select the correct Associate Fields and complete import, saving the Import Setting.

The json settings data appears to be correctly saved to db in _js_res_import.params.

But when running that settings again only the first field is selected. Other dropdowns are set to default 'Do not import anything here' and selections settings arent set as 'selected'in the drop downs. (as per screenshot)

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected?

I do not know anything about that. Id o not have any code that add this. May be this is done somewhere but I am not sure.

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected?

2) About tmp files: As Sackgesicht points out the csv files arent removed - obviously I can remove them, but is it intentional/permanent that they are left in tmp dir?

He mentioned about other files. CSV have to be deleted. Anyways all those fils are stored in tmp folder which you can totally clean from time to time.

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected?

2) About tmp files: As Sackgesicht points out the csv files arent removed - obviously I can remove them, but is it intentional/permanent that they are left in tmp dir?

1) I get a warning for each line in the csv

Looks like files and folders permissions.

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected?

2) About tmp files: As Sackgesicht points out the csv files arent removed - obviously I can remove them, but is it intentional/permanent that they are left in tmp dir?

1) I get a warning for each line in the csv

3) About Import Settings: I select the correct Associate Fields and complete import, saving the Import Setting.

It does remember options here locally. options might be not remembered if you upload different CSV with different set of fields.

ok with the csv files and cleaning out tmp, no problem

I dont think warnings are a perms problem. Or at least everything looks correct (644/755), and the file itself gets uploaded ok

I can see the warning is generated by libs/joomla/filesystem/file.php

707 on tmp doesnt make any difference

777 isnt a good idea

and thats where I got to when I started wondering about the space in filename

By Sergey Romanov Today

for example importfilename.csv becomes importfilename (1).csv

rather than importfilename(1).csv as I would have expected? 

I do not know anything about that. Id o not have any code that add this. May be this is done somewhere but I am not sure.

I think maybe its here in:

media/mint/uploader/UploadHandler.php

    protected function upcount_name_callback($matches)

    {

        $index = isset($matches[1]) ? intval($matches[1]) + 1 : 1;

        $ext = isset($matches[2]) ? $matches[2] : '';

        return '  (' . $index . ')' . $ext;

        //return '(' . $index . ')' . $ext;

    }

around line 440 the return inserts a space before the opening bracket

and then regex in the call to preg_replace_callback in the function below it looks for that space to validate the filename

    protected function upcount_name($name)

    {

        return preg_replace_callback('/(?:(?: \(([\d]+)\))?(\.[^.]+))?$/', array(

        //return preg_replace_callback('/(?:(?:\(([\d]+)\))?(\.[^.]+))?$/', array(

            $this,

            'upcount_name_callback'

        ), $name, 1);

    }

I think maybe, perhaps those commented changes take out the spurious space, but I'm not sure if that breaks something else.

Or if it matters anyway - cos thats not what causes the warnings :-)

OK so I have another question which is tied up with me still looking at importing via cli:

Would it be possible to save the import profile without actually doing the import?

So that in the import screens, at Step 2

At present you can setup a new profile or edit an existing one.

User then clicks 'Next' to 'Finalize Import'.

It might be helpful to have 2 options there instead of just 'Next':

Save Profile and Exit

Continue and Import Data

The object being to save the profile in the db without doing an import and then profile is available for other import options - such as via cli

anyway thanks for considering

777 isnt a good idea

This is what you actually should use. If you care for security just ass .htaccess to that folder that disallow ant php file to be accesses from this folder.

777 isnt a good idea

The object being to save the profile in the db without doing an import and then profile is available for other import options - such as via cli

You are creating CLI based on current import?

This is what you actually should use

I think thats a big debate. I'll just say that in my opinion its what I should not use and I'd prefer to come at it from fixing the problem rather than just set perms to 777

This is what you actually should use

If you care for security

I could suggest that just encouraging users to set 777 on a tmp dir is not caring for security.

But just for the record i tried it and the error is the same.

Anyway the import works despite warning messages, and so this is not a major problem for now.

I'll look at it again later.

You are creating CLI based on current import?

I'm not sure about your meaning of 'current import' so I'll explain...

I'm have to import 1k - 50k records into quite a lot of sites, and then do regular updates that need to be automated. So I am looking at options to do this from cli.

I saw some mentions of cli import somewhere in forum but iirc its not in your plan yet.

I can probably put togther a script to do inserts directly to db for my own case.

But I thought I'd see if I could work something around joomla cli and cobalt's own import stuff that might be generically useable for others.

So since cobalt already has the 'import setting' (or profile/configuration/whateveritscalled) in place I wondered about using the that as basis.

Do you have any thoughts about any of this? Any opinion pointers or tips will be most welcome.

Or even code I can build on? :D

I could suggest that just encouraging users to set 777 on a tmp dir is not caring for security.

I agree. But sometimes it is easiest way. Otherwise you have to now what you are doing. You have to reset all files and folders to FTP user, enable FTP in Joomla config and then 644 files and 755 folders are ok.

Another problem is some PHP methods like imagepng(). This method try to save file under apache or php user. And if everything is reset to FTP it may fail to save file.

This is why tmp folder should be always 777. So I can output anything there and then move to where I want tith JFile::copy() which will use FTP settings.

The only important thing is to prohibit access to this folder from HTTP. But that is tru for any folder in Joomla.

I could suggest that just encouraging users to set 777 on a tmp dir is not caring for security.

Do you have any thoughts about any of this? Any opinion pointers or tips will be most welcome.

Or even code I can build on?

Just look controllers/import.php and you will see all the cords. When you finish send it to me and I'll review code and make my suggestions.

Just look controllers/import.php

yes I already found it

I'm working thru it this morning. I'll let you know how it goes.

a couple points off topic but anyway...

I emailed you before from my subscription email to admin at mintjoomla but you said you didnt receive it.

my forum account email here shows not confirmed but I never received any confirmation email and there doesnt seem to be a way to send again.

a couple points off topic but anyway...

I emailed you before from my subscription email to admin at mintjoomla but you said you didnt receive it. I receive a lot of spam. Around 600 letters a day. Sometimes I jsut cannot check them all. May be it got there.

a couple points off topic but anyway...

I emailed you before from my subscription email to admin at mintjoomla but you said you didnt receive it.

my forum account email here shows not confirmed but I never received any confirmation email and there doesnt seem to be a way to send again.

There should be a button to send confirmation email again.

There should be a button to send confirmation email again

I dont see it? not with ff in fedora anyway

here's a screenshot showing no button and a json error

maybe thats the problem?

If your email is not confirmed, the link to send confirmation email should appear somewhere here