pepperstreet VIP
Total posts: 3,837
09 Jan 2016 13:25

Hello, I am importing older ME-Resources records (.CSV file).
The records‘ modified date (mtime) is always set to the current import date. Why is this!?
I would like to keep the original records infos as they are. How to keep mtime?
Would be useful to have it in fields assignment, too. Currently there is only ctime. :(


Related Idea / Request

In general, it would be nice to see and assign all core fields
i.e. all dates, hits, votes, metadata, alias, author etc.

Last Modified: 26 Mar 2017


Sackgesicht VIP
Total posts: 1,636
09 Jan 2016 19:44

The existing Import feature of Cobalt will not help you with a "Migration" from older ME Resources records.

It just "imports" basic data into limited cobalt fields. You will not be able to keep your core fields as you described.

Also be aware that the import only supports a limited number of records before it times out. You might have to split it in several import sessions.

As far as i remember, i described the migration from ME Resources to Cobalt 7 somewhere here. Then there are other changes again.

To achieve what you want, you need to do a manual upgrade of the ME Resources structure to Cobalt 8.


pepperstreet VIP
Total posts: 3,837
09 Jan 2016 20:17

Hello :)
For the current task, JSmigrate does a pretty good job. At least it is much easier for me ;)
It is not a full migration. I just need a limited set of data from a records archive. The new site does already exist including categories. The "old archive" has many different types, but the crucial data is Title, Category, Dates, Intro/Full texts, 1 Image
In Cobalt, everything goes into 1 type and respective categories.


IMHO, the import feature is too limited with CTIME.
And it lacks BASIC CORE fields. I think the basic record-infos should be supported in any case.
No matter if it is a simple import, or data "migration" from MEresources.

BTW, the ; delimiter option does not work.

Sackgesicht As far as i remember, i described the migration from ME Resources to Cobalt 7 somewhere here. Then there are other changes again.
To achieve what you want, you need to do a manual upgrade of the ME Resources structure to Cobalt 8.

Would like to know more ;)


pepperstreet VIP
Total posts: 3,837
09 Jan 2016 20:24

BTW, I found the params.php file with the CTIME and fields assignment. I guess, the MTIME can be added there. But how and where do I store it in the imported/parsed CSV... (not really my bag)


Sackgesicht VIP
Total posts: 1,636
09 Jan 2016 20:47

JSMigrate is lacking certain structures (if I remember correctly) ...

Example: the "fields" column in #__js_res_record, etc...

Another issue might be the search field results reindexed by the reindex tool, which will timeout/ consume too much memory with a bigger number of records. A better/faster approach might be a custom PHP/SQL reindex script based on a trusted fields column.

Maybe the best approach, if you dont want to do some manual conversion, is to enhance/hack the Cobalt import (which is at the end of the day also a "manual" process) to include everything you need.


pepperstreet VIP
Total posts: 3,837
12 Jan 2016 08:13

Sackgesicht JSMigrate is lacking certain structures (if I remember correctly) ... Example: the "fields" column in #__js_res_record, etc...>

Yes. I am aware of the limitations. It seems to support text/html fields. Also simple picture/image fields are resolved. I can get the plain image filename in CSV file.

I also have "escaped commas", which are not replaced by the Cobalt import. So this have to be done beforehand.


BTW, on JoomlaCode repository I have read a comment about an extended JSmigrate. It was created by a user. Unfortunately the user and the file is not available anywhere? Any clues where to get this version?

pepperstreet BTW, I found the params.php file with the CTIME and fields assignment. I guess, the MTIME can be added there. But how and where do I store it in the imported/parsed CSV... (not really my bag)

Still no success with MTIME.
I just managed to get the form field on the assignement step.
It appears right after CTIME. So far so good.
But I don't know where and how to store the value... ?!?


pepperstreet VIP
Total posts: 3,837
03 May 2016 15:19

pepperstreet

pepperstreet BTW, I found the params.php file with the CTIME and fields assignment. I guess, the MTIME can be added there. But how and where do I store it in the imported/parsed CSV... (not really my bag)

Still no success with MTIME. I just managed to get the form field on the assignement step. It appears right after CTIME. So far so good. But I don't know where and how to store the value... ?!?

Still would like to assign and keep the dates... any clue or code hint?


Sergey
Total posts: 13,748
05 May 2016 14:44

I do not have code hint for it. The only way is for me to add it. But I really do not want to go into importing code.


pepperstreet VIP
Total posts: 3,837
06 May 2016 01:21

:( mmh, still thinking that the current field assignment is missing a lot of basic record infos.
The most important part are the dates, IMHO. I don't get the idea why it just overrides everything(!) with the current date?


If no code hints, could you tell me the most important files that are involved? I was lost at the params.php


Sergey
Total posts: 13,748
06 May 2016 07:10

Еудд me case scenario where modified date is important to keep rather than "I want to keep it"?


pepperstreet VIP
Total posts: 3,837
06 May 2016 11:03

Sergey Еудд me case scenario where modified date is important to keep rather than "I want to keep it"?

Maybe "common sense"? I mean, why is there only Ctime?
I guess, many users have typical core dates. There is no chance to assign these dates.

The only way is to use another extra field. But that would prevent any later usage of these dates with Cobalt core date features?!

Some Examples:

  • Articles archive (initial date of first creation, last modified date...)
  • Products (initial or create date for first time "in-stock" info, expire date for re-order notification)
  • Events (Start and Expire dates)

In general, for FIRST TIME imports a user should be able to import as may core infos as he can.
Later UPDATES might be a different use case and story.


Sackgesicht VIP
Total posts: 1,636
06 May 2016 12:15

pepperstreet The only way is to use another extra field.

Unfortunately the IMPORT does not support DATE fields yet. Workaround is using a text field, then later change field type.

It would be more user friendly if import would support DATE fields. Even just for the default date format as stored in the record_values.

But I agree, core date fields should be available.


Sergey
Total posts: 13,748
10 May 2016 06:40

I added modification time.

And I also added import of single date to datetime field.


Sackgesicht VIP
Total posts: 1,636
10 May 2016 08:25

Sergey I added modification time.

And I also added import of single date to datetime field.

Excellent.

I think we would need a little documentation about import in general and individual fields and the expected content/format to avoid frustration when preparing the CSV file. Will try to outline something for it as a start.


Sergey
Total posts: 13,748
10 May 2016 11:19

Sackgesicht Will try to outline something for it as a start.

That would be awesome.


pepperstreet VIP
Total posts: 3,837
10 May 2016 18:28

Sergey

Sackgesicht Will try to outline something for it as a start. That would be awesome.

You're both...

awesome! :)

Sergey I added modification time.

What about the other core dates?
Is it possible to assign INITIAL-CREATE date and EXPIRE date, too?


Sackgesicht VIP
Total posts: 1,636
11 May 2016 03:51

While trying to come up with some sort of documentation outline, i realized that fields need further improvements to comply with their own settings when importing content.

Example: TEXT field

While it validates the "Regular Expression Validation" parameter from the field settings, it ignores the "Mask Settings", "Allow Raw HTML", "Unique Value", "Default Value" and "Maximum Length" which means at the end of the day that content might come into the system which does not comply with the field settings.

Better to create topics (field import) in the issue tracker for C9 on github and make a proper IMPORT one of the main new features in C9.

It would be good to have some additional parameter for IMPORT behaviour on the field settings.

Example : What to do on import longer than 'Maximum length' --> cut off characters/ignore whole content/replace with default

Optional reporting to an import log when exceptions happen would be another important IMPORT function ....


Sergey
Total posts: 13,748
13 May 2016 11:57

That is right. Import is a whole new field of improvements. Right now there is not validation of field values. Not all fields are importable. Very kimited import logic parameters and so on... It is indeed task for Cobalt 9.


pepperstreet VIP
Total posts: 3,837
14 May 2016 14:50

Sergey I added modification time.

Are we talking about Cobalt 8 ? If yes, is it available in next update release?

pepperstreet What about the other core dates? Is it possible to assign INITIAL-CREATE date and EXPIRE date, too?

Is it modified-date only?


Sergey
Total posts: 13,748
18 May 2016 13:57

pepperstreet Are we talking about Cobalt 8 ? If yes, is it available in next update release?

Yep, Cobalt 8.

pepperstreet Is it modified-date only?

Yes only this one. I do not want overcomplicate import form with tons of fields. Especialy if it is not needed exactly but just in case. Just because this field looks like it might be usefull.


pepperstreet VIP
Total posts: 3,837
20 May 2016 08:46

Sergey

pepperstreet Are we talking about Cobalt 8 ? If yes, is it available in next update release?

Yep, Cobalt 8.

Is there any delayed release this friday? newer than 8.700 ? I would like to test and use the import feature. Any chance to get the updated import files only?

pepperstreet Is it modified-date only?

Yes only this one. I do not want overcomplicate import form with tons of fields. Especialy if it is not needed exactly but just in case. Just because this field looks like it might be usefull.

Archives, Events and any "date-driven" content type needs EXPIRE dates, too.

PS: These are not "tons" of useless information, IMHO these are "basic" core infos.

Powered by Cobalt