Manual import of repositories

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Manual import of repositories

The_Danimal

I have a large number of artifacts to import from another installation of
Artifactory, but don't have the memory needed to import through the web UI
without crashing it with OOMEs. I deployed it as a war in JBoss. Is there a
way to import the data manually through a terminal or something?

Thank you for the time,
Daniel.
--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Manual import of repositories

Yoav Landman
Administrator
Hi Daniel,

High memory peaks on import can be caused by the indexing threads indexing a large amount of data in parallel.
What's the max heap size you have set up? (I assume it crashes on heap not on the perm gen). Also, what's the Artifatory version and are you using MySQL or the built-in Derby?
Repository content can also be imported through the REST API, by using "artadmin import" command (http://wiki.jfrog.org/confluence/display/RTF/The+artadmin+Command+Line+Tool). This will require less memory. Just use a layout similar to a system export - you do not have to have all the repository folders or configuration files present, as long as the repository name you are importing into is already defined in Artifactory (import will be additive).

Thanks,

Yoav

On Mon, Jun 15, 2009 at 6:59 PM, daniel.green <[hidden email]> wrote:

I have a large number of artifacts to import from another installation of
Artifactory, but don't have the memory needed to import through the web UI
without crashing it with OOMEs. I deployed it as a war in JBoss. Is there a
way to import the data manually through a terminal or something?

Thank you for the time,
Daniel.
--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Manual import of repositories

The_Danimal

Hello Yoav,

Thank you for the help. I am currently running Artifactory 2.0.6 (rev.
5625). I was getting other OOMEs, but after I tweaked the settings it's not
crashing on the perm gen, but taking much longer to
do so.

I am using the following to set heap min/max and perm size:

-Xms1024m -Xmx1024m -XX:MaxPermSize=128m

Also, I believe I am simply using Derby as I did not configure MySql ;-).
The server I'm running this on only has 2gig allocated to it.

The wiki mentioned $ARTIFACTORY_HOME, however I deployed Artifactory as a
war. Was this the wrong way to go? I am using JBOSS to host.

Thank you again for the help,
Daniel.


Yoav  Landman wrote:

>
> Hi Daniel,
>
> High memory peaks on import can be caused by the indexing threads indexing
> a
> large amount of data in parallel.
> What's the max heap size you have set up? (I assume it crashes on heap not
> on the perm gen). Also, what's the Artifatory version and are you using
> MySQL or the built-in Derby?
> Repository content can also be imported through the REST API, by using
> "artadmin import" command (
> http://wiki.jfrog.org/confluence/display/RTF/The+artadmin+Command+Line+Tool).
> This will require less memory. Just use a layout similar to a system
> export
> - you do not have to have all the repository folders or configuration
> files
> present, as long as the repository name you are importing into is already
> defined in Artifactory (import will be additive).
>
> Thanks,
>
> Yoav
>
> On Mon, Jun 15, 2009 at 6:59 PM, daniel.green <[hidden email]>
> wrote:
>
>>
>> I have a large number of artifacts to import from another installation of
>> Artifactory, but don't have the memory needed to import through the web
>> UI
>> without crashing it with OOMEs. I deployed it as a war in JBoss. Is there
>> a
>> way to import the data manually through a terminal or something?
>>
>> Thank you for the time,
>> Daniel.
>> --
>> View this message in context:
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
>> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Crystal Reports - New Free Runtime and 30 Day Trial
>> Check out the new simplified licensing option that enables unlimited
>> royalty-free distribution of the report engine for externally facing
>> server and web deployment.
>> http://p.sf.net/sfu/businessobjects
>> _______________________________________________
>> Artifactory-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Artifactory-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24054924.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Manual import of repositories

Yoav Landman
Administrator
Hi Daniel,

1G of heap should be more than enough for the standalone version, but since you are running on JBoss you are already sharing this with existing services and applications, so you might be running short.
It is not mandatory to specify the $ARTIFACTORY_HOME location - when not specified (by setting the ARTIFACTORY_HOME env var or the artifactory.home java system prop), it will be created under the home of the user running Artifactory as a ".artifactory" dir. So, you should find it there.

Hope that helps,

Yoav

On Tue, Jun 16, 2009 at 4:58 PM, daniel.green <[hidden email]> wrote:

Hello Yoav,

Thank you for the help. I am currently running Artifactory 2.0.6 (rev.
5625). I was getting other OOMEs, but after I tweaked the settings it's not
crashing on the perm gen, but taking much longer to
do so.

I am using the following to set heap min/max and perm size:

-Xms1024m -Xmx1024m -XX:MaxPermSize=128m

Also, I believe I am simply using Derby as I did not configure MySql ;-).
The server I'm running this on only has 2gig allocated to it.

The wiki mentioned $ARTIFACTORY_HOME, however I deployed Artifactory as a
war. Was this the wrong way to go? I am using JBOSS to host.

Thank you again for the help,
Daniel.


Yoav  Landman wrote:
>
> Hi Daniel,
>
> High memory peaks on import can be caused by the indexing threads indexing
> a
> large amount of data in parallel.
> What's the max heap size you have set up? (I assume it crashes on heap not
> on the perm gen). Also, what's the Artifatory version and are you using
> MySQL or the built-in Derby?
> Repository content can also be imported through the REST API, by using
> "artadmin import" command (
> http://wiki.jfrog.org/confluence/display/RTF/The+artadmin+Command+Line+Tool).
> This will require less memory. Just use a layout similar to a system
> export
> - you do not have to have all the repository folders or configuration
> files
> present, as long as the repository name you are importing into is already
> defined in Artifactory (import will be additive).
>
> Thanks,
>
> Yoav
>
> On Mon, Jun 15, 2009 at 6:59 PM, daniel.green <[hidden email]>
> wrote:
>
>>
>> I have a large number of artifacts to import from another installation of
>> Artifactory, but don't have the memory needed to import through the web
>> UI
>> without crashing it with OOMEs. I deployed it as a war in JBoss. Is there
>> a
>> way to import the data manually through a terminal or something?
>>
>> Thank you for the time,
>> Daniel.
>> --
>> View this message in context:
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
>> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Crystal Reports - New Free Runtime and 30 Day Trial
>> Check out the new simplified licensing option that enables unlimited
>> royalty-free distribution of the report engine for externally facing
>> server and web deployment.
>> http://p.sf.net/sfu/businessobjects
>> _______________________________________________
>> Artifactory-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Artifactory-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24054924.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Manual import of repositories

The_Danimal

That explains it! Now, will I be able to use the command line tool outside
the context of the standalone version?


Yoav  Landman wrote:

>
> Hi Daniel,
>
> 1G of heap should be more than enough for the standalone version, but
> since
> you are running on JBoss you are already sharing this with existing
> services
> and applications, so you might be running short.
> It is not mandatory to specify the $ARTIFACTORY_HOME location - when not
> specified (by setting the ARTIFACTORY_HOME env var or the artifactory.home
> java system prop), it will be created under the home of the user running
> Artifactory as a ".artifactory" dir. So, you should find it there.
>
> Hope that helps,
>
> Yoav
>
> On Tue, Jun 16, 2009 at 4:58 PM, daniel.green <[hidden email]>
> wrote:
>
>>
>> Hello Yoav,
>>
>> Thank you for the help. I am currently running Artifactory 2.0.6 (rev.
>> 5625). I was getting other OOMEs, but after I tweaked the settings it's
>> not
>> crashing on the perm gen, but taking much longer to
>> do so.
>>
>> I am using the following to set heap min/max and perm size:
>>
>> -Xms1024m -Xmx1024m -XX:MaxPermSize=128m
>>
>> Also, I believe I am simply using Derby as I did not configure MySql ;-).
>> The server I'm running this on only has 2gig allocated to it.
>>
>> The wiki mentioned $ARTIFACTORY_HOME, however I deployed Artifactory as a
>> war. Was this the wrong way to go? I am using JBOSS to host.
>>
>> Thank you again for the help,
>> Daniel.
>>
>>
>> Yoav  Landman wrote:
>> >
>> > Hi Daniel,
>> >
>> > High memory peaks on import can be caused by the indexing threads
>> indexing
>> > a
>> > large amount of data in parallel.
>> > What's the max heap size you have set up? (I assume it crashes on heap
>> not
>> > on the perm gen). Also, what's the Artifatory version and are you using
>> > MySQL or the built-in Derby?
>> > Repository content can also be imported through the REST API, by using
>> > "artadmin import" command (
>> >
>> http://wiki.jfrog.org/confluence/display/RTF/The+artadmin+Command+Line+Tool
>> ).
>> > This will require less memory. Just use a layout similar to a system
>> > export
>> > - you do not have to have all the repository folders or configuration
>> > files
>> > present, as long as the repository name you are importing into is
>> already
>> > defined in Artifactory (import will be additive).
>> >
>> > Thanks,
>> >
>> > Yoav
>> >
>> > On Mon, Jun 15, 2009 at 6:59 PM, daniel.green <[hidden email]>
>> > wrote:
>> >
>> >>
>> >> I have a large number of artifacts to import from another installation
>> of
>> >> Artifactory, but don't have the memory needed to import through the
>> web
>> >> UI
>> >> without crashing it with OOMEs. I deployed it as a war in JBoss. Is
>> there
>> >> a
>> >> way to import the data manually through a terminal or something?
>> >>
>> >> Thank you for the time,
>> >> Daniel.
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
>> >> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>> >>
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> Crystal Reports - New Free Runtime and 30 Day Trial
>> >> Check out the new simplified licensing option that enables unlimited
>> >> royalty-free distribution of the report engine for externally facing
>> >> server and web deployment.
>> >> http://p.sf.net/sfu/businessobjects
>> >> _______________________________________________
>> >> Artifactory-users mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >>
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Crystal Reports - New Free Runtime and 30 Day Trial
>> > Check out the new simplified licensing option that enables unlimited
>> > royalty-free distribution of the report engine for externally facing
>> > server and web deployment.
>> > http://p.sf.net/sfu/businessobjects
>> > _______________________________________________
>> > Artifactory-users mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24054924.html
>> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Crystal Reports - New Free Runtime and 30 Day Trial
>> Check out the new simplified licensing option that enables unlimited
>> royalty-free distribution of the report engine for externally facing
>> server and web deployment.
>> http://p.sf.net/sfu/businessobjects
>> _______________________________________________
>> Artifactory-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Artifactory-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24062782.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Manual import of repositories

Yoav Landman
Administrator
Short answer - yes ;)
But, if you are really short on free heap you might still run into OOME issues. In that case, the only solution would be to break the import into 2 or more smaller chunks.

On Wed, Jun 17, 2009 at 12:00 AM, daniel.green <[hidden email]> wrote:

That explains it! Now, will I be able to use the command line tool outside
the context of the standalone version?


Yoav  Landman wrote:
>
> Hi Daniel,
>
> 1G of heap should be more than enough for the standalone version, but
> since
> you are running on JBoss you are already sharing this with existing
> services
> and applications, so you might be running short.
> It is not mandatory to specify the $ARTIFACTORY_HOME location - when not
> specified (by setting the ARTIFACTORY_HOME env var or the artifactory.home
> java system prop), it will be created under the home of the user running
> Artifactory as a ".artifactory" dir. So, you should find it there.
>
> Hope that helps,
>
> Yoav
>
> On Tue, Jun 16, 2009 at 4:58 PM, daniel.green <[hidden email]>
> wrote:
>
>>
>> Hello Yoav,
>>
>> Thank you for the help. I am currently running Artifactory 2.0.6 (rev.
>> 5625). I was getting other OOMEs, but after I tweaked the settings it's
>> not
>> crashing on the perm gen, but taking much longer to
>> do so.
>>
>> I am using the following to set heap min/max and perm size:
>>
>> -Xms1024m -Xmx1024m -XX:MaxPermSize=128m
>>
>> Also, I believe I am simply using Derby as I did not configure MySql ;-).
>> The server I'm running this on only has 2gig allocated to it.
>>
>> The wiki mentioned $ARTIFACTORY_HOME, however I deployed Artifactory as a
>> war. Was this the wrong way to go? I am using JBOSS to host.
>>
>> Thank you again for the help,
>> Daniel.
>>
>>
>> Yoav  Landman wrote:
>> >
>> > Hi Daniel,
>> >
>> > High memory peaks on import can be caused by the indexing threads
>> indexing
>> > a
>> > large amount of data in parallel.
>> > What's the max heap size you have set up? (I assume it crashes on heap
>> not
>> > on the perm gen). Also, what's the Artifatory version and are you using
>> > MySQL or the built-in Derby?
>> > Repository content can also be imported through the REST API, by using
>> > "artadmin import" command (
>> >
>> http://wiki.jfrog.org/confluence/display/RTF/The+artadmin+Command+Line+Tool
>> ).
>> > This will require less memory. Just use a layout similar to a system
>> > export
>> > - you do not have to have all the repository folders or configuration
>> > files
>> > present, as long as the repository name you are importing into is
>> already
>> > defined in Artifactory (import will be additive).
>> >
>> > Thanks,
>> >
>> > Yoav
>> >
>> > On Mon, Jun 15, 2009 at 6:59 PM, daniel.green <[hidden email]>
>> > wrote:
>> >
>> >>
>> >> I have a large number of artifacts to import from another installation
>> of
>> >> Artifactory, but don't have the memory needed to import through the
>> web
>> >> UI
>> >> without crashing it with OOMEs. I deployed it as a war in JBoss. Is
>> there
>> >> a
>> >> way to import the data manually through a terminal or something?
>> >>
>> >> Thank you for the time,
>> >> Daniel.
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24037446.html
>> >> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>> >>
>> >>
>> >>
>> >>
>> ------------------------------------------------------------------------------
>> >> Crystal Reports - New Free Runtime and 30 Day Trial
>> >> Check out the new simplified licensing option that enables unlimited
>> >> royalty-free distribution of the report engine for externally facing
>> >> server and web deployment.
>> >> http://p.sf.net/sfu/businessobjects
>> >> _______________________________________________
>> >> Artifactory-users mailing list
>> >> [hidden email]
>> >> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >>
>> >
>> >
>> ------------------------------------------------------------------------------
>> > Crystal Reports - New Free Runtime and 30 Day Trial
>> > Check out the new simplified licensing option that enables unlimited
>> > royalty-free distribution of the report engine for externally facing
>> > server and web deployment.
>> > http://p.sf.net/sfu/businessobjects
>> > _______________________________________________
>> > Artifactory-users mailing list
>> > [hidden email]
>> > https://lists.sourceforge.net/lists/listinfo/artifactory-users
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Manual-import-of-repositories-tp24037446p24054924.html
>> Sent from the Artifactory-Users mailing list archive at Nabble.com.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Crystal Reports - New Free Runtime and 30 Day Trial
>> Check out the new simplified licensing option that enables unlimited
>> royalty-free distribution of the report engine for externally facing
>> server and web deployment.
>> http://p.sf.net/sfu/businessobjects
>> _______________________________________________
>> Artifactory-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Artifactory-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/artifactory-users
>
>

--
View this message in context: http://www.nabble.com/Manual-import-of-repositories-tp24037446p24062782.html
Sent from the Artifactory-Users mailing list archive at Nabble.com.


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users