Artifactory 4.1.0 artifacts size vs. binaries size

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Artifactory 4.1.0 artifacts size vs. binaries size

moberberger
Hey,

maybe a stupid question, but as far as I understand the documentation, the binaries size should never be bigger than the artifacts size.

At our artifactory I see something like this:
Binaries Size: 177.11 GB
Artifacts Size: 29.26 GB

but how can that be?

Thanks for your help.

--
Best regards
Max
Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

shayb
In case that the Binaries Size is bigger than the Artifacts size, it usually means that the Artifactory Garbage Collection (which responsible on deleting binaries which has no-more references in the DB) has not run yet.
You can force the Garbage Collection to run (and delete all the binaries which has no reference within the DB).from the Web-UI -> Admin tab -> Maintenance screen.

Hope that helps,
Shay.



On Fri, Dec 4, 2015 at 11:51 AM, moberberger [via Artifactory] <[hidden email]> wrote:
Hey,

maybe a stupid question, but as far as I understand the documentation, the binaries size should never be bigger than the artifacts size.

At our artifactory I see something like this:
Binaries Size: 177.11 GB
Artifacts Size: 29.26 GB

but how can that be?

Thanks for your help.

--
Best regards
Max


If you reply to this email, your message will be added to the discussion below:
http://forums.jfrog.org/Artifactory-4-1-0-artifacts-size-vs-binaries-size-tp7580636.html
To start a new topic under Artifactory - Users, email [hidden email]
To unsubscribe from Artifactory, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

moberberger
Hey,

thanks for the answer. I tried it several times, but still the same issue. The "Binaries Size" is much bigger than the "Artifacts Size".

I also tried "Cleanup Unused Cached Artifacts", "Cleanup Virtual Repositories" and "Prune Unreferenced Data", but unfortunately no success.

Is there maybe anything else I can try? Or should I start from 0 with a new artifactory?

--
Best regards
Max
Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

shayb
Do you actually see the GC (Garbage Collection) summary in the artifactory.log file that indicates that the GC actually finished? it should look like that:
2015-12-07 10:50:14,387 [art-exec-53] [INFO ] (o.a.s.b.GarbageCollectorInfo:78) - Storage garbage collector report:
Number of binaries:      2,458
Total execution time:    1.25 minutes
Candidates for deletion: 2,294
Checksums deleted:       2,294
Binaries deleted:        2,294
Total size freed:        269.50 MB
Current total size:      384.20 MB

If the GC indeed finished, then the binaries size will not be bigger than the artifacts size.
In case you did not saw the summary yet, there could be two reasons for that:
First reason: The GC still in progress. You can verify that by adding the bellow trace logger into the $ARTIFACTORY_HOME/etc/logback.xml file.
<logger name="org.artifactory.storage.db.binstore.service">
   <level value="trace"/>
</logger>

Applying the log does not requires restarting the Artifactory server. Once applying the log, if the GC is in the middle of a run, you should see similar log entries as below:
2015-12-07 10:50:00,155 [art-exec-53] [TRACE] (o.a.s.d.b.s.BinaryStoreImpl:437) - Candidate for deletion: {0d691a0f03d29be835c900e903206c3881240dd8,1adaa5676a4f877cf3695089512c83d9,9669080}
2015-12-07 10:50:00,159 [art-exec-53] [TRACE] (o.a.s.d.b.s.BinaryStoreImpl:684) - Deleted 0d691a0f03d29be835c900e903206c3881240dd8 record from binaries table
...

Second reason: In case that you don't see the above log entries after applying the trace loggers, it could be that for some reason, the GC does not starting it's run.
You haven't mentioned the version of your Artifactory server, but you might be interested to know that in the past, with older versions of Artifactory, there were few reports on GC that wont start for a long period of time, see here for example (there were a few other relevant Jira issues). If you are running into similar behavior when the GC never runs and you are running an old version of Artifactory, I will first recommend to upgrade to one of the most recent versions. You might be interested to know that at these days, except restarting the server (which helped to overcome this state that the GC never runs), re-saving the config descriptor from the Web-UI also helped an after saving it, users were able to trigger the GC to run again.

I hope that helps,
Shay




On Mon, Dec 7, 2015 at 10:34 AM, moberberger [via Artifactory] <[hidden email]> wrote:
Hey,

thanks for the answer. I tried it several times, but still the same issue. The "Binaries Size" is much bigger than the "Artifacts Size".

I also tried "Cleanup Unused Cached Artifacts", "Cleanup Virtual Repositories" and "Prune Unreferenced Data", but unfortunately no success.

Is there maybe anything else I can try? Or should I start from 0 with a new artifactory?

--
Best regards
Max


If you reply to this email, your message will be added to the discussion below:
http://forums.jfrog.org/Artifactory-4-1-0-artifacts-size-vs-binaries-size-tp7580636p7580643.html
To start a new topic under Artifactory - Users, email [hidden email]
To unsubscribe from Artifactory, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

moberberger
I see the GC summary in the log file, but I think it does not look like expected:
2015-12-07 08:28:38,193 [art-exec-1041880] [INFO ] (o.a.s.b.GarbageCollectorInfo:78) - Storage garbage collector report:
Number of binaries:      15,141
Total execution time:    5.18 secs
Candidates for deletion: 1,273
Checksums deleted:       5
Binaries deleted:        5
Total size freed:        3.56 KB
Current total size:      180.35 GB

The artifactory server is: artifactory-tomcat-7.0.64

Before the GC finished, I see a lot of these messages:

2015-12-07 08:28:33,496 [art-exec-1041880] [INFO ] (o.a.s.d.b.s.BinaryStoreImpl:451) - Found 1273 candidates for deletion
.....
2015-12-07 08:28:33,620 [art-exec-1041880] [INFO ] (o.a.s.d.b.s.BinaryStoreImpl:717) - Binary 7f284ecfc9d9c0491a2997cf60cdfb4f1235329a is being read! Not deleting.
2015-12-07 08:28:33,620 [art-exec-1041880] [INFO ] (o.a.s.d.b.s.BinaryStoreImpl:717) - Binary 1891d9593508e28481006a3f8c1a7f4692b10d30 is being read! Not deleting.
2015-12-07 08:28:33,620 [art-exec-1041880] [INFO ] (o.a.s.d.b.s.BinaryStoreImpl:717) - Binary 3a0913c9b27dfef1031dc17e9ec10b5e468ce0cd is being read! Not deleting.
2015-12-07 08:28:33,620 [art-exec-1041880] [INFO ] (o.a.s.d.b.s.BinaryStoreImpl:717) - Binary 495ae7498ab1ec63f8d91cf51221db9832932b3c is being read! Not deleting.
.....

Is this normal, or not?

--
Best regards
Max
Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

pwalsh
Hi,

In case it helps in raising this as an issue with JFrog, we have seen a similar issue just today in my company, on Artifactory 4.1.3 OSS.  The efficiency value had climbed to 226%, and when I tried running the Storage Garbage Collection, I got similar errors to the previous user about files still being read, and thus not being deleted.  Solution in our instance was to restart Artifactory, and immediately run the Storage GC, at which point 216 GB was freed (>50% of the storage used!), and no such errors appeared.  We never had this happen until now, and we had not restarted the application since upgrading to 4.1.3 in late October.

Thus this looks to be a bug in artifactory 4.1.x whereby the storage GC simply doesn't work correctly, and registers files for deleted artifacts as still being accessed, meaning the space is not freed.  This seems to get reset on restart, and freed up.  I can't find any ticket open in the artifactory Jira suggesting this is fixed in a later release yet either, so for the time being we will likely have to arrange preventative restarts every 4-6 weeks.

Best Regards,
Paddy
Reply | Threaded
Open this post in threaded view
|

Re: Artifactory 4.1.0 artifacts size vs. binaries size

moberberger
Hey,

many thanks. That really solved the issue for now :)

I think we will add something to our monitoring to get that status as early as possible... And hopefully they will fix it in the future...

--
Best regards
Max