Quantcast

How to deploy c++ include/lib filesystem artifacts

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to deploy c++ include/lib filesystem artifacts

Hoobajoob
Completely new to artifactory, trying to get our first c++ projects successfully building and resolving dependencies through TeamCity + Artifactory OSS, version 4.7.4.

Some projects are multi-platform, built using gradle's native binary support, but most are just straightforward windows-only binaries with header file and static library dependencies built from Visual Studio solution and project files.

For my first candidate project, I have a few different third party c++ prebuilt libraries that the project requires as dependencies that consist of a header file and static library file system. They're not built from source, only rarely get updated.

I was hoping I could just manually deploy a zip file containing the header & library file system and start building against that, but I'm not seeing what I'm expecting, so I'm not sure I'm setting it up or using it correctly, or if I have the wrong expectations.

I'd prefer, during a build of a dependent project using this library, not to have to first retrieve the dependency as a zip file which I then have to unzip into a folder before I can start the build of the dependent component. I'd like to just retrieve the filesystem intact for each such dependency, then start the build.

I'd also like to retrieve the entire filesystem with one logical operation on Artifactory, not one retrieval operation per folder/file.

I'm not yet understanding whether my expectations are misaligned with what Artifactory OSS supports, or if I can easily achieve this through the use of the right Artifactory plugin. At the moment, if I upload artifacts as a bundle, it's looking to me like the dependent project has to issue per-file requests and manually reconstruct the entire file system on the build agent this way, which seems completely impractical.

What am I missing? Any guidance or suggestions?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to deploy c++ include/lib filesystem artifacts

HughG_TMVSE
Hi there,

sadly, there seems to be no standard way to do this.  There are projects like conan.io which provide dependency management for C++, but only at the source level and only for C++.  There are NuGet native modules but they were not great when I looked last year -- poor support for Platform/Configuration variation; multiple copies of files if you use the same dependency in multiple projects -- and of course its Windows-only.

We started trying to do this in-house about 3-4 years ago and ended up building a number of plugins on top of Gradle: code at https://bitbucket.org/nm2501/holy-gradle-plugins, doc at http://holygradle.bitbucket.org/.  They are somewhat tied to Windows (but only in a couple of places), based on an *old* version of Gradle (1.4), and skewed to requirements of particular teams here, but generally works really well for us.  It doesn't require Artifactory Pro, either, just OSS :-)  The basic functionality is as follows.

* Writing/reading credentials from the Windows Credential Store, instead of putting them in gradle.properties.

* Creating sets of related configurations (e.g., "build_{Win32,x64}_{Release,Debug}") and chaining them together across modules.  (If I were doing this again now, I'd tie it to the Gradle native C++ stuff -- but what's there now could be initialised from the Gradle model.)

* Downloading artifacts which are ZIPs, unzipping them to a cache folder, then putting a symlink (or directory junction) to that into your project dir.

* Zipping up files to be published as artifacts.

* Basic and somewhat ugly support for chaining build and test tasks across multiple projects in the same source checkout.

A more esoteric feature is the ability to do "source code sub-repos" in your build.gradle, so it doesn't matter if some source is in Mercurial and some in Subversion, for example.  (No Git support yet but it would be trivial to add.)  We discovered recently that our plugins only support multi-project builds cleanly if you have one project per repo -- something to fix one day ...

An even more out-there feature, not merged to default yet, is the ability to change the link for a binary dependency to point to some locally-built source ... and still make sure the complete transitive dependency hierarchy is consistent.  (There's something like this in Gradle 2.x, but I think ours is more flexible, if not quite as convenient.)


If you want to do your own, I'd suggest the following which we discovered as we went ...

* You may want to shell out to 7z.exe to unzip if available, because it's much faster than the built-in Gradle unzip for large ZIPs (e.g., boost), at least on Java 7 / Gradle 1.4.

* If you are a Windows local admin, symlinks require you to turn UAC down/off on Win 7 and need admin elevation on Windows 8+, but directory junctions don't.  We used JNA, which required some code duplication because Gradle's stuck on an old verson, but it works; or you could run "cmd /c mklink /j ...".

* Come up with a standard configuration naming scheme and make everyone stick to it.

* You may want to use configuration.resolutionStrategy.failOnVersionConflict(), because C++ libraries tend to be less forgiving of drop-in replacement of newer versions of DLLs, than Java is of upgraded JARs.

* Don't allow people to create links to the unpack cache in their project dir wherever they want them.  If you do, you end up with assumptions about the layout which may not be true for the consumers of that module.  You basically want one "link folder" folder per configuration (just as Gradle gives you one Java classpath per configuration), with a flat folder structure of all dependencies inside that.  If several configurations have the same set of dependencies, then you can share the same "link folder".  Also, you can use something like Gradle's "Sync" task to set up your link folder -- but only if that folder only contains links to the unpack cache and not any files you want to keep, because Sync will delete everything else!

** OR, if you prefer to have a unified folder hierarchy of "bin/, include/{A, B, C, ...}/, lib/", I'd suggest using the same unpack cache idea, then. copying/syncing them all into the unified hierarchy.  If you try to merge them all by just unzipping one on top of the other, you'll have no cheap way to upgrade just some dependencies.

* Consider having one unpack cache per volume, if you think you might be copying some of the dependency files (e.g., DLLs) to output folders.  If you do that, you can use hard links instead of copying, and save disk space if you have multiple builds of source with the same dependencies.

* When you package things, use the same layout in the package as in your original source, otherwise you can't easily swap in the source later.


Hope that helps ... and that it doesn't discourage you completely!  We've found it hard work at times but well worth the investment.

Regards,

Hugh Greene, Senior Software Developer
Toshiba Medical Visualization Systems Europe, Ltd
Bonnington Bond, 2 Anderson Place, Edinburgh EH6 5NP, UK
Tel + 44 (0)131 472 4792 / Fax + 44 (0) 131 472 4799
http://www.tmvse.com / mailto:[hidden email]

DISCLAIMER
Unless indicated otherwise, the information contained in this message is privileged and confidential, and is intended only for the use of the addressee(s) named above and others who have been specifically authorized to receive it. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message and/or attachments is strictly prohibited. The company accepts no liability for any damage caused by any virus transmitted by this email. Furthermore, the company does not warrant a proper and complete transmission of this information, nor does it accept liability for any delays. If you have received this message in error, please contact the sender and delete the message.


-----Original Message-----
From: Hoobajoob [mailto:[hidden email]]
Sent: 25 April 2016 12:18
To: [hidden email]
Subject: [Artifactory-users] How to deploy c++ include/lib filesystem artifacts

Completely new to artifactory, trying to get our first c++ projects successfully building and resolving dependencies through TeamCity + Artifactory OSS, version 4.7.4.

Some projects are multi-platform, built using gradle's native binary support, but most are just straightforward windows-only binaries with header file and static library dependencies built from Visual Studio solution and project files.

For my first candidate project, I have a few different third party c++ prebuilt libraries that the project requires as dependencies that consist of a header file and static library file system. They're not built from source, only rarely get updated.

I was hoping I could just manually deploy a zip file containing the header & library file system and start building against that, but I'm not seeing what I'm expecting, so I'm not sure I'm setting it up or using it correctly, or if I have the wrong expectations.

I'd prefer, during a build of a dependent project using this library, not to have to first retrieve the dependency as a zip file which I then have to unzip into a folder before I can start the build of the dependent component.
I'd like to just retrieve the filesystem intact for each such dependency, then start the build.

I'd also like to retrieve the entire filesystem with one logical operation on Artifactory, not one retrieval operation per folder/file.

I'm not yet understanding whether my expectations are misaligned with what Artifactory OSS supports, or if I can easily achieve this through the use of the right Artifactory plugin. At the moment, if I upload artifacts as a bundle, it's looking to me like the dependent project has to issue per-file requests and manually reconstruct the entire file system on the build agent this way, which seems completely impractical.

What am I missing? Any guidance or suggestions?




--
View this message in context: http://forums.jfrog.org/How-to-deploy-c-include-lib-filesystem-artifacts-tp7580913.html
Sent from the Artifactory - Users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com ______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to deploy c++ include/lib filesystem artifacts

greenaj
In reply to this post by Hoobajoob
What I've done over the last few years is use IVY style repositories. That lets you store SEVERAL artifacts for a module in a repository. Use Gradle to save to the repo and retrieve. Don't mess with all those little IVY.xml files. It works great this way. You save upstream artifacts into the repository that downstream builds can use. Also, with Gradle, you can easily rename the artifact after downloading. This is useful if you save a static lib or shared object (.so or .dll) in Artifactory. The version goes into the artifact name as per Java naming conventions, but you don't want it in the retrieved artifact that your project is linking against. Just string out the version as you download in Gradle.

There is not standard way of artifact manage yet for C++ projects, but this has worked.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to deploy c++ include/lib filesystem artifacts

jbaruch
Administrator
This post has NOT been accepted by the mailing list yet.
Starting Artifactory 4.15 we'll support Conan – a first-class dependency manager for C/C++. This should solve the problem :)
Loading...