PyPi repo layout and ci/releases repos with promotion

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

PyPi repo layout and ci/releases repos with promotion

XabierD
This post was updated on .

Q1:

I have a PyPi enabled repo (my-pypi-repo) where I can publish my packages. When uploading via python setup.py -sdist,I get a structure like this:
my-pypi-repo|
            |my_package|
                       |x.y.z|
                             |my_package-x.y.z.tar.gz

The problem is this structure will not match any "allowed" repo layout in Artifactory, since [org] or [orgPath] are mandatory: Pattern '[module]/[baseRev]/[module]-[baseRev].[ext]' must at-least contain the tokens 'module', 'baseRev' and 'org' or 'orgPath'. I managed to publish to a path by 'hacking' the package name to 'myorg/my_package', but then
pip
cannot find it, so it's pretty useless

Q2:

Has anyone tried the "ci-repo" and "releases-repo" with promotion for Python? What I would like to achieve:
CI repo:
my_package-1.2.3+build90.tar.gz (I don't care that much about the [org] When this artifact gets promoted (build metadata gets dropped):
Releases repo:
my_package-1.2.3.tar.gz I can achieve this via repo layouts (providing I resolve Q1). The problem is how to deal with the "embedded" version inside my Python script. I'd rather not rebuild the package again, for best practices.
Reply | Threaded
Open this post in threaded view
|

Re: PyPi repo layout and ci/releases repos with promotion

drohm
I am running into the same issue in regards to your first question/problem.  When configuring my system to publish to artifactory using pip, it uses the format of:

pypi-repo
            |package_name
                                |x.y.z
                                       |package_file.zip

The [org] or [orgPath] is mandatory and this basically breaks all the REST API functionality, like searching for latest version, etc.  I'm currently using this as my Artifact Path Pattern:

  [org]/[module]/[baseRev].([fileItegRev])/[module]-[baseRev].([fileItegRev]).[ext]

The problem is that pip doesn't understand the concept of [org] in this case.  I'm temporarily using a python script to publish my packages to Artifactory to get around this.  Hopefully this is something that can be addressed by the jFrog team?
Reply | Threaded
Open this post in threaded view
|

Re: PyPi repo layout and ci/releases repos with promotion

XabierD
Thanks for your reply.
Could you elaborate about your Python script to publish?
Do you mean that, with your script, the REST API functionality works?
Can you use pip to install packages not in the root folder?

Thanks

(Have the same question posted in Stackoverflow here:
http://stackoverflow.com/questions/29102415/artifactory-pypi-repo-layout-with-build-promotion)
Reply | Threaded
Open this post in threaded view
|

Re: PyPi repo layout and ci/releases repos with promotion

drohm
The python script simply uses Artifactory's REST API to publish to my local pypi repository, tacking on a few properties so that some of the REST API functions work properly, like 'Artifact Latest Version Search Based on Properties' which you can read here:

https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-ArtifactLatestVersionSearchBasedonProperties

I need to be able to use that call because we're using Chef in-house and we use that method to get the latest version.  The 'pypi.version' property that gets added when publishing via 'python setup.py sdist upload -r local' doesn't work with the REST API so I have to manually add the 'version' property.  Painful to be honest since we can't add properties when using the upload option with setup.py.  Ideally I'd like to be able to do everything using pip, but at the moment this isn't possible.

I'm using the requests package and the method in the documentation here:

https://www.jfrog.com/confluence/display/RTF/Using+Properties+in+Deployment+and+Resolution

Here is the function that publishes the package and adds a few properties (you can add more):

def _publish_artifact(name, version, path, summary):
    base_url = 'http://server:8081/artifactory/{0}'.format(PYPI_REPOSITORY)
    properties = ';version={0};pypi.name={1};pypi.version={0};pypi.summary={2}'\
                 .format(version, name, summary)
    url_path = '/Company/{0}/{1}/{0}-{1}.zip'.format(name, version)
    url = '{0}{1}{2}'.format(base_url, properties, url_path)

    dist_file = r'{0}\dist\{1}-{2}.zip'.format(path, name, version)
    files = {'upload_file': open(dist_file, 'rb')}
    s = requests.Session()
    s.auth = ('username', 'password')
    reply = s.put(url, files=files)
    logger.info('HTTP reply: {0}'.format(reply))

Hope that helps and I hope jFrog is reading this. It's nice what they've added so far, they just need a few more tweaks and this will be great.