Docker broken - again - in 4.4.0

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Docker broken - again - in 4.4.0

Lars Skjærlund
Just upgraded to 4.4.0 and Docker has stopped working.

An attempt to login to a repository ends like:

las@las-docker:~$ docker login https://docker.dbc.dk
Username (las): las
Error response from daemon: no successful auth challenge for https://docker.dbc.dk/v2/ - errors: [token auth attempt for registry https://docker.dbc.dk/v2/: http://docker.dbc.dk/v2/token?account=las&service=docker.dbc.dk request failed with status: 404 Not Found]
las@las-docker:~$

I've created a test environment with a virtual machine and a snapshot: Docker works reliably with 4.3.2, fails when upgraded to 4.4.0, works again when downgraded through a snapshot restore.

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

mori
From version 4.4.0 of Artifactory, the ‘X-Artifactory-Override-Base-Url’ header is required on a reverse proxy configuration for Docker. The Docker login and other Docker commands will fail to work if the header below is not configured correctly (e.g. missing a port).
 
The line needs to include the server port number, as such:
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port/<public context>;
 
For example,
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port
OR
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:444/artifactory
 
You can see https://www.jfrog.com/confluence/display/RTF/Docker+Repositories#DockerRepositories-1.SettingupNGINXasaReverseProxy for more info.

Thanks
Mor
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Alix Lourme
Hi,

For Apache proxy, the below directive is needed:
---
RequestHeader set X-Artifactory-Override-Base-Url http[s]://TheArtifactoryUrl
---

With that "docker login" works (but not dockerHub remote for me ... investigations is progress).

------------------------------------------------------------------------------
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
Hi Alix,

Thank you so much. We use Apache as proxy and your tip did the trick.

Why aren't criminal changes like this documented in the release notes?

It's in the documentation - but it is not highlighted as a change since last version: Are you expected to read the entire documentation for every minor upgrade and figure out changes since last version by yourself?

If that's the case, I don't think Artifactory will survive for long within our organization...

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Alix Lourme
Hi Lars,

I'm not in JFrog team, so I can't answer concretely to your question.
But I have signified this lack (about release note v4.4.0 & Apache config) to the JFrog support, I have received that these points will be improved.

About my "dockerHub remote fail ... investigations is progress" in my previous post: The repositories remotes works fine on an Artifactory v4.4.0 "clean" installation.
Perhaps I do something not clear in the upgrade process.
In all case, for users who have the same problem, I advise to redefine the company proxy and association with remote repositories.

Best regards.


------------------------------------------------------------------------------
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
Hi Alix,

I understand you're not with JFrog - so I can only raise my thumbs once again for your help!

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

dariusjs
Lars, did you ever manage to sort this out fully? I also got caught out with this update where everything else worked perfectly with the old method. I used to have artifactory.local and artifactory2.local, the latter would proxy to  192.168.100.10:8081/artifactory/api/docker/docker-local/

Since 4.3 or 4.4 this is now changed to only work with ports or subdomains. I am trying to work which the ports for simplicity with certs. I managed to get docker login to work as well as docker pull but docker push gives me TLS issues.

The push refers to a repository [artifactory.local:6555/demo/gowebserver] (len: 1)
adcb5120aceb: Pushing 1.024 kB
tls: oversized record received with length 20527
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
Hi dariusjs,

I never tried 4.4.1, but it works again with 4.4.2 - without the extra header.

The header caused endless troubles for us as the Docker client all of a sudden would try to connect to port 8081 (unencrypted) instead of 443 (encrypted). Port 8081 is closed in our internal firewall, so we could not update production servers from our internal Docker repository.

A real mess.

Fortunately, it works again. Let's hope it stays that way for a long time.

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

dariusjs
Lars,

Which method are you using? I upgraded to 4.4.2 and still haven't managed to
get "docker push" to actually work against artifactory. I am also an apache
user.

The old method as of 4.2.x was you only simply had to have the below but
this doesn't work at all.
        ProxyPass         /
http://192.168.100.10:8081/artifactory/api/docker/docker-local/
        ProxyPassReverse  /
http://192.168.100.10:8081/artifactory/api/docker/docker-local/

But this now results in:
Error response from daemon: no successful auth challenge for
https://artifactory2.local/v2/ - errors: [token auth attempt for registry
https://artifactory2.local/v2/:
https://artifactory.local/api/docker/docker-local/v2/token?account=uploader&service=artficatory.local
request failed with status: 401 Unauthorized]


With the new method
https://www.jfrog.com/confluence/display/RTF/Docker+Repositories#DockerRepositories-ApacheConfiguration
I have more success but of course not with docker push at all so our
workflow is broken.



--
View this message in context: http://forums.jfrog.org/Docker-broken-again-in-4-4-0-tp7580708p7580800.html
Sent from the Artifactory - Users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

dariusjs
In reply to this post by Lars Skjærlund
Lars,

Which method are you using? I upgraded to 4.4.2 and still haven't managed to get "docker push" to work since the update. I am also an apache user.

The old method as of 4.2.x was you only simply had to have the below but this doesn't work anymore.
        ProxyPass         / http://192.168.100.10:8081/artifactory/api/docker/docker-local/
        ProxyPassReverse  / http://192.168.100.10:8081/artifactory/api/docker/docker-local/

The above config now results in:
Error response from daemon: no successful auth challenge for https://artifactory2.local/v2/ - errors: [token auth attempt for registry https://artifactory2.local/v2/: https://artifactory.local/api/docker/docker-local/v2/token?account=uploader&service=artficatory.local request failed with status: 401 Unauthorized]


With the new method https://www.jfrog.com/confluence/display/RTF/Docker+Repositories#DockerRepositories-ApacheConfiguration I have more success but of course not with docker push at all so our workflow is broken. Ve also experienced what you may have where docker login and docker pull seem to work bot docker push doesn't like the certificate setup even though the chain is unchanged from 4.2
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
We're using a slightly more complicated setup:

SetEnv                  proxy-sendcl 1

ProxyRequests           off
ProxyPreserveHost       on

ProxyPass               / http://localhost:8081/artifactory/api/docker/docker-test/
ProxyPassReverse        / http://localhost:8081/artifactory/api/docker/docker-test/

Whilst most of these options are mentioned in the Artifactory documentation, the "proxy-sendcl 1" is not as far as I can see. It is important, though, if you want to use your repository with Docker clients < 1.8.

I'm going to test it in production today.

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
One more thing: Artifactory recommends setting the header

RequestHeader set X-Artifactory-Override-Base-Url "http://artifactory.organization.com:8081/artifactory"

I do not: If you add the header, the Docker client will try to connect directly to the Artifactory server at port 8081. This causes two problems: The traffic is unencrypted, and you need to open port 8081 in the firewall.

If you omit the header, all traffic is forced through the proxy at the SSL port. This gives better security and a port less to administrate in the firewall setup.

The hack was needed with Artifactory 4.4.0, but it appears to work without it again in 4.4.2.

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

dariusjs
Lars, no matter what I tried I could not get my setup to work. I ended up installing a clean Ubuntu server with 4.3.2 from the Jfrog deb packages and only then was I able to get docker push to work but only with nginx as a reverse proxy. No matter what I try when using apache I get the below error. It must be something I am missing, I've tried your suggestions and some historical ones I found for the docker private registry as well as artifactory.

This resulted in me having to change a lot of the workflows over to the nginx server.

tls: oversized record received with length 20527
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

Lars Skjærlund
Well - it works for me, so I'm afraid I cannot help you any further. It's hard to debug an error you cannot reproduce.

BTW, I'm testing 4.4.3 right now and it works with my Apache setup as well.

Regards,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

dariusjs
No, I inderstand. Every setup is different, I'll keep at it. I've also gone to 4.4.3 with the new install as the release notes mentioned fixing issues with docker push which seems to have helped my installation just not the Apache part yet.

On Tue, 9 Feb 2016 at 15:34, Lars Skjærlund [via Artifactory] <[hidden email]> wrote:
Well - it works for me, so I'm afraid I cannot help you any further. It's hard to debug an error you cannot reproduce.

BTW, I'm testing 4.4.3 right now and it works with my Apache setup as well.

Regards,
Lars


If you reply to this email, your message will be added to the discussion below:
To unsubscribe from Docker broken - again - in 4.4.0, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

zanbel
You can try and generate the reverse proxy configuration using the Reverse Proxy mechanism in Artifactory.

It supports both NGINX and Apache and can help you generate the relevant configuration when working in ports method, or sub domain with wildcard certificate.
Reply | Threaded
Open this post in threaded view
|

Re: Docker broken - again - in 4.4.0

webmutation
In reply to this post by Lars Skjærlund
I'm on 4.5.2 now upgraded from 3.7 and nothing works now LOL

I have my Apache httpd reverse proxy setup with the generated configuration, but when I try to connect to the machine using docker login I get

v2 ping attempt failed with error: Get https://docker.atmyorg:4443/v2/: x509: certificate signed by unknown authority

and this is after i add the --insecure-registry docker.atmyorg:4443

If I use curl I can get the authorization token, really dont know why docker login cant login