Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Jerry C
This post was updated on .
Hi,

When pushing a docker image with size > 600MB to Artifactory docker v2 registry behind Apache HTTPD, it failed with message below.  The same image can be push successfully through Nginx reverse proxy.  Does anything have experience with Apache HTTPD as reverse proxy help?

Artifactory version: 4.0.2 rev 40009

Apache HTTPD version:
[root@tsunami conf.d]# httpd -v
Server version: Apache/2.4.6 (CentOS)
Server built:   Aug 24 2015 18:11:25

Docker Registry version: v2

docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64
 

[otter@volcano Desktop]$ docker push tsunami:443/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin:1.0.0
The push refers to a repository [tsunami:443/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin] (len: 1)
cb73be4d3636: Image already exists
217d94368d2a: Image push failed
Error pushing to registry: Server error: 502 trying to push sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin blob - sha256:54878006f7a40a46fdb9953d7bdf76eecb616e8b9a6aa5edab2a6d4357ce4db1

The following errors in Artifactory request log:

20150911120949|1|REQUEST|10.0.150.212|otter|POST|/api/docker/registry_v2/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/|HTTP/1.1|202|0
20150911120950|3|REQUEST|10.0.150.212|otter|PUT|/api/docker/registry_v2/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/cede11b9-7b7b-4090-851b-85eabe01e487|HTTP/1.1|404|-1
20150911122240|42|REQUEST|fe80::250:56ff:fe31:914f|docker|GET|/api/docker/registry_v2/v2/auth|HTTP/1.1|200|0
20150911122246|9|REQUEST|fe80::250:56ff:fe31:914f|docker|GET|/api/docker/registry_v2/v2/_catalog|HTTP/1.1|404|0

In httpd error log:
[Fri Sep 11 12:09:50.129662 2015] [proxy:error] [pid 7405] (104)Connection reset by peer: [client 10.0.150.212:50481] AH01084: pass request body failed to [fe80::250:56ff:fe31:914f]:8081 (tsunami)
[Fri Sep 11 12:09:50.129871 2015] [proxy_http:error] [pid 7405] [client 10.0.150.212:50481] AH01097: pass request body failed to [fe80::250:56ff:fe31:914f]:8081 (tsunami) from 10.0.150.212 ()

Log message from docker:
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.063015047-04:00" level=debug msg="rendered layer for 217d94368d2a580dcc238b590c83063b895977f30923241a5d8dc70614640e9a of [2311185] size"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.063121464-04:00" level=debug msg="[registry] Calling \"POST\" https://tsunami:443/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.063157432-04:00" level=debug msg="Using cached token for otter"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.063191072-04:00" level=debug msg="hostDir: /etc/docker/certs.d/tsunami:443"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.110972279-04:00" level=debug msg="[registry] Calling \"PUT\" https://tsunami:443/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/aae108d2-95a8-46e1-bfa9-843bf80ee593"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.111050754-04:00" level=debug msg="Using cached token for otter"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.111090876-04:00" level=debug msg="hostDir: /etc/docker/certs.d/tsunami:443"
Sep 11 11:32:19 volcano docker: time="2015-09-11T11:32:19.203683945-04:00" level=debug msg="Unexpected response from server: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>502 Bad Gateway</title>\\n</head><body>\\n

Bad Gateway

\\n<p>The proxy server received an invalid\\r\\nresponse from an upstream server.<br />\\r\\n</p>\\n</body></html>\\n\" http.Header{\"Content-Length\":[]string{\"232\"}, \"Content-Type\":[]string{\"text/html; charset=iso-8859-1\"}, \"Date\":[]string{\"Fri, 11 Sep 2015 15:32:19 GMT\"}, \"Server\":[]string{\"Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}}"


Here is my Apache httpd reverse proxy configuration:
LoadModule ssl_module modules/mod_ssl.so


<VirtualHost 0.0.0.0:443>
  ServerName tsunami
 
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
 
  SSLEngine on
  SSLCertificateFile      /home/otter/certs/registry/tsunami.crt
  SSLCertificateKeyFile   /home/otter/certs/registry/tsunami.key

  Header always set "Docker-Distribution-Api-Version" "registry/2.0"
  Header onsuccess set "Docker-Distribution-Api-Version" "registry/2.0"

  RequestHeader set X-Forwarded-Proto "https"

  ProxyRequests off
  ProxyPreserveHost on
 
  ProxyPass         / http://tsunami:8081/artifactory/api/docker/registry_v2/
  ProxyPassReverse  / http://tsunami:8081/artifactory/api/docker/registry_v2/
</VirtualHost>



Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Nihal Choudhary
Hi Jerry,

This issue seems to happen as a combination of Apache and an old version of a docker client. It seems that the older versions of docker client does not send the content length header with the request, so when the requests get to Artifactory, it returns 404 response which Apache translate to 502 error. We have manage to reproduce similar behavior locally.
We have also tested it with other reverse proxy server (nginx) which seems to be 'smart enough' and calculate the gzip stream length and add it as an header, so when the request got to Artifactory it worked.

With newer client (docker 1.8 and above) docker seems to send the content-length header as it should and it seems to work with no issues. Can you check this with the latest version of Docker?

On Fri, Sep 11, 2015 at 10:02 AM, Jerry C <[hidden email]> wrote:
Hi,

When pushing a docker image with size > 600MB to Artifactory docker v2
registry behind Apache HTTPD, it failed with message below.  The same image
can be push successfully through Nginx reverse proxy.  Does anything have
experience with Apache HTTPD as reverse proxy help?

Artifactory version: 4.0.2 rev 40009

Apache HTTPD version:
[root@tsunami conf.d]# httpd -v
Server version: Apache/2.4.6 (CentOS)
Server built:   Aug 24 2015 18:11:25

Docker Registry version: v2

docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64


[otter@volcano Desktop]$ docker push
tsunami:443/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin:1.0.0
The push refers to a repository
[tsunami:443/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin]
(len: 1)
cb73be4d3636: Image already exists
217d94368d2a: Image push failed
Error pushing to registry: Server error: 502 trying to push
sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin blob -
sha256:54878006f7a40a46fdb9953d7bdf76eecb616e8b9a6aa5edab2a6d4357ce4db1

The following errors in Artifactory request log:

20150911120949|1|REQUEST|10.0.150.212|otter|POST|/api/docker/registry_v2/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/|HTTP/1.1|202|0
20150911120950|3|REQUEST|10.0.150.212|otter|PUT|/api/docker/registry_v2/v2/sigma/com.sigma.sigmaadmin.demo.node.catalogservices-admin/blobs/uploads/cede11b9-7b7b-4090-851b-85eabe01e487|HTTP/1.1|404|-1
20150911122240|42|REQUEST|fe80::250:56ff:fe31:914f|docker|GET|/api/docker/registry_v2/v2/auth|HTTP/1.1|200|0
20150911122246|9|REQUEST|fe80::250:56ff:fe31:914f|docker|GET|/api/docker/registry_v2/v2/_catalog|HTTP/1.1|404|0

In httpd error log:
[Fri Sep 11 12:09:50.129662 2015] [proxy:error] [pid 7405] (104)Connection
reset by peer: [client 10.0.150.212:50481] AH01084: pass request body failed
to [fe80::250:56ff:fe31:914f]:8081 (tsunami)
[Fri Sep 11 12:09:50.129871 2015] [proxy_http:error] [pid 7405] [client
10.0.150.212:50481] AH01097: pass request body failed to
[fe80::250:56ff:fe31:914f]:8081 (tsunami) from 10.0.150.212 ()

Here is my Apache httpd reverse proxy configuration:
LoadModule ssl_module modules/mod_ssl.so


<VirtualHost 0.0.0.0:443>
  ServerName tsunami

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  SSLEngine on
  SSLCertificateFile      /home/otter/certs/registry/tsunami.crt
  SSLCertificateKeyFile   /home/otter/certs/registry/tsunami.key

  Header always set "Docker-Distribution-Api-Version" "registry/2.0"
  Header onsuccess set "Docker-Distribution-Api-Version" "registry/2.0"

  RequestHeader set X-Forwarded-Proto "https"

  ProxyRequests off
  ProxyPreserveHost on

  ProxyPass         /
http://tsunami:8081/artifactory/api/docker/registry_v2/
  ProxyPassReverse  /
http://tsunami:8081/artifactory/api/docker/registry_v2/
</VirtualHost>







--
View this message in context: http://forums.jfrog.org/Push-large-size-docker-image-to-Artifactory-Docker-v2-registry-behind-Apache-httpd-tp7580464.html
Sent from the Artifactory - Users mailing list archive at Nabble.com.

------------------------------------------------------------------------------
_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users


------------------------------------------------------------------------------

_______________________________________________
Artifactory-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/artifactory-users
Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Jerry C
This post was updated on .
Hi Nihal,

Thank you for your reply.  I tried with docker client 1.8.2 but I got another error (see below) when pushing through Apache HTTPD reverse proxy.  Again pushing through Nginx has not problem.

Do you have any idea?

regards,
Jerry

[docker@torvm-eng-tools ~]$ docker push artifactory.sigma-systems.com/external/registry:2
The push refers to a repository [artifactory.sigma-systems.com/external/registry] (len: 1)
1e847b14150e: Image already exists
e024fb496e6b: Image already exists
6228a99f9630: Image already exists
76b7062ceb9a: Image already exists
e4aee72fc6c3: Pushing [==================================================>]    290 B/290 B
Error parsing HTTP response: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>405 Method Not Allowed</title>\n</head><body>\n

Method Not Allowed

\n<p>The requested method PATCH is not allowed for the URL /v2/external/registry/blobs/uploads/ddca08b5-0f36-44c9-af07-ed271a4db242.</p>\n</body></html>\n"

The following message in docker log file:
10.0.110.168 - - [22/Sep/2015:11:31:43 -0500] "PATCH /v2/external/registry/blobs/uploads/ddca08b5-0f36-44c9-af07-ed271a4db242 HTTP/1.1" 405 294 "-" "docker/1.8.2 go/go1.4.2 git-commit/0a8c2e3 kernel/3.10.0-123.el7.x86_64 os/linux arch/amd64"
Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Aaron Rhodes
Hi Jerry,

Could you show us your Apache HTTP configuration? We have a sample here that you can use to compare:

Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

eguibib
In reply to this post by Jerry C
Hi Jerry,

We are facing similar situation, these is are findings if this might help:

For Docker client 1.7.X., you need to force apache to send content-length header. This can be done thru parameter: proxy-sendcl
http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html
But be aware of performance limitations that are associated!!!

With this you will still get 502 on bigger layers, this looks to be related with a bug in Apache 2.2.X (https://bz.apache.org/bugzilla/show_bug.cgi?id=56541). Workaround, in our case, was to increase the keepAliveTimeout (or connectionTineout) in Tomcat to avoid 502 errors:
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Again it might comes with some performance limitations.
The bug looks to be fix in Httpd 2.4 (I have not tested that), there is also a patch for 2.2 (not tested on my side also).

For Docker 1.8+ clients, this is not needed. The error you have, if my guess is right, is due to the use of AJP protocol. Indeed Docker uses a specific HTTP command (PATCH) that is not supported by AJP (causing 405 errors).

I hope this helps,
Thanks,
Regards,
Guillaume
Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Jerry C
This post was updated on .
Aaron, Guillaume, thank you for your reply.  I managed to get passed the above issue.  But when I build image using Spotify Maven docker plugin, I run into the following error from time-to-time (I got this error 3 out of 4 times)

[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.2.13:build (default) on project sigmaadminui: Exception caught: Error: Status 400 trying to pull repository sigma_external/centos-java8: "{\n  \"errors\" : [ {\n    \"status\" : 400,\n    \"message\" : \"Unsupported docker v1 repository request for 'sigma-docker'\"\n  } ]\n}" -> [Help 1]

The httpd log shows
10.0.110.168 - - [23/Sep/2015:14:07:40 -0500] "GET /v2/ HTTP/1.1" 401 77 "-" "docker/1.8.2 go/go1.4.2 git-commit/0a8c2e3 kernel/3.10.0-123.el7.x86_64 os/linux arch/amd64"
10.0.110.168 - - [23/Sep/2015:14:07:40 -0500] "GET /v2/ HTTP/1.1" 401 77 "-" "docker/1.8.2 go/go1.4.2 git-commit/0a8c2e3 kernel/3.10.0-123.el7.x86_64 os/linux arch/amd64"
10.0.110.168 - - [23/Sep/2015:14:07:40 -0500] "GET /v1/repositories/sigma_external/centos-java8/images HTTP/1.1" 400 124 "-" "docker/1.8.2 go/go1.4.2 git-commit/0a8c2e3 kernel/3.10.0-123.el7.x86_64 os/linux arch/amd64"

The docker log shows:
Sep 28 16:01:01 torvm-eng-tools systemd: Created slice user-0.slice.
Sep 28 16:01:01 torvm-eng-tools systemd: Starting Session 1726 of user root.
Sep 28 16:01:01 torvm-eng-tools systemd: Started Session 1726 of user root.
Sep 28 16:01:13 torvm-eng-tools docker: time="2015-09-28T17:01:13.545686349-05:00" level=debug msg="Calling POST /build"
Sep 28 16:01:13 torvm-eng-tools docker: time="2015-09-28T17:01:13.545751935-05:00" level=info msg="POST /build?t=artifactory.sigma-systems.com/sigma_foundation/sigmaadminui:1.0.1-SNAPSHOT"
Sep 28 16:01:34 torvm-eng-tools docker: time="2015-09-28T17:01:34.676671065-05:00" level=debug msg="hostDir: /etc/docker/certs.d/artifactory.sigma-systems.com"
Sep 28 16:01:34 torvm-eng-tools docker: time="2015-09-28T17:01:34.677216633-05:00" level=debug msg="Trying to pull artifactory.sigma-systems.com/sigma_external/centos-java8 from https://artifactory.sigma-systems.com v2"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.196817818-05:00" level=debug msg="Pulling tag from V2 registry: \"centos7\""
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.197064153-05:00" level=debug msg="Error trying v2 registry: Get https://artifactory.sigma-systems.com/v2/sigma_external/centos-java8/manifests/centos7: no basic auth credentials"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.197103614-05:00" level=debug msg="Trying to pull artifactory.sigma-systems.com/sigma_external/centos-java8 from https://artifactory.sigma-systems.com v1"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.198992179-05:00" level=debug msg="hostDir: /etc/docker/certs.d/artifactory.sigma-systems.com"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.199122805-05:00" level=debug msg="attempting v2 ping for registry endpoint https://artifactory.sigma-systems.com/v2/"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.219771446-05:00" level=debug msg="[registry] Calling GET https://artifactory.sigma-systems.com/v1/repositories/sigma_external/centos-java8/images"
Sep 28 16:01:35 torvm-eng-tools docker: time="2015-09-28T17:01:35.249717091-05:00" level=debug msg="Not continuing with error: Error: Status 400 trying to pull repository sigma_external/centos-java8: \"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 400,\\n    \\\"message\\\" : \\\"Unsupported docker v1 repository request for 'sigma-docker'\\\"\\n  } ]\\n}\""


Does anyone have idea why something the pull succeeded and something it failed with "no basic auth credentials"?

Here is my httpd config:
LoadModule ssl_module modules/mod_ssl.so

<VirtualHost *:443>
  ServerName artifactory.sigma-systems.com

  ErrorLog ${APACHE_LOG_DIR}/registry_error.log
  CustomLog ${APACHE_LOG_DIR}/registry_access.log combined

  SSLEngine on
  SSLCertificateFile      /data/users/docker/certs/smp_cert.pem
  SSLCertificateKeyFile   /data/users/docker/certs/smp_cert_key.pem
  SSLCertificateChainFile /data/users/docker/certs/gdig2.crt
  SSLProxyEngine on

  Header set Host "artifactory.sigma-systems.com"
  Header always set "Docker-Distribution-Api-Version" "registry/2.0"
  Header onsuccess set "Docker-Distribution-Api-Version" "registry/2.0"

  ProxyRequests off
  ProxyPreserveHost on

  ProxyPass         / http://artifactory:8081/artifactory/api/docker/sigma-docker/
  ProxyPassReverse  / http://artifactory:8081/artifactory/api/docker/sigma-docker/
</VirtualHost>
Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Jerry C
This post was updated on .
I found that if I reload my httpd server by
sudo systemctl reload httpd.service
The pull right after the reload usually succeeded.

Does anyone have idea what went wrong?

thanks,
Jerry
Reply | Threaded
Open this post in threaded view
|

Re: Push large size docker image to Artifactory Docker v2 registry behind Apache httpd

Jerry C
Here is what I found in Artifactory request log
For a failed run:
20151001123016|1|REQUEST|10.0.150.212|non_authenticated_user|GET|/api/docker/registry_v2/v2/|HTTP/1.1|401|0
20151001123016|1|REQUEST|10.0.150.212|non_authenticated_user|GET|/api/docker/registry_v2/v2/|HTTP/1.1|401|0

20151001123016|4|REQUEST|10.0.150.212|non_authenticated_user|GET|/api/docker/registry_v2/v1/repositories/sigma_external/centos-java8/images|HTTP/1.1|400|0

For a successful run:
20151001121529|335|REQUEST|10.0.150.212|non_authenticated_user|GET|/api/docker/registry_v2/v2/|HTTP/1.1|200|0
20151001121530|317|REQUEST|10.0.150.212|non_authenticated_user|GET|/api/docker/registry_v2/v2/sigma_external/centos-java8/manifests/centos7|HTTP/1.1|200|25646
20151001121530|134|REQUEST|10.0.150.212|non_authenticated_user|HEAD|/api/docker/registry_v2/v2/sigma_external/centos-java8/blobs/sha256:4a4adf5f2dbd3fd262fa5415c4e9b6f278c9dc90cd65a334688e19b80f96cf6d|HTTP/1.1|200|0

There is no change between the two runs.  But the http get (see bold lines above) return 401 and 200.  Does anyone have any idea?

thanks,
Jerry