# Private Docker Registry

Private Docker Registry

1. Docker registry Images 가져오기<span class="anchor" id="bkmrk-"></span><span class="anchor" id="bkmrk--1"></span><span class="anchor" id="bkmrk--2"></span>```
     docker pull registry:latest
    ```
    
    <span class="anchor" id="bkmrk--3"></span>
2. docker images<span class="anchor" id="bkmrk--4"></span>
3. hyunsu@kubemaster:/etc/docker/registry$ sudo mv ./config.yml ./config.yml.20201022<span class="anchor" id="bkmrk--5"></span>
4. config<span class="anchor" id="bkmrk--6"></span><span class="anchor" id="bkmrk--7"></span><span class="anchor" id="bkmrk--8"></span><span class="anchor" id="bkmrk--9"></span><span class="anchor" id="bkmrk--10"></span><span class="anchor" id="bkmrk--11"></span><span class="anchor" id="bkmrk--12"></span><span class="anchor" id="bkmrk--13"></span><span class="anchor" id="bkmrk--14"></span><span class="anchor" id="bkmrk--15"></span><span class="anchor" id="bkmrk--16"></span><span class="anchor" id="bkmrk--17"></span><span class="anchor" id="bkmrk--18"></span><span class="anchor" id="bkmrk--19"></span><span class="anchor" id="bkmrk--20"></span><span class="anchor" id="bkmrk--21"></span><span class="anchor" id="bkmrk--22"></span><span class="anchor" id="bkmrk--23"></span><span class="anchor" id="bkmrk--24"></span><span class="anchor" id="bkmrk--25"></span><span class="anchor" id="bkmrk--26"></span><span class="anchor" id="bkmrk--27"></span><span class="anchor" id="bkmrk--28"></span>
    
    ```
    version: 0.1
    log:
      fields:
        service: registry
    storage:
      cache:
        blobdescriptor: redis
      filesystem:
        rootdirectory: /data/registry
    redis:
      addr: redis:6379
    http:
      addr: :5000
      headers:
        X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3
    ```
    
    <span class="anchor" id="bkmrk--29"></span>
5. docker run -d -p 9002:5000 -e REGISTRY\_STORAGE\_DELETE\_ENABLED=true --restart=always --name [JoangPrivateDocker](http://web.joang.com:9000/jcook/JoangPrivateDocker) -v /home/hyunsu/config.yml:/home/hyunsu/config.yml registry:2  
    : -e REGISTRY\_STORAGE\_DELETE\_ENABLED=true 옵션은 이미지를 삭제를 할 수 있다는 의미로 아래 DELETE 명령이 동작한다.   
    안하면 {"errors":\[{"code":"UNSUPPORTED","message":"The operation is unsupported."}\]} 오류가 발생 <span class="anchor" id="bkmrk--30"></span>
6. docker ps -a<span class="anchor" id="bkmrk--31"></span>
7. docker ps -l (최근)<span class="anchor" id="bkmrk--32"></span>
8. netstat -an | grep 9002 (확인)<span class="anchor" id="bkmrk--33"></span><span class="anchor" id="bkmrk--34"></span>

---

<span class="anchor" id="bkmrk--36"></span>Private Docker 사용<span class="anchor" id="bkmrk--37"></span>

1. docker build -t tomcat-meta:0.1 .<span class="anchor" id="bkmrk--38"></span>
2. docker tag tomcat-meta:0.1 web.joang.com:9002/tomcat-meta:0.1<span class="anchor" id="bkmrk--39"></span>
3. docker tag를 이용하여 docker registry에 push<span class="anchor" id="bkmrk--40"></span><span class="anchor" id="bkmrk--41"></span><span class="anchor" id="bkmrk--42"></span><span class="anchor" id="bkmrk--43"></span><span class="anchor" id="bkmrk--44"></span><span class="anchor" id="bkmrk--45"></span>```
    cat /etc/docker/daemon.json
    {
     "insecure-registries" : ["192.168.0.130:9002"]
    }
    ```
    
    <span class="anchor" id="bkmrk--46"></span>systemctl restart docker<span class="anchor" id="bkmrk--47"></span>
4. docker push web.joang.com:9002/tomcat-meta:0.1<span class="anchor" id="bkmrk--48"></span>
5. 확인 [http://192.168.0.100:9002/v2/\_catalog](http://192.168.0.100:9002/v2/_catalog)<span class="anchor" id="bkmrk--49"></span>
6. [http://192.168.0.100:9002/v2/meta-meta/tags/list](http://192.168.0.100:9002/v2/meta-meta/tags/list)<span class="anchor" id="bkmrk--50"></span><span class="anchor" id="bkmrk--51"></span><span class="anchor" id="bkmrk--52"></span>

주의 : 기본적으로 https를 사용해야 하는데 https를 쓰는 걍우 push 에서 오류가 난다.   
The push refers to repository \[192.168.0.100:9002/tomcat-meta-batch\]  
Get https://192.168.0.100:9002/v2/: http: server gave HTTP response to HTTPS client  
The push refers to repository \[192.168.0.100:9002/tomcat-meta-batch\]  
Get https://192.168.0.100:9002/v2/: http: server gave HTTP response to HTTPS client  
\---&gt; 따라서 보안을 http접근을 하용 해야 한다.   
sudo vi /etc/docker/daemon.json 를 아래와 같이 수정

```dart
{
 "insecure-registries" : ["web.joang.com:9002"]
}

```

IP로 하니까 오류 발생 !!!!!

**<span style="color: rgb(241, 196, 15);">127.0.1.1 web.joang.com 추가 !</span>**

```dart
127.0.0.1       localhost
127.0.1.1       kubemaster
127.0.1.1       web.joang.com

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

```

20231003 오류 발생   
\[Docker\] private registry http: server gave HTTP response to HTTPS client 해결 방법

- 상기한 sudo vi /etc/docker/daemon.json 조치
- /etc/containerd/config.toml에 아래 내용 추가   
    ```yaml
    ...
    ...
    
        [plugins."io.containerd.grpc.v1.cri".registry]
          config_path = ""
    
          [plugins."io.containerd.grpc.v1.cri".registry.auths]
    
          [plugins."io.containerd.grpc.v1.cri".registry.configs]
            [plugins."io.containerd.grpc.v1.cri".registry.configs."web.joang.com:9002".tls]
             insecure_skip_verify = true
    
          [plugins."io.containerd.grpc.v1.cri".registry.headers]
    
          [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
            [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
             endpoint = ["https://registry-1.docker.io"]
            [plugins."io.containerd.grpc.v1.cri".registry.mirrors."web.joang.com:9002"]
             endpoint = ["http://web.joang.com:9002"]
    
    ... 
    ```
- 작업 순서   
    hyunsu@2-kworker1:/etc/containerd$ <span style="color: rgb(255, 255, 255);">sudo vi /etc/hosts </span>  
     : 192.168.0.130 reg.joang.com  
    hyunsu@2-kworker1:/etc/containerd$ <span style="color: rgb(255, 255, 255);">sudo vi /etc/containerd/config.toml</span>  
    hyunsu@2-kworker1:/etc/containerd$ <span style="color: rgb(255, 255, 255);">sudo vi /etc/docker/daemon.json</span>  
    hyunsu@2-kworker1:/etc/containerd$ <span style="color: rgb(255, 255, 255);">sudo systemctl restart containerd</span>  
    hyunsu@2-kworker1:/etc/containerd$ <span style="color: rgb(255, 255, 255);">sudo systemctl restart docke</span>r
- yaml에서 이미지 full 시 아래와 같이 표시

---

<span class="anchor" id="bkmrk--55"></span><span class="anchor" id="bkmrk--56"></span>

#### **1.** 리포지토리 조회

\- Usage : curl -X GET &lt;Repository URL/v2/\_catalog&gt;

[http://web.joang.com:9002/v2/\_catalog](http://web.joang.com:9002/v2/_catalog)

#### **2.** 삭제할 리포지토리의 Tag 조회

\- Usage : curl -X GET &lt;Repository URL/v2/&lt;repository 이름&gt;/tag/list

[http://web.joang.com:9002/v2/tomcat-meta/tags/list](http://web.joang.com:9002/v2/tomcat-meta/tags/list)

#### **3.** content digest(hash) 조회(registry 컨테이너가 작동중인 노드에서 실행)

\- Usage : curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET &lt;Repository URL&gt;/v2/**&lt;Repository 이름&gt;**/manifests/**&lt;Tag&gt;** 2&gt;&amp;1 | grep Docker-Content-Digest | awk '{print ($3)}'

예) curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET http://192.168.0.100:9002/v2/joang-mediawiki/manifests/1 2&gt;&amp;1 | grep Docker-Content-Digest | awk '{print ($3)}'

\--&gt; 결과 sha256:e9c342dfa34bf2c3cf58503db8bc9a1298e233fadfbd6551ecea83aca80d701a

예) curl -XGET -v -H "Accept: application/vnd.docker.distribution.manifest.v2+json" [https://registry.hoya.com/v2/ubuntu/manifests/17.04](https://registry.hoya.com/v2/ubuntu/manifests/17.04)

#### **4.** manifest 삭제

\- Usage 1 : curl -X DELETE &lt;Repository URL&gt;/v2/**&lt;Repository 이름&gt;**/manifests/&lt;content digest&gt;

예) curl -X DELETE [https://registry.hoya.com/v2/ubuntu/manifests/sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459](https://registry.hoya.com/v2/ubuntu/manifests/sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459)

#### **5.** GC(Garbage Collection) 실행 : Garbage 이미지 삭제

\- Usage : docker exec -it registry bin/registry garbage-collect /etc/docker/registry/config.yml

docker exec -it JoangPrivateDocker registry garbage-collect /etc/docker/registry/config.yml

#### **6.** 레지스트리 서버 재시작

docker stop registry

docker start registry

#### 예) Registry 파일시스템내에서 파일 삭제

```bash
shell> curl -X GET https://registry.hoya.com/v2/_catalog 
{"repositories":["debian","ubuntu"]}

shell> curl -X GET https://registry.hoya.com/v2/ubuntu/tags/list 
{"name":"ubuntu","tags":["17.04","18.04"]}

shell> curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://registry.hoya.com/v2/ubuntu/manifests/17.04 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'
sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459

shell> docker exec -it registry sh      => registry container에 shell로 접속
/ # cd /var/lib/registry/docker/registry/v2
/var/lib/registry/docker/registry/v2 # rm -rf ./repositories/ubuntu/_manifests/tags/17.04  
/var/lib/registry/docker/registry/v2 # rm -rf ./repositories/ubuntu/_manifests/revisions/sha256/<content-digest>

shell> docker exec -it registry  bin/registry garbage-collect  /etc/docker/registry/config.yml 

shell> docker stop registry 
shell> docker start registry
```

#### 리포지토리 삭제

예시) ubuntu 리파지토리 삭제

```bash
shell> curl -X GET https://registry.hoya.com/v2/_catalog
{"repositories":["debian","ubuntu"]}

shell> docker exec -it registry sh      => registry container에 shell로 접속
/ # cd /var/lib/registry/docker/registry/v2
/var/lib/registry/docker/registry/v2 # rm -rf ./repositories/ubuntu/   => 레파지토리 삭제
/var/lib/registry/docker/registry/v2 # exit

shell> docker exec -it registry  bin/registry garbage-collect  /etc/docker/registry/config.yml 

shell> docker stop registry
shell> docker start registry
```

#### **TroubleShooting**

**증상 )** curl 명령어를 이용하여 삭제시 아래(붉은 글씨) 와 오류가 발생할 경우

<table border="1" id="bkmrk-%3E-delete-%2Fv2%2Fubuntu%2F"><tbody><tr><td>&gt; DELETE /v2/ubuntu/manifests/sha256:e5dd9dbb37df5b731a6688fa49f4003359f6f126958...............   
&gt; User-Agent: curl/7.29.0  
&gt; Host: registry.hoya.com  
&gt; Accept: application/vnd.docker.distribution.manifest.v2+json  
&gt;  
&lt; **HTTP/1.1 405 Method Not Allowed**  
&lt; Content-Type: application/json; charset=utf-8  
&lt; Docker-Distribution-Api-Version: registry/2.0  
&lt; X-Content-Type-Options: nosniff  
&lt; Date: Thu, 02 Apr 2020 03:24:55 GMT  
&lt; Content-Length: 78  
&lt;  
**{"errors":\[{"code":"UNSUPPORTED","message":"The operation is unsupported."}\]}**  
\* Connection #0 to host registry.hoya.com left intact</td></tr></tbody></table>

**원인)**

registry 시작시 환경변수 -e REGISTRY\_STORAGE\_DELETE\_ENABLED=true 를 지정하지 않았을 경우 DELETE 메소드가 허용되지 않는다.

**조치)**

Registry 시작시 "-e REGISTRY\_STORAGE\_DELETE\_ENABLED=true" 환경변수를 추가해서 서비스를 시작한다.

**- TroubleShooting**

**증상)** curl 명령어 실행시 아래와 같은 오류 발생

\- 데비안, 우분투

<table border="1" id="bkmrk-shell%3E-curl--x-get%C2%A0h"><tbody><tr><td>shell&gt; curl -X GET https://registry.hoya.com/v2/\_catalog   
curl: (60) SSL certificate problem: self signed certificate in certificate chain   
More details here: https://curl.haxx.se/docs/sslcerts.html   
  
curl failed to verify the legitimacy of the server and therefore could not   
establish a secure connection to it. To learn more about this situation and   
how to fix it, please visit the web page mentioned above.   
shell&gt;</td></tr></tbody></table>

\- CentOS

<table border="1" id="bkmrk-shell%3E-curl--x-get%C2%A0h-1"><tbody><tr><td>shell&gt; curl -X GET https://registry.hoya.com/v2/\_catalog   
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.   
More details here: http://curl.haxx.se/docs/sslcerts.html   
  
curl performs SSL certificate verification by default, using a "bundle"   
 of Certificate Authority (CA) public keys (CA certs). If the default   
 bundle file isn't adequate, you can specify an alternate file   
 using the --cacert option.   
If this HTTPS server uses a certificate signed by a CA represented in   
 the bundle, the certificate verification probably failed due to a   
 problem with the certificate (it might be expired, or the name might   
 not match the domain name in the URL).   
If you'd like to turn off curl's verification of the certificate, use   
 the -k (or --insecure) option.   
shell&gt;</td></tr></tbody></table>

**원인)**

registry 컨터이너가 사설 인증서를 사용해서 서비스를 할경우 curl 에서 인증서 오류 발생

**조치)**

 **1.** 데비안, 우분투

/usr/local/share/ca-certificates 디렉토리에 사설 rootca 인증서를 등록후 update-ca-certificates 명령어 실행

<table border="1" id="bkmrk-shell%3E%C2%A0cp%C2%A0rootca.crt"><tbody><tr><td>shell&gt; cp rootca.crt /usr/local/share/ca-certificates  
shell&gt; update-ca-certificates  
Updating certificates in /etc/ssl/certs...  
1 added, 0 removed; done.  
Running hooks in /etc/ca-certificates/update.d...  
done.  
shell&gt;</td></tr></tbody></table>

 **2.** CentOS

/etc/pki/ca-trust/source/anchors/ 디렉토리에 사설 rootca 인증서를 등록후 update-ca-trust명령어 실행

<table border="1" id="bkmrk-shell%3E-cp-rootca.crt"><tbody><tr><td>shell&gt; cp rootca.crt /etc/pki/ca-trust/source/anchors/  
shell&gt; update-ca-trust</td></tr></tbody></table>

 3. curl 명령어에 **-k** 또는 **--inscure** 옵션 사용

<table border="1" id="bkmrk-shell%3E-curl--k%C2%A0https"><tbody><tr><td>shell&gt; curl -k https://www.domain.com   
*OR* shell&gt; curl --insecure https://www.domain.com</td></tr></tbody></table>

Private image delete<span class="anchor" id="bkmrk--58"></span>

1. curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET [http://192.168.56.3:9002/v2/tomcat-synapse/manifests/0.1](http://192.168.56.3:9002/v2/tomcat-synapse/manifests/0.1) 2&gt;&amp;1 | grep Docker-Content-Digest | awk '{print ($3)}'<span class="anchor" id="bkmrk--59"></span>
2. curl -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X DELETE [http://192.168.56.3:9002/v2/tomcat-synapse/manifests/sha256:65336b7ee5a56dc2a7294c02fbb515542e5212a7ea193943160d9ecbb4ca0f62](http://192.168.56.3:9002/v2/tomcat-synapse/manifests/sha256:65336b7ee5a56dc2a7294c02fbb515542e5212a7ea193943160d9ecbb4ca0f62)<span class="anchor" id="bkmrk--60"></span>
3. GC(Garbage Collection)<span class="anchor" id="bkmrk--61"></span><span class="anchor" id="bkmrk--62"></span><span class="anchor" id="bkmrk--63"></span>```
    docker exec -it JoangPrivateDocker registry garbage-collect /etc/docker/registry/config.yml
    ```
    
    <span class="anchor" id="bkmrk--64"></span>
4. Image 정리<span class="anchor" id="bkmrk--65"></span><span class="anchor" id="bkmrk--66"></span><span class="anchor" id="bkmrk--67"></span>```
    docker image prune -f
    ```
    
    <span class="anchor" id="bkmrk--68"></span><span class="anchor" id="bkmrk--69"></span>