카타코더 - Get Started with Istio and Kubernetes

1 개요[ | ]

카타코더 - Get Started with Istio and Kubernetes

2 Launch Kubernetes Cluster[ | ]

root@master:~# launch.sh
Waiting for Kubernetes to start...
Kubernetes started
root@master:~# kubectl cluster-info
Kubernetes master is running at https://172.17.0.13:6443
KubeDNS is running at https://172.17.0.13:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

3 Deploy Istio[ | ]

  • Install CLI tooling
root@master:~# curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.0.0 sh -
...
Istio 1.0.0 Download Complete!

Istio has been successfully downloaded into the istio-1.0.0 folder on your system.

Next Steps:
See https://istio.io/docs/setup/kubernetes/install/ to add Istio to your Kubernetes cluster.

To configure the istioctl client tool for your workstation,
add the /root/istio-1.0.0/bin directory to your environment path variable with:
         export PATH="$PATH:/root/istio-1.0.0/bin"

Begin the Istio pre-installation verification check by running:
         istioctl verify-install

Need more information? Visit https://istio.io/docs/setup/kubernetes/install/
root@master:~# export PATH="$PATH:/root/istio-1.0.0/bin"
root@master:~# cd /root/istio-1.0.0
root@master:~/istio-1.0.0# kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml -n istio-system
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/policies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/meshpolicies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/bypasses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/circonuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/deniers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/fluentds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/kubernetesenvs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listcheckers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/memquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/noops.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/opas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/prometheuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/redisquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicecontrols.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/signalfxs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/solarwindses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stackdrivers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/statsds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stdios.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/apikeys.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/authorizations.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/checknothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/kuberneteses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/logentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/edges.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/metrics.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/reportnothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicecontrolreports.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/tracespans.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created
root@master:~/istio-1.0.0# kubectl apply -f install/kubernetes/istio-demo-auth.yaml
namespace/istio-system created
configmap/istio-galley-configuration created
configmap/istio-grafana-custom-resources created
configmap/istio-statsd-prom-bridge created
configmap/prometheus created
configmap/istio-security-custom-resources created
configmap/istio created
configmap/istio-sidecar-injector created
serviceaccount/istio-galley-service-account created
serviceaccount/istio-egressgateway-service-account created
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/istio-grafana-post-install-account created
clusterrole.rbac.authorization.k8s.io/istio-grafana-post-install-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-grafana-post-install-role-binding-istio-system created
job.batch/istio-grafana-post-install created
serviceaccount/istio-mixer-service-account created
serviceaccount/istio-pilot-service-account created
serviceaccount/prometheus created
serviceaccount/istio-cleanup-secrets-service-account created
clusterrole.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
job.batch/istio-cleanup-secrets created
serviceaccount/istio-security-post-install-account created
clusterrole.rbac.authorization.k8s.io/istio-security-post-install-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-security-post-install-role-binding-istio-system created
job.batch/istio-security-post-install created
serviceaccount/istio-citadel-service-account created
serviceaccount/istio-sidecar-injector-service-account created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io configured
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io configured
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io configured
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io configured
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io configured
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/bypasses.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/circonuses.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/deniers.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/fluentds.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/kubernetesenvs.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/listcheckers.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/memquotas.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/noops.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/opas.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/prometheuses.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/rbacs.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/redisquotas.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/servicecontrols.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/signalfxs.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/solarwindses.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/stackdrivers.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/statsds.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/stdios.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/apikeys.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/authorizations.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/checknothings.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/kuberneteses.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/listentries.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/logentries.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/edges.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/metrics.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/quotas.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/reportnothings.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/servicecontrolreports.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/tracespans.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io configured
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io configured
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io configured
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io configured
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io configured
clusterrole.rbac.authorization.k8s.io/istio-galley-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-mixer-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrole.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-sidecar-injector-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-galley-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-mixer-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-sidecar-injector-admin-role-binding-istio-system created
service/istio-galley created
service/istio-egressgateway created
service/istio-ingressgateway created
service/grafana created
service/istio-policy created
service/istio-telemetry created
service/istio-statsd-prom-bridge created
deployment.extensions/istio-statsd-prom-bridge created
service/istio-pilot created
service/prometheus created
service/istio-citadel created
service/servicegraph created
service/istio-sidecar-injector created
deployment.extensions/istio-galley created
deployment.extensions/istio-egressgateway created
deployment.extensions/istio-ingressgateway created
deployment.extensions/grafana created
deployment.extensions/istio-policy created
deployment.extensions/istio-telemetry created
deployment.extensions/istio-pilot created
deployment.extensions/prometheus created
deployment.extensions/istio-citadel created
deployment.extensions/servicegraph created
deployment.extensions/istio-sidecar-injector created
deployment.extensions/istio-tracing created
gateway.networking.istio.io/istio-autogenerated-k8s-ingress created
horizontalpodautoscaler.autoscaling/istio-egressgateway created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-pilot created
service/jaeger-query created
service/jaeger-collector created
service/jaeger-agent created
service/zipkin created
service/tracing created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
attributemanifest.config.istio.io/istioproxy created
attributemanifest.config.istio.io/kubernetes created
stdio.config.istio.io/handler created
logentry.config.istio.io/accesslog created
logentry.config.istio.io/tcpaccesslog created
rule.config.istio.io/stdio created
rule.config.istio.io/stdiotcp created
metric.config.istio.io/requestcount created
metric.config.istio.io/requestduration created
metric.config.istio.io/requestsize created
metric.config.istio.io/responsesize created
metric.config.istio.io/tcpbytesent created
metric.config.istio.io/tcpbytereceived created
prometheus.config.istio.io/handler created
rule.config.istio.io/promhttp created
rule.config.istio.io/promtcp created
kubernetesenv.config.istio.io/handler created
rule.config.istio.io/kubeattrgenrulerule created
rule.config.istio.io/tcpkubeattrgenrulerule created
kubernetes.config.istio.io/attributes created
destinationrule.networking.istio.io/istio-policy created
root@master:~/istio-1.0.0# kubectl get pods -n istio-system
NAME                                        READY     STATUS      RESTARTS   AGE
grafana-86645d6b4d-2n72z                    1/1       Running     0          1m
istio-citadel-55d9bb9b5f-6vjs5              1/1       Running     0          1m
istio-cleanup-secrets-5qz8v                 0/1       Completed   0          1m
istio-egressgateway-6fc78c57c8-mh4rg        1/1       Running     0          1m
istio-galley-d4bc6c974-8r5gv                1/1       Running     0          1m
istio-grafana-post-install-zrvcz            0/1       Completed   0          1m
istio-ingressgateway-744c6b94d7-2pw42       1/1       Running     0          1m
istio-pilot-7ccb57d8b7-mg948                2/2       Running     0          1m
istio-policy-7bff8b7d65-rwp9j               2/2       Running     0          1m
istio-security-post-install-c72sl           0/1       Completed   0          1m
istio-sidecar-injector-854f6498d9-lxck4     1/1       Running     0          1m
istio-statsd-prom-bridge-549d687fd9-8mkjd   1/1       Running     0          1m
istio-telemetry-d57b9ccb4-s9chk             2/2       Running     0          1m
istio-tracing-7596597bd7-zn44t              1/1       Running     0          1m
prometheus-6ffc56584f-cbz4v                 1/1       Running     0          1m
servicegraph-7bdb8bfc9d-lfq28               1/1       Running     0          1m
root@master:~/istio-1.0.0# kubectl apply -f /root/katacoda.yaml
service/katacoda-servicegraph created
service/katacoda-grafana created
service/katacoda-jaeger-query created
service/katacoda-prometheus created
service/istio-ingressgateway configured

4 Istio Architecture[ | ]

Istio-arch1.png

5 Deploy Sample Application[ | ]

root@master:~/istio-1.0.0# kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml)
service/details created
deployment.extensions/details-v1 created
service/ratings created
deployment.extensions/ratings-v1 created
service/reviews created
deployment.extensions/reviews-v1 created
deployment.extensions/reviews-v2 created
deployment.extensions/reviews-v3 created
service/productpage created
deployment.extensions/productpage-v1 created
root@master:~/istio-1.0.0# kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yamlgateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created
root@master:~/istio-1.0.0# kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
details-v1-7b5f4fc575-zffd6      2/2       Running   0          31s
productpage-v1-c86d5ddf9-ldvtj   2/2       Running   0          31s
ratings-v1-58d59948f5-275n4      2/2       Running   0          31s
reviews-v1-6d9787f845-87rp5      2/2       Running   0          31s
reviews-v2-84b4476485-54pf8      2/2       Running   0          31s
reviews-v3-55d6499f7c-4dltn      2/2       Running   0          31s
root@master:~/istio-1.0.0# kubectl apply -f samples/bookinfo/networking/destination-rule-all-mtls.yaml
destinationrule.networking.istio.io/productpage created
destinationrule.networking.istio.io/reviews created
destinationrule.networking.istio.io/ratings created
destinationrule.networking.istio.io/details created

6 Bookinfo Architecture[ | ]

Bookinfo-noistio.svg

7 Control Routing[ | ]

  • User Based Testing / Request Routing
root@master:~/istio-1.0.0# cat samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1
root@master:~/istio-1.0.0# kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
virtualservice.networking.istio.io/reviews created
  • Traffic Shaping for Canary Releases
root@master:~/istio-1.0.0# cat samples/bookinfo/networking/virtual-service-reviews-50-v3.yamlapiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 50
    - destination:
        host: reviews
        subset: v3
      weight: 50
root@master:~/istio-1.0.0# kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-50-v3.yaml
virtualservice.networking.istio.io/reviews configured
  • New Releases
root@master:~/istio-1.0.0# cat samples/bookinfo/networking/virtual-service-reviews-v3.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v3
root@master:~/istio-1.0.0# kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-v3.yaml
virtualservice.networking.istio.io/reviews configured
  • List All Routes
root@master:~/istio-1.0.0# istioctl get virtualservices
VIRTUAL-SERVICE NAME   GATEWAYS           HOSTS     #HTTP     #TCP      NAMESPACE   AGE
bookinfo               bookinfo-gateway   *             1        0      default     2m
reviews                                   reviews       1        0      default     1m
root@master:~/istio-1.0.0# istioctl get virtualservices -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"default"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"exact":"/productpage"}},{"uri":{"exact":"/login"}},{"uri":{"exact":"/logout"}},{"uri":{"prefix":"/api/v1/products"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}}
  creationTimestamp: null
  name: bookinfo
  namespace: default
  resourceVersion: "2159"
spec:
  gateways:
  - bookinfo-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"reviews","namespace":"default"},"spec":{"hosts":["reviews"],"http":[{"route":[{"destination":{"host":"reviews","subset":"v3"}}]}]}}
  creationTimestamp: null
  name: reviews
  namespace: default
  resourceVersion: "2439"
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v3
---

8 Access Metrics[ | ]

  • Generate Load
root@master:~/istio-1.0.0# while true; do
>   curl -s https://2886795277-80-frugo01.environments.katacoda.com/productpage > /dev/null
>   echo -n .;
>   sleep 0.2
> done
........................
Grafana https://2886795277-3000-frugo01.environments.katacoda.com/dashboard/db/istio-mesh-dashboard
Jaeger https://2886795277-16686-frugo01.environments.katacoda.com/
Service Graph https://2886795277-8088-frugo01.environments.katacoda.com/dotviz
  • Ctrl+C 눌러 스크립트 중단

9 Visualise Cluster using Weave Scope[ | ]

  • Deploy Scope
root@master:~/istio-1.0.0# kubectl create -f 'https://cloud.weave.works/launch/k8s/weavescope.yaml'
namespace/weave created
serviceaccount/weave-scope created
clusterrole.rbac.authorization.k8s.io/weave-scope created
clusterrolebinding.rbac.authorization.k8s.io/weave-scope created
deployment.apps/weave-scope-app created
service/weave-scope-app created
deployment.apps/weave-scope-cluster-agent created
daemonset.extensions/weave-scope-agent created
root@master:~/istio-1.0.0# kubectl get pods -n weave
NAME                                        READY     STATUS    RESTARTS   AGE
weave-scope-agent-pqbpr                     1/1       Running   0          11s
weave-scope-agent-vdps9                     1/1       Running   0          11s
weave-scope-app-6d6db5db98-22qv2            1/1       Running   0          11s
weave-scope-cluster-agent-b994546c9-d25f5   1/1       Running   0          11s
  • Make Scope Accessible
root@master:~/istio-1.0.0# pod=$(kubectl get pod -n weave --selector=name=weave-scope-app -ojsonpath={.items..metadata.name})
root@master:~/istio-1.0.0# kubectl expose pod $pod -n weave --external-ip="172.17.0.13" --port=4040 --target-port=4040
service/weave-scope-app-6d6db5db98-22qv2 exposed
root@master:~/istio-1.0.0# while true; do
>   curl -s https://2886795277-80-frugo01.environments.katacoda.com/productpage > /dev/null
>   echo -n .;
>   sleep 0.2
> done
..............

10 같이 보기[ | ]

11 참고[ | ]

문서 댓글 ({{ doc_comments.length }})
{{ comment.name }} {{ comment.created | snstime }}