31/12/2018

[2018 Mini Pycon] 파이썬 기반서비스의 구성과 이해 - 이새로찬

Download Presentation File

Python Seminar Pycon
2/11/2018

Install Parity to Ubuntu

Install parity

$ bash <(curl https://get.parity.io -L) -r stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   170    0   170    0     0    122      0 --:--:--  0:00:01 --:--:--   122
100  3856  100  3856    0     0   2182      0  0:00:01  0:00:01 --:--:-- 1892k
Release selected is: stable
Upgrading parity from 0.0.0 to 2.0.8

systemd

  • systemd unit file - /lib/systemd/system/parity.service
[Unit]
Description=Parity Ethereum Daemon
After=network.target

[Service]
# run as root, set base_path in config.toml
ExecStart=/usr/bin/parity --config /etc/parity/config.toml
# To run as user, comment out above and uncomment below, fill in user and group
# picks up users default config.toml in $HOME/.local/share/io.parity.ethereum/
# User=username
# Group=groupname
# ExecStart=/usr/bin/parity
Restart=on-failure

# Specifies which signal to use when killing a service. Defaults to SIGTERM.
# SIGHUP gives parity time to exit cleanly before SIGKILL (default 90s)
KillSignal=SIGHUP

[Install]
WantedBy=default.target

parity configuration

데이터 경로를 백업볼륨 경로로 선정. 자세한 설정은 해당 링크 참고. Configuring-Parity-Ethereum

  • parity config file - /etc/parity/config.toml
[parity]
base_path = "/var/lib/parity"
db_path = "/var/lib/parity/chains"
keys_path = "/var/lib/parity-data/keys"
...

# log option
[misc]
logging = "info"
log_file = "/var/lib/parity/parity.log"

run parity

parity 를 실행하기 위한 기본적인 셋업은 끝났으며, 정상작동을 확인해본다.

$ sudo systemctl start parity
# 실행 상태 확인
$ sudo systemctl status parity
# 로그 확인
$ sudo tail -f /var/lib/parity/parity.log

logrotate

parity 로그 파일을 logroate사용하여 관리한다.

  • logroate conf file - /etc/logrotate.d/wallet
/data/log/wallet.log { 
    daily
    compress
    rotate 365
    missingok
    notifempty
    dateext
    copytruncate
}
$ sudo logrotate -f /etc/logrotate.d/wallet
$ sudo gzip -d wallet.log-20201010.gz
$ sudo cat wallet.log-20201010

Reference

parity-ethereum github

22/10/2018

Flask gunicorn nginx docker

Best practice of dockerizing flask gunicorn nginx.

PS. If you want to see the sample code. Click this link GitHub Repository

Concept

Starting with the installation of flask application, stack it step by step with Bottom Up.

PS. This following descriptions may not be accurate and in such cases pleas read and understand the code in the repository

main app versions

  • python/3.6.4
  • nginx/1.15.5

Flask Web Application

$ python -m venv venv
$ . venv/bin/activate
$ python -V
$ pip install --upgrade pip
$ pip install Flask
$ pip list
# main.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return "HELLO, WORLD!"


@app.route('/ping')
def ping():
    return "PONG"
$ FLASK_APP=main.py flask run
$ curl 127.0.0.1:5000/ping
PONG
$ curl 127.0.0.1:5000/
HELLO, WORLD!

debug or development mode

  • it activates the debugger
  • it activates the automatic reloader
  • it enables the debug mode on the Flask application.
$ export FLASK_ENV=development
$ export FLASK_DEBUG=1
$ export FLASK_APP=main.py
$ flask run

PS. This makes it a major security risk and therefore it must never be used on production machines.

performance test for flaks standalone

$ flask run
$ ab -n 32 -c 16 127.0.0.1:5000/sleep/10
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        Werkzeug/0.14.1
Server Hostname:        127.0.0.1
Server Port:            5000

Document Path:          /sleep/10
Document Length:        19 bytes

Concurrency Level:      16
Time taken for tests:   20.031 seconds
Complete requests:      32
Failed requests:        0
Total transferred:      5536 bytes
HTML transferred:       608 bytes
Requests per second:    1.60 [#/sec] (mean)
Time per request:       10015.349 [ms] (mean)
Time per request:       625.959 [ms] (mean, across all concurrent requests)
Transfer rate:          0.27 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing: 10006 10012   2.9  10012   10017
Waiting:    10005 10011   2.8  10011   10017
Total:      10006 10012   2.9  10012   10017

Percentage of the requests served within a certain time (ms)
  50%  10012
  66%  10013
  75%  10015
  80%  10015
  90%  10016
  95%  10017
  98%  10017
  99%  10017
 100%  10017 (longest request)

gunicorn

$ pip install gunicron
$ gunicorn -w 3 --threads 3 wsgi:app -b 0.0.0.0:5000
[2018-10-09 19:30:57 +0900] [79365] [INFO] Starting gunicorn 19.9.0
[2018-10-09 19:30:57 +0900] [79365] [INFO] Listening at: http://0.0.0.0:5000 (79365)
[2018-10-09 19:30:57 +0900] [79365] [INFO] Using worker: threads
[2018-10-09 19:30:57 +0900] [79368] [INFO] Booting worker with pid: 79368
[2018-10-09 19:30:57 +0900] [79369] [INFO] Booting worker with pid: 79369
[2018-10-09 19:30:57 +0900] [79370] [INFO] Booting worker with pid: 79370

main configuration

  • bind
  • workers
  • worker_class
  • threads
    • This setting only affects the Gthread worker type.
  • access_logfile
    • - means log to stdout
  • access_logformat
    • default: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  • error_logfile
    • - means log to stderr
# TCP SOCKET BINDING
$ gunicorn -w 3 --access-logfile - -k gevent wsgi:app -b 0.0.0.0:5000
...
# UNIX SOCKET BINDING
$ gunicorn -w 3 --access-logfile - -k gevent wsgi:app -b unix:/tmp/gunicorn.sock
...

# Check The Socket
$ echo -e "GET /ping HTTP/1.0\r\n\r\n" | nc -q 2 -U /tmp/gunicorn.sock
PONG
$ gunicorn --config gunicorn_config.py wsgi:app

Nginx

mac os var, etc path

/usr/local/etc/nginx/
/usr/local/var/log/nginx

gunicorn-nginx.conf

/usr/local/etc/nginx/servers/gunicorn-nginx.conf

upstream gunicorn-app {
    server unix:/tmp/gunicorn.sock fail_timeout=0;
}


server {
    listen 80;
    server_name 0.0.0.0;

    client_max_body_size 5M;

    access_log /usr/local/var/log/nginx/access.log combined;
    error_log /usr/local/var/log/nginx/error_log.log warn;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        proxy_pass http://gunicorn-app;
    }
}
sudo nginx -c [config_path]
sudo nginx -s stop

Docker

When I dockerize, first setup the tools and environment necessary for dockerizing image. Then, enter console of the container, check the configuration, and run each application to verify normal operation. Afrter verifcation is completed, write CMD or ENTRYPOINT COMMAND.

Dockerfile
FROM python:3.6.6-jessie

MAINTAINER cgex

ENV SRVHOME=/srv/app

WORKDIR $SRVHOME

COPY ./ $SRVHOME

RUN apt-get update &amp;&amp; apt-get install -y \
    nginx \
    supervisor

COPY ./nginx/nginx.conf /etc/nginx/
COPY ./nginx/gunicorn-app.conf /etc/nginx/conf.d/

COPY ./gunicorn_config.py /etc/gunicorn/

COPY supervisord.conf /etc/supervisor/

RUN pip install --upgrade pip
RUN pip install -r requirements.txt

EXPOSE 80
EXPOSE 8080

# for receving stream of log 
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
    &amp;&amp; ln -sf /dev/stderr /var/log/nginx/error.log

CMD ["/usr/bin/supervisord"]

REFERENCES

gunicorn vs uwsgi flask-concurrency-test

12/2/2018

WEB Framework Check Template

글에서 말하는프레임웍은 REST API 중심의프레임웍을 말한다. 해당 글의 작성 목적은프레임웍을 선택 혹은 구축하는데 있어 고려해볼사항들에 대해서 포괄적으로 언급한다. 파이썬 진형의 대표적인프레임웍 장고에서 많은 개념을 참고했다.

주요 고려 항목

  • Authentication
    • OAuth2, JWT
  • Explicit Project Structure
  • Architecture
    • Design Pattern. MVC Pattern
    • 장고의 경우 View, Model, Serializer 와 같은모듈의 명확한 역할 정의. 균일한 데이터 흐름을 갖게 하는게 매우 중요.
  • Middleware
    • Throttling
    • 필요하다면 Device 분기
    • Logging
    • 현재 서버 환경별 Request, Response Handling
      • Beta 환경인 경우. 토큰발행등으로 Beta 서버 추가 인증.
  • Dynamic and Elastic Settings Load. e.g) local, development, alpha, beta, production …
  • DDD or Adaptable MSA
    • 장고의 경우 앱을 Service 중심으로 분류. 서비스간 단일 인터페이스를 통해 의존관계 설립.
  • REST API URL일반화 가능여부.
    • 장고의 경우 function view => class view
      • class view경우 API 별 permission, authentication, throttling 관리가 매우 편해짐.
  • Exception Handling Centralization
  • Test Environment
    • Test Framework
  • Cache Backend & Engine
    • Redis, Memcached …
  • Storage Engines
    • MySQL, MSSQL, PostgreSQL …
  • ORM Engine
  • API Versioning
    • /v1/
  • API Documentation
    • Swagger
  • Web 혹은 Admin 필요시 Template Engine
  • Runtime Debugger
    • django-debbug-toolbar
  • Serialization

ON DEMAND

  • Asynchronous task queue
    • Celery
    • Email, SMS 발송 같은 서비스
  • Support DATA Team
    • DATA HOOK
  • Log Centralization
Web Django
7/2/2018

[Django] contenttypes framework

부제: django.contrib.contenttypes 은 무엇인가?

장고 프로젝트를 생성하고, settings.py 파일을 보면 기본값 설정들이 되어있고, INSTALLED_APPS 하위에 이미 어떤 장고 어플리케이션들이 선언되어있다. 해당 어플리케이션인들이 프로젝트에 판단하기 위해 역할을 알아볼 것이다. 이 포스트에서는 django.contrib.contenttypes대해 살펴볼것이다.

django.contrib.contenttypes

네이밍이 contenttypes 이여서 HTTP Header의 속성 Content-Type 에 관련된 앱인줄 알았다. 하지만 설명을 읽어보니 전혀 상관없었다.

Django includes a contenttypes application that can track all of the models installed in your Django-powered project, providing a high-level, generic interface for working with your models.

설명을 보아하니, 장고 프로젝트안에서 선언된 모델을 추적하며, 높은수준의 사용하기 편한 인터페이스를 제공한단다. 목적은 파악되었다. 아직 필요한지 제거해도 되는지 판단이 서지 않아서 조금알아보겠다.

It’s generally a good idea to have the contenttypes framework installed; several of Django’s other bundled applications require it:

The admin application uses it to log the history of each object added or changed through the admin interface. Django’s authentication framework uses it to tie user permissions to specific models.

몇몇 장고의 번들 어플리케이션을 사용할때 contenttypes framework필요하단다… (몇몇!!!??? 말이여 방구여 그냥 쓰라는 의미로 받아들이자.) 장고 인증 프레임웍과 어드민에 의존 어플리케이션이란다. 더 이상 읽을 필요가 없다. 사용하기로 결정!!

PS. 장고에서 코드를 작성할때, 모델 사용을 참조 형식이 아닌, 문자열 형식으로 선언하고 런타임 환경에서 Lazy Loading 형식으로 구현하는 경우가 많다. 추축이지만 아마 contenttypes framework 때문에 가능한것 같다. 코드를 까서 살펴보면 되지만, 시간이 없다. 조사는 여기까지 그만 짜이찌엔!!

Django Python
31/1/2018

이벤트 기반의 이해

마이크로서비스 아키텍처 구축살펴보던이벤트 기반(Event Driven)대해서 언급이 나왔다. 읽기만 해서는 이해가 되지 않아 코드를 작성해보았다. 물론 아직도 이해가안된다.

오케스트레이션(Orchestration)

중앙에서 이벤트를 모두 관리처리

class Customrer():
    def __init__(self, name, address, email):
        self.name = name
        self.address = address
        self.email = email


class CustomerService():

    def signup(self, customrer):
        # 계좌 개설
        BankService().create_account(customrer)

        # 환영 메일 발송
        MailService().send(customrer.email)

        # 가입 패키지 발송
        PostService().send(customrer.address)


class BankService():
    def create_account(self, customrer):
        print("Service Name: %s, Customrer Name: %s" %(self.__class__.__name__, customrer.name))


class MailService():
    def send(self, to):
        print("Service Name: %s, To: %s" % (self.__class__.__name__, to))


class PostService():
    def send(self, to):
        print("Service Name: %s, Address: %s" % (self.__class__.__name__, to))


if __name__ == '__main__':
    customer = Customrer(name='SELO', address='Seoul, Korea', email='rochan87@gmail.com')
    CustomerService().signup(customer)

코레오그래피(Choreography)

발행/구독 패턴을 통한 느슨한 결합

class Customrer():
    def __init__(self, name, address, email):
        self.name = name
        self.address = address
        self.email = email

EventBook = {
    # example
    'main_event': {
        'events': ['dependent_event', ],
    }
}


def register_event(pub_event_name, sub_event):
    try:
        pub_event = EventBook[pub_event]
        pub_event.append(sub_event)
    except KeyError:
        EventBook[pub_event] = [sub_event]


def pub(event_name):
    def add(func):
        def __wrapper(*args, **kwargs):
            _r = func(*args, **kwargs)
            sub_events = EventBook[event_name]
            for sub_event in sub_events:
                sub_event(*args, **kwargs)
            return _r
        
        if event_name not in EventBook:
            EventBook[event_name] = []

        return __wrapper
    return add


def sub(event_name):
    def register(func):
        def __wrapper(*args, **kwargs):
            return func(*args, **kwargs)

        if event_name not in EventBook:
            EventBook[event_name] = []

        EventBook[event_name].append(__wrapper)
        return __wrapper
    return register


SIGNUP = 'signup'
UPDATE = 'update'


class CustomerService:
    @pub(SIGNUP)
    def signup(self, customrer):
        print("SignUp\n")

    @pub(UPDATE)
    def update(self, customrer):
        print("Update\n")


class BankService:
    @sub(SIGNUP)
    def create_account(self, customrer):
        print("Service Name: %s, Customrer Name: %s" %(self.__class__.__name__, customrer.name))


class MailService:

    # 아래와 같이 구독하고 싶은 이벤트만 추가해서 원하는 동작을 수행한다.
    @sub(SIGNUP)
    @sub(UPDATE)
    def send(self, customrer):
        print("Service Name: %s, To: %s" % (self.__class__.__name__, customrer.email))


class PostService:
    @sub(SIGNUP)
    def send(self, customrer):
        print("Service Name: %s, Address: %s" % (self.__class__.__name__, customrer.address))



if __name__ == '__main__':
    customer = Customrer(name='SELO', address='Seoul, Korea', email='rochan87@gmail.com')
    CustomerService().signup(customer)
    CustomerService().update(customer)
MSA MicroserviceArchitecture EventDriven