기록's

개요

로컬에 fluentd에 의해 efs에 적재되는 로그들에 대해 압축하여 s3로 Upload하는 작업이다. (tmi:efs보다 s3가 비용저렴하다)

aws role 관련 문제에 의해 aws mv가 안 먹히는 부분으로 cp로 진행 하였으며, 모든 복사가 끝나면 삭제 되도록 되어 있다. 적재 되는 로그 상위 폴더의 Naming이 불규칙적이기 때문에 순서로 커트 하였다.

#!/bin/bash

s3Path=s3://test-logs.s3.test.com/fluentd/

/usr/bin/find /test/efs/ -name '*.log' -not -name 'buffer.*' -mtime +0 -exec gzip -9 {} +;
/usr/bin/find /test/efs/ -name '*.log.*' -not -name 'buffer.*' -mtime +0 -exec gzip -9 {} +;

uploadList=(`/usr/bin/find /test/efs -name '*.gz' -not -name 'buffer.*' | cut -f4,5 -d '/'`)
uploadFile=(`/usr/bin/find /test/efs -name '*.gz' -not -name 'buffer.*'`)

/usr/bin/touch /usr/share/script/logUpload.log
/usr/bin/chown -R test. /usr/share/script/logUpload.log

for u in "${!uploadFile[@]}"; do
        /usr/bin/sudo -u test /usr/bin/aws s3 cp ${uploadFile[$u]} s3://test-logs.s3.test.com/fluentd/${uploadList[$u]} >> /usr/share/script/logUpload.log
done

catLog=`cat /usr/share/script/logUpload.log | grep 'upload' | wc -l`

if [ ${catLog} -gt 0 ]
then
   /usr/bin/rm -f ${uploadFile[@]}
   /usr/bin/rm -f /usr/share/script/logUpload.log
else
  echo "not found log" >> /usr/share/script/logUpload.log
fi

PostgreSQL 문에서 Select으로 리스트를 뽑을 때 Json 형식으로 Row를 Export 하는 방식이다.

row_to_json으로 사용하는 것으로 생각하면 된다.

특정 알파뱃 제외 구문이나 인코딩 방식 등 포함 되어 있으니 보고하실거면 주의 필요함

#-*- coding: euc-kr -*-
import json
import psycopg2
import requests
import sys
import chardet

sys.stdout = open('/usr/share/openldap-servers/userList.json', 'w')

try:
    conn_string = "host='dbsvr.com' dbname='gooddb' user='test' password='test12!@' port='5432'"
    conn = psycopg2.connect(conn_string)
    cur = conn.cursor()
    qry_origin = "select userid, status, empno, username, mailaddr, dutycode, positionname, deptcode, deptname, dutyname, updated_at from view_user where updated_at >= current_timestamp + '-120 minute' and deptcode not like 'E%' and dutyname is not null order by updated_at desc"
    #qry_origin = "select userid, status, empno, username, mailaddr, dutycode, positionname, deptcode, deptname, dutyname, updated_at from view_user where status='ONLINE'
    qry = u"select row_to_json(tmp) from ("+ qry_origin +") tmp;"
    cur.execute(qry)
    rownum = int(cur.rowcount)
    i=0
    while i < rownum:
        result = cur.fetchone()[0]
        print(json.dumps(result, ensure_ascii=False).encode('utf8'))
        i = i + 1
    cur.close()
    conn.close()
except psycopg2.DatabaseError as db_err:
    print('!!! not connected !!!', db_err)

sys.stdout.close()

sys.stdout = open('/usr/share/openldap-servers/userStopList.json', 'w')

try:
    conn_string = "host='dbsvr.com' dbname='gooddb' user='test' password='test12!@' port='5432'"
    conn = psycopg2.connect(conn_string)
    cur = conn.cursor()
    qry_origin = "select userid, status, empno, username, mailaddr, dutycode, positionname, deptcode, deptname, dutyname from view_user where status='STOP'"
    qry = u"select row_to_json(tmp) from ("+ qry_origin +") tmp;"
    cur.execute(qry)
    rownum = int(cur.rowcount)
    i=0
    while i < rownum:
        result = cur.fetchone()[0]
        print(json.dumps(result, ensure_ascii=False).encode('utf8'))
        i = i + 1
    cur.close()
    conn.close()
except psycopg2.DatabaseError as db_err:
    print('!!! not connected !!!', db_err)

sys.stdout.close()

#!/bin/sh
ap=`hostname`

for i in {1..20} ; do

    healthStatus=`curl -LI http://localhost:55001/health -o /dev/null -w '%{http_code}' -s`
    echo "[$ap] health status : $healthStatus"

    if [ "200" == "$healthStatus" ]; then
        break
    fi

    sleep 5
done

#!/bin/bash


#SLACK_CHANNEL="#채널명"
SLACK_USERNAME="[Bot] send from Bash shell"
SLACK_URL='https://hooks.slack.com/services/1234"
STRING="test"
#curl -X POST --data-urlencode "payload={\"channel\": \"$SLACK_CHANNEL\", \"username\": \"$SLACK_USERNAME\", \"text\": \"$STRING\"}" $SLACK_URL
curl -X POST --data-urlencode "payload={\"username\": \"$SLACK_USERNAME\", \"text\": \"$STRING\"}" $SLACK_URL

# 쉘 경로 지정
vi /etc/makeloglink.sh
 
# 쉘 내용
#!/bin/sh
LOGFILE=`ls -t /home/test/logs/log.* | head -n 1`
if [ -f "${LOGFILE}" ];then
        rm -f /tmp/test.log
        ln -s ${LOGFILE} /tmp/test.log
fi
 
# 쉘 권한
chmod 755 /etc/makeloglink.sh
 
# 쉘 크론탭 등록
1 0 * * * /etc/makeloglink.sh