用Shell脚本实现自动从NewSmth.net的MyPhoto版下载照片
2008-03-22 23:53
886 查看
写了一个脚本,用于自动从水木社区(Newsmth.net)的MyPhoto版自动下载图片到本地。
运行该脚本后,将自动从水木社区的MyPhoto版自动下载最近N(默认为3,通过参数1指定)天内的所有图片到本地的photo目录下。
用法:
1、把以下代码保存为autoPicSmth.sh
2、为脚本增加可执行权限,并运行脚本。
CHEYO:~/auto # chmod +x autoPicSmth.sh
CHEYO:~/auto # ./autoPicSmth.sh
脚本写得比较粗糙,欢迎优化改进。
源码:
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#!/bin/bash
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#####################################################################
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Script: autoPicSmth.sh
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Author: cheyo
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Email: icheyo at Gmail dot com
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# From: www.icheyo.net
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Date: 2008-02-22
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Description:
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# This script is used for downloading pictures from the MyPhoto
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# board in the newsmth.net automatically.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#####################################################################
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Usage: autoPicSmth.sh [days]
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# days: download all pictures of recent /days/ days
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# For Example: ./autoPicSmth.sh 3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
WORKING_DIR=working
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_OUT_DIR=photo
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
DAYS_TO_DOWN=3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
QUERY_FILE=QueryResult.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
THREAD_FILE=ThreadUrl.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FORMAT_FILE=ThreadInfo.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
CURR_THREAD_FILE=CurrThread.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_URL_FILE=PicUrl.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_DOWN_LOG=PicDown.log
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE1=PicInfo1.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE2=PicInfo2.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE3=PicInfoFinal.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ------------------------------------------------------------------ #
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ShowUsage()
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Show the usage of this script
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ------------------------------------------------------------------ #
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ShowUsage()
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
{
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "This script is used for automatic downloading pictures from MyPhoto board in the newsmth.net"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " Usage: autoPicSmth.sh [days]"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " days: download all pictures of recent /days/ days. 3 for default."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " Example: ./autoPicSmth.sh 3"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# check arguments
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ $# -gt 1 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ShowUsage
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
exit 1;
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
elif [ $# -eq 1 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
DAYS_TO_DOWN=$1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mkdir -p $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cd $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Get the thread search result HTML page to local
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
SearchUrl="http://bbs4.newsmth.net/bbsbfind.php?q=1&board=MyPhoto&dt=${DAYS_TO_DOWN}&ag=1"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "${SearchUrl}" -o ${QUERY_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all Thread URLs
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
egrep "<a href="bbscon.php?bid=" $QUERY_FILE | awk -F[<>"] '{print "http://bbs4.newsmth.net/"$9}' > $THREAD_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ThreadCount=`cat $THREAD_FILE | wc -l`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "Total ${ThreadCount} threads are found."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all BoardId and ThreadId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk -F[=&] '{print $2,$4}' $THREAD_FILE > $FORMAT_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to sotre all pictures infomation
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Format: BoardId ArticleId FileName FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "# BoardId ArticleId FileName FileSize FileId" > $PIC_INFO_FILE1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cat $FORMAT_FILE | while read BoardId ArticleId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ThreadUrl=`echo "http://bbs4.newsmth.net/bbscon.php?bid=$BoardId&id=$ArticleId"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "$ThreadUrl" -o $CURR_THREAD_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
grep "attach" $CURR_THREAD_FILE | tr ");" ") " | grep "attach" | awk -F[' ,)] -v BoardId=$BoardId -v ArticleId=$ArticleId '{print BoardId,ArticleId,$2,$5,$7}' >> $PIC_INFO_FILE1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all pictures info with file extention name
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# but not full file name.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Format: BoardId ArticleId FileExt FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# echo "# BoardId ArticleId FileExt FileSize FileId" > $PIC_INFO_FILE2
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk -F[. ] '$0~/^[^#]/ {print $1,$2,$4,$5,$6}' $PIC_INFO_FILE1 >> $PIC_INFO_FILE2
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove the records which don't contain enough info.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# in normal case, it should be 5 columns in the file.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk '$5~/^[^$]/ {print $0}' $PIC_INFO_FILE2 > $PIC_INFO_FILE3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all picture url
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
grep ^[^#] $PIC_INFO_FILE3 | while read BoardId ArticleId FileExt FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ $FileSize -gt 51200 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileType="p"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
else
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileType="s"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicUrl=`echo "http://att.newsmth.net/att.php?$FileType.$BoardId.$ArticleId.$FileId.$FileExt"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "$PicUrl" >> $PIC_URL_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove all duplicted URL from the file
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE} ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
sort -dfu ${PIC_URL_FILE}.tmp > ${PIC_URL_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove the URLs which have been downed before
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ -f "../${PIC_OUT_DIR}/${PIC_DOWN_LOG}" ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cp ../$PIC_OUT_DIR/${PIC_DOWN_LOG} .
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk '{print $3}' ${PIC_DOWN_LOG} > ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
sort -dfu ${PIC_URL_FILE}.history > ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE}.tmp ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
comm -1 -3 ${PIC_URL_FILE}.history ${PIC_URL_FILE} > ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE}.tmp ${PIC_URL_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Download all pictures from server to local
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicCount=`wc -l $PIC_URL_FILE | awk '{print $1}'`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicIndex=1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mkdir -p ../$PIC_OUT_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "Total number of pictures to be downloaded: $PicCount"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cat $PIC_URL_FILE | while read CurrUrl
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileName=`echo "$CurrUrl" | awk -F[?] '{print $2}'`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "[$PicIndex/$PicCount] Start to download $CurrUrl"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "$CurrUrl" -o ../$PIC_OUT_DIR/$FileName
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Write download log to log file
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
CurrTime=`date +"%Y-%m-%d %H:%M:%S"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "$CurrTime $CurrUrl" >> "../$PIC_OUT_DIR/$PIC_DOWN_LOG"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "[$PicIndex/$PicCount] Download finished."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo ""
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicIndex=`expr $PicIndex + 1`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#mv $PIC_URL_FILE ../$PIC_OUT_DIR/PicUrl.list
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#mv $PIC_INFO_FILE3 ../$PIC_OUT_DIR/PicInfo.list
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cd ..
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm -r $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "All Pictures Downloading finished."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
运行该脚本后,将自动从水木社区的MyPhoto版自动下载最近N(默认为3,通过参数1指定)天内的所有图片到本地的photo目录下。
用法:
1、把以下代码保存为autoPicSmth.sh
2、为脚本增加可执行权限,并运行脚本。
CHEYO:~/auto # chmod +x autoPicSmth.sh
CHEYO:~/auto # ./autoPicSmth.sh
脚本写得比较粗糙,欢迎优化改进。
源码:
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#!/bin/bash
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#####################################################################
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Script: autoPicSmth.sh
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Author: cheyo
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Email: icheyo at Gmail dot com
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# From: www.icheyo.net
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Date: 2008-02-22
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Description:
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# This script is used for downloading pictures from the MyPhoto
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# board in the newsmth.net automatically.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#####################################################################
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Usage: autoPicSmth.sh [days]
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# days: download all pictures of recent /days/ days
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# For Example: ./autoPicSmth.sh 3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
WORKING_DIR=working
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_OUT_DIR=photo
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
DAYS_TO_DOWN=3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
QUERY_FILE=QueryResult.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
THREAD_FILE=ThreadUrl.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FORMAT_FILE=ThreadInfo.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
CURR_THREAD_FILE=CurrThread.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_URL_FILE=PicUrl.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_DOWN_LOG=PicDown.log
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE1=PicInfo1.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE2=PicInfo2.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PIC_INFO_FILE3=PicInfoFinal.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ------------------------------------------------------------------ #
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ShowUsage()
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Show the usage of this script
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# ------------------------------------------------------------------ #
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ShowUsage()
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
{
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "This script is used for automatic downloading pictures from MyPhoto board in the newsmth.net"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " Usage: autoPicSmth.sh [days]"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " days: download all pictures of recent /days/ days. 3 for default."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo " Example: ./autoPicSmth.sh 3"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# check arguments
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ $# -gt 1 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ShowUsage
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
exit 1;
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
elif [ $# -eq 1 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
DAYS_TO_DOWN=$1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mkdir -p $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cd $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Get the thread search result HTML page to local
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
SearchUrl="http://bbs4.newsmth.net/bbsbfind.php?q=1&board=MyPhoto&dt=${DAYS_TO_DOWN}&ag=1"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "${SearchUrl}" -o ${QUERY_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all Thread URLs
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
egrep "<a href="bbscon.php?bid=" $QUERY_FILE | awk -F[<>"] '{print "http://bbs4.newsmth.net/"$9}' > $THREAD_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ThreadCount=`cat $THREAD_FILE | wc -l`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "Total ${ThreadCount} threads are found."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all BoardId and ThreadId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk -F[=&] '{print $2,$4}' $THREAD_FILE > $FORMAT_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to sotre all pictures infomation
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Format: BoardId ArticleId FileName FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "# BoardId ArticleId FileName FileSize FileId" > $PIC_INFO_FILE1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cat $FORMAT_FILE | while read BoardId ArticleId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
ThreadUrl=`echo "http://bbs4.newsmth.net/bbscon.php?bid=$BoardId&id=$ArticleId"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "$ThreadUrl" -o $CURR_THREAD_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
grep "attach" $CURR_THREAD_FILE | tr ");" ") " | grep "attach" | awk -F[' ,)] -v BoardId=$BoardId -v ArticleId=$ArticleId '{print BoardId,ArticleId,$2,$5,$7}' >> $PIC_INFO_FILE1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all pictures info with file extention name
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# but not full file name.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Format: BoardId ArticleId FileExt FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# echo "# BoardId ArticleId FileExt FileSize FileId" > $PIC_INFO_FILE2
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk -F[. ] '$0~/^[^#]/ {print $1,$2,$4,$5,$6}' $PIC_INFO_FILE1 >> $PIC_INFO_FILE2
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove the records which don't contain enough info.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# in normal case, it should be 5 columns in the file.
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk '$5~/^[^$]/ {print $0}' $PIC_INFO_FILE2 > $PIC_INFO_FILE3
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Create a file to store all picture url
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
grep ^[^#] $PIC_INFO_FILE3 | while read BoardId ArticleId FileExt FileSize FileId
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ $FileSize -gt 51200 ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileType="p"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
else
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileType="s"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicUrl=`echo "http://att.newsmth.net/att.php?$FileType.$BoardId.$ArticleId.$FileId.$FileExt"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "$PicUrl" >> $PIC_URL_FILE
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove all duplicted URL from the file
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE} ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
sort -dfu ${PIC_URL_FILE}.tmp > ${PIC_URL_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Remove the URLs which have been downed before
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
if [ -f "../${PIC_OUT_DIR}/${PIC_DOWN_LOG}" ]; then
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cp ../$PIC_OUT_DIR/${PIC_DOWN_LOG} .
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
awk '{print $3}' ${PIC_DOWN_LOG} > ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
sort -dfu ${PIC_URL_FILE}.history > ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE}.tmp ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
comm -1 -3 ${PIC_URL_FILE}.history ${PIC_URL_FILE} > ${PIC_URL_FILE}.tmp
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mv ${PIC_URL_FILE}.tmp ${PIC_URL_FILE}
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm ${PIC_URL_FILE}.history
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
fi
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Download all pictures from server to local
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicCount=`wc -l $PIC_URL_FILE | awk '{print $1}'`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicIndex=1
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
mkdir -p ../$PIC_OUT_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "Total number of pictures to be downloaded: $PicCount"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cat $PIC_URL_FILE | while read CurrUrl
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
do
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
FileName=`echo "$CurrUrl" | awk -F[?] '{print $2}'`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "[$PicIndex/$PicCount] Start to download $CurrUrl"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
curl "$CurrUrl" -o ../$PIC_OUT_DIR/$FileName
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
# Write download log to log file
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
CurrTime=`date +"%Y-%m-%d %H:%M:%S"`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "$CurrTime $CurrUrl" >> "../$PIC_OUT_DIR/$PIC_DOWN_LOG"
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "[$PicIndex/$PicCount] Download finished."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo ""
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
PicIndex=`expr $PicIndex + 1`
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
done
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#mv $PIC_URL_FILE ../$PIC_OUT_DIR/PicUrl.list
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
#mv $PIC_INFO_FILE3 ../$PIC_OUT_DIR/PicInfo.list
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
cd ..
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
rm -r $WORKING_DIR
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
echo "All Pictures Downloading finished."
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
![](http://images.csdn.net/syntaxhighlighting/OutliningIndicators/None.gif)
相关文章推荐
- Linux系统下查看文件调用情况Shell命令
- VipShell驱动隐藏模块
- U盘病毒及其相关资源的分析(patch shell32.dll)
- 从卡巴漏洞管窥内核模式Shellcode的编写
- 系统级ring3后门——byshell v0.64编程与应用
- to bashman
- SHELL编程基础 By jackie
- 编写启动程序和停止程序的shell脚本
- Linux shell 中参数变量&shell中的双引号
- 常用shell脚本命令总结(添加中。。。)
- shell的基本概念
- 用户账号管理基本概念
- Shell变量
- awk教程入门与实例练习(一)
- sed教程入门与实例练习(一)
- 用shell进行日期合法性校验
- 2008脚本大赛PowerShell初级组Event 10解题及分析
- 运行.vbs脚本时,无法找到名为“wscript.shell”的automation类
- bash命令处理的12个步骤[zt]
- tamarin系列之5] 植入本地方法实现