Upgrading R from 340 to 352

necessary to install packages below manually.

install.packages("xts")
install.packages("quantmod")
install.packages("vars")
install.packages("mondate")
install.packages("RMySQL")
install.packages("forecast")
install.packages("beepr")

TIPS 20180805

警告の抑止(未検証)　

options(warn=-1)
library(・・・)
# 読み込んだよーというメッセージがでなくなる

警告の抑止(検証済)  その2

        sink(file="/tmp/r.log")　# console output is redirected to "/tmp/r.log"
     
        sink()  # reset to the console.

警告の抑止(未検証)　その３

        options("getSymbols.yahoo.warning"=FALSE).

ヒストグラムの出力

  # kikan like kikan or "2018::06-21::2018-07-13"
  # len is a number of samples.
  # loc_x is x-axis position kikan 0 to 1.
  # loc_y is y-axis.
  # br is breaks for hist()
  # ymax is max value of y-axis.
  # xmin and xmax are for x-axis
  # color is from 1 to 9?
  hist(as.vector(last(par_xts[kikan][,highlow],n=len)),breaks=br,xlim=c(xmin,xmax),ylim=c(0,ymax),col=color)
  axis(side=2, pos=round(mean(par_xts[kikan][,highlow])),labels=F)
 #
 # par + text to output overlay messages on the graph.
 # graph's dimension data to graph_dim
 # "graph_dim[1] + graph_dim[2] *  loc_x " is 50% of horizontal loc
 #
  graph_dim <- par('usr')
  text( graph_dim[1] + graph_dim[2] *  loc_x   ,(graph_dim[4] - graph_dim[3]) * (loc_y+0) + graph_dim[3] ,paste("#",len,sep="="),adj=c(0,0))

垂直線をxtsグラフに引く

events <- xts(c("natrix","weight"),as.Date(c("2018-06-20", "2018-07-14")))
addEventLines(events, srt=90, pos=2,col=10)

   未検証だが以下もOKなはず。

addEventLines(as.xts(c("natrix","weight"),as.Date(c("2018-06-20", "2018-07-14"))), srt=90, pos=2,col=10)

多面プロットのやり方(plot when multi.panel=T and yaxis.same=F)

plot(merge(to.monthly(N225["2007-01::"])[,4],as.vector(residuals(result)/to.monthly(N225)[,4])),multi.panel=TRUE,yaxis.same=FALSE)

季節調整　case shiller 10 city composite.

CS.stl <- stl(ts(as.numeric(CS),frequency=12), s.window="periodic")
last(merge(CS,as.numeric(CS.stl$time.series[,2]),suffixes = c("","season")),n=24)
plot(last(merge(CS,as.numeric(CS.stl$time.series[,2])),n=48)[,1] - last(merge(CS,as.numeric(CS.stl$time.series[,2])),n=48)[,2])

chartSeries(
  weekly_pf,
  show_grid=TRUE,
  type="candlesticks"
)

ffmpegの使い方　再生できないファイルがある場合

再生できないファイルがある場合、mp4に変換してやるとうまくvlcで再生できることが多い。以下はcodecを変更しないでコンテナだけをmp4に変えてやる使い方。以下はaviファイルでオーディオストリームに問題があるらしい場合のエラーメッセージ。

[avi @ 0x7ffaf7800000] Could not find codec parameters for stream 1 (Audio: mp3 (mp3float) (U[0][0][0] / 0x0055), 48000 Hz, 2 channels, fltp, 128 kb/s): unspecified frame size


で、オーディオストリームはそのままにしてコンテナだけをmp4に変更する。

# no conversion just switch container. highly reliable.

ffmpeg -i <INPUT>.avi -codec:v copy -codec:a copy <OUTPUT>.mp4

OR

ffmpeg -i <INPUT>.avi -codec:a copy out.mp4

こちらは ビデオストリームをコピーする例。やはり、コンテナはmp4に変更する。

ffmpeg -i <INPUT>.mkv -vcodec copy <OUTPUT>.mp4

OR

ffmpeg -i sample_input.mkv -vcodec copy sample_output.mp4

for文とカウンタ in bash

j=0;for i in  *.mkv; do echo $i;  let ++j ; echo $j ; ffmpeg -y -i $i -vcodec copy $j.mp4; done

• ;を書く文の最後に
• let をつかってカウンタを加算する
• ffmpeg -yで既存ファイルがあっても強制上書き
• forはdo とdoneで受ける
• 変数は初期化すること

以下と等価となる。

j=0;for i in  *.mkv; 

> do echo $i

> let ++j 

> echo $j 

> ffmpeg -y -i $i -vcodec copy $j.mp4

> done

j=0;for i in *.mp4; do let ++j; mv $j.mp4 hibike_euphonuim_S2E$j.mp4; done

• -e 大事。これがないと\1 が使えない。(間違い。なくてもOK)
• さいごの「２」で出現順序を指定して置換する。
• \(<patter>\)で指定して\1で受けてる。

Zero Padding by sed

ただし、awkのgensub関数を使ったほうが良いと思う

Hibike Euphonium 1 [BD 720p]$ ls *.mp4 | awk '{ print "mv "$1" "$1}' | gsed -e 's/E\([1-9]\)\./E0\1./2'
mv hibike_euphonuim_S1E1.mp4 hibike_euphonuim_S1E01.mp4
mv hibike_euphonuim_S1E10.mp4 hibike_euphonuim_S1E10.mp4
mv hibike_euphonuim_S1E11.mp4 hibike_euphonuim_S1E11.mp4

awk のgensub関数を使えば、sedは使わなくて良いはず。第二引数の"g"はグローバル？例えば2を指定すると2番めにヒットした正規表現だけを処理してくれるらしい。(要研究)

ls *.mp4 | awk '{ print "ls "$1" "gensub("0([1-9])","1\\1","g",$1)}' 

gensub関数 in awk

gensub(regexp, replacement, how [, target])

gensubは汎用的な置換関数である。subやgsubのように 対象文字列targetから正規表現regexpに マッチする部分を検索する。subやgsubと違うのは、 関数の戻り値として置換が行われた文字列を返し、 元の文字列を変更しないという点である。

gensub を使うとこうなる。

K-On!$ ls | awk '{print "ffmpeg -i \""$0"\" K-OnS1E" gensub(/(\y.\y)/,"0\\1",1,$3)".mp4"}'

ffmpeg -i "K-ON! Ep 01 - Dissolution!.mkv" K-OnS1E01.mp4
ffmpeg -i "K-ON! Ep 02 - Instruments!.mkv" K-OnS1E02.mp4
ffmpeg -i "K-ON! Ep 03 - Special Lessons!.mkv" K-OnS1E03.mp4
ffmpeg -i "K-ON! Ep 04 - Training Camp!.mkv" K-OnS1E04.mp4
ffmpeg -i "K-ON! Ep 05 - Advisor!.mkv" K-OnS1E05.mp4
ffmpeg -i "K-ON! Ep 06 - School Festival!.mkv" K-OnS1E06.mp4
ffmpeg -i "K-ON! Ep 09 - New Club Member!.mkv" K-OnS1E09.mp4
ffmpeg -i "K-ON! Ep 10 - Another Training Camp!.mkv" K-OnS1E10.mp4
ffmpeg -i "K-ON! Ep 11 - Crisis!.mkv" K-OnS1E11.mp4
ffmpeg -i "K-ON! Ep 12 (Season Finale) - Light Music!.mkv" K-OnS1E12.mp4
ffmpeg -i "K-ON! Ep 13 (Extra) - Winter Days!.mkv" K-OnS1E13.mp4
ffmpeg -i "K-ON! Ep 14 (OVA) - Live House!.mkv" K-OnS1E14.mp4
ffmpeg -i "K-ON! Ep 7 - Christmas!.mkv" K-OnS1E07.mp4
ffmpeg -i "K-ON! Ep 8 - Freshman Reception!.mkv" K-OnS1E08.mp4

gawk　正規表現

\w

これは単語を構成する任意のキャラクタ、つまり 文字、数字、それとアンダースコアにマッチする演算子である。 これは [[:alnum:]_] の簡潔な表現とみなして良い。

\W

これは単語を構成する要素にならない任意のキャラクタにマッチする 演算子である。これは [^[:alnum:]_] の簡潔な表現とみなして良い。

\<

これは単語の先頭にある空文字列にマッチする演算子である。 例えば、/\<away/は`away'にマッチするが、 `stowaway'にはマッチしない。

\>

これは単語の末尾にある空文字列にマッチする演算子である。 例えば、/stow\>/は`stow'にマッチするが、 `stowaway'にはマッチしない。

\y

これは単語の先頭、あるいは末尾の空文字列とマッチする演算子である (つまり語の区切りとマッチするということである)。 例えば、`\yballs?\y'は独立した単語として `ball' にも `balls'にもマッチする。注意！　他のGNU softwareは単語の区切りに\bを使うらしい。

\B

この演算子は単語中の空文字列にマッチする。言い換えると、`\B'は二つ の単語の構成要素文字の間にある空文字列にマッチするということである。例え ば、/\Brat\B/ は`crate'にマッチする。しかし、`dirty rat' にはマッチしない。`B'は簡単にいうと`\y'の反対語である。

\( .... \)　() でマッチの塊を扱う

\n 　\n は n 番目の () に対応 ([a-z]*) \1*

Calculate your mortgage.

CHECK THIS AS WELL

For the case interest rate is 0.875%, months to pay is 26 full years, and the remaining principle is 50,814,556JPY.

See aDFyear for each year and aDFmonth for each month.

> mortgage(P=50814556, I=0.875, L=(26*12), amort=T, plotData=T)

The payments for this loan are:


Monthly payment: ￥182,154.3 (stored in monthPay)
Total cost: ￥56,832,131

The amortization data for each of the 312 months are stored in "aDFmonth".

The amortization data for each of the 312 years are stored in "aDFyear".

> aDFyear
   Amortization Annual_Payment Annual_Principal Annual_Interest Year
1      50814556        2185851          1748224       437627.33    1
2      49066332        2185851          1763582       422268.88    2
3      47302750        2185851          1779076       406775.49    3
4      45523674        2185851          1794705       391146.00    4

                                     <skip>

25      4332107        2185851          2156580        29270.81   25
26      2175526        2185851          2175526        10324.87   26

Upgrade packages in R

When unable to upgrade packages as they are already loaded, check "package" panel and uncheck loaded ones.

Unchecking the mark is equal to the command "detach("package:RMySQL", unload=TRUE)".
Do same for "DBI".
*Note: Skip "restart R session before upgrade" as it will bring back unloaded packages.

R_proj$cat ~/.Rprofile 

source( "~/R_proj/startup.R" )

R_proj$cat ~/R_proj/startup.R

library(xts)

library(quantmod)

library(vars)

library(mondate)

# library(RMySQL) #junbi

library(forecast)

library(beepr)

Don't forget to comment and uncomment packages in initial start files as required.

.emacs

Don't forget good old days.

(global-set-key "\C-h" 'delete-backward-char)
(setenv "PATH" "/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin")
(setq exec-path (parse-colon-path (getenv "PATH")))
; (add-to-list 'load-path "~/.emacs.d")                                       
(global-visual-line-mode)
(autoload 'wikipedia-mode "wikipedia-mode.el"
   "Major mode for editing documents in Wikipedia markup." t)
(require 'epg)
(setq epa-armor t)

; 緯度経度は小数点一桁までで指定@東京駅
(setq calendar-latitude 35.7)  ;
(setq calendar-longitude 139.8)  ;
(setq calendar-location-name "Tokyo,Japan")

Calculate a number of the iteration, which records decline for each month from 1950-01 to now.

for(i in seq(1,12,1)){print(length(((SP5[,4]-SP5[,1])/SP5[,1])[seq(i,length(SP5[,4]),12)][((SP5[,4]-SP5[,1])/SP5[,1])[seq(i,length(SP5[,4]),12)] < 0]))}

> for(i in seq(1,12,1)){print(length(((SP5[,4]-SP5[,1])/SP5[,1])[seq(i,length(SP5[,4]),12)][((SP5[,4]-SP5[,1])/SP5[,1])[seq(i,length(SP5[,4]),12)] < 0]))}
[1] 28
[1] 30
[1] 25
[1] 21
[1] 28
[1] 33
[1] 30
[1] 31
[1] 38
[1] 26
[1] 23
[1] 17

January records 28 times decline since 1950. September is the worst to do 38 times, which is more than 50% probability, while December is the best month to invest S&P 500.

Automatic save before quit

.Last <- function() {

  # save.image(file=paste(getwd(),Sys.time(),sep="/"))
  save.image(file=gsub(" ","",paste(getwd(),Sys.time(),sep="/")))

  cat("bye bye...\n")

}

quit("yes")

Backup files are created under the current working directory with the time stamp.



save.image(file=paste(getwd(),format(Sys.time(), "%b%d%X"),sep="/"))

or

save.image(file=gsub(" ","",paste(getwd(),Sys.time(),sep="/"))

Will eliminate unnecessary space in the file names.

> gsub(" ","",paste(getwd(),Sys.time(),sep="/")
+ )
[1] "/Users/honomoto/2017-12-1210:08:23"

Below will improve the readability.

> gsub(" ","",paste(getwd(),gsub(" ","-",Sys.time()),sep="/"))
[1] "/Users/honomoto/R_proj/tmp1/2018-08-14-15:00:34"


> format(Sys.time(), "%b%d%X")
[1] "121210時03分53秒"
> Sys.setlocale("LC_ALL",'C')
[1] "C/C/C/C/C/ja_JP.UTF-8"
> format(Sys.time(), "%b%d%X")
[1] "Dec1210:04:28"

花粉集計 by awk

get 2017 data from http://nagakura-ac.com/2017-kafun-info/index.shtml

when inputs are as below

2月1日 0 0 0 晴 13.8 2.0 明けましておめでとうございます。今年も花粉情報をお伝えします。
2月2日 10 0 0 晴 13.3 3.8
2月3日 20.3 0 0 晴 13.7 3.5 温かい穏やかなお正月を迎え、最高気温が高い影響を受け、スギ花粉が0.3個測定されており。１月３日が今年のスギ花粉初観測日となりました。これは例年より早めで、昨年は１月１９日、一昨年は11日でした。
               <SKIP>
3月4日 30 0 0 晴 14 3.6 花粉症症状を自覚している方も出始めています。
3月5日 20.3 0 0 晴 10.4 3.7 スギ花粉が0.3個測定されています。
3月6日 0 0 0 晴 8.8 1.5

1. Calculate the total of each February, March, and April. 
2. Print results at the end.

awk '/^2/{feb=feb+$2;print $1" "$2}/^3/{mar=mar+$2;print $1" "$2}/^4/{apr=apr+$2;print $1" "$2}END{print "######\n feb="feb"  \n mar="mar"  \n apr="apr}' < kafun

Normalize Historical EPS data from S&P.

S&P500 eps data comes from here http://us.spindices.com/documents/additional-material/sp-500-eps-est.xlsx?force_download=true

When inputs are as below.


09/30/2016,$28.69,$25.39
06/30/2016,$25.70,$23.28
  <SKIP>
06/30/1988,$6.05,$6.22
03/31/1988,$5.48,$5.53

awk to remove "$" mark in 2nd and 3rd columns is

~$ awk  -F, '{gsub("\\$","",$2);gsub("\\$","",$3);print $1","$2","$3}' < inputfile 

results are,

09/30/2016,28.69,25.39
06/30/2016,25.70,23.28
      <SKIP>
06/30/1988,6.05,6.22
03/31/1988,5.48,5.53

don't forget to use "\\"  not "\". it is because "\" itself requires own escape sequence. for the case above, "^." might work in the same way.

furthermore.

Below will remove the 1st field, print header in the first line and reverse order for latter usages.

awk  -F, '{gsub("\\$","",$2);gsub("\\$","",$3);print $2","$3}' < inputfile | tail -r | awk 'BEGIN{print "ope,rep"}{print $0}'

ope,rep
5.48,5.53
6.05,6.22
6.22,6.38
6.37,5.62
6.41,6.74

Manipulate columns

When data day_xts has multiple columns and you like to delete or pick up one of them, writing a simple condition clause is enough to do that job.
However, in the case to deal with more than one column, it doesn't work and you need other solution.


> head(day_xts)
           new_call new_mail abandon inc
2013-01-01       35       28    12.9  63
2013-01-02       55       21    31.1  76
2013-01-03       86       20    76.7 106


> head(day_xts[,colnames(day_xts) != "inc"])
           new_call new_mail abandon
2013-01-01       35       28    12.9
2013-01-02       55       21    31.1
2013-01-03       86       20    76.7


Using "grep" with regular expression picks up columns either "inc" or "abandon".

> head(day_xts[,grep('inc|abandon',colnames(day_xts))])
           abandon inc
2013-01-01    12.9  63
2013-01-02    31.1  76
2013-01-03    76.7 106


"invert" argument does reverse the result. The columns which are NOT designated in "grep" will be chosen.

> head(day_xts[,grep('inc|abandon',colnames(day_xts),invert=T)])
           new_call new_mail
2013-01-01       35       28
2013-01-02       55       21
2013-01-03       86       20

Edit lines by AWK - line number, field number, gsub() and sub()

0. skip line number 9.
1. omit data after "-" in 2nd field.
2. delete first character in 1st field.
3. replace all 0x80 with "@"
3. print line number, the second last and last field.


cat t  | nkf | awk  '{if(NR != 9){sub("-.*","",$2); sub("^.","",$1);gsub("\x80","="); print NR," x ",$(NF-1)," x ",$NF;}}'

# gsub("[\x80\xe3]","<any character sequence>") 

calculate total file size.

Sum up file size for specific type of files.

 $ find ./<directory_name>  -print | grep -E  "mp4$|wmv$|avi$|jpg$|png$|mp3$|mkv" | awk '{print "ls  -l \""$0"\""}' > <tmporary_file_name>

The content of size_t_file is as below

ls -l "./<directory_name>/<subdirectory_name>/001.jpg"
ls -l "./<directory_name>/<subdirectory_name>/002.jpg"

Then

$ sh <tmporary_file_name> | awk '{total = total + $5}END{print "total is "total}'

This will count all designated type's file size and calculate the summation.

$find ./<directory_name>  -print | grep -vE  "mp4$|wmv$|avi$|jpg$|png$|mp3$|mkv$|JPG$|MP4$|AVI$"

The command above will omit all output which ends with motion and still picture file type. Use option "-v" for this purpose.

sort out call data with "unique" function.

when call data is as below

       number status
1  0975357001  FDISC
2  0975357001  FDISC
3  0975357001  FDISC
4 08037573628  FDISC
5  0458583634  FDISC
6  0454315424  FDISC
     <skipped>
695  0999255908   ABAN
696  0266735287    ANS
697  0995252908   ABAN
698 08035447511  FDISC
699 08043657563  FDISC
700 09027935501  FDISC

> for(i in unique(calldata$number)){cat(i);cat(" ");
cat(calldata$status[calldata$number==i]);
cat("\n")}

will return the sequence of returncode with the unique call number.

0727229518 1
32072 6 6
09086909233 2
09077857629 1
08035447613 5
09025935528 5

> for(i in unique(calldata$number)){cat(i);print(calldata$status[calldata$number==i])}

will return status code in the original data instead of numeric ones.

09086909233[1] ANS
Levels: ABAN ANS CONN FBUSY FDISC OTHER
09077857629[1] ABAN
Levels: ABAN ANS CONN FBUSY FDISC OTHER
08035447613[1] FDISC
Levels: ABAN ANS CONN FBUSY FDISC OTHER
09025935528[1] FDISC
Levels: ABAN ANS CONN FBUSY FDISC OTHER

Use list as associative array part 2

This one is better than the previous one.

> test2_list <- list(list("Mon","Tue","Wed","Thu","Fri","Sat","Sun"),list(1,2,3,4,5,6,7))
> for(i in test2_list[[2]]){
+   # if(match(weekdays(as.Date(ISOdate(year(Sys.Date()),1,1)),abbreviate = TRUE),test_list[[1]][i],nomatch = FALSE))
+   print(paste(">>",test2_list[[1]][i]))
+ }
[1] ">> Mon"
[1] ">> Tue"
[1] ">> Wed"
[1] ">> Thu"
[1] ">> Fri"
[1] ">> Sat"
[1] ">> Sun"
> 

Use list as associative array.

plase also see part2.

utilize a list as associative array.

1)create a list which each element contains a pair of information.

> test_list <- list(list("Mon",1),list("Tue",2),list("Wed",3),list("Thu",4),list("Fri",5),list("Sat",6),list("Sun",7))

2)when weekday returns day of the first day of the year, numeric seq number which is associated with weekday's return will be printed.

> for(i in 1:length(test_list)){
   if(match(weekdays(as.Date(ISOdate(year(Sys.Date()),1,1)),abbreviate = TRUE),test_list[[i]][1],nomatch = FALSE))
     {print(test_list[[i]][2])}
 }
[[1]]
[1] 5

Calculate day dependent data with less workload - 2 - test 2 strings are equal or not

Use match function to test 2 strings are same or not.

Below is a normal case and easy.

> weekdays(as.Date("2016-01-01"),abbreviate = TRUE)
[1] "Fri"
> match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Fri")
[1] 1

In other cases, match return NA not 0 by default and this causes a problem in if function. I see this is very strange, however, it is just one more option to go.

> match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Mon")
[1] NA

The parameter nomatch will give an option to set a value for the case. with "nomatch=0" you can booleanize the return value

> match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Fri", nomatch=0)
[1] 1
> match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Thur", nomatch=0)
[1] 0

Then now it's possible to use if-then-else clause without problem.

> if(match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Mon",nomatch=0)){print("金曜日")}else{print("other days...")}
[1] "other days..."
> if(match(weekdays(as.Date("2016-01-01"),abbreviate = TRUE),"Fri",nomatch=0)){print("金曜日")}else{print("other days...")}
[1] "金曜日"