onom: 8月 2016

2016年8月29日月曜日

sort out call data with "unique" function.

when call data is as below

number status
1 0975357001 FDISC
2 0975357001 FDISC
3 0975357001 FDISC
4 08037573628 FDISC
5 0458583634 FDISC
6 0454315424 FDISC
<skipped>
695 0999255908 ABAN
696 0266735287 ANS
697 0995252908 ABAN
698 08035447511 FDISC
699 08043657563 FDISC
700 09027935501 FDISC

> for(i in unique(calldata$number)){cat(i);cat(" ");
cat(calldata$status[calldata$number==i]);
cat("\n")}

will return the sequence of returncode with the unique call number.

0727229518 1
32072 6 6
09086909233 2
09077857629 1
08035447613 5
09025935528 5

> for(i in unique(calldata$number)){cat(i);print(calldata$status[calldata$number==i])}

will return status code in the original data instead of numeric ones.

09086909233[1] ANS
Levels: ABAN ANS CONN FBUSY FDISC OTHER
09077857629[1] ABAN
Levels: ABAN ANS CONN FBUSY FDISC OTHER
08035447613[1] FDISC
Levels: ABAN ANS CONN FBUSY FDISC OTHER
09025935528[1] FDISC
Levels: ABAN ANS CONN FBUSY FDISC OTHER

2016年8月19日金曜日

revisiting perl

CAUTION!

Perl list seq number starts from ZERO, then some_list[size_of_list] returns nothing. when access to the last element, use some_list[size_of_list -1].

this is not R.

The input file format is as below.

[[2016-01-01,111],[2016-01-02,222], <skip>,[2016-09-03,999]]

Contents comes as a single line.
Remove 2 brackets at the beginning and the end of the line.
Disassemble the line at "],[", thus each entry contains the pair of data and its date.
Dissasseble each entries at ",". Do this from the first entry to the last.
Save the date of the first entry.
Concatenate retrieved data with ",". Thus construct the sequence which start with the data of the first date till the last date.
Use the date of the first entry and the last entry to create the necessary date sequence such as "seq(as.Date(\"start_date\"),as.Date(\"end_date\"),1)"

#! /usr/bin/perl

# $file="./testdata";
system("wget http://10.251.66.58/kljstatistics/php/dashboard/maindashboard/dailynew2016.php");
$file="./dailynew2016.php";
$outfile="> ./dataget.r";

open(IN,$file) or die "$!";
open(OUT,$outfile) or die "$!";
print("# start\n");
while(<IN>){
# data is expected to come in a sigle line.
# remove 2 brackets at the start and the end of the input.
$buff = substr($_,2,length($_));
$buff = substr($buff,0,length($buff)-2);
# print $buff; # for debug only
# split data at the sequence of "],["
@data = split(/\],\[/, $buff);
}
print "# file read end. the size of data is ";
# get the length of the list.
$size = @data;
print "# ";
print $size;
print "\n\n\n";

$count = 0;
# continue process from the start till the end of the list
# prepare start of the output statement.
$output = "w <- c(); w <- append(w,xts(c(";
while($count < $size){
# break up the element by comma.
@xts = split(/,/,$data[$count]);
# for the first element of the list.
if($count == 0){
# save start date for the later process.
$startdate = $xts[0];
}
# concatenate incident number with comma.
$output =$output.$xts[1].",";
$count++;
}
# remove comma at the end of $output
$output = substr($output,0,length($output)-1);
# add date sequence with start date and end date. end date comes from the last data element.
$output = $output."),seq(as.Date(\"$startdate\"),as.Date(\"$xts[0]\"),1)))";
print OUT $output;
print "\n\n\n";

$len = scalar(@data);
print "# $len th entry is $data[$len-1]\n";
# remove input file. don't forget this otherwise wget stores data in a different filename.
system("rm dailynew2016.php");
print("# from $startdate til $xts[0].\n");
print "# the end of the process.\n";

close(IN);
close(OUT);

2016年8月12日金曜日

Exhaust options

Before initiating complex processes, always calculate the theoretical maximum number of options, no matter of how much impractical ones. Un-matured minds often confuse between what is possible to exist and practical to implement. Knowing the theoretical limit will help to remove unnecessary bias and to reach at optimal solutions.

2016年8月10日水曜日

Find common ground and pursue common interest

Always look for common ground and interest with other people. The differences are easier to spot and more likely to cause problems and to be a blocking factor. From fear for them, being ashamed or given penalty, people's attention is taken away and stacked into the maze of the struggle over differences.

Common ground and/or interest are the key driver to find your alliance and sometimes finding them is the most critical and essential factor of success. The things you can achieve are very limited if you are alone.

2016年8月9日火曜日

New forecast of S&P 500

You are able to get the latest S&P 500 estimate by the model. The parameter "last_GDP" was calculated from FRED's last update GDP data of 2016-04-01 and GDPnow.

>last_GDP <- 16732.53
>last_UNDCONTSA <- 1015
>last_PAYEMS <- 144448

>3.358e+05-2.632e+00*last_PAYEMS-3.034e+02*last_UNDCONTSA-2.205e+01*last_GDP+2.405e-03*last_PAYEMS*last_UNDCONTSA+1.737e-04*last_GDP*last_PAYEMS+1.923e-02*last_UNDCONTSA*last_GDP-1.531e-07*last_GDP*last_PAYEMS*last_UNDCONTSA
[1] 2150.232

2016年8月7日日曜日

New GDP/S&P500 model without boolean switch

>start_date <- "2001-04-01"
>end_date <- "2016-04-01"

>summary(lm(to.quarterly(SP5[paste(start_date,end_date,sep='::')])[,4] ~
to.quarterly(PAYEMS[paste(start_date,end_date,sep='::')])[,4] *
to.quarterly(UNDCONTSA[paste(start_date,end_date,sep='::')])[,4] *
GDPC96[paste(start_date,end_date,sep='::')]))

Call:
lm(formula = to.quarterly(SP5[paste(start_date, end_date, sep = "::")])[,
4] ~ to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[,
4] * to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[,
4] * GDPC96[paste(start_date, end_date, sep = "::")])

Residuals:
Min 1Q Median 3Q Max
-193.065 -65.262 9.311 77.450 178.759

Coefficients:
Estimate
(Intercept) 3.358e+05
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4] -2.632e+00
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] -3.034e+02
GDPC96[paste(start_date, end_date, sep = "::")] -2.205e+01
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 2.405e-03
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 1.737e-04
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 1.923e-02
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] -1.531e-07
Std. Error
(Intercept) 6.689e+04
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4] 5.037e-01
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 6.614e+01
GDPC96[paste(start_date, end_date, sep = "::")] 4.428e+00
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 4.988e-04
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 3.329e-05
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 4.386e-03
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 3.301e-08
t value
(Intercept) 5.020
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4] -5.226
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] -4.588
GDPC96[paste(start_date, end_date, sep = "::")] -4.980
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 4.821
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 5.218
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 4.384
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] -4.639
Pr(>|t|)
(Intercept) 6.18e-06
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4] 2.98e-06
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 2.78e-05
GDPC96[paste(start_date, end_date, sep = "::")] 7.13e-06
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4] 1.24e-05
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 3.07e-06
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 5.54e-05
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")] 2.34e-05

(Intercept) ***
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4] \
***
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]
***
GDPC96[paste(start_date, end_date, sep = "::")]
***
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]
***
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")]
***
to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")]
***
to.quarterly(PAYEMS[paste(start_date, end_date, sep = "::")])[, 4]:to.quarterly(UNDCONTSA[paste(start_date, end_date, sep = "::")])[, 4]:GDPC96[paste(start_date, end_date, sep = "::")]
***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 95.52 on 53 degrees of freedom
Multiple R-squared: 0.9374, Adjusted R-squared: 0.9291
F-statistic: 113.4 on 7 and 53 DF, p-value: < 2.2e-16