Distributed and Cloud Computing Notes 1

Reasons for Distributed Systems

• Functional Separation
  • Different Capabilities and purposes
• Inherent Distribution
  • Information
  • People
• Power imbalance and load variation
• Reliability
• Economies

Consequences of Distributed Systems

• Concurrency - Each computer is autonomous
  • Carry our tasks independently
  • Tasks coordinate their actions by exchanging messages
  • System capacity can be increased by adding more resources
• No global clock
• Independent Failures

Motivation of Distributed Systems

• To share resource and information

Trends in Distributed Systems

• The emergence of pervasive networking technology
• The emergence of mobile and ubiquitous computing
• The increasing demand for multimedia services
• The view of distributed systems as a utility

Maintenance of intranet

• No rick if no connection to internet
• Firewalls are used to limit services from/to an intranet
  • Limit FTP/Remote Desktop etc.

Mobile computing: Performing computing tasks while the user is on the move, away from his/her usual environment

Eight forms of transparency

• Access transparency
• Location transparency
• Concurrency transparency
• Replication transparency
• Failure transparency
• Mobility transparency
• Performance transparency
• Scaling transparency

List of Challenge

• Heterogeneity
• Security
  • Confidentiality
    • Protection against disclosure to unauthorized individual information
  • Integrity
    • Protection against alteration or corruption
  • Availability
    • Protection against interference targeting access to the resources
    • DDoS
  • Authenticity or Non-repudiation
    • Proof of sending / receiving an information
    • digital signature

Failure

Availability =MTTF/(MTTF+MTTR)

• Mean time to failure(MTTF)
  • The average time of normal operation before the system fails
• Mean time to repair (MTTR)
  • The average time it takes to repair the system and restore it to working condition

Single point failure

Single hardware/Software component failures cause the whole system crash. The key approach to enhancing availability is to make as many as possible partial failures by removing single points of failure

Checkpointing

• The process of periodically saving the stage of an executing program to stable storage, from which the system can recover after a failure.
  • Each program stae saved is called a Checkpoint .
  • Checkpointing can be realized by operating system at kernel level/Third party library/by the application itself.

Jobs

• Serial Jobs: Run on a single node
• Parallel jobs: use multiple nodes
• Interactive jobs: require fast turnaround time, and their input/output is directed to a termainal
• Batch jobs: need more resources and don’t need immediate responses. Scheduled jobs.

job Management System

• A user server: Let user submit jobs.
• A job scheduler: performs job scheduling
• A resource manager: allocates and monitors resources. Enforces scheduling policies, and collects accounting information.

Security Mechanisms

• Encryption(AES, RSA)
• Authentication(Password, Public key)
• Authorization(access control)

• Concurrency
  • Fair scheduling
  • Preserve dependencies
  • Avoid deadlocks
  • Object locking, data consistency, semaphores
• Fault tolerance (No failure despite faults)
  • Fault detection
    • Checksums
    • Heartbeat
  • Fault masking
    • Retransmission of corrupted messages
    • Redundancy
  • Fault toleration
    • Exception handling
    • Timeouts
  • Fault recovery
    • Rollback mechanisms
• Scalability
• Openness
• Distribution transparency <= Do not let other touch

Information Security Notes 1 - Classical Crypto System

What is Encryption

Encryption is composed of a key and an Encryption algorithm.

1. Type of operations used for transforming plaintext to ciphertext
2. The number of keys used
3. The way in which the plaintext is processed

Encryption and Decryption

• Unencrypted message = plaintext/message
• Encrypted message = cipher/ciphertext

Cryptanalysis

Means attack

• Brute-force
  • Tries every possible
• Breaking the algorithm
  • Tries to exploit the weakness of the encryption algorithm

How to measured by the following dimensions

• Attacker models
  • How strong is the attacker
  1. Ciphertext only attacks
  2. Known plaintext attacks
  3. Chosen plaintext attacks(Attacker can choose plaintext on his own)
  4. Chosen ciphertext attacks(Attacker can choose the cipher and obtain the plaintext)
• Security Goal
  • What Goals does your attacker wants to achieve
  1. Computationally secure: The cost of breaking the cipher exceeds the value of the encryptited information
  2. Unconditionally secure: No matter how much time as opponent has, it is impossible for people decrypt. (secure against brute-force)
• Assumptions:
  • What is the computational limitation
  • Always better to over-estimate the ability of your attackers
  1. Computation: attacker might have many computing resource(super-computers)
  2. Network: attacker might have control over the network/communication channel, they can send/drop/inject/view your packet
  3. Some problems are hard(NP=/=P), no polynomial time solutions
  4. We generally assume computation requiring 2^80 is unsolvable

Brute-force attack

• Attackers try all possible sets of keys
• By probability, it has to try at least half of them
• We generally assume computation requiring 2^80 is unsolvable

Comments

Pros

1. Cheap, Super Cheap
2. guaranteed performance
3. Decent Provider
4. World Wide Locations

Cons

1. Buggy panel
2. Delay delivery
3. M247 network

Information of the Deal

1
2
3
4
5

2 CPU core (50% dedicated, burstable up to 200%)
8 GB RAM 
40 GB RAID-10 NVMe
5 TB bandwidth (1 TB in Hong Kong and Sydney)
$30 per year

Bench

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

---------------------------------
Processor  : Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
CPU cores  : 2 @ 2999.998 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ❌ Disabled
RAM        : 7.8 GiB
Swap       : 4.0 GiB
Disk       : 39.1 GiB

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 193.83 MB/s  (48.4k) | 2.60 GB/s    (40.7k)
Write      | 194.34 MB/s  (48.5k) | 2.62 GB/s    (40.9k)
Total      | 388.18 MB/s  (97.0k) | 5.22 GB/s    (81.7k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 4.26 GB/s     (8.3k) | 4.92 GB/s     (4.8k)
Write      | 4.49 GB/s     (8.7k) | 5.25 GB/s     (5.1k)
Total      | 8.75 GB/s    (17.1k) | 10.18 GB/s    (9.9k)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed     
                |                           |                 |                
Clouvider       | London, UK (10G)          | busy            | busy           
Online.net      | Paris, FR (10G)           | 431 Mbits/sec   | 217 Mbits/sec  
WorldStream     | The Netherlands (10G)     | busy            | 281 Mbits/sec  
Biznet          | Jakarta, Indonesia (1G)   | 118 Mbits/sec   | 440 Mbits/sec  
Clouvider       | NYC, NY, US (10G)         | 451 Mbits/sec   | 365 Mbits/sec  
Velocity Online | Tallahassee, FL, US (10G) | 59.1 Mbits/sec  | 397 Mbits/sec  
Clouvider       | Los Angeles, CA, US (10G) | busy            | busy           
Iveloz Telecom  | Sao Paulo, BR (2G)        | busy            | busy           

Geekbench 5 Benchmark Test:
---------------------------------
Test            | Value                         
                |                               
Single Core     | 606                           
Multi Core      | 1149                          
Full Test       | https://browser.geekbench.com/v5/cpu/5694857

Network

Looking Glass
Seems everything M247, just like other location of HH. With $30, what ever.

Tips for using HostHatch’s broken Panel

Their panel is very buggy, it seems that centos7 and ubuntu are bad, but when I reinstalled to debian10 then all functions are normal
So here’s what I did

1. reinstall to Debian
2. Mount the netboot.xyz
3. reinstall the os you want from ISO

This is only my personal solution, not guaranteed to work

Ubuntu

1. Download the ubuntu-mainline-kernel.sh Bash script utility. Open up your terminal and enter:

wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh

2. install the previously downloaded script into our executable path:

sudo install ubuntu-mainline-kernel.sh /usr/local/bin/

3. All is ready to upgrade the Ubuntu kernel to the latest version. To do so run do following command:

ubuntu-mainline-kernel.sh -i

The above command will search for the latest kernel version available. To continue with the kernel installation answer y.

4. Finally, reboot your system

reboot

Debian

1. Update the system
   apt update && apt full-upgrade -y
apt install lsb-release

2. Add the repo to the sources list

1
2
3
4
5
6
7
8

cat > /etc/apt/sources.list << EOF
deb http://cdn-aws.deb.debian.org/debian $(lsb_release -sc) main contrib non-free
deb http://cdn-aws.deb.debian.org/debian-security $(lsb_release -sc)/updates main contrib non-free
deb http://cdn-aws.deb.debian.org/debian $(lsb_release -sc)-updates main contrib non-free
deb http://cdn-aws.deb.debian.org/debian $(lsb_release -sc)-backports main contrib non-free
deb http://cdn-aws.deb.debian.org/debian $(lsb_release -sc)-proposed-updates main contrib non-free
# deb http://cdn-aws.deb.debian.org/debian $(lsb_release -sc)-backports-sloppy main contrib non-free
EOF

3. Do the update again
   apt update

4. upgrade your kernel
   apt install -t $(lsb_release -sc)-backports linux-image-$(dpkg --print-architecture) linux-headers-$(dpkg --print-architecture) --install-recommends -y
update-grub
reboot

Algorithm - Dynamic Programming

What is Dynamic Programming?

Principle of Optimality (Bellman, 1957):
An optimal sequence of decisions has the property that whatever the initial state and decision are, the remaining decisions must constitute an optimal decision with regard to the state resulting from the first decision.

The Other Algorithmic Design Philosophy

1. Divide-and-Conquer:
   The problem is divided and the subproblems are processed in a recursive manner, but the solutions of Divide-and-Conquer subproblems are usually not repeated, and when they are repeated, the same subproblems are usually recalculated.

2. Greedy Approach:
   At each stage, starting from a certain starting point, each input is checked one by one to see if it is suitable to be added to the answer. If it is not possible to find a selection procedure to check one by one for the optimization problem to be handled, we will discuss it later.

Divide and Conquer and Dynamic Programming are very similar. The difference is that Dynamic Programming’s subproblems have many overlaps, which can be stored in a table without recalculation, exchanging space for time.

Internetional Relationship - Tools and Target

Tools

1. Political Diplomacy
2. Economy and Trade
3. Military
4. Cultural Promotion
5. Spy

Target

1. Attitude
2. Weakening Enemy
3. Seize benefits
4. Eliminate the opponent

Sails.js is a MVC Framework for Node.js

Create a new sails project

sails new {your-app-name}

Create a new Controller(apis)

sails generate api {generator-name}

Master Theorem for Exam

VirMach, A ColoCrossing reseller. Good? Not bad. Cheap? Yes!

SmallWooden 6.2USD/Annually

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #
#              Yet-Another-Bench-Script              #
#                     v2020-12-07                    #
# https://github.com/masonr/yet-another-bench-script #
# ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## #

Fri 18 Dec 2020 09:37:31 AM EST

Basic System Information:
---------------------------------
Processor  : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
CPU cores  : 1 @ 2499.998 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ❌ Disabled
RAM        : 987.5 MiB
Swap       : 975.0 MiB
Disk       : 18.6 GiB

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 42.54 MB/s   (10.6k) | 682.43 MB/s  (10.6k)
Write      | 42.61 MB/s   (10.6k) | 686.02 MB/s  (10.7k)
Total      | 85.16 MB/s   (21.2k) | 1.36 GB/s    (21.3k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 957.74 MB/s   (1.8k) | 1.04 GB/s     (1.0k)
Write      | 1.00 GB/s     (1.9k) | 1.11 GB/s     (1.0k)
Total      | 1.96 GB/s     (3.8k) | 2.15 GB/s     (2.1k)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed     
                |                           |                 |                
Clouvider       | London, UK (10G)          | 614 Mbits/sec   | 162 Mbits/sec  
Online.net      | Paris, FR (10G)           | 670 Mbits/sec   | 22.8 Mbits/sec 
WorldStream     | The Netherlands (10G)     | 440 Mbits/sec   | 123 Mbits/sec  
Biznet          | Jakarta, Indonesia (1G)   | 130 Mbits/sec   | 43.7 Mbits/sec 
Clouvider       | NYC, NY, US (10G)         | 809 Mbits/sec   | 686 Mbits/sec  
Velocity Online | Tallahassee, FL, US (10G) | 709 Mbits/sec   | 247 Mbits/sec  
Clouvider       | Los Angeles, CA, US (10G) | 551 Mbits/sec   | 198 Mbits/sec  
Iveloz Telecom  | Sao Paulo, BR (2G)        | 47.5 Mbits/sec  | 100 Mbits/sec  

Geekbench 5 Benchmark Test:
---------------------------------
Test            | Value                         
                |                               
Single Core     | 336                           
Multi Core      | 345                           
Full Test       | https://browser.geekbench.com/v5/cpu/5399053

DryKey 4.5USD/Annually

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

---------------------------------
Processor  : QEMU Virtual CPU version (cpu64-rhel6)
CPU cores  : 1 @ 2499.998 MHz
AES-NI     : ❌ Disabled
VM-x/AMD-V : ❌ Disabled
RAM        : 731.6 MiB
Swap       : 256.0 MiB
Disk       : 14.5 GiB

fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 41.09 MB/s   (10.2k) | 519.01 MB/s   (8.1k)
Write      | 41.18 MB/s   (10.2k) | 521.75 MB/s   (8.1k)
Total      | 82.28 MB/s   (20.5k) | 1.04 GB/s    (16.2k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 845.56 MB/s   (1.6k) | 694.81 MB/s    (678)
Write      | 890.49 MB/s   (1.7k) | 741.09 MB/s    (723)
Total      | 1.73 GB/s     (3.3k) | 1.43 GB/s     (1.4k)

iperf3 Network Speed Tests (IPv4):
---------------------------------
Provider        | Location (Link)           | Send Speed      | Recv Speed     
                |                           |                 |                
Clouvider       | London, UK (10G)          | 2.13 Gbits/sec  | 945 Mbits/sec  
Online.net      | Paris, FR (10G)           | 2.41 Gbits/sec  | 663 Mbits/sec  
WorldStream     | The Netherlands (10G)     | 2.01 Gbits/sec  | 541 Mbits/sec  
Biznet          | Jakarta, Indonesia (1G)   | 550 Mbits/sec   | 22.1 Mbits/sec 
Clouvider       | NYC, NY, US (10G)         | 5.86 Gbits/sec  | 1.07 Gbits/sec 
Velocity Online | Tallahassee, FL, US (10G) | 2.83 Gbits/sec  | 1.35 Gbits/sec 
Clouvider       | Los Angeles, CA, US (10G) | 2.64 Gbits/sec  | 874 Mbits/sec  
Iveloz Telecom  | Sao Paulo, BR (2G)        | 877 Mbits/sec   | 586 Mbits/sec  

Geekbench releases can only be downloaded over IPv4. FTP the Geekbench files and run manually.

VirMach use Track - How to enable AES-NI without opening a ticket

You may already find that two box above, one is AES-NI enabled but another one not.

When you reinstall your OS in VirMach’s SolusVM:

You can see some operating system is end with a gen2
If you install these gen2 template, your box will likely to enable the AES-NI. Even you reinstall to other OS later, the AES are still enabled.

Personlly, I install Debian 8 x 86_64 Minimal v1 (gen 2)

Since I recently learned about Nodejs, and I will probably focus on nodejs in the foreseeable future, I migrated my blog from Wordpress to Hexo.
Here is a little note about how to install a hexo blog.

Install VSCode

This part you need to do it yourself.

Install scoop

Scoop is a package manager on Windows. Highly suggested.

Type the following into PowerShell to ensure that local scripts are allowed to execute.

set-executionpolicy remotesigned -scope currentuser

Then install Scoop by executing the following command.

iex (new-object net.webclient).downloadstring('https://get.scoop.sh')

Just wait for the script to finish, and once it’s installed successfully, let’s try the following

scoop help

Finally install nodejs with scoop.

scoop install nodejs

Install hexo and git

Use npm to install hexo on your laptop:

npm install -g hexo-cli

Open the folder you want to store your blog data, then create a blog by using the following command:

hexo init [folder] // [folder] = your blog’s path

Open the project folder with VSCode:

npm install

Finally,

hexo s

Open your browser and localhost:4000 to see your website

Create a github repo

Follow the command provided by github:

Push your blog to Github

Every git-push process need the following three commands:

1
2
3

git add .
git commit -m "Your comment"
git push

You can open terminal on VSCode by clicking Ctrl+`

Host on Netlify for free

Well, I am lazy… and Netlify already did the job

To be continues…