Code Snapshot: Connection Borrowing Logic

Here’s a key piece of HikariCP internals when a thread tries to borrow a connection from the pool:

1
2
3
4
5
6
7
8

// ②
// Get a connection from the pool, with a timeout
final PoolEntry poolEntry = connectionBag.borrow(timeout, MILLISECONDS);

// The borrow method returns null only if it times out
if (poolEntry == null) {
  break; // We timed out... break and throw exception
}

This code attempts to borrow a connection from the internal connectionBag. If it doesn’t succeed within the specified timeout, it returns null, and the calling code exits the loop and throws an exception.

Behind the Scenes: What’s connectionBag?

The connectionBag is a custom concurrent structure used by HikariCP to manage connections. Internally, it uses a CopyOnWriteArrayList to store available PoolEntry objects.

Why Use CopyOnWriteArrayList?

CopyOnWriteArrayList is a thread-safe variant of ArrayList where all mutative operations (like add, remove) are implemented by making a fresh copy of the underlying array. It shines in situations where:

• Reads are far more frequent than writes.
• Thread safety is critical, but locking overhead must be minimized.

This fits HikariCP’s use case perfectly—connections are borrowed and returned frequently under high concurrency, and most operations are reads (checking for available connections).

What Happens During borrow()?

The borrow() method performs the following steps:

1. Iterates over the CopyOnWriteArrayList of available connections.
2. Tries to atomically claim one via compareAndSet.
3. If no connection is immediately available, it waits until:
   • A connection is returned.
   • The timeout expires.

Thanks to CopyOnWriteArrayList, multiple threads can safely iterate and borrow connections without the risk of ConcurrentModificationException or complex locking strategies.

Timeout Behavior

If no connection is available within the timeout window:

1
2
3

if (poolEntry == null) {
  break; // We timed out... break and throw exception
}

The system recognizes that it’s better to fail fast than to block indefinitely. This ensures predictability and avoids resource starvation under load.

Trade-offs of CopyOnWriteArrayList

While CopyOnWriteArrayList is great for safe, lock-free reads, it does have drawbacks:

• Writes (adds/removes) are costly since the array is copied.
• It’s not ideal if the list is modified very frequently.

In HikariCP’s case, connection availability doesn’t change every millisecond—so this trade-off is acceptable and even advantageous.

Takeaways

• CopyOnWriteArrayList plays a crucial role in enabling fast, concurrent access to connection entries in HikariCP.
• It ensures safety and performance without heavyweight synchronization.
• The timeout logic provides a safety net to prevent system hangs under high load.

Final Thoughts

This case study shows how a seemingly simple collection choice—like CopyOnWriteArrayList—can dramatically influence the performance and reliability of a high-throughput system like HikariCP. It’s a perfect example of using the right tool for the job in a multithreaded environment.

HikariCP Case Study: Understanding FAUX_LOCK

HikariCP, a high-performance JDBC connection pool, is renowned for its minimalist design and efficient concurrency handling. One of its clever optimizations is the FAUX_LOCK, a no-op (no operation) implementation of the SuspendResumeLock class. In this short case study, we’ll explore the purpose of FAUX_LOCK, its implementation, and how it leverages JIT (Just-In-Time) compilation to boost performance.

What is FAUX_LOCK?

The SuspendResumeLock class in HikariCP manages the suspension and resumption of connection acquisition, typically during pool maintenance or shutdown. The FAUX_LOCK is a static instance of SuspendResumeLock that overrides its methods—acquire, release, suspend, and resume—to do nothing:

1
2
3
4
5
6
7
8
9
10

public static final SuspendResumeLock FAUX_LOCK = new SuspendResumeLock(false) {
    @Override
    public void acquire() {}
    @Override
    public void release() {}
    @Override
    public void suspend() {}
    @Override
    public void resume() {}
};

This “fake” lock acts as a placeholder when actual locking is unnecessary, minimizing overhead in high-performance scenarios.

Why Use FAUX_LOCK?

HikariCP is designed for speed, and every cycle matters in high-throughput applications. The FAUX_LOCK is used when the pool is configured to operate without suspension or locking, specifically when allowPoolSuspension is false (the default). Its key purposes are:

1. Single-Threaded or Non-Suspended Pools: When pool suspension is disabled, there’s no need for lock operations. FAUX_LOCK eliminates synchronization overhead.
2. Simplified Code Path: Using FAUX_LOCK avoids conditional logic to check whether locking is needed, maintaining a consistent SuspendResumeLock interface.
3. Performance Optimization: By providing empty method implementations, FAUX_LOCK reduces the cost of lock operations to zero.

JIT Optimization: The Hidden Benefit

So, what’s the real advantage of this approach? When pool suspension is disabled, FAUX_LOCK provides an empty implementation, with the expectation that the JVM’s Just-In-Time (JIT) compiler will optimize it away. Each call to acquire, release, suspend, or resume is an empty method that does nothing. After the code runs multiple times, the JIT compiler may recognize these methods as no-ops and inline or eliminate them entirely.

This means that, over time, the overhead of calling these methods disappears. When acquiring a connection, the application skips the token acquisition step entirely, as the JIT-optimized code bypasses the empty method calls. This results in significant performance savings, especially in high-concurrency scenarios where connection acquisition is frequent.

When is FAUX_LOCK Used?

FAUX_LOCK is employed when allowPoolSuspension is false. In this mode, HikariCP does not support suspending the pool for tasks like shrinking or reaping idle connections. By using FAUX_LOCK, calls to lock-related methods become no-ops, allowing HikariCP to focus solely on connection management. For example, in a web application with a fixed pool size and no need for suspension, FAUX_LOCK ensures minimal overhead.

Benefits of FAUX_LOCK

• Zero Overhead: Empty methods eliminate lock-related costs, and JIT optimization may remove them entirely.
• Code Simplicity: A consistent SuspendResumeLock interface avoids complex branching logic.
• Flexibility: Supports both high-performance (with FAUX_LOCK) and maintenance-friendly modes (with a real lock).
• Performance Boost: JIT-eliminated method calls reduce connection acquisition time.

Considerations

FAUX_LOCK is ideal for performance-critical applications but unsuitable when pool suspension is needed (e.g., for dynamic resizing). Enabling allowPoolSuspension requires a real SuspendResumeLock, and misconfiguration could disrupt pool maintenance.

Conclusion

The FAUX_LOCK in HikariCP is a brilliant optimization that showcases how small design choices can yield big performance gains. By providing a no-op lock and leveraging JIT compilation to eliminate method call overhead, FAUX_LOCK ensures HikariCP remains blazingly fast in non-suspended pools. For developers, this underscores the importance of aligning HikariCP’s configuration with application requirements to unlock its full potential.

When configuring your HikariCP pool, check if allowPoolSuspension is necessary. If not, FAUX_LOCK and JIT optimization will work behind the scenes to make your application faster and more efficient.

HikariCP Case Study: Understanding the getConnection Semaphore

One of its key mechanisms for managing connections efficiently is the use of a Semaphore in the getConnection method. In this case study, we’ll dive into how HikariCP leverages Semaphore to manage database connections, ensuring thread safety and optimal resource utilization.

Background on HikariCP

HikariCP is a JDBC connection pool designed for speed and simplicity. Unlike traditional connection pools that may rely on heavy synchronization or complex locking mechanisms, HikariCP uses modern concurrency utilities from Java’s java.util.concurrent package, such as ConcurrentBag and Semaphore, to achieve low-latency connection management.

The getConnection method is the primary entry point for applications to acquire a database connection from the pool. This method must balance speed, thread safety, and resource constraints, especially under high concurrency. The use of a Semaphore in this context is critical to controlling access to the finite number of connections.

The Role of Semaphore in getConnection

In HikariCP, a Semaphore is used to limit the number of threads that can simultaneously attempt to acquire a connection from the pool. A Semaphore is a concurrency primitive that maintains a set of permits. Threads must acquire a permit to proceed, and if no permits are available, they block until one is released.

Here’s how HikariCP employs a Semaphore in the getConnection process:

1. Connection Acquisition Limit: The Semaphore is initialized with a number of permits corresponding to the maximum pool size (maximumPoolSize). This ensures that no more than the configured number of connections are ever allocated.

2. Thread Safety: When a thread calls getConnection, it must first acquire a permit from the Semaphore. This prevents excessive threads from overwhelming the pool or attempting to create new connections beyond the pool’s capacity.

3. Timeout Handling: HikariCP’s getConnection method supports a timeout parameter (connectionTimeout). If a thread cannot acquire a permit within this timeout, the Semaphore’s tryAcquire method fails, and HikariCP throws a SQLException, informing the application that no connection is available.

4. Efficient Resource Management: Once a connection is acquired or created, the thread proceeds to use it. After the connection is returned to the pool (via close), the permit is released back to the Semaphore, allowing another thread to acquire a connection.

This approach ensures that HikariCP remains both thread-safe and efficient, avoiding the overhead of traditional locking mechanisms like synchronized blocks.

Case Study: High-Concurrency Scenario

Let’s consider a real-world scenario where a web application handles thousands of concurrent requests, each requiring a database connection. Without proper concurrency control, the application could exhaust the database’s connection limit, leading to errors or crashes. Here’s how HikariCP’s Semaphore-based getConnection handles this:

Setup

• HikariCP Configuration:
  • maximumPoolSize: 20
  • connectionTimeout: 30000ms (30 seconds)
  • minimumIdle: 5
• Application: A Java-based REST API using Spring Boot, handling 1000 concurrent requests.
• Database: PostgreSQL with a maximum of 100 connections.

Observations

1. Initial State: The pool starts with 5 idle connections (as per minimumIdle). The Semaphore has 20 permits available, corresponding to maximumPoolSize.

2. Spike in Requests: When 1000 requests hit the API simultaneously, each thread calls getConnection. The Semaphore ensures that only 20 threads can proceed at a time. Other threads wait for permits to become available.

3. Connection Reuse: As threads complete their database operations and return connections to the pool, permits are released. Waiting threads acquire these permits and reuse existing connections, preventing the need to create new ones unnecessarily.

4. Timeout Behavior: If the pool is fully utilized and no connections are available within 30 seconds, threads that cannot acquire a permit receive a SQLException. This allows the application to gracefully handle overload scenarios, perhaps by retrying or returning an error to the client.

Results

• Stability: The Semaphore prevented the pool from exceeding 20 connections, avoiding overwhelming the PostgreSQL server.
• Performance: Connection reuse and efficient concurrency control minimized latency, with most requests served within milliseconds.
• Error Handling: Threads that timed out received clear exceptions, allowing the application to implement fallback logic.

Code Example

Below is a simplified view of how HikariCP’s getConnection logic might look, focusing on the Semaphore usage:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

import java.sql.Connection;
import java.sql.SQLException;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;

public class HikariPool {
    private final Semaphore connectionSemaphore;
    private final int maxPoolSize;
    private final long connectionTimeout;

    public HikariPool(int maxPoolSize, long connectionTimeoutMs) {
        this.maxPoolSize = maxPoolSize;
        this.connectionTimeout = connectionTimeoutMs;
        this.connectionSemaphore = new Semaphore(maxPoolSize, true);
    }

    public Connection getConnection() throws SQLException {
        try {
            // Attempt to acquire a permit within the timeout
            if (!connectionSemaphore.tryAcquire(connectionTimeout, TimeUnit.MILLISECONDS)) {
                throw new SQLException("Connection timeout after " + connectionTimeout + "ms");
            }
            // Logic to acquire or create a connection from the pool
            return acquireConnection();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new SQLException("Interrupted while waiting for connection", e);
        } finally {
            // Release the permit back to the semaphore after returning the connection
            connectionSemaphore.release();
        }
    }

    private Connection acquireConnection() {
        // Placeholder for actual connection acquisition logic
        return null;
    }
}

This example illustrates the Semaphore’s role in controlling access to the connection pool. In the actual HikariCP implementation, additional optimizations like the ConcurrentBag for connection storage and housekeeping threads for pool maintenance further enhance performance.

Advantages of Using Semaphore

• Lightweight Concurrency: Compared to traditional locks, Semaphore provides a more flexible and lightweight mechanism for controlling access.
• Fairness: HikariCP’s Semaphore is configured to be fair, ensuring that threads are served in the order they request permits, reducing starvation.
• Timeout Support: The ability to specify a timeout for permit acquisition aligns with HikariCP’s focus on predictable behavior under load.
• Scalability: The Semaphore scales well under high concurrency, allowing HikariCP to handle thousands of requests efficiently.

Challenges and Considerations

While the Semaphore-based approach is highly effective, there are some considerations:

1. Configuration Tuning: The maximumPoolSize and connectionTimeout must be carefully tuned based on the application’s workload and the database’s capacity. Setting maximumPoolSize too high can overwhelm the database, while setting it too low can lead to timeouts.

2. Timeout Handling: Applications must be prepared to handle SQLExceptions caused by timeouts, possibly with retry logic or user-friendly error messages.

3. Monitoring: Under high load, monitoring the pool’s metrics (e.g., active connections, wait time) is crucial to detect bottlenecks or misconfigurations.

Conclusion

HikariCP’s use of a Semaphore in the getConnection method is a brilliant example of leveraging Java’s concurrency utilities to build a high-performance connection pool. By limiting concurrent access to connections, enforcing timeouts, and ensuring thread safety, the Semaphore enables HikariCP to deliver reliable and efficient database access in demanding environments.

For developers and architects, understanding this mechanism provides valuable insights into designing scalable systems. Properly configuring HikariCP and monitoring its behavior can make the difference between a sluggish application and one that performs flawlessly under pressure.

If you’re using HikariCP in your projects, take the time to review your pool configuration and consider how the Semaphore-based concurrency control impacts your application’s performance. With the right setup, HikariCP can be a game-changer for your database-driven applications.

HikariCP Source Code Analysis: HikariPool Initialization

HikariCP is a high-performance JDBC connection pool framework, and one of its core components is the HikariPool class. This article dives into the initialization process of HikariPool, focusing on the following line of code:

1

pool = fastPathPool = new HikariPool(this);

This line appears in the initialization flow of HikariDataSource or related configuration logic, serving as a critical step in creating the HikariCP connection pool. Below, we’ll analyze its meaning, context, and implementation details from the source code perspective.

1. Context: Background of HikariPool Creation

In HikariCP, HikariPool is the core class responsible for managing database connections, including their creation, recycling, borrowing, and destruction. When an application starts and configures a HikariDataSource, HikariCP initializes a HikariPool instance based on the provided configuration.

The line of code in question typically appears in the initialization logic of HikariDataSource, such as:

1
2
3
4
5

private void initializePool() {
    if (pool == null) {
        pool = fastPathPool = new HikariPool(this);
    }
}

Here, pool and fastPathPool are member variables of HikariDataSource, both pointing to the same HikariPool instance. Let’s break down what this code does.

2. Code Analysis: pool = fastPathPool = new HikariPool(this)

2.1 Key Components

• pool: A member variable in HikariDataSource that stores the HikariPool instance. It serves as the primary entry point for interacting with the connection pool.
• fastPathPool: Another member variable pointing to the same HikariPool instance. The name fastPathPool suggests a potential performance optimization (more on this- new HikariPool(this): Creates a new HikariPool instance, passing the current HikariDataSource (or its configuration object) as a parameter to the HikariPool constructor.
• this: Refers to the HikariDataSource or its related configuration object (e.g., HikariConfig), used to pass configuration details to the pool.

2.2 Why Two Variables?

Assigning the same HikariPool instance to both pool and fastPathPool may seem redundant, but it reflects a design choice for flexibility:

• pool: Acts as the primary reference to the connection pool, used in most scenarios.
• fastPathPool: Indicates a potential performance-optimized path (fast path). While fastPathPool currently points to the same object as pool, this design allows HikariCP to potentially switch to a more optimized pool implementation in specific scenarios without altering the external interface.

This approach provides HikariCP with the flexibility to evolve its internal implementation while maintaining compatibility.

3. HikariPool Constructor Analysis

To understand what new HikariPool(this) does, let’s examine the HikariPool constructor (simplified version):

1
2
3
4
5
6
7
8
9
10
11

public HikariPool(final HikariConfig config) {
    super(config);
    this.connectionTimeout = config.getConnectionTimeout();
    this.validationTimeout = config.getValidationTimeout();
    this.maxLifetime = config.getMaxLifetime();
    this.idleTimeout = config.getIdleTimeout();
    this.leakDetectionThreshold = config.getLeakDetectionThreshold();
    this.poolName = config.getPoolName();
    // Initialize other properties...
    initializeConnections();
}

3.1 Main Tasks of the Constructor

1. Inheritance and Configuration Setup:

   • HikariPool extends PoolBase, which handles foundational operations like creating and closing connections.
   • The constructor takes a HikariConfig object, extracts configuration parameters (e.g., maximum pool size, minimum idle connections, connection timeout), and assigns them to HikariPool member variables.

2. Connection Pool Initialization:

   • Calls initializeConnections() to create the initial set of database connections and populate the pool.
   • Starts background threads (e.g., HouseKeeper) to periodically check connection health, recycle idle connections, and perform other maintenance tasks.

3. Performance Optimization:

   • Uses efficient data structures like ConcurrentBag to manage connections, ensuring high concurrency and low-latency operations for borrowing and returning connections.

3.2 Role of the this Parameter

The this parameter (typically HikariDataSource or HikariConfig) provides the configuration details, such as:

• Database URL, username, and password
• Maximum pool size (maximumPoolSize)
• Minimum idle connections (minimumIdle)
• Connection timeout (connectionTimeout)
• Advanced settings (e.g., connection validation query, leak detection)

HikariPool uses these settings to determine how to initialize and manage connections.

4. Potential Role of fastPathPool

Although fastPathPool currently points to the same object as pool, its naming and design suggest performance optimization possibilities. Here are some speculations and insights:

• Fast Path Optimization: HikariCP might intend to use a specialized pool implementation in certain scenarios, potentially skipping checks (e.g., connection validation) for better performance.
• Dynamic Switching: The existence of fastPathPool allows HikariCP to dynamically switch to a more efficient pool implementation based on runtime conditions or configuration.
• Backward Compatibility: By maintaining both pool and fastPathPool, HikariCP can introduce new pool implementations without breaking existing code.

While fastPathPool’s full potential is not yet utilized, its design leaves room for future enhancements.

5. Conclusion

The line pool = fastPathPool = new HikariPool(this); is a pivotal part of HikariCP’s connection pool initialization. It creates a HikariPool instance and assigns it to both pool and fastPathPool, setting up the core component for managing database connections. The HikariPool constructor handles configuration parsing, pool initialization, and background maintenance tasks.

This code reflects HikariCP’s key strengths:

• High Performance: Efficient data structures and optimized logic ensure low latency and high throughput.
• Flexibility: The fastPathPool design allows for future performance enhancements.
• Simplicity: The initialization logic is clear and maintainable.

By analyzing this code, we gain insight into HikariCP’s connection pool creation process and appreciate its forward-thinking design. For those interested in diving deeper, exploring components like ConcurrentBag or HouseKeeper in the HikariCP source code can reveal even more about its robust implementation.

HikariCP case study 1 Thread Safety

HikariDataSource is a high-performance JDBC connection pooling library widely used in Java applications to manage database connections efficiently. This case study explores a critical aspect of HikariDataSource’s implementation: thread safety, focusing on how it ensures consistent behavior in high-concurrency environments.

Thread Safety in HikariDataSource

A key piece of code in HikariDataSource prevents the use of the connection pool after it has been closed:

1
2
3

if (isClosed()) {
    throw new SQLException("HikariDataSource " + this + " has been closed.");
}

This code checks whether the connection pool is closed. If isClosed() returns true, it throws an exception to prevent further operations. While this appears to be a simple check, it reveals important design considerations for thread safety.

The isClosed() Method

The isClosed() method is implemented as:

1

return isShutdown.get();

Here, isShutdown is a field defined as:

1

private final AtomicBoolean isShutdown = new AtomicBoolean();

The use of AtomicBoolean ensures that the isShutdown state is thread-safe, meaning its value remains consistent across multiple threads, even in high-concurrency scenarios. Java’s Atomic classes, such as AtomicBoolean, AtomicInteger, and AtomicLong, provide atomic operations that guarantee thread safety without explicit synchronization.

This design ensures that when the connection pool is closed, all threads can reliably detect this state, preventing race conditions or inconsistent behavior.

Why Thread Safety Matters

To understand why AtomicBoolean is necessary, we need to explore the root cause of thread safety issues.

Modern CPUs have multiple levels of caching: L1, L2, and L3 caches, which are exclusive to each CPU core, and main memory, which is shared across all cores. When a CPU core performs a computation, it loads data from main memory into its L1 cache for faster access. However, this caching mechanism can lead to inconsistencies across cores.

For example, if one thread updates the isShutdown value on one CPU core, that update may remain in the core’s L1 cache and not immediately propagate to other cores. As a result, other threads running on different cores might read an outdated value of isShutdown, leading to thread-unsafe behavior.

How AtomicBoolean Ensures Thread Safety

AtomicBoolean addresses this issue through the use of a volatile field:

1

private volatile int value;

The value field stores the boolean state (0 for false, 1 for true). The volatile keyword plays a crucial role in ensuring thread safety by enforcing the following:

1. Write Synchronization: When a thread modifies the value, the change is immediately written to main memory, bypassing the CPU cache.
2. Read Synchronization: When a thread reads the value, it always fetches the latest value from main memory, not from the CPU cache.

This ensures that all threads see a consistent value for isShutdown, regardless of which CPU core they are running on.

The Trade-Off of volatile

While volatile guarantees thread safety, it comes with a performance cost. Reading from and writing to main memory is significantly slower than accessing CPU caches. Therefore, using volatile introduces latency, which can impact performance in high-throughput systems.

This trade-off highlights an important lesson: volatile should only be used when thread safety is critical. In cases where a state variable is rarely updated or does not require real-time consistency, a non-volatile field might suffice to avoid the performance overhead.

Lessons from HikariCP’s Source Code

HikariCP’s use of AtomicBoolean demonstrates a careful consideration of thread safety in a high-performance system. However, this is just one example of the library’s low-level optimizations. Other aspects of HikariCP’s design include:

• Bytecode Size Control: HikariCP minimizes bytecode size to improve JVM optimization and reduce overhead.
• Concurrency Patterns: HikariCP employs advanced concurrency techniques, similar to those found in frameworks like Disruptor, which is known for its CPU cache-aware design and exceptional performance.

These optimizations show how understanding low-level details, such as CPU caching and memory synchronization, can lead to more efficient code. For developers, studying frameworks like HikariCP and Disruptor offers valuable insights into writing high-performance applications.

Takeaways

Reading HikariCP’s source code can feel like a deep dive into computer science fundamentals, from CPU caches to JVM optimizations. It serves as a reminder that the abstractions we use in high-level programming are built on intricate low-level mechanisms. As developers, investing time in understanding these details can help us write better, more efficient code.

Reflecting on this, I can’t help but think: All those naps I took in university lectures on operating systems and computer architecture? It’s time to pay them back by diving into the source code!

By learning from frameworks like HikariCP, we can bridge the gap between high-level programming and low-level optimizations, ultimately becoming better engineers.

In Java, there are two main types of problems that can occur during the execution of a program: Errors and Exceptions.

1. Errors

Errors are serious issues that occur beyond the control of the application. These are typically problems related to the Java Virtual Machine (JVM) itself, such as:

• StackOverflowError
• OutOfMemoryError
• VirtualMachineError

These errors are often unrecoverable and should not be handled in the code. When an error occurs, it’s best to let the system crash or shut down gracefully, as the environment may no longer be stable.

2. Exceptions

Exceptions are issues that arise during the normal operation of a program and can usually be anticipated and handled. For example:

• Trying to read a file that doesn’t exist.
• Invalid user input.
• Attempting to divide by zero.

Java provides a robust mechanism to handle exceptions using try-catch-finally blocks. Exceptions are further categorized into two types:

a. Checked Exceptions

These are exceptions that are checked at compile time. The compiler requires the developer to handle these exceptions explicitly, either by using a try-catch block or by declaring them in the method signature using the throws keyword.

Examples:

• IOException
• SQLException

b. Unchecked Exceptions

These are exceptions that are not checked at compile time. They usually indicate programming bugs, such as logic errors or improper use of an API. These exceptions inherit from RuntimeException.

Examples:

• NullPointerException
• ArrayIndexOutOfBoundsException
• IllegalArgumentException

Summary

Understanding the difference between errors and exceptions—and between checked and unchecked exceptions—helps in writing more robust and fault-tolerant Java applications.

Let me know if you’d like a more casual tone or if you want to turn this into a tutorial-style post!

Introduction

I am working with a login api, and therefore I have some notes about Session and Token (JWT - Json web token).

Session

The general practice of a login system should be to verify that the customer’s login information is correct. Then add a logged in attribute to the client’s session if it is correct. There are usually some tools that help us doing that. Generally the default name of the session(cookie) is “JSESSIONID”; Stored in the client’s cookie, so we don’t have to write any more complicated operations in the program.

Each time the Client Side send a request, we bring the session id along with it. Server side will take the session ID and find out the specific session from the many sessions stored in Server.
There it is, if there are 10000 user online, server need to store 10000 different session in the database. Which is a very high IO, also, there is also the problem of how to share sessions between multiple hosts.

To solve this problem, we normally use Redis.

JWT token

It is very popular to use JWT as a Token instead of session. jwt is a string encrypted by the server and issued to the client.
After receiving the token, the client sends a request with the token in case of need, so that the Server can decrypt and verify the identity.
Because the token itself stores the authentication information of the client. In general, the Server will no longer store the token after it is issued.
Note that, the token can actually be stored in a cookie.

JWT implementation

There are three part of a JWT, header, payload, signature

The whole thing will use base64 encode

Header

• alg: Cryptographic algorithms used
• typ: JWT

Payload

• iss: Issuer
• sub: subject, can be the key value such as account no.
• exp: expiration time

Signature

sign(hash(header+payload))

The signature also certifies that only the party holding the private key is the one that signed it.

Generating JWT

1
2
3
4
5
6
7
8
9

// JWT code here
Date expireDate = new Date(System.currentTimeMillis()+30*60*1000);
                    
String jwtToken = Jwts.builder().setSubject(email)
    .setExpiration(expireDate)
    .signWith(SignatureAlgorithm.HS512, "secret")
    .compact();
                    
return jwtToken;

Check the token

• notes 1 : Whenever the user wants to access a protected route or resource, the user agent should send the JWT, typically in the Authorization header using the Bearer schema. The content of the header should look like the following: Authorization: Bearer <token>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

public class AuthorizationCheckFilter extends OncePerRequestFilter{

    @Override
    `protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain) throws ServletException, IOException {
                if(!req.getServletPath().equals("/api/v1/user/login")){
                    
                    
                    String authorHeader =  req.getHeader(AUTHORIZATION);
                    String bearer ="Bearer "; // notes 1

                    if(authorHeader!= null && authorHeader.startsWith(bearer)){
                        try{
                        String token = authorHeader.substring(bearer.length());
                        Claims claims = Jwts.parser().setSigningKey("MySecret")
                        .parseClaimsJws(token).getBody();

                        System.out.println("JWT payload:"+claims.toString());

                        chain.doFilter(req, res);
                        
                        }catch(Exception e){
                            System.err.println("Error : "+e);
                            res.setStatus(FORBIDDEN.value());
                            
                            Map<String, String> err = new HashMap<>();
                            err.put("jwt_err", e.getMessage());
                            res.setContentType(APPLICATION_JSON_VALUE);
                            new ObjectMapper().writeValue(res.getOutputStream(), err);
                        }
                    }else{
                        res.setStatus(UNAUTHORIZED.value());
                    }
                }else{
                    chain.doFilter(req, res);
                }
        
    }

}

The jwt implementation of nestjs
https://github.com/etklam/nestjs-jwt-implementation

Cryptography

• Symmetric encryption (base on substitution and permutation): a secret key and an encryption algorithm
• Asymmetric encryption (base on exponential): public key and private key system.
• Public-key: provide confidentiality(encryption/decryption) and authentication (signature)
• Both of them are secure and useful in diff scenarios
• RSA algorithm is based on the difficulty of factoring problem.
• Hash function can be applied to any sized message and produce fixed length message digest.
• Sign the message digest instead of the message itself.
  • first hash the message
  • Then encrypt(sign) the hashed value
  • The message usually longer than the key size
• DSA signature is based on the difficulty of discrete logarithms problem.

AES

permutation (shift rows)

AES key will be expanded(x11)
16bytes -> 176bytes
The chiper consists of N rounds, N depends on the key length:

• 16bytes: 10rounds
• 24bytes: 12rounds
• 14bytes: 14rounds

Certification Authority (CA)

Sign certificate that bind subscriber’s name and his public key.
Indicates that the subscriber has sole control and access to the corresponding private key.

Public Key Infarstructure (PKI)

A set of policy, processes, server platforms, software etc…
to administer certificates

• issue
• maintain
• revoke

Bitcoin

• Merkle Tree Root is public for Verification
• Merkle Tree Root for txs, and store in the block header
• non-singular elliptic curve is the set of points and the point at infinity O
• The point at infinity O is the identity elements
• Bitcoin use Mudulo p - secp256k1, ECDSA

Bitcoin structure

Header:

• Version Number
• Hash of prev block header (by SHA256 double hash)
• Hash of Transactions(merkle tree)
• timestamp
• Threshold(difficulty)
• Nonce any value

Body:

• Number of TXs
• Coinbase
• Regulars etc

Hash function Requirment

• Easy to compute but diffcult to invert
• Collision resistant

New diffculty caluate

Bitcoin create once every 10 minutes
Update the diffculty every 2016 blocks
T-new = T-sum/(2016*10*60) * T

Simple Payment Verification Node (SPV)

only stores the block header, contact full nodes when information needed.

Mining

read my code

Probability that the block hash falls below the target threshold T:
p= T+1/2^256

Lock Time

>= 510^8, it is a Unix time
< 510^8, it is a block blockHeight

• Bitcoin prevents double spending(verifty every single node) and tampering(unless 51%)
• Mining difficulty adjucted to regulate coin supply
• Bitcoin address are shared over the internet

ETH

Account based model instead of UTXO. Main a global state to record the account balance

Accounts in ETH

Externally owned Accounts(EOA)

• Controlled by private key
• Has an Ether balance
• no code

Contract Accounts

• Has balance
• Has code (smart contract)
• has own permanent state

ETH Contact Transaction

1. Create new contract
2. Message the contract to execute it

Gas fee, Gas limit

Gas fee is the price per gas unit. Different operation cost different unit of gas.
Gas limit is the most you are willing to paid. The remainer will refund

ETH storage management - Radix Trie and Patricia Trie

—— Skip ——

• State Trie
• Transaction Trie
• Receipts Trie

ETH consensus - simpler GHOST

Uncle Block: floked block. Give reward to honest but unlucky minor.

intrinsic reward = 5

If include a uncle block, minor can get extra 1/32 intrinsic reward.
Uncle can get depands on block height
(Uncle + 8 - block that include uncle)* intrinsic reward/8

Incentive: reward unluck but honest miners. Make it more fair.

Solidity simplest form notes

• require The require function call defines conditions that reverts all changes if not met
• emit an event after successful money transfer

Token

can be programmed to provide different functions

Initial Coin Offering (ICO)

raise funds for a company to create a new coins. similar to IPO

Consensus

PoW

• the longest chain wins
• the one growing fastest will be the longest and most trustworthy
• take a lot of time to generate a block
• if too easy, the chain can be DDoS attack
• Huge Energy Consumption

PoS

• creator of a block chosen in a random way, depending on the user’s wealth
• In order to validate, forger must first put their own coin at “stake”.
• When folk, pos vote

Randomized Block Selection

randomizaion to generate the following forger. Not true random in computer world. So is usually able to predict which user will be selected to forge the next block.

Coin Age Based selection

coin age = time * amount

Target * CoinAge = the hash difficulty.

To join the PoS, you might load your coins to other or join the pool youself.

Nothing at stake

when folk, vote for both because it gains most benefits. Always win, nothing to lose.
The blockchain might never reach Consensus
Use casper: punish

DPoS Delegated Proof of stake

vote to elect witnesses
21-100 elected witnesses in a DPoS. time slots are given to each witness to publish their block.
Longest chain wins
Much faster than POW and POS

Byzantine Fault Torlerance

1. Commander -> all traitor
2. traitor boardcast -> other traitor
3. consensus

Consensus if at least 3m+1 nodes can achieve consensus. M is malicious nodes

Oral Message

1. All messages are delivered correctly
2. Know who this message is from
3. Missing messages can be detected

Permissioned Blockchain

• Regulation
• Complete control of their data
• Can be fully centralizated. Members negotiate.

Membership Service Providers (MSPs)

PKI and CA

Endorsement and validation policy can be adjusted as need.

1. Client -> Endorser (proposal)
2. Endorser check the certificate and others to validate the transactions
3. Executes the chaincode
4. Enderser -> Client
5. Client -> Orderer
6. Orderer include the transaction and generate blocks
7. Orderer -> Anchor
8. Anchor -> boardcast the block
9. Peer verify the new block
10. Peer -> Client

Channel

Maintan a Ledger, only nodes in channel can access this ledger.

Collection

The data on the chain can be said to be “permanent” and “public” to be shared among the participants. However, in the real application scenario, many data are not “publicly” stored in the blockchain due to privacy.
Only the header is stored on chain for verification.

Introduction

I am responsible for the transaction part in a group project about blockchain and bitcoin, and here is my code. Just for archive.

library

Bitcoin is using elliptic and secp256k1

1
2
3
4

import { SHA256 } from 'crypto-js';

var EC = require('elliptic').ec;
var ec = new EC('secp256k1');

1
2
3
4
5
6
7
8
9
10
11
12

export class Signature {
    static sign(priKey: string, msg: string): string {
        const key = ec.keyFromPrivate(priKey, 'hex');
        const signature = key.sign(msg).toDER();
        return signature;
    }

    static verify(pubKey: string, sig: string, msg: string): boolean {
        const key = ec.keyFromPublic(pubKey, 'hex');
        return key.verify(msg, sig);
    }
}

Class in a Transaction

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

export class RegularTxIn extends TxIn {
    txOutId: string;
    txOutIndex: number;
    signature: string;

    constructor(txOutId: string, txOutIndex: number, priKey) {
        super();
        this.txOutId = txOutId;
        this.txOutIndex = txOutIndex;
        this.signature = this.createSig(priKey, this.msgHash());
    }

    createSig(priKey: string, msg: string): string {
        return Signature.sign(priKey, msg);
    }

    msgHash(): string {
        return SHA256(SHA256(this.txOutId + this.txOutIndex)).toString();
    }
}

export class CoinbaseTxIn extends TxIn {
    public blockHeight: number;

    constructor(blockHeight: number) {
        super();
        this.blockHeight = blockHeight;
    }
}

export class RegularTxIn extends TxIn {
    txOutId: string;
    txOutIndex: number;
    signature: string;

    constructor(txOutId: string, txOutIndex: number, priKey) {
        super();
        this.txOutId = txOutId;
        this.txOutIndex = txOutIndex;
        this.signature = this.createSig(priKey, this.msgHash());
    }

    createSig(priKey: string, msg: string): string {
        return Signature.sign(priKey, msg);
    }

    msgHash(): string {
        return SHA256(SHA256(this.txOutId + this.txOutIndex)).toString();
    }
}

export class CoinbaseTxIn extends TxIn {
    public blockHeight: number;

    constructor(blockHeight: number) {
        super();
        this.blockHeight = blockHeight;
    }
}

export class TxOut {
    address: string; //public key
    amount: number;

    constructor(address: string, amount: number) {
        this.address = address;
        this.amount = amount;
    }
}

export class UTXO {
    txId: string;
    txOut: TxOut;
    txIndex: number;

    constructor(txId: string, txOut: TxOut, txIndex: number) {
        this.txId = txId;
        this.txOut = txOut;
        this.txIndex = txIndex;
    }
}

Transaction Class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

export class Transaction {
    type: string;
    id: string;
    txIns: TxIn[];
    txOuts: TxOut[];

    constructor(ins: TxIn[], outs: TxOut[]) {
        this.txIns = ins;
        this.txOuts = outs;
        this.id = this.setId();
    }

    setId(): string {
        let txInContent: string;
        if (this instanceof RegularTx) {
            txInContent = this.txIns
                .map((regularTxIn: RegularTxIn) => regularTxIn.txOutId + regularTxIn.txOutIndex)
                .reduce((a, b) => a + b, '');
        } else {
            txInContent = this.txIns
                .map((coinbaseTxIn: CoinbaseTxIn) => coinbaseTxIn.blockHeight)
                .reduce((a, b) => a + b, '');
        }

        const txOutContent: string = this.txOuts
            .map((txOuts: TxOut) => txOuts.address + txOuts.amount)
            .reduce((a, b) => a + b, '');

        return SHA256(SHA256(txInContent + txOutContent)).toString();
    }

    public static createRegularTx(
        senderPubKey: string,
        senderPriKey: string,
        receiverPubKey: string,
        receiveAmount: number,
        fee: number,
    ) {
        const utxo = this.findUTXO(senderPubKey);
        let sumUTXO = 0;
        const txIns = [];
        const txOuts = [];
        let i = 0;
        utxo.forEach((val) => {
            //the sum of UTXO of a pubkey
            sumUTXO += val.amount;
            // Create input object for each UTXO, sign the input by user private key
            i++;
            txIns.push(new RegularTxIn(val.id, i, senderPriKey));
        });
        const totalAmountToSpend = receiveAmount + fee;
        if (sumUTXO < totalAmountToSpend) {
            // Not enough money
            return; //exception
        }
        for (let n = 0; n < txIns.length; n++) {
            // verify the input by signature
            const checker = Signature.verify(utxo[i].address, txIns[i].signature, txIns[i].msgHash());
            if (!checker) {
                return; //exception
            }
        }
        //Create out put to receiver by PP2K
        txOuts.push(new TxOut(receiverPubKey, receiveAmount));
        //return change to the sender
        const change = sumUTXO - receiveAmount - fee;
        if (change > 0) {
            txOuts.push(new TxOut(senderPubKey, change));
        }
        const tx = new Transaction(txIns, txOuts);
        // tx.setId()
        return tx;
    }

    public static findUTXO(senderPubKey) {
        const allBlock = [];
        const allTxOut = [];
        const allTxIn = [];
        allBlock.forEach((block) => {
            const txs = block.txs;
            txs.forEach((tx) => {
                const txOuts = tx.txOuts;
                txOuts.forEach((out) => {
                    if (out.address == senderPubKey) {
                        allTxOut.push(out);
                    }
                });
            });
        });
        return [];
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

export class RegularTx extends Transaction {
    type: 'regular';
    txIns: RegularTxIn[];

    constructor(ins: RegularTxIn[], outs: TxOut[]) {
        super(ins, outs);
    }
}

export class CoinBaseTx extends Transaction {
    type: 'coinbase';
    txIns: CoinbaseTxIn[];

    constructor(ins: CoinbaseTxIn, outs: TxOut[]) {
        super([ins], outs);
    }
}

Create Transaction and UTXO

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

public async createTx(
    senderPubKey: string,
    senderPriKey: string,
    receiverPubKey: string,
    receiveAmount: number,
    fee: number,
) {
    const utxos = await this.getUXTO(senderPubKey);
    let sumUTXO = 0;

    const txIns = utxos.map((tx) => {
        sumUTXO += tx.txOut.amount;
        return new RegularTxIn(tx.txId, tx.txIndex, senderPriKey);
    });

    const txOuts = [];

    const totalAmountToSpend = receiveAmount + fee;

    if (sumUTXO < totalAmountToSpend) {
        // Not enough money
        throw new Error('Insufficient Balance'); //exception
    }

    for (let i = 0; i < txIns.length; i++) {
        // verify the input by signature
        // TODO: check if utxos[i].txOut.address is correct or not
        const checker = Signature.verify(utxos[i].txOut.address, txIns[i].signature, txIns[i].msgHash());

        if (!checker) {
            throw new Error('Invalid txIns'); //exception
        }
    }

    //Create out put to receiver by PP2K
    txOuts.push(new TxOut(receiverPubKey, receiveAmount));
    //return change to the sender
    const change = sumUTXO - receiveAmount - fee;
    txOuts.push(new TxOut(senderPubKey, change));

    const tx = new RegularTx(txIns, txOuts);
    tx.setId();

    this.broadcast(tx);
    await this.transactionPoolService.addTransaction(tx);

    return tx;
}

public async getAllUTXO(): Promise<UTXO[]> {
    const outs: UTXO[] = [];
    // loop all the blocks
    for (let i = 0; i < this.blockService.getBlockHeight(); i++) {
        const currentBlockHash = this.blockService.getBlockHash(i);
        const currentBlock = await this.blockService.getBlock(currentBlockHash);
        const currentTx = currentBlock.data.transactions; // need to convert to list of Transaction

        // loop all the transaction
        currentTx.forEach((currentTx) => {
            const txID = currentTx.id;
            const txOuts = currentTx.txOuts;
            const txIns = currentTx.txIns;

            // Create UTXO object for each TxOutPut, push to UTXO
            txOuts.forEach((txOut, i) => {
                outs.push(new UTXO(txID, txOut, i)); // create UTXO obj that store tx, txid and index
            });

            // Remove the spent money
            txIns.forEach((tx) => {
                if (tx instanceof RegularTxIn) {
                    const outID = tx.txOutId;
                    const outIndex = tx.txOutIndex;
                    const indexOfTx = outs.findIndex((obj) => {
                        return obj.txId == outID && obj.txIndex == outIndex;
                    });

                    if (indexOfTx >= 0) {
                        outs.splice(indexOfTx, 1);
                    }
                }
            });
        });
    }
    return outs;
}

public async getUXTO(pubKey: string): Promise<UTXO[]> {
    const allUtxo = await this.getAllUTXO();

    return allUtxo.filter((tx) => {
        return tx.txOut.address === pubKey;
    });
}

Operation Control

Segregation of Duties

Avoid single person could be responsible for diverse and critical functions. Otherwise, error or misappropriation could occur and not be detected in a timely manner and in normal course of business processes.

Incident handling

identify when where whole

Shadow IT: IT users at an organisation electing to use tools and services that have not been officially sanctioned by said organisation.

• Converage - insurance?
• Action - what to do?
• Evidence
• Tasks to do during recovery

Management of removable media and system documentation

Monitoring

• audit logging
• Clock Synchronize

Logical Controls

Concurrent Sign-on Session

can be very useful, but also a control weaknesses

Suggestion:

• No or only few user can have concurrent
• No more than two
• Logged and reviewed

Remote access Control

• Deducated leased liveness
• VPN

1. Identification process (username?)
2. Authentication process (password?)
3. Permitted/denied

Input Control

source document design - arrange fields for ease of use.

Software development Control

• Business realization: 個system點幫到公司
• project management
  • Cost and resource/ Deliverable/Time(Duration)
• System development approach SDLC approaches
  • SDLC: 流水線