DKProbes

Dependency Injection

2024-07-13T00:00:00Z

Dependency Injection (DI) is a design pattern that allows an object to receive its dependencies from an external source rather than creating them itself. This pattern promotes loose coupling and makes your code more modular and testable, and less error prone.

I had an opportunity to refactor DI implemention at my workplace.

What is Dependency Injection?#

Dependency Injection is a technique where the dependencies (objects) of a class are provided (injected) by Spring, typically through constructors, setters, or interfaces. DI helps in separating the creation of dependencies from the business logic, thereby adhering to the principle of Inversion of Control (IoC).

Types of Dependency Injection#

Constructor Injection: Dependencies are provided through a class constructor.
Setter Injection: Dependencies are provided through setter methods.
Field Injection: Dependencies are directly injected into the class fields using annotations.

Problem statement#

Our first implementation relied on a tightly coupled instantiation of services into a component

public class Subsidiary {
    String name;
    Integer partyId;
    List<String> ratings;
    public void updateSubsidiary(List<String> ratings) {
        this.ratings = ratings;
        // update ratings in db
    }
}


public class UpdateSub {
    private SubsidiaryService subsidiaryService;
    public UpdateSub() {
        this.subsidiaryService = new SubsidiaryService();
    }

    public void processUpdates(List<String> ratings) {
        subsidiaryService.updateSubsidiary(ratings);
    }
}

// Main.java
public class Main {
    public static void main(String[] args) {
        UpdateSub UpdateSub = new UpdateSub();
        UpdateSub.processUpdates(["AA+", "BB-", "CCC"]);
    }
}

As visible here, we are creating an object of SubsidiaryService inside UpdateSub and instantiating it.

Challenges with the above approach?#

Tight coupling : Since we create and instantiate the object of SubService manually, it is coupled to the business logic of the UpdateSub itself. Should SubService be made into an interface, we'd need to update the logic inside UpdateSub to instantiate the impl of the interface.
Challenge during testing : When writing junits, we don't need to actually create database connections, rather, just mock them. However, in this case, when we call UpdateSub, it'd end up updating the database connection and it won't be possible to mock it

Stage 1 of solution : Field injection#

To cater to the above limitations, we decided to implement dependency injection. But we decided to go with a type called field injection.

As the name suggests, we inject dependencies as a field of the class.

In code, it looked something like this

public class UpdateSub {
    @Autowired
    private SubsidiaryService subsidiaryService;
    public UpdateSub() {
    }

    public void processUpdates(List<String> ratings) {
        subsidiaryService.updateSubsidiary(ratings);
    }
}

Just by writing the @Autowired annotation, we were able to inject the SubService dependency. Now, our junits could mock SubService using @InjectMocks or @Mock from Mockito.

Stage 2 : Limitations of @Autowired#

There are a couple of limitations with this approach

You cannot make the injected service immutable

Since @Autowired will inject the service after the instantiation of UpdateSub, setting it as final will throw a compile time error. This is a challenge when we want to make sure our injections aren't overridden

Chances of NPE

Again, owing to the above reason that injection happens after the root class instantiation, we found null pointer exceptions because we were trying to access the method of a service that spring hadn't yet been able to instantiate

The partial accuracy of inject mocks

In the junits for UpdateSub, if we want to mock UpdateSub, we'd need to mock SubService and pass it along to UpdateSub. We achieved this by @Mocking SubService and @InjectMocking the mock into UpdateSub, but that didn't feel like the right approach

Solution - Constructor injection#

We therefore decided to move to Constructor injection

As the name suggests, we inject services into the constructor, rather than as a field.

Here is how it looked like in code :


public class UpdateSub {
    
    SubsidiaryService subsidiaryService;

    @Autowired
    public UpdateSub(Subsidiaryservice subsidiaryService) {
        System.out.println("Update Sub");
    }

    public void processUpdates(List<String> ratings) {
        subsidiaryService.updateSubsidiary(ratings);
    }
}

Here, we autowire the constructor and pass the dependency as a param. The advantage of this is that the dependency will be initialized when object of UpdateSub is created, thus solving the null pointer concerns of above

We can even do away with the explicit @Autowired annotation when there is just one constructor, as is the case above, since Spring handles the initiation during the constructor invocation.

This, considering all factors, seems to us, the most useful and recommended implementation of Dependency injection

Advantages of Dependency Injection#

To summarize, following are the advantages of DI

Loose Coupling: DI reduces the coupling between classes, making the system more flexible and easier to maintain.
Easier Testing: Dependencies can be easily mocked or stubbed during unit testing, leading to more isolated and reliable tests.
Improved Code Readability: DI promotes clean code practices by clearly defining dependencies and their relationships.
Enhanced Maintainability: Changes in dependencies require minimal changes in the dependent classes, making the system more maintainable.
Increased Reusability: DI encourages the use of interfaces and abstract classes, enhancing the reusability of components.

Conclusion#

Dependency Injection is a powerful design pattern that improves the modularity, testability, and maintainability of your code.

Google Cloud VPC

2024-06-15T00:00:00Z

In the world of cloud computing, a Virtual Private Cloud (VPC) is a private network within a public cloud that allows organizations to isolate their resources and manage them securely. Google Cloud Platform (GCP) offers a robust VPC service that provides scalable and flexible networking capabilities. In this blog, we'll delve into the concept of VPCs in GCP, explore their features, and guide you through setting up a VPC with snapshots from the GCP platform.

What is a Virtual Private Cloud (VPC)?#

A Virtual Private Cloud (VPC) is a logically isolated section of a public cloud where you can launch resources in a virtual network that you define. A VPC provides the ability to:

Isolate resources within the cloud environment.
Control network settings such as IP address ranges, subnets, and route tables.
Secure communication between resources using firewalls and security groups.

Key Features of GCP VPC#

Global Scope: GCP VPCs are global resources that span all the regions, allowing you to create subnets in any region without creating multiple VPCs.
Flexible Subnetworks: Subnets can be defined per region, allowing for more granular control over your network.
Custom Routes and Firewalls: VPCs come with default route tables and firewall rules that you can customize to control traffic flow.
Private Google Access: VPCs can enable private access to Google services, ensuring secure communication without exposing traffic to the internet.
VPC Peering: Connect multiple VPCs together to share resources across different projects or organizations.

Setting Up a VPC in GCP#

Step 1: Create a VPC#

Navigate to the VPC Network Section:
Create a New VPC:
- Go to the GCP Console.
- Navigate to the "VPC network" section under the "Networking" category.
- Click on "Create VPC network".
Configure the VPC:
- Provide a name for your VPC.
- Choose an automatic or custom subnet creation mode. For this example, select "Custom" to define subnets manually.
- Click "Create".

Step 2: Create Subnets#

Add Subnet:
- In the "Create a subnet" section, provide a name for the subnet.
- Select the region where the subnet will be located.
- Specify the IP address range for the subnet (e.g., 10.0.0.0/24).
- Click "Add subnet".
Repeat for Additional Subnets:
- Add more subnets as needed for different regions or availability zones.

Step 3: Configure Firewall Rules#

Navigate to Firewall Rules:
- Under the "VPC network" section, click on "Firewall rules".
Create Firewall Rule:
- Click on "Create firewall rule".
- Provide a name for the firewall rule.
- Define the targets, source IP ranges, and protocols/ports.
- Click "Create".

Step 4: Enable Private Google Access#

Private Google Access:
- Navigate to the "Subnets" section under the "VPC network".
- Select a subnet and edit it.
- Enable "Private Google Access" to allow instances in the subnet to access Google APIs and services using internal IP addresses.

Advantages of Using GCP VPC#

Global Connectivity: GCP VPC allows you to connect resources across regions without needing multiple VPCs.
Scalability: Easily scale your network by adding subnets and configuring routes and firewalls as needed.
Security: Implement granular security controls using firewall rules, private access, and custom routes.
Flexibility: Create custom subnet configurations and manage IP address ranges to suit your specific needs.
Integration: Seamlessly integrate with other GCP services such as Cloud Interconnect, Cloud VPN, and more.

Conclusion#

Understanding and utilizing VPCs in Google Cloud Platform is essential for creating a secure and scalable cloud infrastructure. By leveraging GCP VPCs, you can isolate your resources, manage network configurations, and ensure secure communication within your cloud environment. The step-by-step guide provided in this blog, along with the snapshots from the GCP platform, should help you get started with setting up and configuring your own VPC in GCP.

Understanding GraphQL Mutations

2024-07-08T00:00:00Z

GraphQL has revolutionized the way we interact with APIs by providing a flexible and efficient approach to querying and mutating data. While queries are used to fetch data, mutations are the means to modify it. In this blog, we'll dive deep into GraphQL mutations, explore the concept of transactional updates, and discuss how to implement rollbacks to ensure data integrity.

What are GraphQL Mutations?#

GraphQL mutations are operations that allow you to create, update, or delete data. Unlike queries, which are idempotent (they can be called multiple times without changing the result), mutations are meant to cause side effects on the server.

Basic Mutation Example#

Let's start with a simple example of a mutation to update a financial transaction:

mutation UpdateTransaction($id: ID!, $amount: Float!, $status: String!) {
  updateTransaction(id: $id, amount: $amount, status: $status) {
    id
    amount
    status
  }
}

In this mutation, we pass the transaction ID, amount, and status as arguments to update the transaction details. The response includes the updated transaction information.

Implementing Mutations in a Server#

Here’s how you can implement the above mutation in a Java server using Spring Boot and a mock data source:

// Transaction.java - Entity Class
@Entity
public class Transaction {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private double amount;
    private String status;

    // Getters and Setters
}

// TransactionRepository.java - Repository Interface
public interface TransactionRepository extends JpaRepository<Transaction, Long> {}

// TransactionService.java - Service Class
@Service
public class TransactionService {
    @Autowired
    private TransactionRepository repository;

    @Transactional
    public Transaction updateTransaction(Long id, double amount, String status) {
        Transaction transaction = repository.findById(id)
                .orElseThrow(() -> new ResourceNotFoundException("Transaction not found"));
        transaction.setAmount(amount);
        transaction.setStatus(status);
        return repository.save(transaction);
    }
}

// TransactionResolver.java - GraphQL Resolver
@Component
public class TransactionResolver implements GraphQLMutationResolver {
    @Autowired
    private TransactionService service;

    public Transaction updateTransaction(Long id, double amount, String status) {
        return service.updateTransaction(id, amount, status);
    }
}

// schema.graphqls - GraphQL Schema
type Transaction {
    id: ID!
    amount: Float!
    status: String!
}

type Mutation {
    updateTransaction(id: ID!, amount: Float!, status: String!): Transaction
}

type Query {
    transaction(id: ID!): Transaction
}

Transactional Updates#

In a production environment, mutations often need to be part of a transaction to ensure data consistency. A transaction is a sequence of operations performed as a single logical unit of work. If any operation within the transaction fails, the entire transaction is rolled back, leaving the database in a consistent state.

Transaction Example with Spring Boot#

Spring Boot provides strong support for transactions, making it easy to implement transactional updates in your GraphQL mutations:

// TransactionService.java - Service Class with Transaction Management
@Service
public class TransactionService {
    @Autowired
    private TransactionRepository repository;

    @Transactional
    public Transaction updateTransaction(Long id, double amount, String status) {
        Transaction transaction = repository.findById(id)
                .orElseThrow(() -> new ResourceNotFoundException("Transaction not found"));
        transaction.setAmount(amount);
        transaction.setStatus(status);
        return repository.save(transaction);
    }
}

In this example, we wrap the mutation in a transaction. If any error occurs during the update, the transaction is rolled back to ensure data consistency.

Rollbacks in GraphQL#

Rollbacks are crucial for maintaining data integrity, especially in scenarios where multiple mutations are involved. Implementing rollbacks in GraphQL involves using transactions provided by the database or ORM.

Handling Rollbacks#

To handle rollbacks, ensure that each mutation is wrapped in a transaction. Here’s a more complex example involving multiple updates:

// TransactionService.java - Service Class with Complex Transaction Management
@Service
public class TransactionService {
    @Autowired
    private TransactionRepository repository;

    @Transactional
    public Transaction updateTransactionAndLog(Long transactionId, double amount, String status, Long logId, String logMessage) {
        Transaction transaction = repository.findById(transactionId)
                .orElseThrow(() -> new ResourceNotFoundException("Transaction not found"));
        transaction.setAmount(amount);
        transaction.setStatus(status);

        Log log = logRepository.findById(logId)
                .orElseThrow(() -> new ResourceNotFoundException("Log not found"));
        log.setMessage(logMessage);

        repository.save(transaction);
        logRepository.save(log);

        return transaction;
    }
}

// TransactionResolver.java - GraphQL Resolver
@Component
public class TransactionResolver implements GraphQLMutationResolver {
    @Autowired
    private TransactionService service;

    public Transaction updateTransactionAndLog(Long transactionId, double amount, String status, Long logId, String logMessage) {
        return service.updateTransactionAndLog(transactionId, amount, status, logId, logMessage);
    }
}

// schema.graphqls - GraphQL Schema Update
type Mutation {
    updateTransactionAndLog(transactionId: ID!, amount: Float!, status: String!, logId: ID!, logMessage: String!): Transaction
}

In this example, we update both a transaction and a log entry within a single transaction. If either update fails, the transaction is rolled back, ensuring that partial updates do not occur.

Conclusion#

Understanding GraphQL mutations, transactional updates, and rollbacks is essential for building robust and reliable applications. By leveraging transactions, you can ensure data consistency and integrity, even in the face of errors. Implementing these practices in your GraphQL server can help you avoid common pitfalls and provide a better experience for your users.

The challenges of Database Migration

2024-07-08T00:00:00Z

Database migration is a critical task that involves transferring data from one database to another. This process is often necessary when upgrading systems, consolidating databases, or changing database vendors. However, database migration comes with its own set of challenges and potential pitfalls. In this blog, I’ll share insights from our recent database migration project at my workplace, highlighting the challenges we faced and how we overcame them.

Understanding Database Migration#

Database migration involves moving data from a source database to a target database. This can include migrating the database schema, data, and sometimes even the database engine. Successful migration requires careful planning, execution, and validation to ensure data integrity and minimal downtime.

Common Challenges and Pitfalls#

1. Data Integrity and Consistency#

Challenge: Ensuring that the data remains intact and consistent during and after the migration is paramount. Any loss or corruption of data can have significant consequences.

Pitfall: Inconsistent data formats, incompatible data types, and schema differences can lead to data integrity issues.

Solution: Thoroughly analyze the source and target databases to identify and address any discrepancies. Use data validation techniques and tools to verify data integrity before, during, and after migration.

// Example of data validation in Java
public boolean validateData(String sourceData, String targetData) {
    return sourceData.equals(targetData);
}

2. Downtime Management#

Challenge: Minimizing downtime during migration is crucial, especially for applications that require high availability.

Pitfall: Prolonged downtime can disrupt business operations and lead to customer dissatisfaction.

Solution: Plan the migration during off-peak hours and implement a phased or incremental migration approach. Use techniques like database replication and shadow databases to minimize downtime.

-- Example of using replication to minimize downtime
CREATE PUBLICATION my_publication FOR ALL TABLES;
CREATE SUBSCRIPTION my_subscription CONNECTION 'dbname=mydb' PUBLICATION my_publication;

3. Performance Issues#

Challenge: The performance of the target database can be affected due to differences in indexing, query optimization, and hardware configurations.

Pitfall: Poor performance can lead to slow application response times and increased resource consumption.

Solution: Optimize the target database for performance by analyzing and tuning queries, indexing, and database configurations. Perform load testing to identify and address performance bottlenecks.

// Example of indexing in SQL
CREATE INDEX idx_user_name ON users (name);

4. Compatibility Issues#

Challenge: Migrating between different database systems can lead to compatibility issues with SQL syntax, stored procedures, and database features.

Pitfall: Incompatible SQL queries and database functions can cause errors and application failures.

Solution: Rewrite SQL queries and stored procedures to be compatible with the target database. Use database migration tools that offer compatibility checks and automated code conversion.

-- Example of converting SQL syntax for compatibility
-- Source (MySQL)
SELECT * FROM users WHERE DATE(created_at) = CURDATE();

-- Target (PostgreSQL)
SELECT * FROM users WHERE created_at::date = CURRENT_DATE;

5. Data Volume#

Challenge: Migrating large volumes of data can be time-consuming and resource-intensive.

Pitfall: Insufficient planning for data volume can lead to extended migration times and potential failures.

Solution: Use data chunking and parallel processing to handle large volumes of data efficiently. Consider using cloud-based migration services that offer scalability.

// Example of data chunking in Java
public void migrateDataInChunks(int chunkSize) {
    for (int i = 0; i < totalDataSize; i += chunkSize) {
        // Migrate data chunk
    }
}

6. Security Concerns#

Challenge: Ensuring the security of data during migration is critical, especially for sensitive and confidential information.

Pitfall: Data breaches and unauthorized access during migration can have severe consequences.

Solution: Implement strong encryption and access control measures during migration. Use secure connections and data masking techniques to protect sensitive information.

// Example of encrypting data during migration
public String encryptData(String data) {
    // Encryption logic
    return encryptedData;
}

7. Testing and Validation#

Challenge: Thorough testing and validation are essential to ensure the success of the migration.

Pitfall: Inadequate testing can lead to undetected issues that surface post-migration.

Solution: Develop a comprehensive testing plan that includes unit tests, integration tests, and user acceptance tests. Validate the migrated data and application functionality to ensure everything works as expected.

// Example of unit testing in Java
@Test
public void testMigration() {
    String sourceData = "source";
    String targetData = "target";
    assertTrue(validateData(sourceData, targetData));
}

Conclusion#

Database migration is a complex and challenging process that requires careful planning, execution, and validation. By understanding and addressing the common challenges and pitfalls, you can ensure a smooth and successful migration. Our recent migration project at my workplace taught us valuable lessons that can help others navigate this intricate process.

Grafana

2024-07-08T00:00:00Z

Observability has become a crucial aspect of modern software systems. It enables developers and operations teams to understand the internal state of a system based on the data it produces. At my workplace, we recently implemented Grafana to enhance our observability capabilities. This blog will guide you through the basics of observability, why we chose Grafana, and how we implemented it to gain deeper insights into our applications.

What is Observability?#

Observability refers to the ability to measure the internal states of a system by examining its outputs. The three key pillars of observability are:

Metrics: Quantitative data about the system's performance.
Logs: Detailed records of events that occur within the system.
Traces: A record of the journey of a request through the system.

Why Grafana?#

Grafana is a powerful open-source platform for monitoring and observability. It allows you to query, visualize, alert on, and understand your metrics no matter where they are stored. Here's why we chose Grafana:

Extensibility: Grafana supports a wide range of data sources and plugins.
Customizable Dashboards: Create interactive and visually appealing dashboards.
Alerting: Set up alert rules to notify you when certain conditions are met.
Ease of Use: User-friendly interface for setting up and managing observability.

Setting Up Grafana#

Step 1: Install Grafana#

First, we need to install Grafana. You can install Grafana on various platforms. Here’s an example of installing Grafana on Ubuntu:

sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Step 2: Configure Data Sources#

Once Grafana is installed, configure the data sources. Grafana supports various data sources like Prometheus, InfluxDB, Elasticsearch, etc. In our setup, we used Prometheus.

Navigate to the Grafana UI (http://localhost:3000).
Log in with the default credentials (username: admin, password: admin).
Go to Configuration > Data Sources.
Add Prometheus as a data source by providing the URL of your Prometheus server.

Step 3: Create Dashboards#

Next, we create dashboards to visualize our metrics.

Go to Create > Dashboard.
Add a new panel and configure the query to fetch data from Prometheus.
Customize the visualization type (e.g., Graph, Gauge, Heatmap) and panel settings.

Here’s an example query to display CPU usage:

rate(node_cpu_seconds_total{job="node_exporter",mode="idle"}[5m])

Step 4: Set Up Alerts#

Alerts are crucial for proactive monitoring. In Grafana, you can set up alerts based on specific conditions.

In the panel editor, go to the Alert tab.
Create a new alert rule with conditions (e.g., CPU usage > 80%).
Configure notification channels (e.g., email, Slack).

Here's a sample configuration for setting up an alert:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'localhost:9093'

Step 5: Explore Logs and Traces#

Grafana also supports log aggregation and tracing. Integrate with Loki for logs and Tempo for tracing to gain a comprehensive view of your system's behavior.

logcli query '{job="varlogs"} | logfmt'
tempo query 'span_id=12345'

Advantages of Using Grafana#

Unified View: Grafana provides a single-pane-of-glass view of your metrics, logs, and traces.
Proactive Monitoring: With alerting, you can detect and respond to issues before they impact users.
Historical Analysis: Grafana allows you to explore historical data, aiding in troubleshooting and capacity planning.
Customization: Tailor dashboards and visualizations to meet specific needs.

Conclusion#

Implementing Grafana at my workplace has significantly enhanced our observability capabilities. We can now monitor our systems in real-time, set up alerts for critical conditions, and analyze logs and traces for in-depth insights. Grafana’s extensibility and ease of use make it an excellent choice for any organization looking to improve its observability practices.

Implementing the Command Design Pattern

2024-07-10T00:00:00Z

At my workplace, we often deal with complex business logic that involves multiple operations. To maintain a clean and maintainable codebase, we decided to implement the Command Design Pattern. This pattern not only improved our code structure but also enhanced its extensibility and scalability. In this blog, I'll walk you through the Command Design Pattern, compare code written without and with this pattern, and discuss its advantages using a real-world example of updating organization details.

What is the Command Design Pattern?#

The Command Design Pattern is a behavioral design pattern that turns a request into a stand-alone object that contains all information about the request. This transformation allows us to parameterize methods with different requests, delay or queue a request's execution, and support undoable operations.

Code Without Command Pattern#

Let's consider a simple example where we need to update the details of an organization. Here's how the code might look without using the Command Pattern:

// Organization class
public class Organization {
    private String name;
    private String address;

    public Organization(String name, String address) {
        this.name = name;
        this.address = address;
    }

    public void updateName(String newName) {
        this.name = newName;
        System.out.println("Organization name updated to: " + newName);
    }

    public void updateAddress(String newAddress) {
        this.address = newAddress;
        System.out.println("Organization address updated to: " + newAddress);
    }

    // Getters for name and address
}

// OrganizationService class
public class OrganizationService {
    private Organization organization;

    public OrganizationService(Organization organization) {
        this.organization = organization;
    }

    public void updateDetails(String newName, String newAddress) {
        organization.updateName(newName);
        organization.updateAddress(newAddress);
    }
}

// Main class
public class Main {
    public static void main(String[] args) {
        Organization org = new Organization("Old Name", "Old Address");
        OrganizationService orgService = new OrganizationService(org);

        orgService.updateDetails("New Name", "New Address");
    }
}

In this implementation, the OrganizationService class directly depends on the Organization class and its specific methods. This tight coupling makes the code difficult to extend and maintain, especially when new operations are introduced.

Code With Command Pattern#

By implementing the Command Pattern, we can decouple the invoker (organization service) from the receiver (organization) and encapsulate the request as an object. Here's how we can refactor the above code:

// Command interface
public interface Command {
    void execute();
}

// Organization class
public class Organization {
    private String name;
    private String address;

    public Organization(String name, String address) {
        this.name = name;
        this.address = address;
    }

    public void updateName(String newName) {
        this.name = newName;
        System.out.println("Organization name updated to: " + newName);
    }

    public void updateAddress(String newAddress) {
        this.address = newAddress;
        System.out.println("Organization address updated to: " + newAddress);
    }

    // Getters for name and address
}

// Concrete Command classes
public class UpdateNameCommand implements Command {
    private Organization organization;
    private String newName;

    public UpdateNameCommand(Organization organization, String newName) {
        this.organization = organization;
        this.newName = newName;
    }

    @Override
    public void execute() {
        organization.updateName(newName);
    }
}

public class UpdateAddressCommand implements Command {
    private Organization organization;
    private String newAddress;

    public UpdateAddressCommand(Organization organization, String newAddress) {
        this.organization = organization;
        this.newAddress = newAddress;
    }

    @Override
    public void execute() {
        organization.updateAddress(newAddress);
    }
}

// OrganizationService class
public class OrganizationService {
    private Command command;

    public void setCommand(Command command) {
        this.command = command;
    }

    public void executeCommand() {
        command.execute();
    }
}

// Main class
public class Main {
    public static void main(String[] args) {
        Organization org = new Organization("Old Name", "Old Address");

        Command updateName = new UpdateNameCommand(org, "New Name");
        Command updateAddress = new UpdateAddressCommand(org, "New Address");

        OrganizationService orgService = new OrganizationService();

        orgService.setCommand(updateName);
        orgService.executeCommand();

        orgService.setCommand(updateAddress);
        orgService.executeCommand();
    }
}

In this refactored implementation, we have introduced a Command interface and concrete command classes (UpdateNameCommand and UpdateAddressCommand). The OrganizationService class now uses a command object to perform operations, which decouples it from the specific implementations of those operations.

Advantages of Using the Command Pattern#

Decoupling of Invoker and Receiver: The invoker (organization service) does not need to know the specifics of the receiver (organization). It only interacts with the command interface, making the code more flexible and easier to extend.
Extensibility: Adding new commands is straightforward. We just need to implement a new command class without modifying existing code.
Support for Undo Operations: By storing executed commands, we can implement undo functionality. Each command can have an unexecute method to reverse its action.
Queue and Log Requests: Commands can be queued or logged for future execution, enabling features like request logging, job scheduling, and task retry mechanisms.
Promotes Reusability: Common commands can be reused across different parts of the application, reducing code duplication.

Conclusion#

Implementing the Command Design Pattern at my workplace has significantly improved our code's maintainability and extensibility. By decoupling the invoker from the receiver and encapsulating requests as objects, we have made our codebase more flexible and easier to manage. If you're dealing with complex operations in your projects, consider using the Command Pattern to achieve a cleaner and more modular design.

Can database consistency, exception handling and Angular popups come together

2024-06-24T00:00:00Z

Having prided myself on my full stack skills, I was tested of my bonafides with a rather interesting and critical problem at work, which involved me to understand Oracle SQL database writes, GraphQL mutations, Java collections, run time exception handling, and RxJS - pretty much the entire full stack.

System architecture -

My system writes updates into an Oracle SQL DB, via GraphQL.This GraphQL mutation logic is called from a Java service, driven by Command and Builder patterns, and if it fails, should be handled appropriately through rollback mechanisms.

Mutations(write operations) are batched together to ensure readability and consistency.

An Angular MVC takes care of the UI end.

Context :

My system works as a group of parties. There's a root party A, and subordinate parties A1, A2 and so on. Changes to A, propagate to all its subordinates. --> Java principle of hybrid inheritance.

The problem -

The user updated a single subordinate party (A4), and all the rest of the parties started messing up. In fact, my framework was built to handle corruption oso well that data corruption with one party won't affect the rest. However, bang's the problem

The UX -

The user saw a big exception thrown to the screen in the form of an popup, and the user was transfixed, as he hadn't even touched anything except to load the application. How can something break, when you haven't even started working at it.

The analysis -

Common sense would've had me check the origination of the request and see what went missing. However, software engineers often don't go by common sense. So, I ended up checking logs for the approval of the last request (A4).

Why do that? Since ALL requests on other parties were corrupted, I figured there'd a common mess up between them. This mess up can not happen at an initiation - which is an operation on each party, but on updation - where common attributes might get changed.

Next up - what could've changed? And how did that happen?

The code intricacy -

We implemented the command, builder and factory patterns in our Spring Boot - GraphQL codebase, to follow a highly modularized and extensible approach. We first build a structure, run what we 'premutation commands' followed by the actual update of the party (the actual mutation), and finally 'post mutation commands'

Codewise,

PartyUpdateBuilder = party.addPartyId()
                            .addPartyName()
                            .addPartyStatus()
                            .build();

Followed by

PartyUpdateCommandBuilder.execute();

which actually writes the data into the tables.

Mutation is GraphQL version of an update query - it updates the database. It looks something like this

mutation ($params: Params!, $partyId: PartyId!) {
    updateData(params: $params, partyId: $partyId) {
        status
        log
        created
    }
}

{
    variables: {
        params: {
            "partyId": 1234,
            "partyName": "ABC"
        }
    }
}

Implications of the structure

The framework we created ensured that even if multiple tables were being updated via our mutation, we remained fully ACID compliant by batching the entire update in a single mutation that was run centrally

Services written by different developers only need to call this central source, and all fields would be updated.

While the mutation was running, there is an inbuilt system of preemptive locking, to avoid stale data being overwritten in the microseconds it takes to update the data.

Addtionally, the entire operation is maintained as a business change log in a central database, for tracking.

Our update strategy was inspired from this

Now, in such an apparent 'fail safe' framework, there was corruption happening en masse. The question was how?

Stay tuned for part 2

Stopgapping

2023-09-02T00:00:00Z

Stopgapping as a strategy is rather underrated. Often times, when a severely life changing event occurs, we can't immediately find a path to walk on. More often than not, we're indecisive, and torn. Indecision is one of the worst states one can be in. And in this state, a major choice might tend to cause regret and fear.

In such a case, we stopgap - we find out a minimalist set of steps so that we are not arresting all momentum, yet at the same time, give our body the time to get used to the new reality.

Let's consider an example. You were working at a job and one fine day, realize you've been laid off. Your world turns upside down. A stable income you'd planned on for years vanishes instantly. And most people don't look for jobs immediately on getting laid off. In this time, a pragmatic approach would be to do activities that can contribute to the overall financial and mental stability of your life, and at the same time, not place you under duress of rushing. In this case, you can choose to follow a daily routine of job shortlisting, meditation, and a set number of topics you upskill on each day. None of these are large enough to cause a major paradigm shift in your thinking, yet considerable enough to give you momentum at a time when you fear you've come to a standstill.

When I'd ended a serious relationship, my goals and priorities all went up in smoke. I chose to follow daily rituals of upskilling, finding a new hobby, and meditation to ensure that I was moving ahead and on, yet at the same time, not making a hard choice I'd regret.

Not all major decisions have to be that way - some, you just need time on, for new circumstances to pop through, and the best thing you can do, in the moment, is keep going, without regretting.

Monorepo architecture

2023-09-02T00:00:00Z

Since the advent of the microservices concept, most people are fans of distributed architectures. You don't want to have single points of failure, have autonomy across teams, and want to customize tech stack by service.

This concept has propagated into domains other than services too.

At work, we had three different repos catering to one single application - two libraries, and one repo for the actual configuration which just mapped components from the libraries onto the actual Angular app.

The idea behind this was primarily, the separation of concerns. Repo A included fundamental components and styling, say, dialog boxes, text editors, toasts and their corresponding styles. Repo B included actual application components organized on the basis of business logic and their occurrence on the application. Both A and B were built, deployed and their prod build versions injected as npm packages into C.

Now, for an application as large and diverse as ours, it kinda made sense. If you just had to make a config change, we wouldn't really want to rebuild fundamental CSS all over again. Different teams could own these repos differently and you could get the latest working version of any of these repos by picking the last build version from the common artifactory.

Now, however, there come the pitfalls :

Cumbersome fixing and testing : If I have to make a fundamental CSS change, in repo A, I need to fix it in A, test it in B, and then finally in C. I essentially have to setup and run all three repos for a minor CSS change. Because the repos were owned separately, there's a fair chance they'd have their separate requirements in terms of setup, dependencies and run commands. How much of an overhead for such a little change? Wouldn't it be better to just have one repo, fix something, and voila, see the change?
Inconsistent design : If you want to make a CSS change to a component, do you make it in A or B? It's a subjective question and varies by use case, so most people just did what they felt was right, meaning half the changes were in one repo, half in another. One actual example - our dialog boxes were styled from Repo A in two of our application tabs and from Repo B in the remaining 3. Who'd remember where the styles are coming from then?
Versioning : Some change works on x.1 version of Repo A, y.2 version of B and z.3 version of C. Now, everytime, we have to check this version compatibility. Changing one of the versions could adversely impact the rest.

-- work in progress--

Action method

2021-12-24T00:00:00Z

The Action method encourages you to look at everything in your life as a project, with a set of actionable items, organized by priority, and associated references. Completion of all the action items will signify completion of the event/project. The advantage of this method is that converting seemingly subjective items like events/meetings into actionable steps will prompt you into taking the next small step, and get you started on tasks that you'd otherwise have procrastinated.

Every major life item you have is considered a project, and you break it into action steps, references and backburners

The action steps are as they sound like - a progession of doable items that will lead to achievement of the project goal.

References are materials, resources and information that will aid in the achievement of the actions steps. It is stuff which is related to the project, but not directly actionable. This includes URLs to necessary references, some go to reference books/articles for the project, and so on.

Backburner items refer to items that might be important at later stages, but can be put aside at the moment. Entire projects can be backburners too, meaning that a project need not be taken up at the present moment, because you have other, more important projects on your plate.

What's the advantage of this method?

'Action is the greatest motivation'. The most common reason for procrastination is the lack of the next small step towards a goal. If that next small todo can be found and completed, it's enough to get the ball rolling. Each action item is the next small step towards achieving a large project.
Looking at every item in your life as a project gives you an objective vision into what you'd actually need to do. Meetings and events are otherwise subjective and abstract events - converting these into projects gives you action items for before, during and post the meetings, and thus, you know what would make an event a success.
Unlike other todo lists, this method differentiates between actionable items and non actionable items, that complement the former, and provides a way to keep track of both.
The concept of backburner items and projects allows you to prioritize projects and action items based on the impact they have on your life and project, without worrying about forgetting them later. Once you're done with the action items of your project, you can pull items from backburner and take them up as action items.

How do you make ACTION method work with Routine#

Routine's flexibility can be leveraged to use it to implement the ACTION method for some/all of your projects/tasks. Each task can be opened as a document which can include everything that we need - first, markdown, so that we can create and distinguish between the sections, second, the embed feature to embed resources, and finally, checkboxes which will make every checkbox item a task, which can be scheduled on the Routine calendar just like any other task.

Let's see an example. I create a new task - which will represent my project, let's say Blog writing.

Double clicking on it opens up the task as a document

Now, click on add subtasks, and add a few tasks

Next, create the sections we'd need - action items, references and backburners. Go to a new line, and press '/', which will give you the list of possible markdown options - choose H2, and create the three headings.

Now, drag and drop the immediate tasks you'd need doing into action items, and the others, into backburners

Next, I want to embed a YouTube video I wish to refer to. Under the references heading, I select 'embed', and paste the video link, and there you have it

And finally, to schedule our tasks - when you hover over a task, a calendar icon is highlighted - click on it, and give it a date and time - the great bit is you can just write it in words and Routine will intellisense it into a schedule.

Bingo, you see your task in the upcoming tasks

There you have it - a complete action system in Routine to help you on the road to getting that project done.

Creating a full stack app using AWS Amplify

2021-12-12T00:00:00Z

Overview#

Amplify is an offering by AWS that lets you develop and deploy full stack applications by only focusing on the business logic, with all the configuration being handled behind the scenes.

In this tutorial, we'll understand what Amplify is, how it works, and finally, set up a Todo list application with a GraphQL backend and a React frontend using Amplify

Prerequisites#

You'll need to have Node/NPM and Git installed on your local systems.

You should have an AWS account. Some knowledge of AWS concepts like IAM roles will come in handy, since we'll have to setup an IAM user for connecting our app.

It'll also be useful to have some knowledge of React, since we'll be adding some code for the UI. GraphQL code will also be used, but since it'll be autogenerated, it isn't absolutely necessary for you to know that.

Introduction to AWS Amplify#

Building full stack applications is no small task. And a lot of time is spent on writing boilerplate code that's already been written previously, and not enough effort can be put in developing the business logic of the application. Moreover, once the app has been built, deploying and scaling it is another major blocker for development teams to handle.

Amplify tries to alleviate these use cases. It abstracts away some core functionalities by leveraging existing AWS services and code, and allows developers to only add the business logic of the application and configuring the rest of the application intelligently.

Some of the existing AWS services that are leveraged by Amplify include AppSync for GraphQL. Cognito to handle authentication and DynamoDB for database.

Amplify also provides other features like REST APIs, Lambda functions support and adding prebuilt Figma components into the frontend of the app, all of which are very frequent use cases

The process we'll follow#

We'll first setup the Amplify CLI on our local systems. We'll then setup the AWS profile to connect our app to AWS. We'll then add the frontend and the GraphQL code respectively, to get our app running.

Setting up Amplify on local#

Create a new folder called amplify-app. Open the command line and navigate to this folder

We'll start with installing the Amplify CLI. It's the command line application for Amplify that'll allow us to configure our app using commands. Use the following command to install Amplify

npm install -g @aws-amplify/cli

Next, we'll be configuring Amplify by creating an AWS IAM user.

Enter

amplify configure

You'll be prmoted to enter a username and select a region. You can choose anything you wish for both, just make sure to remember it.

You'll then be prompted to sign in to your AWS account on the browser

Once you're signed in, you'll have to create an IAM(Identity and Access Management) user. It's this user whose credentials will be used for the app.

The username will have been auto populated.

Check the password and custom password option and add a custom password, since it's easier than auto generated password. Do note that the access keys option should remain checked.

Then keep hitting next until you get the button to Create the user and click it

Your user will be created with an access key id and a secret key. Keep the window open since you'll be needing the details.

Come back to the terminal and press enter

You'll be prompted to add, first the access key id and then the secret. Copy and paste both of them.

If you're prompted to add a profile name, add a random one.

With this, our AWS profile setup is complete

Setting up React app#

Use the following command to set up the default react application and name it todo-amplify

npx create-react-app todo-amplify

cd todo-amplify

npm run start

This will start the sample React app on localhost:3000.

Close the app and keep it on hold. We'll come back to the frontend in a bit

Initialize backend#

Type

amplify init

to start the setup for the backend

You'll be asked for some configuration options like this :

Enter a name for the project (react-amplified)

# All AWS services you provision for your app are grouped into an "environment"
# A common naming convention is dev, staging, and production
Enter a name for the environment (dev)

# Sometimes the CLI will prompt you to edit a file, it will use this editor to open those files.
Choose your default editor

# Amplify supports JavaScript (Web & React Native), iOS, and Android apps
Choose the type of app that you're building (javascript)

What JavaScript framework are you using (react)

Source directory path (src)

Distribution directory path (build)

Build command (npm run build)

Start command (npm start)

# This is the profile you created with the `amplify configure` command in the introduction step.
Do you want to use an AWS profile

Keep hitting enter to choose all the default options. For the AWS profile, choose the one you'd created previously. The setup will eventually finish in a few seconds

So what exactly happens#

When you initialize a new Amplify project, a few things happen:

It creates a top level directory called amplify that stores your backend definition. During the tutorial you'll add capabilities such as a GraphQL API and authentication. As you add features, the amplify folder will grow with infrastructure-as-code templates that define your backend stack. Infrastructure-as-code is a best practice way to create a replicable backend stack.
It creates a file called aws-exports.js in the src directory that holds all the configuration for the services you create with Amplify. This is how the Amplify client is able to get the necessary information about your backend services.
It modifies the .gitignore file, adding some generated files to the ignore list
A cloud project is created for you in the AWS Amplify Console that can be accessed by running amplify console. The Console provides a list of backend environments, deep links to provisioned resources per Amplify category, status of recent deployments, and instructions on how to promote, clone, pull, and delete backend resources

Back to the frontend#

We'll install the 2 packages we're going to need for the project, using :

npm install aws-amplify @aws-amplify/ui-react@1.x.x

Next, we'll update our client with the backend configuration stuff. Open src/index.js of your React app and add the following code at the top

import Amplify from "aws-amplify";
import awsExports from "./aws-exports";
Amplify.configure(awsExports);

And that's all it takes to configure Amplify. As you add or remove categories and make updates to your backend configuration using the CLI, the configuration in aws-exports.js will update automatically.

Finally, update your src/App.js with the logic for Todo :

/* src/App.js */
import React, { useEffect, useState } from 'react'
import Amplify, { API, graphqlOperation } from 'aws-amplify'
import { createTodo } from './graphql/mutations'
import { listTodos } from './graphql/queries'

import awsExports from "./aws-exports";
Amplify.configure(awsExports);

const initialState = { name: '', description: '' }

const App = () => {
  const [formState, setFormState] = useState(initialState)
  const [todos, setTodos] = useState([])

  useEffect(() => {
    fetchTodos()
  }, [])

  function setInput(key, value) {
    setFormState({ ...formState, [key]: value })
  }

  async function fetchTodos() {
    try {
      const todoData = await API.graphql(graphqlOperation(listTodos))
      const todos = todoData.data.listTodos.items
      setTodos(todos)
    } catch (err) { console.log('error fetching todos') }
  }

  async function addTodo() {
    try {
      if (!formState.name || !formState.description) return
      const todo = { ...formState }
      setTodos([...todos, todo])
      setFormState(initialState)
      await API.graphql(graphqlOperation(createTodo, {input: todo}))
    } catch (err) {
      console.log('error creating todo:', err)
    }
  }

  return (
    <div style={styles.container}>
      <h2>Amplify Todos</h2>
      <input
        onChange={event => setInput('name', event.target.value)}
        style={styles.input}
        value={formState.name}
        placeholder="Name"
      />
      <input
        onChange={event => setInput('description', event.target.value)}
        style={styles.input}
        value={formState.description}
        placeholder="Description"
      />
      <button style={styles.button} onClick={addTodo}>Create Todo</button>
      {
        todos.map((todo, index) => (
          <div key={todo.id ? todo.id : index} style={styles.todo}>
            <p style={styles.todoName}>{todo.name}</p>
            <p style={styles.todoDescription}>{todo.description}</p>
          </div>
        ))
      }
    </div>
  )
}

const styles = {
  container: { width: 400, margin: '0 auto', display: 'flex', flexDirection: 'column', justifyContent: 'center', padding: 20 },
  todo: {  marginBottom: 15 },
  input: { border: 'none', backgroundColor: '#ddd', marginBottom: 10, padding: 8, fontSize: 18 },
  todoName: { fontSize: 20, fontWeight: 'bold' },
  todoDescription: { marginBottom: 0 },
  button: { backgroundColor: 'black', color: 'white', outline: 'none', fontSize: 18, padding: '12px 0px' }
}

export default App

Setting up API and Database#

The API you will be creating in this step is a GraphQL API using AppSync and DynamoDB database

Use the following CLI command to initialize the API creation :

amplify add api

You'll be prompted through a list of options. Keep hitting enter to choose the default ones

Once it's complete, we'll push the changes using

amplify push

Once it completes, it gives you the endpoint and an API key

Once it's done, run the React app again, and going to localhost:3000, you should see your todo app

Deploying your app to Amplify cloud#

Additionally, you can also deploy your todo application to Amplify cloud using the following commands :

amplify add hosting

amplify publish

Conclusion#

Thus, with this, you've completed developing an entire full stack application, with the only code you had to write being the business logic for the todo. Imagine the time and effort saved when all the GraphQL code, and the connections came up magically outta nowhere!

References#

AWS Amplify Docs

AWS Lambda vs ECS

2021-12-06T00:00:00Z

Overview#

In this tutorial, we'll be taking a deep dive into the differences between AWS Lambda and AWS ECS. We'll be setting up sample applications using each of those and then contrasting the different use cases they have.

What is AWS Lambda#

Lambda uses resources that are the same that a server-driven deployment would've given us - EC2 instances, coupled with load balancers, security groups, auto-scaling services. However, unlike the latter, these resources are configured entirely on the backend, away from the user, and automatically scaled up/down as per traffic. All the user needs to do is provide the code, and let Lambda take care of ensuring it runs.

The following Block diagram describes how lambda works

What is ECS#

ECS stands for Elastic Container Service and is a container orchestration solution - meaning, it allows deployment and management of applications which are containerized using tools like Docker.

The following block diagram gives an explanation of how it all comes together

What is AWS Fargate#

We'll be using AWS Fargate in our ECS example. Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers. Unlike EC2, you don't actually have to worry about setting up and provisioning the servers. You only provide a containerized application, and Fargate handles the hosting based on the resources you require

Setting up Lambda#

Go to aws.amazon.com and sign up for an account if you don't already have one.

Once you're signed in, search 'Lambda' in the search bar. You should be redirected to the Lambda dashboard

Before you create a Lambda function, you need to identify its inputs and triggers, choose a runtime environment, and decide what permissions and role the service will use.

Lambda functions accept JSON input and JSON output. Your function’s input and output contents are closely tied to the event source that will trigger your function.

An event source is usually a web request, that'll cause the execution of the function code

You also need to select a runtime for your function. We'll be using Node.js

Finally, your function will need an AWS role, that defines the entitlements the function has within the AWS platform.

Click on Create function

Keep the default 'Author from scratch' option selected

Give your function a name as you wish, and leave everything else as it is.

Click on Create function at the bottom of the page

You'll be redirected to the function configuration page, that looks something like this

You'll first have to add a trigger for your lambda function. Click on add trigger.

You'll then be asked to choose a trigger - select API Gateway. An API Gateway essentially lets you create, deploy and monitor APIs. In our case, we'll be able to use our function like an API - when we hit the deployed URL, it'll trigger our function.

Choose API type as REST API, security as Open, and leave the rest as it is. Finally, click Add

You'll see that the trigger is added.

Next, you are given a code source window with an integrated code editor, where you can add/edit code and files.

A sample code snippet is provided. You can choose to modify the message to something you wish, and keep the rest of the code as it is for now.

Testing the function#

Next, we'll test if the function works as expected. Go to the test tab.

Here, you're given an option to create an event. An event is a happening that triggers the function. It has a JSON input. Since we're not actually using the input in any way, it's not much to us. However, when the lambda function is deployed as a service to some application, there'll be inputs coming in that the function will use. Those inputs can be given here to test if they give the required outcome.

Leave everything unchanged, and click Test.

It'll run the test using the event config, and will pass with the following message in a second or two.

Understanding the result#

The details show the function output. In our case, the status code and the message body.

The summary tab has a few important fields. The duration denotes the time it took for the lambda to run, which is an important pointer when we are running a production grade application and are likely to get timeout/performance issues

The billed duration is another important indicator - you only pay for what you use. Unlike the EC2 instance, where you were charged for the server just being on, irrespective of whether or not anything was running on it, Lambda only charges you for the times your function runs. Thus, being an obvious cost advantage

And the field one of the most significant to our discussion - Resources configured. 128 MB in our case. Do you remember configuring anything at all, apart from the function code itself? Nope. So where did the 128 MB come from? That's the magic - by just telling Lambda what code you need to run, it automatically provisions the resources needed to run it, saving considerable bandwidth of the developers that would've otherwise gone in getting the servers configured.

Deploying the Lambda function#

Go back to the code tab, and click on Deploy

Now, click on API Gateway in Function Overview.

It'll give you the API endpoint. Copy it, and paste it in a new browser tab.

Sure enough, you'll see the learning lambda message on the screen.

Come back to the lambda dashboard and go to the monitor tab. Here, you'll be able to monitor the calls being made to your API. Refresh the page of the API a few times, and you'll see the requests being shown on the graphs

Notice the usefulness of the graphs - The invocations show you how many times the API was invoked.

The error count and success rate let you track if the function is facing downtime/run time errors.

Setting up ECS#

Next, we'll setup and configure an ECS application using AWS Fargate

Go to AWS dashboard and search for ECS. You'll be taken to the ECS dashboard, that looks like this

Click on Get Started

We'll be selecting an Nginx container

Next, you'll be prompted to add a service, which ensures that the defined task instances are maintained. If not, a new task instance is created.

Next, you'll be asked to configure your cluster details - keep them as they are.

Finally, click create.

You can see the status of the resources being provisioned :

Finally, your service will be active

Go to task definitions

Copy the public IP and paste it in a new browser tab

You'll see that the default nginx screen opens up

Refresh it a few times

Come back to the ECS dashboard and go to logs. You'll see that for every refresh, a log entry is created

Difference between Lambda and ECS#

Thus, you created and deployed sample services using both Lambda and ECS(via Fargate).

At the first glance, these two look similar - both of them are serverless solutions that configure the server resources based on the configuration that your application needs, and work on a pay-per-use model. They both also provide monitoring and logs in a similar fashion

However, there are a few subtle differences - Lambda essentially allows you to run tiny functions -they can of course be as gigantic as applicaations themselves, but that's not what it's meant for. Isolated services that can then be plugged into existing applications via triggers like the API Gateway we used ensure that your services work in isolation, and the downtime of one doesn't affect the other.

ECS is a container orchestrator, and is principally meant for running 'containerized applications'. There's some configuration needed for you to define when setting up the resources, where in Lambda, it was handled in its entirety by AWS itself. ECS is mainly meant for larger applications but with a flexibility of not having to manage compute instances yourself.

Consider Lambda over ECS when#

You have a smaller application that runs on-demand in 15 minutes or less.
You don’t need advanced EC2 instance configuration. Lambda manages, provisions, and secures EC2 instances for you, along with providing target groups, load balancing, and auto-scaling. It eliminates the complexity of managing EC2 instances.
You want to pay only for capacity used. Lambda charges are metered by milliseconds used and the number of times your code is triggered. Costs are correlated to usage. Lambda also has a free usage tier.

Consider ECS over Lambda when#

You are running Docker containers. While Lambda now has Container Image Support, ECS is a better choice for a Docker ecosystem, especially if you are already creating Docker containers.
You want flexibility to run in a managed EC2 environment or in a serverless environment. You can provision your own EC2 instances or Amazon can provision them for you. You have several options.
You have tasks or batch jobs running longer than 15 minutes. Choose ECS when dealing with longer-running jobs, as it avoids the Lambda timeout limit above.
You need to schedule jobs. ECS provides a service scheduler for long running tasks and applications, along with the ability to run tasks manually.

Conclusion#

Thus, in this tutorial, you got an introduction to AWS Lambda, AWS ECS and Fargate. You understood the similarities among them by setting up sample applications using each. You then created distinctions between them, and hands on checklists as to when one would be preferred over the other

References#

Intro to Serverless

2021-12-05T00:00:00Z

Overview#

In this section, we'll get a deep understanding of what it means to have 'serverless' applications - most importantly, why it's a misnomer. We'll understand the use case of using this paradigm, how it's implemented on the ground and finally, take up a hands on example to create a sample NodeJS service using AWS Lambda.

Introduction#

Web applications and services need servers to run on. These servers can be custom on-premise servers that large companies themselves own, or cloud servers by providers like EC2 by AWS. We've used the latter in a few tutorials in the past.

While the cloud servers leave out the complexity of server maintenance, we still need to manually configure load balancing and track usage. We'll be charged for all the time the server's up, irrespective of whether or not we're the server's being used at all or not. This is suboptimal for many small organizations, who not only want to minimize cloud costs, but also can't spare enough manpower on customizing load balancing and server instance uptime.

Thus came the concept of 'Serverless'. First things first, it's NOT like there's no server at all. It's just that we aren't granted access to an entire server like we were for EC2. Instead, we just give the cloud provider the application code we need to run, and then it's their job to run the code, ensure that it scales up/down based on traffic, allowing us to focus on the application itself.

How exactly does this work?#

The following Block diagram describes how lambda works

What we'll be doing#

We'll be setting up a NodeJS service using AWS Lambda, configuring the triggers that would cause it to run, and then hitting those triggers to run it, and tracking the logs as the function runs.

Setting up AWS lambda#

Go to aws.amazon.com and sign up for an account if you don't already have one.

Once you're signed in, search 'Lambda' in the search bar. You should be redirected to the Lambda dashboard

Before you create a Lambda function, you need to identify its inputs and triggers, choose a runtime environment, and decide what permissions and role the service will use.

Lambda functions accept JSON input and JSON output. Your function’s input and output contents are closely tied to the event source that will trigger your function.

An event source is usually a web request, that'll cause the execution of the function code

You also need to select a runtime for your function. We'll be using Node.js

Finally, your function will need an AWS role, that defines the entitlements the function has within the AWS platform.

Click on Create function

Keep the default 'Author from scratch' option selected

Give your function a name as you wish, and leave everything else as it is.

Click on Create function at the bottom of the page

You'll be redirected to the function configuration page, that looks something like this

You'll first have to add a trigger for your lambda function. Click on add trigger.

Choose API type as REST API, security as Open, and leave the rest as it is. Finally, click Add

You'll see that the trigger is added.

Next, you are given a code source window with an integrated code editor, where you can add/edit code and files.

A sample code snippet is provided. You can choose to modify the message to something you wish, and keep the rest of the code as it is for now.

Testing the function#

Next, we'll test if the function works as expected. Go to the test tab.

Leave everything unchanged, and click Test.

It'll run the test using the event config, and will pass with the following message in a second or two.

Understanding the result#

The details show the function output. In our case, the status code and the message body.

Deploying the Lambda function#

Go back to the code tab, and click on Deploy

Now, click on API Gateway in Function Overview.

It'll give you the API endpoint. Copy it, and paste it in a new browser tab.

Sure enough, you'll see the learning lambda message on the screen.

Notice the usefulness of the graphs - The invocations show you how many times the API was invoked.

The error count and success rate let you track if the function is facing downtime/run time errors.

All of this, without having to configure any of it - that's the beauty of Lambda

Adding further code#

Now that your Lambda function is up and running, you can add further code to create actual services, connect it to databases and more.

Conclusion#

Thus, in this tutorial, we got introduced to what serverless means, and how it is beneficial over the traditional server-driven model. We used AWS lambda to setup and configure a NodeJS service, set up a trigger using the API Gateway, and monitored our service, all while having to configure little beyond our business logic.

References#

AWS Lambda official docs

Demystifying procrastination

2021-12-20T00:00:00Z

The biggest threat to productivity is procrastination - the wilful(?) destruction of a more structured life by giving in to short term pleasures over long term contentment. Notice the '?' after 'wilful'.

Is procrastinating wilful? Do we, who have big aims and aspirations of a better life, CHOOSE to derail our progress on those goals, WHILST being aware that it could potentially be the death knell for the consistency we'd maintained so far? I mean, no sane person would kill one's own desires so willingly, right?

The subject of procrastination has been under medical research for years, and while it's unnecessary for us to deep dive into the intricacies of the flashing neurons, it helps to know a few superficial facts (I am no more a medical guy than Jackie Chan is a ballerina, so do not kill me over the medical accuracy of what I write - it's been vastly simplified for ease of understanding) - a section of our brain, called the prefrontal cortex, can be thought of as the logical part - which makes you do stuff that make 'logical sense', making goals like exercising, personal projects, reading and so on.

And there's this other dude called the limbic system, which has more to do with the emotional and instinctive stuff. And no surprises, it's this little son of a jumbled mass of neurons that makes you procrastinate - it's responsible for short term pleasures that your brain seeks, and fights with the prefrontal cortex for dominance over your body. Whenever the PC wins, you're productive. When it's the limbic system who comes out on top, #NetflixBinge

And thus, our quest to cutting at our procrastination would be to ensure that our prefrontal cortex wins more often.

Superb. How do we make that happen?

The Productivity Project author Chris Bailey calls out six traits that usually occur in various quantities in almost all tasks we procrastinate. The intensity and quantity of each of these traits in a task determines how likely we are to procrastinate it.

These are :

Boring
Frustrating
Difficult
Unstructured or ambiguous
Lacking in personal meaning
Lacking in intrinsic rewards

Let's take an activity that's pretty common a procrastinated one for many of us - tidying our rooms, and rate it on a scale of 1 to 10 in all 6 of these.

Boring - yeah, a bit. 6/10.

Frustrating? Often - you know it's gonna be the same old mess within a week at most, and thus makes you wonder why do it at all. 9/10.

Difficult? Umm, not so much, unless your room is a palace. 2/10.

Unstructured or ambiguous? Yes, absolutely. When do you decide if it's 'clean enough'? Where do you start cleaning? Do you clean the insides of the cupboards too? 10/10

Lacking in personal meaning? Unless you're a Monica from Friends, definitely yes. It doesn't give the kicks, and contributes little to personal goals. 9/10

Lacking in intrinsic rewards? Again, yes. No direct benefit to me that I can see. 9/10

And there, we have it. While thinking of rating the intensity of each of these traits on the task of cleaning the room, we thought about the negative aspects of the task that made us procrastinate it. And once you know where you're going wrong with a problem, the problem's half solved.

Boring? -> Play your fav music as you clean.

Unstructured? -> Create a weekwise plan beforehand as to what part of the room you'll clean the coming day/week, and then, tackle only that, not worrying about the others.

Lacking in intrinsic rewards? -> If the 'feeling of accomplishment' isn't a good enough reward, you may create a reward for yourself - 10 minutes of binging on something you like if you clean the room.

And thus, by quantifying and categorizing some of the 'procrastin-able' aspects of a task, you make plans to systematically limit/eliminate those, and make it harder for the limbic system dude to come out on top.

It may seem like a pain, definitely, to think and plan so much before all your procrastinable tasks, and might make you wonder - should I just have gritted myself and got done with the task in the time, rather than planning like a military general for it? Well, the very reason you're planning on the task is BECAUSE you could not grit yourself and get done with it. The planning will get it done. And once you've gotten a hang of it, the eliminations will come instinctively and faster

Productivi-TEA - Time, Energy, Attention

2021-12-29T00:00:00Z

When starting with a new productivity goal, we often expect from ourselves, something that's entirely alien to human nature - that, irrespective of our body's responses, we're constantly able to attack our day's plans with the same zeal, zest and energy throughout the day. Most of us who started on a sunny day with the motivation to blast their productivity through the roof, started with dumping tasks and plans on every second of the day, and then watched helplessly as the scheduled stuff came, but the body didn't respond with the same energy, or if our attention went into scrolling through some extremely relatable memes on IG.

Productivity is a function of Time, Energy and Attention, all of which, we have in finite availability. Think of it like money - if you only have a 100 bucks, you'd rather spend 90 of it on what's going to help the most in your survival - food, water and clothing, rather than buying a Netflix subscription. Similarly, the best of our Time, Energy and Attention has to be devoted to the tasks that are the most meaningful to us, from which, we hope to derive the maximum output.

And that's why, the name of this article - ProductiviTEA - the TEA are like caffeine. The optimal usage of the TEA can give you a boost in your life.

So, how do you ensure that the best of your TEA goes into your most important tasks? And how can Routine help you in your journey?

Tracking your TEA#

You can only improve if you know where you are lacking. Tracking your Time, energy and attention throughout the day will give you insights on what are your peak moments, and thus, how you can leverage those.

Routine's essence, is its calendar-drag-and-drop, and you can utilize this core feature to track your traits, for a week.

To do that, schedule a task every waking hour on the Routine calendar. This task can be an actual work task, or just be anything that you intend to do at that particular time, including watching TV, or propping your feet onto the table and staring at the ceiling. You just need to track what you're doing at that time.

Double click on a task to open it as a doc, and add two points - Energy and Attention

For each hour, track your energy level out of 100. It may be tough to quantify it at first, but after a few trials, you'll be to able to put in a number relative to what you put in the past.

Also, for each hour, try and track how many times you felt your attention wandering from what you were meant to do in the past hour. You need not keep a strict count of this - a rough approximation works, to begin with.

Follow this ritual for about a week, and you'll begin to notice some patterns - Your energy and attention levels are high at certain times of the day. For early morning birds, it's usually the morning hours, likewise for night owls. These reflect what's called your Biological Prime Time. Note, manually enforced energy and attention like caffeine induced, pressure of deadlines, do not count here. Your BPT is based on your natural body clock, and your habits - when you're naturally the most prone to attention and action. During these BPT high times, perform your most important tasks, usually the ones that you're highly likely to procrastinate on.

At your worst BPT, schedule tasks that require the least bit attention, such as tracking emails.

Thus, by actively managing your BPT, you can get more done, without forcing your body

CI CD using Github Actions and Netlify

2021-12-06T00:00:00Z

Overview#

In this tutorial, we'll be taking a ground-up understanding of what Continuous Integration-Delivery-Deployment means, why it's so useful in modern software development and DevOps. We'll then use a hands on example of configuring a CI pipeline for a sample React application using Github actions, understanding some syntax along the way. We'll then use netlify to connect our repo to Netlify and configuring a CD pipeline.

Prerequisites#

You'll need a Github account. We'll be using a sample React application to set up the workflow, and it might help to understand how to run a React app, although any detailed understanding is not necessary, since we won't be adding any React code in this tutorial.

You'll also need an account on netlify.com, which we'll be connecting with the Github account to set up a CD pipeline. All of this is entirely free.

Introduction to CI CD#

Disclaimer : Some of the practices might seem vague or overkill right now, especially for those who have not had experience working in large teams. However, CI CD was developed keeping in mind software development for large, distributed teams.

In any team delivering software for a client, it is not enough to just push your code along to a remote repository and be done with it. There's an entire process that happens once you're done coding, and it's fraught with complications.

There'll be tens or hundreds of developers making changes to the same codebase, all with different coding styles. Your code might not work with the most recent push made by another developer. Your code might not be good quality, which might make it difficult for other developers to understand it/build up on it. Your code might 'work on your machine', but it might not work in the higher environments.

All of these things can go wrong, and they do, so much so that they forced the industry pioneers to come up with an approach to ensure that any new code that was being pushed followed a set of guidelines, and that it went through a series of steps before it finally got merged into the main codebase. This process was rote enough to not be done manually each time somebody pushed something, and thus, there were tools developed to automate the checks and steps that needed to be taken.

This process, is called Continuous Integration. Your code is continuously 'integrated' into the application, AFTER ensuring that automated tests and other scripts run onto it confirm that it doesn't break some existing feature and is of good quality.

A sample CI workflow looks something like this (It differs by team - this is just a sample one) :

Developer pushes the code into her/his feature branch. No one pushes code directly into master branch in a development team.
Developer seeks code reviews from teammates and raises a pull request
As soon as the PR is raised, a step in the CI workflow is triggered and a build starts using the new code, on a build automation tool like Jenkins or Teamcity. If the build fails, it's pointless to carry on to further steps, and the code is reverted to the developer asking her/him to check why it failed and make the changes.
If the build passes, the reviewers manually check the changes made by the dev and approve the PR.
Once the necessary number of approvals have been granted, the next workflow step gets triggered, wherein automated tests are run on the code to ensure the functionality is working as expected
Further checks MIGHT be made by automated tools checking for code quality or test coverage using tools like SonarQube(SonarLint) or Codecov. These tools raise flags if the new code does not follow some coding standards configured by the team. The developer has to rectify those and restart the workflow.
Once the checks are complete, the code then 'tries' to get merged onto the main branch. If there's some other code commit made on the same lines as this push has, there is an automatic merge failure that the developer has to resolve manually.
If not, the code gets merged into the main branch.

This might sound like a lot of work, but in a complex project, it's critical to ensure that any new change is the 'right change', or it could take weeks to unroll if it passes undetected. Moreover, with the practice of automated CI, almost all the steps are done automatically, without the need of someone to manually push the code along to the next workflow step.

Thus, CI is about pushing code in small increments as frequently as possible, ensuring that it's bug free and follows best practices, and finally merging it into the main code.

CD is a term that can refer to Continuous Deployment, and/or Continuous Delivery, usually referring to both, first Delivery, then Deployment. Atlassian describes the difference between Delivery and Deployment as while Delivery requires a manual intervention for pushing to a production environment, deployment automates that step as well.

Once your code is pushed into the master branch at the end of a CI workflow, it needs to now go through various testing environments where further tests like FT(Functional testing), SIT(System Integration Testing) and UAT(User Acceptance Testing) are run to ensure the application is working as expected. And once it's gone through all the testing environments, the final release to production can be done manually(c. delivery), or automatically(c. deployment)

Intro to GitHub Actions#

GitHub actions is a tool provided by Github that helps you create and run the workflows for CI/CD. By creating a simple workflow file, you can ensure that once your code is committed to GitHub, it'll get released to your production environment entirely on its own without requiring any effort from you.

GitHub actions is an extremely popular tool for beginners, since, unlike other CI tools like Jenkins, it's extremely simple to set up and start, and abstracts away a lot of the setup that newbies need not bother themselves with.

How does it work?#

Actions uses code packages in Docker containers, which run on GitHub servers and which, in turn, are compatible with any programming language. There are tons of preconfigured workflows available across frameworks like Node, Python, Java, which we can pick and customize for our application, and that's precisely what we're going to do when we get to the hands on.

Terms#

There are a few terms that will be used in the configuration files that we need to look through. Fortunately, they're more or less exactly like they sound :

Step : A set of tasks that need to be performed. They can be commands like run:npm ci or other actions, like checking out a specific branch
Work : It is a set of steps that run-in runner of our process. The works can be executed independently or sequentially depending on whether the success of our work depends on the previous one.
Workflow : This is what we'll be creating as our end goal - a workflow. It is an automated procedure composed of one or more jobs that is added to a repository and can be activated by an event. They are defined by YAML files and with it you can build, test, package, relay or deploy a project.
Event : These are specific activities that trigger the execution of a workflow. For instance, committing to a specific branch, a new PR and so on.
Action : Smallest building block of a workflow and can be combined as steps to create a job.
Runner : It is a machine with the GitHub Actions application already installed, whose function is to wait for the work to be available and then be able to execute the actions and report the progress and the results.

Introduction to Netlify#

Netlify is a platform that allows you to host and deploy frontend applications. It is an extremely popular tool for newbies because it takes no more than a few clicks to deploy an application code on Github direct to Netlify.

Sample application to set up workflow#

We'll be setting up the workflow using a simple React application - https://github.com/dkp1903/react-github-actions. You need not clone it to your local, since we will not be making any code changes to the app. Instead, you'll have to fork the repo to your own github using the fork button available on the top right corner.

Once it's done, go to the Actions tab on Github, and select the NodeJS workflow.

It'll create a file called node.js.yml with some prewritten configuration, like this :

# This workflow will do a clean install of node dependencies, cache/restore them, build the source code and run tests across different versions of node
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions

name: Node.js CI

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [12.x, 14.x, 16.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js $NaN
      uses: actions/setup-node@v2
      with:
        node-version: $NaN
        cache: 'npm'
    - run: npm ci
    - run: npm run build --if-present
    - run: npm test

The 'on' field describes when this particular workflow will be run. Right now, it's set to all pushes to and pull requests on, the main branch.

'runs-on' desribes the environment the code will be run on, on Github servers. It's Ubuntu 20, and we'll leave it at that.

The node versions to be checked against are 12, 14 and 16, so we'll have three different jobs running parallely when the workflow gets triggered. We'll leave this one as it as well.

The 'run' fields signify the commands to be run, first the npm ci(clean install). We'll change that to npm i for ease of understanding.

Then the npm run build with the --if-present flag, which means that the build script will run only if the build script is present. Fortunately, our app does have a build script, so we'll leave this as wlel

Finally, the npm test command will run the test file we have(App.test.js) which contains just a single test.

Click on the start commit button on the top right. Once you do, the workflow will automatically be triggered.

There will be three jobs running in parallel, one each for Node versions 12, 14 and 16. The jobs will all be successful in a few minutes.

Open the build 12, and look at the steps that were followed. If you open the npm test one, it will see that there's one test, which had passed.

We'll soon mess with that.

We've set up the CI part of the CI/CD pipeline. Now, for the CD, we'll connect our repository to Netlify, where we'll host our react application code.

Go to netlify.com and sign up using Github, using the same github account as your react-github-actions repo is on.

Now, click on New site. Select provider as Github. Search for the react-github-actions repo and add it.

You'll be asked for some details.

Changed the build command to npm run build

Click on Deploy site.

Once you do, the deploy will be auto triggered.

So, now that we've everything running smoothly, we'll make a mess of things by introducing a breaking change, because, things don't work at one go in software development

Go to Github, and edit the App.js file by removing the words 'React App'.

Add the commit message as 'added-breaking-change', and instead of pushing directly to the master branch, click on create a new branch. You may name it anything.

Now, we'll make a PR into the master branch.

If you remember, we'd configured the CI pipeline to work in two cases : One, if a push was made to master, and two, if there was a PR raised.

So this time, as soon as we raise the PR, the workflow should be triggered.

Sure enough, you'll see the steps being run.

We see that the checks fail.

If you go to the Actions tab and check the CI logs, you'll see that the test failed, which is what we'd expected.

Go to Netlify and confirm that no deployment has started.

Now, add the 'React App' back into the file and make a commit into the same branch.

The tests will now run again, and you'll see that they pass now. Once the tests pass, you can merge the Pull request

And going to Netlify, you'll see that a deploy has been triggered

Conclusion#

Thus, you understood the concepts of CI/CD, how they work in a production environment. You set up an application configured CI on it using Github actions and CD using Netlify. You confirmed the flow by purposely failing the CI test and ensured that an incorrect deployment did not get triggered.

References#

(CI vs CD vs CD)[https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment]

Optimal timeblocking

2021-11-26T00:00:00Z

Time blocking has been much acclaimed as a wonderful means to get things done by not giving our mind an alternative - if it's there on the calendar, you do it. Come. What. May.

The premise is this - instead of dumping a list of tasks on the Todo list and getting to them when you 'feel like it', if you instead assign a time you'll do each task, you'll not give your brain a chance to procrastinate.

It takes various forms - Elon Musk plans his entire day in five minute chunks, with not a minute of his waking day left unscheduled, whereas many others only schedule the most important and unmissable events and tasks, and handle the rest on a 'will be taken up as possible' basis

In such a case, how do we make sure that we block times 'optimally', so that we get the tasks done, and at the same time, keep enough leeway for interruptions?

Here are a few steps you can take to ensure you're timeblocking 'optimally' :

Block incrementally. A large number of us, in an initial spur of motivation, end up blocking the day down to the minute, only for the entire schedule to unravel at the first overrun task, or the first distraction. To steel your brain into following a calendar is a challenge, and it takes time to get used to it. Thus, start with blocking unavoidable meetings/events/tasks - these have the highest likelihood of not being procrastinated. Once your brain gets into the habit of checking your calendar before picking up a task, then start adding further tasks slowly - start with the ones you're most likely to procrastinate on, and then go to the relatively easy ones. The moment you feel an urge to NOT do a task inspite of it being on the calendar, take a day's break, without adding any further tasks, until you can steel yourself to stick to it. Reason being - the mind should see the calendar as sacred and unmodifiable. If you start pushing around tasks, you'll start doing that with every task on there, in no time.
Block time, for not just work, but also 'non work' : 'Spending time with family' may not figure on many of our todolists, however, it does take time. Block time for stuff that's not a direct task/meeting, but is anyway gonna take time - otherwise, it'll feel like you had an empty calendar, and still got nothing done.
Optimal duration : How much time should you give to a task? On the one hand, there's an idea that says that work expands to take up as much time as you allot to it, but it misses the fine print that there's definite upper and lower limits. You can't cook dinner in 2.5 minutes, no matter how motivated you are. Giving too little of time to a task will make you feel demotivated at being unable to meet the deadline. And at the same time, giving way too much time to a task will make you procrastinate - the very thing we're trying to avoid. Thus, spend a few extra seconds planning the optimal duration for each slot you block on the calendar. As a rule of thumb, always plan a few more minutes for a task than you think you'll need, since humans have a tendency to overestimate themselves and underestimate the challenges. If the task/meeting involves other people, make sure you finalize the agenda and the duration in advance, since it can otherwise derail very easily.
Padding : Add a few minutes of padding after a task - say 15 minutes for every one hour. This is to ensure that you can take a break before getting on with the next task. This break is necessary, because no matter how motivated, humans' attention span for deep work is low, and needs constant replenishment. Moreover, you can utilize this
Scheduling breaks : No, all the white space on the calendar is NOT a break. You MUST schedule break times on your calendar, wherein you can actually rejuvenate. And it doesn't mean scrolling socials. Your mind needs a break, your eyes need a break, and your body needs movement - so give it that.
Rescheduling : No, you can't work without having to reschedule at least once a week. But at the same time, 'not feeling like it' isn't a valid excuse for pushing a task to the next day. Rescheduling has to follow the same discipline that you followed when scheduling, or you'll eventually end up rescheduling all the tasks you don't want to do. Rescheduling has to follow a careful evaluation process;

one, reschedule a task only if circumstances entirely out of your control come in and threaten to take up more than 50% of the time you initially allotted for the task. Otherwise, just push the task a bit and see it through.
Second, reschedule the task to a time that you KNOW you'll be able to do it at. Pushing a task away to a random slot just to get it out of the way from the moment means that you're going to have to reschedule the task at least once more, and that'll kill off the motivation you have for doing it.
Three, if you have to reschedule the same task more than twice, reevaluate it - is it really unavoidable circumstances, or are you just finding excuses to delay the inevitable?
Finally, If you end up with a lot of rescheduling done over the week, your scheduling wasn't good enough to begin with - so rethink your scheduling strategy.

Flexibility : This may seem counterintuitive, because the tone of this article has been to force your mind. However, flexibility does not mean to reschedule and reprioritize tasks at will. Instead, it's the freedom to reevaluate your scheduling strategies based on the insights you derive from your present schedule. For instance, you observe over a week that 9 PM - 10 PM is a super productive time for you, but your calendar is filled with relatively unimportant tasks in that slot - change it in your next schedule. If you observe your tasks often overshoot, reevaluate how you estimate the time block for each task.

Timeblocking is an effective way to avoid procrastinating on necessary tasks by leaving choice out of the equation, and if done the right way, can greatly boost net productivity

Lessons learnt from a year long experiment on productivity

2022-01-15T00:00:00Z

For over the past 52 weeks, I've invested in learning more about productivity patterns, and how I could possibly meet the goals I set for myself, get over my ADHD and do a decent job at work, without burning myself out in the process.

This article reflects my major findings - all of which I've tried and tested on myself.

1. Action is the greatest motivation#

10 minutes of actually working on a task that's important for you creates a motivation boost to complete the rest of it, far more than any planning ever will. This is how most habits begin to develop - we start at 1%, and the action becomes the motivation for further action. Thus, next time you procrastinate, just get started on one tiny bit of task, and it'll boost you to continue.

2. Conservative time blocking#

Timeblocking is a well recognized technique where you schedule time for a task, and in that duration, just work on that task, thereby eliminating the need for the mind to will itself into picking a task off the todo list.

However, if done wrongly, timeblocking can end up being as unproductive as all other todo lists and way more demoralizing. Timeblocking every minute of the day without considering your energy levels and other distractions can make the exercise futile. Thus, when you start, block time only for the most essential tasks that you dare not skip. Once you get into the habit, only then schedule your time more. This incremental approach will trick your brain into believing, that if it's on the calendar, it's sacred, and can NOT be missed, come what may.

3. Part-day planning#

Most productivity gurus talk about planning one day ahead. However, early on in our careers, we have very little control over our time at work, and thus, our day plans can get disrupted if we're pushed into an energy draining task that we hadn't expected. You have considerable more control over only the next 6 hours of the day, at any given point. Thus, divide your day into 3 halves, and only plan for the next 6 hours. At 8 AM, plan for your 8-1. At 1:30, plan for your 2-7 PM, and at 7:30 PM, plan for your 8 PM - 1 AM. You'll be able to gauge your energy levels and calendar blockers much better this way

4. Biological Prime Time#

As the name suggests, it refers to a few times of the day when you're at your highest energy. Schedule your most energy draining tasks around your BPT to increase your chances of getting them done.

How do you find out your BPT? For one/two weeks, keep track of how motivated and mentally fresh you feel at every hour of the day. After the duration, you'll see a recurring high at some common time slots. For me, it usually comes between 7 AM - 9 AM in the morning, 5 PM - 7 PM in the afternoon, and 9:30 - 10:30 PM at night.

At your lowest energy levels, either switch off entirely from doing anything, or if that's unavoidable(like, if it falls during work hours), do your least energy consuming, 'maintenance' tasks like checking mail, cleaning up your workspace etc, which require very little mental presence.

5. Objectivize#

A lot of what we do is based on subjective decisions and considerations - I'll consider this website complete, when I feel it's 'good enough', or this task will take 'some time', or I have a goal to 'get 6 pack abs eventually'.

These subjective connotations means that your mind has to work at 'interpreting' what they mean, before you actually do something about them - your mind has to define when you feel good enough for your website, or how much time is 'some time', or what you should do at this moment to 'get 6 pack abs eventually'

Instead, creating objective, measurable checklists for your tasks and milestones makes it infinitely easier for you to track them, and remove the barrier for your brain to expend energy every time into defining the criteria for completion. You can just kick into auto gear mode. In the above examples, 'I'll consider this website done, once the header has a gradient, I have done the three body sections and added 5 links in the footer', 'completing these 3 checkpoints in the task will take a total of 90 mins', 'I'll do 60 situps and 80 leg rotations every alternate day'.

Intro to Terraform

2021-11-23T00:00:00Z

Overview#

Terraform is an Infrastructure as Code (IaC) tool, used to deploy and manage infrastructure (cloud servers, cloud DB instances etc) using code, rather than a GUI. In this tutorial, we'll look at what IaC is, why it's thought to be a better idea than using a GUI, and how Terraform achieves it. We'll then implement a rather unique project, to create a Spotify playlist using Terraform!

We'll use an AWS EC2 instance for the tutorial because it's much faster and straightforward than to fight with our miserly personal laptop RAMs. Instructions to set up an AWS EC2 instance can be found here

Introduction to IaC#

Infrastructure refers to everything that's used in the deployment of an application - including the server configurations, load balancers, access groups, VPCs and a zillion other things. As beginners, we often use GUIs to configure this infrastructure - such as the AWS EC2 setup you'd have done - configuring the security groups, storage etc by clicking away at the console.

This practice however, is often not optimal when you're working with hundreds of instances which need very precise configurations, and are worked on by hundreds of developers. In this case, we take a trick off the old hat - just like how our normal application code changes are managed and maintained using version control, we use code to configure our infrastructure, and deploy that configuration to version control so that other developers can see it, edit it and use it.

How exactly does that work? The configurations that we do work via APIs that modify and manage the resources and infrastructure. When we use a GUI like the EC2 dashboard, it's the UI that's making the calls to the APIs for modifying the infrastructure. The same APIs can also be accessed via code, to give the same result. And that's precisely what IaC is.

Intro to Terraform#

Terraform is the tool to bring IaC to reality. It has a configuration language using which you can interact with the infrastructure platform APIs, like AWS EC2 APIs, to add, update and remove resources.

These configuration files can be deployed to version control, meaning that other developers on the team can refer to these or update them as required, without the intervention of the person who first set it up.

So how exactly does it all come together in practice?

1. Making configuration edits#

The developers first make the changes to the infrastructure in the configuration language

2. Execution plans#

Terraform then generates an execution plan based on the configuration changes you made, and asks you for your approval to ensure there're no unexpected changes. You wouldn't want a semi colon removed by the ill famed intern to bring down your primary server, would you?

3. Resource graph#

Infrastructure takes time to setup and configure, especially when there's tons of it involved with specifics in each. Thus, it creates a resource graph to allow it to build and provision independent resources in parallel to save time

4. Change automation#

When you make changes to your infrastructure, Terraform applies those changes with as much efficiency as possible, and with minimal human intervention required.

Now that we're clear with the concepts, let's get out hands dirty by setting up Terraform

Setting up Terraform#

As discussed, we'll be using an EC2 instance to set up and configure Terraform and the other necessary dependencies, since it has a much better RAM and doesn't heat your laptop to 10 million degrees.

You can follow the instructions in the article linked in the overview for setting up an EC2 instance. If not, you can continue on your personal laptops. OS wise instructions can be found here

Once you're logged into the EC2 terminal, we'd first need a few packages that Terraform uses. Execute the following commands on the terminal

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -

Finally, to install terraform (We do the apt-get update to update the repository we installed in the previous step):

sudo apt-get update && sudo apt-get install terraform

Once complete, type terraform -help and a list of options as below will indicate that the installation has been successful.

Setting up Docker Engine#

Now that we're complete with installing Terraform, the next step is to set up Docker Engine, since we'll be using a Docker image for our project.

First, we update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt-get update

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

Next, we add Docker's official GPG key. GPG stands for GNU Privacy Guard and is essentially an encryption mechanism to keep your docker images and installations secure.

 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

Next, we add the stable repository

 echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Now, we'll install the docker engine

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

In case you're wondering, apt-get update downloads the package lists from the repositories and "updates" them to get information on the newest versions of packages and their dependencies.

Finally, to ensure if Docker has been installed successfully, run the following hello world image.

sudo docker run hello-world

Configuring Spotify#

Next, we'll set up Spotify developer dashboard. Go to https://developer.spotify.com/dashboard and login/signup. Once you do, you should see a dashboard like this :

Click the Create an App button, and enter details like so :

and click create

Once the application is created, click the green Edit Settings button on the top right side.

Go to the redirect_uris section and add a URL - http://localhost:27228/spotify_callback. Click on add and then save at the bottom. Do not forget to save - it can be easily missed.

This URL is what we'll be redirected to, once we're authenticated by Spotify, to have rights to create the playlist.

One question you might have - we're using an EC2 instance for the Terraform setup. Why did we add a localhost link there? We'll come to that answer in a bit.

Now, since we're dealing with a port that's expected to have some traffic, we'll need to add it to inbound rules of our AWS security group for our instance to avoid getting a failed request. If you don't know how, follow the instructions here

Now, we'll have to add the redirect URL as an environment variable to our EC2 instance. Go to the terminal and enter the following

export SPOTIFY_CLIENT_REDIRECT_URI=http://localhost:27228/spotify_callback

Next, we'll create a .env file to host our Spotify app credentials.

Type

nano .env

to create a .env file and open it in the nano text editor.

We'll be adding two variables, the client ID and the client secret :

SPOTIFY_CLIENT_ID=
SPOTIFY_CLIENT_SECRET=

For these values, go to the Spotify developer dashboard and copy the client ID and secret and paste them here

And now, the moment of truth - we'll use the docker image of the application to run it and see if we're able to authenticate ourselves. In the terminal, enter the following command

docker run --rm -it -p 27228:27228 --env-file ./.env ghcr.io/conradludgate/spotify-auth-proxy

You shoul see an output like this

Unable to find image 'ghcr.io/conradludgate/spotify-auth-proxy:latest' locally
latest: Pulling from conradludgate/spotify-auth-proxy
5843afab3874: Pull complete
b244520335f6: Pull complete
Digest: sha256:c738f59a734ac17812aae5032cfc6f799e03c1f09d9146edb9c2836bc589f3dc
Status: Downloaded newer image for ghcr.io/conradludgate/spotify-auth-proxy:latest
APIKey: xxxxxx...
Token:  xxxxxx...
Auth:   http://localhost:27228/authorize?token=xxxxxx...

Copy the http://localhost url and paste it in a new browser tab.

Well?

Did you get a 'Site can't be reached' page? Of course you did. Wonder why?

Your server is running on EC2, not on localhost, as we'd noted earlier. So, in the URL, replace the localhost with the Public IPv4 address of your EC2 instance. Once you do that, the page would load into an authorization page like this :

Click on agree, and you'll be redirected to the localhost link you'd given as the redirect url. Again, replace the localhost with the ip of the server, and you'll be able to see this message :

And your terminal will be updated as follows :

Keep the server up and running. Open a new terminal and ssh into it once again to connect to the EC2 instance - we need this one for setting up the terraform configuration.

Now, we'll be working on the terraform configuration we'd need for our app. Use this command to clone a repo that contains the Tf configuration that searches for songs by Dolly Parton and creates a playlist out of them.

git clone https://github.com/hashicorp/learn-terraform-spotify.git

And cd into the directory

cd learn-terraform-spotify

Do an ls command, and you'll see three files in the repo

Enter cat main.tf to open the file. The content will be something like this

terraform {
  required_providers {
    spotify = {
      version = "~> 0.1.5"
      source  = "conradludgate/spotify"
    }
  }
}

variable "spotify_api_key" {
  type = string
}

provider "spotify" {
  api_key = var.spotify_api_key
}

resource "spotify_playlist" "playlist" {
  name        = "Terraform Summer Playlist"
  description = "This playlist was created by Terraform"
  public      = true

  tracks = [
    data.spotify_search_track.by_artist.tracks[0].id,
    data.spotify_search_track.by_artist.tracks[1].id,
    data.spotify_search_track.by_artist.tracks[2].id,
  ]
}

data "spotify_search_track" "by_artist" {
  artists = ["Dolly Parton"]
  #  album = "Jolene"
  #  name  = "Early Morning Breeze"
}

output "tracks" {
  value = data.spotify_search_track.by_artist.tracks
}

The first terraform block contains the terraform configuration, followed by the provider details. Here, we'll enter the spotify API key, which'll allow us to access the developer account and add the song details

Then come the details of the playlist itself - we search the artist Dolly Parton, and (commented out) the album and name of the song.

Next, Rename the terraform.tfvars.example file terraform.tfvars so that Terraform can detect the file, using the following command :

mv terraform.tfvars.example terraform.tfvars

Next, open the above file using nano and add the API key which you'd copied earlier from the running Docker container. Remember to keep the quotes there.

Next, we'll initialize terraform, which will install the Spotify provider, using the following command :

terraform init

Now, enter

terraform apply

to apply the configuration you have made. You'll see a confirmation with the details you've entered, like so :

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # spotify_playlist.playlist will be created
  + resource "spotify_playlist" "playlist" {
      + description = "This playlist was created by Terraform"
      + id          = (known after apply)
      + name        = "Terraform Summer Playlist"
      + public      = true
      + snapshot_id = (known after apply)
      + tracks      = [
          + "2SpEHTbUuebeLkgs9QB7Ue",
          + "4w3tQBXhn5345eUXDGBWZG",
          + "6dnco8haegnJYtylV26cBq",
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + playlist_url = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Enter yes, and the playlist will be created

  Enter a value: yes

spotify_playlist.playlist: Creating...
spotify_playlist.playlist: Creation complete after 1s [id=40bGNifvqzwjO8gHDvhbB3]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

playlist_url = "https://open.spotify.com/playlist/40bGNifvqzwjO8gHDvhbB3"

And there you have it. You can open the link in the browser.

Conclusion#

Thus, in this tutorial, we understood what IaC is, and the use cases for it, and how it's better than the GUI based configuration. We got introduced to Terraform and how it works.

We then set up a Spotify playlist using Terraform, getting a decent overview of how it works in the process.

References#

Terraform Docs

AWS Cloudwatch

2021-12-08T00:00:00Z

Overview#

In this tutorial, we'll understand the requirements of metrics for our server instances. We'll set up an EC2 instance and configure cloudwatch to track metrics for the instance and set up alerts when certain criteria are met

Prerequisites#

You'll need an AWS account. If you do not have one, sign up on aws.amazon.com.

Metrics, and why we need them#

Every server instance we use has a finite amount of load it can hold, the CPU power, the number of reads/writes, and so on. In case of excessive load, the server might crash, leading to service disruption for users. While it doesn't sound like a big deal when working on personal projects, it can have serious business consequences when working with actual users. Remember the time Google went down for just 45 mins? The world practically came to a standstill. To avoid this, we use metrics that track server activity - how many requests it's handling, the CPU being used, and so on. If we see traffic hitting the server's limits, we can configure additional instances and balance load across these

Introduction to AWS Cloudwatch#

AWS Cloudwatch is a service that tracks various metrics for your AWS resources, including EC2 instances, S3 buckets, lambda functions, EBS and more.

You can create dashboards to track the metrics over time and can take intelligent decisions regarding scaling up your server capacity

Most importantly, it also allows to setup alarms when certain critical thresholds are hit, so that you can take action immediately without any service disruption

Getting hands on with Cloudwatch#

In this tutorial, we'll be setting up an EC2 instance and run a simple React app on it. We'll then track the metrics of the instance as we make hits to the server and set up an alarm when the requests cross a certain threshold

The following section describes the steps to set up and run a React app on an EC2 instance. If you already have one running, skip this section and go to the next one.

Setting up a react app on an EC2 instance#

Next, let’s set up a remote EC2 server instance. As said before, you’ll need an AWS account for the same. If you don’t already have one, you’d need to create it. Remember, it’ll ask you for debit/credit card credentials, but as long as you follow the steps in this tutorial, you will not get charged for it.

To set up an AWS account, go to https://aws.amazon.com and follow the steps to set up an account. You’ll get a confirmatory mail once your account is set up and ready.

Once you login to the account, you should see a screen similar to this

Click on the blue ‘Launch a virtual machine’ line, and you’ll be taken to the EC2 setup screen, wherein you’d have to select an AMI, an Amazon Machine Image.

An AMI describes the configuration of the server you’d be using to host your application, including the OS configuration - Linux, Ubuntu, Windows etc. If you have been following tech news, a Mac version was also released for the first time in early 2021.

We’ll be going with Ubuntu server 20.04. You may choose another, but the rest of the steps might vary slightly. Also, do NOT choose an option that doesn’t have the ‘Free tier eligible’ tag, otherwise, you’ll be having to sell off some jewellery to pay the AWS bill.

The next step is choosing an instance type. This describes the server configuration, including CPU, memory, storage, and so on.

Here, we’ll pick the t2.micro instance types, which is the only one available in the free tier. You’ll need larger ones as your application size and requirements in RAM or processing speed increase. In case you’re not clear with any of the column fields, click the information icon next to the headings to get a description of what it means.

Once this is done, click on Next: Configure Instance Details

Here, you’re asked the number of server instances you wish to create and some properties regarding them. We only need one server instance. The rest of the properties are auto filled based on the configuration we selected in earlier steps and/or default values, and thus, should be kept as it is.

Next, click on Add storage

As the name suggests, storage refers to the amount of storage in our server. Note that this isn’t the storage you’d consider for storing databases. This is temporary storage that will last only as long as the instance lasts, and thus, can be used for things like caching. A size of 8GB, that’s part of the free tier, and is the default, suffices our purpose.

Next, we’d be adding a tag for our instance. It is a key:value pair that describes an instance. Since we only have a single instance right now, it is not very useful, but when you are working with multiple instances and instance volumes, as will be the case when the application scales, it is used to group, sort and manage these instances.

Next, we’ll be adding a security group to our instance. A SG is practically a firewall for your instance, restricting the traffic that can come in, what ports it can access, called inbound, and the traffic that can go out, called outbound. There’s further options to restrict the traffic based on IP. For instance, your application will run on port 3000, and thus, that’s a port you’d want all your users to be able to access. Compare that to a Postgres database service running on port 5432. You don’t want anyone else but you meddling with that, so you’ll restrict the IP of that port to only you.

Create a new security group. Next, we have to add the rules for the group, describing what ports are accessible to the outside world, and who they are accessible to. Note that outbound traffic has no restrictions by default, meaning that your application can send a request to anywhere without any restriction from the SG unless you choose to restrict it. As for inbound, we’ll first add HTTP on port 80 and HTTPS on port 443. Next, we’ll add an SSH rule for port 22. SSH stands for Secure Socket Shell and will allow you to connect to your instance, as we’ll soon see in the coming section. Finally, we’ll add a custom TCP rule for the port our application is going to expose - port 3000.

For simplicity, we’ll keep the sources of all of those at ‘anywhere’. Ideally, SSH should be limited only to those you want to allow to connect to your instance, but for the sake of the tutorial, we’ll keep it at anywhere.

Once the rules are set, click on Review and Launch. You’ll be shown the configurations you’ve selected to ensure you didn’t make a mistake anywhere. Once you hit launch, you’ll be asked to create/select a key pair. As the name suggests, it’s a pair of keys - one held by AWS, and the other by you, that acts as a sort of password for you to connect to your instance. Anyone wishing to SSH into this instance must have access to this key file or they won’t be able to.

The content of the file is RSA encrypted, which uniquely determines your access to the instance. Click on create new, give it a name(that you must remember), and download it.

It’s recommended that you download the .pem key file to C:/Users/Home directory on Windows( /home/usr or similar for Linux and Mac), to avoid any access issues.

Once the file is downloaded, you’ll get a prompt that your instance is starting, and after a few minutes, your instance will be started. Your EC2 home page should look like this. Note the Name : Main(tag), the Instance type t2.micro that we selected when we were setting up the instance.

Next, select the instance, and click on Connect on the top bar. It’ll open this page :

This lists a few ways in which you can connect to the instance. Go to the SSH client tab. Now, we’ll be using the terminal to connect to your instance(remote server). For that, open a new terminal as administrator(superuser or sudo for linux), and navigate to the directory where you stored the .pem key file.

First, we’ll run the chmod 400 keyfilename.pem command to allow read permission on that file, and remove all other permissions. Note that if the key file gets overwritten, you’ll lose SSH access to that instance forever, and you’ll have to recreate the instance, since you won’t get the .pem file to download again.

And once you’re done with that, it’s time for the high jump - connecting via a simple command to a remote computer thousands of miles away. The command to run will be on the AWS page as shown above - the ssh -i … one

It means that we’re ssh-ing into the instance defined by the DNS(the .amazonaws.com thing), and proof that we’re authorized to do it, is in the pem file.

It’ll ask a confirmation prompt that you have to type yes to, and if all works well, you should see a welcome to Ubuntu text as shown above, which means that you’re now logged into the instance.

Great going.

Now, our next step is to bring the code into our instance and run it. To do that, we’ll do a git clone exactly the same way we cloned the repo on our local system, using the git clone command.

Once you’re done cloning the repo, the next step is to install the dependencies and start the application. Navigate to the repo directory and try running

npm install

Did you get an error? Ofcourse you did. You need to install NodeJS on the instance. How do you do that? The answer’s in the error itself :

sudo apt install nodejs

This will take a few minutes to complete. Once it’s done, try running npm install again, and you’ll see that this time, you’re able to.

Finally, the moment of truth - run

npm run start

Once you see the application live on localhost:5000 written on the terminal, you’ll have to navigate to the server IP to check if it works.

This IP can be found from the AWS instance details - Public IPV4 address. Copy that, and paste it onto a browser tab, and add :3000 after it.

If the application did work correctly - you should be able to see the same screen that you were able to see locally on your machine.

Setting up cloudwatch#

Now that you have a working application, we'll setup Cloudwatch.

Go to the search bar and type Cloudwatch. You'll get the service option there like so

Click on it, and you'll be taken to the Cloudwatch home page. Look at the navigation tab carefully - it has options for Logs, events, metrics, dashboards and so on.

Click on the Create dashboard button, and give it a name of your choice

Next, you'll be prompted for the widget type you want to add, line graph/cumulative/alarm, etc

We'll pick the line graph. We can always add more widgets later

Next, you'll be asked where should this graph's data come from - the metrics, or the logs? We'll pick metrics since that's what we want to track

Next, you'll get a screen like this, and a list of services which you can track. Click on EC2, and use the select all button to have all EC2 metrics showing up on the widget.

Finally click Create widget, and you'll be able to see the widget on the dashboard

Similarly, you can add another widget for numeric data like so :

Finally, we'll be trying to set up an alarm.

Click on add new widget and select alarm

You'll be redirected to the alarms dashboard

Click on Create alarm

We'll be asked to select the metric on which we want to set an alarm. Search, and select CPUUtilization

You'll then be asked to specify the conditions for the alarm - we'll set the alarm when the CPUUtilization is Greater than 0.6. (It's a pretty low number, but since we wish to see the alarm triggered while not having that amount of utilization, we're keeping it this way)

You'll then be prompted to configure notifications - we choose to get notified 'in alarm', that is, when the threshold has been breached

Next, we are asked to select an SNS topic. SNS stands for Simple notification service, an AWS service used to send alerts to users. We'll click on creating a new topic, and add our email ID in the email endpoint

Click on create topic

Finally, you'll be asked to enter the name of the alarm. And then, the alarm will be created

You'll get a notification on the top stating that you'd need to verify your subscription to the SNS. Go to the email ID you'd entered in the alarm page, and you'll see a mail from AWS with a confirmation link, like this :

If you do not see it, check spam.

Once you hit the confirm link, you're then confirmed to start receiving the notification messages.

Now, go to your SSH terminal,and run the following command, to trigger the CPU usage.

sudo npm i -g pm2

Within a few seconds, you'll see the state of the alarm change to 'In alarm', and you'll have received an email from AWS with the alert

Conclusion#

Thus, in this tutorial, you understood why metrics are important, and how we can use AWS's cloudwatch service to setup and track metrics for your instances. We set up an EC2 instance, and configured cloudwatch to track metrics on it.

You can further expand on this knowledge and track metrics across your projects to drive improvements

References#

AWS Cloudwatch official tutorials

Setting up load balancing using Nginx

2021-11-14T00:00:00Z

This post has been written in collaboration with BacktoBackSWE.com, a portal for interview preparation.

Overview#

In this tutorial, we'd be understanding some core concepts of load balancing - what it is, why we need it using a practical example. We'll then be using setting up three server instances using AWS EC2. We'll then understand what Nginx is, and configure it on the servers so that one of them acts as a load balancer and directs requests to the other two

Prerequisites#

A basic understanding of AWS will be helpful - what is an instance, what is SSH etc. You'll need an AWS account to set up the servers. If you don't have one, you'll have to sign up on https://aws.amazon.com. You'll be asked for Credit/Debit card details, but as long as you stick to the instructions in this tutorial, you won't be charged.

Introduction to load balancing#

Very few things in software engineering sound like they are. Fortunately, load balancing is one of them. Let's consider Uber - an application that sees varying loads in a day based on the time of day - if it's rush hour, the application will be overloaded with requests from the thousands of folks who need to get to their offices on time. In contrast, in the middle of the night, the number of requests will be way lesser.

To handle such scenarios, what does Uber do? They keep multiple servers - each with the same application as their sister server, and all of these sister servers are connected to a main load balancer, not directly to the outside world. Now, when the requests for booking a ride come in, they go to the load balancer, which redirects the requests to any of the sister servers. The LB also keeps track of how many requests are being processed by each server, so that any one server doesn't get overwhelmed and die of exhaustion, while the others sit around swatting flies. This way, the 'load' - the number of requests coming in, gets 'balanced' across the servers, and thus, allows all users to have a smooth experience.

That's the core concept of load balancing.

Introduction to AWS hosting services and EC2#

AWS isn’t something you’re new to, or you won’t be reading this tutorial, but a one liner for it is that it’s a cloud hosting solutions provider by Amazon that allows you to host, manage and scale applications. For the sake of this tutorial, AWS will provide you the remote server where your React app will eventually run. The server itself will be located in some Amazon Data center, but you’d be able to access it remotely from your PC via a set of commands. We’ll be using the EC2 service of AWS. EC2 stands for Elastic Compute Cloud, and it does what we described above - lets you access a remote server and host applications on it

Setting up an AWS EC2 instance#

Next, let’s set up a remote EC2 server instance. You’ll need an AWS account for the same. If you don’t already have one, you’d need to create it. Remember, it’ll ask you for debit/credit card credentials, but as long as you follow the steps in this tutorial, you will not get charged for it.

To set up an AWS account, go to https://aws.amazon.com and follow the steps to set up an account. You’ll get a confirmatory mail once your account is set up and ready.

Once you login to the account, you should see a screen similar to this

Click on the blue ‘Launch a virtual machine’ line, and you’ll be taken to the EC2 setup screen, wherein you’d have to select an AMI, an Amazon Machine Image.

The next step is choosing an instance type. This describes the server configuration, including CPU, memory, storage, and so on.

Once this is done, click on Next: Configure Instance Details

Here, you’re asked the number of server instances you wish to create and some properties regarding them. We'll be going with 3 instances - 2 as server instances, and third a load balancer. They'll be ditto copies of each other for now, until we configure one of them.

Next, click on Add storage

Once the rules are set, click on Review and Launch. You’ll be shown the configurations you’ve selected to ensure you didn’t make a mistake anywhere.

Once you hit launch, you’ll be asked to create/select a key pair. As the name suggests, it’s a pair of keys - one held by AWS, and the other by you, that acts as a sort of password for you to connect to your instance. Anyone wishing to SSH into this instance must have access to this key file or they won’t be able to.

The content of the file is RSA encrypted, which uniquely determines your access to the instance. Click on create new, give it a name(that you must remember), and download it.

It’s recommended that you download the .pem key file to C:/Users/Home directory on Windows( /home/usr or similar for Linux and Mac), to avoid any access issues.

Once the file is downloaded, you’ll get a prompt that your instances are starting, and after a few minutes, they'll be started. Your EC2 home page should look like this (Three running instances. Ignore the fourth terminated one you can see here. It's an old one):

For easier understanding, let's rename our instances. If you hover around their names, you'll see a pencil icon - you can click on it to rename the instances - Server-A, Server-B and Load-Balancer, like so :

Now that our instances are running, we have to connect to each one of them. We'll connect to them via the SSH command line, the terminal. For easy access, we'll stay connected to all three of them via three separate terminals

Select one of the instances, and click on Connect. You'll be taken to another page.

It means that we’re ssh-ing into the instance defined by the DNS(the .amazonaws.com thing), and proof that we’re authorized to do it, is in the pem file.

It’ll ask a confirmation prompt that you have to type yes to, and if all works well, you should see a welcome to Ubuntu text as shown above, which means that you’re now logged into the instance.

Repeat the exact same process for the other two servers in two separate command prompts

If all goes well, you should have the three cmds open, looking like this

Great going.

Now, we'll be installing Nginx onto each of the three servers, to permit us to load balance

Intro to Nginx#

Nginx is a lot of things. Primarily, it's a web server - it takes requests for applications hosted on it, and returns the corresponding files as response to the requests. What does it look like? It's essentially a software that you download and setup on a machine. It has configurations, that once setup, will allow the host machine to accept incoming requests, process them, and send out the outputs.

This request-response ability of Nginx can be put to other uses as well - such as load balancing, reverse proxying, and so on. Load balancing is what we're going to use it for, in this tutorial.

Since Nginx has the ability to accept requests, we can also configure it to accept requests, and based on preset rules, direct those requests to other Nginx servers.

See the reason for the three servers now? Each of those will have nginx set up on them, and thus, all of them can accept incoming requests and return the corresponding responses. We'll configure one of them to work as a load balancer, such that all it does is accept the traffic, and redirect it to either of the two other servers.

Now that we're clear with the theory, let's see how we can set up our servers for the task

Configuring the servers#

Go to the server A command prompt, and type the following command

sudo apt-get update

Once that's done, this command :

sudo apt-get install nginx

Now, go to the EC2 instance dashboard, select Server A, copy its public IPv4 DNS from the details below(remember, copy it - directly opening the URL might lead to unexpected errors) and paste it in a new browser window.

You should see a plain HTML page like so :

Repeat the exact same procedure for servers B and Load Balancer, and ensure that you see the Welcome to Nginx page on the public DNS links for both of these as well.

Next, let's try to edit this page so that we can uniquely identify the server the page is on just by looking at it.

As you might've guessed, the content comes from a simple index.html page that comes with the nginx installation.

In the terminal for server B, we'll go into the directory that houses the index.html page using the following command :

cd /var/www/html

Type

ls -l

to list the files inside the directory, and sure enough, you'll see a file named something like index.nginx-debian.html (The nginx-debian thing refers to the nginx version tells that we have the Debian distribution of nginx downloaded - Debian is a Linux distribution, like Ubuntu and Fedora)

This is the file whose contents we'll have to edit to customize them for the server we're on.

Type

sudo nano index.nginx-debian.html

which will open the file in the Nano editor - a text editor for Ubuntu. And sure enough, you can see the Welcome to nginx content in the file that you are able to see on the public DNS.

Replace the content of the file like this (for server B).:

Once that's done, do a command/Ctrl + X to exit the editor. The terminal will prompt if you want to save it - type Y and hit enter to return back to the terminal.

Repeat the exact same process for Server A.

Configuring load balancer#

Now is the main configuration change - configuring the load balancer to manage the requests going to A and B by routing through it. Based on the ratio we decide, x% of the requests will be going to server A, and the remaining to B.

This configuration is done in the nginx.conf file. To go there

cd /etc/nginx

Then, to open the file

sudo nano nginx.conf

Do NOT forget the sudo since you'd otherwise not be able to save the file after editing it - editing a configuration file requires superuser permission.

You'll see some pre-written content in the file already. Clear all of it, and paste the following content in there :

http {
        upstream myapp
        {
                server <Server_1_Address> weight=1;
                server <Server_2_Address> weight=1;
        }
        
        server {
                listen 80;
                location / {
                        proxy_pass http://myapp
                }
        }
}

And replace the <Server_1_Address> by the Public IPv4 address of Server A, and similarly, for B.

Since we updated the configuration file, we'd need to restart nginx, which we do by this command :

sudo systemctl restart nginx

Note that we didn't have to restart the service after updating the index.html file, since we didn't change any Nginx configuration when we edited the file.

Now, if you go to the public DNS of Load balancer and refresh it - you'll see Server A. Refresh it again - Server B, and this alternates each time.

So, what just happened? And what's all the gobbledygook we wrote in the conf file?

The first http {} reflects the type of requests we'll be accepting - HTTP requests. Upstream means that the requests will be sent FROM the load balancer, to the other servers. What other servers? The servers defined inside that block - defined by their IP addresses. 'myapp' is the name of the group of servres. We then have the server addresses, and weights for each. What do the weights represent? The ratio of the requests - right now, it's 1:1, that's why we see requests going between A and B alternately. You may tweak the weights to see the corresponding changes. In real life, some servers are often larger and can handle more requests, and thus, are allotted more weight.

The server{} block shows the port number the requests should be listened for on(80 - the HTTP port). The remaining line is the most crucial - it essentially says, whenever you encounter the route '/', replace it with http://myapp, aka, our server group. That one line is responsible for directing requests to the respective servers.

Thus, this is how we've successfully setup a load balancing system using three AWS servers

References#

Intro to Server security

2021-11-21T00:00:00Z

This post has been written in collaboration with BacktoBackSWE.com, a portal for interview preparation.

Introduction#

None of us would want an attack on our servers - neither us, who actually own the applications being run on the servers and whose bread and butter depend on the application running smoothly, nor the cloud provider - AWS, Azure, GCP, who actually own the server and whose bread and butter depend on our continuing to use the server.

However, server attacks, breaches and data thefts have been as old as the concept of shared servers itself. And interestingly, the job of ensuring security is equally distributed between the cloud provider, and the user.

Why, you ask. The cloud provider owns the server, and therefore, it ought to be their responsibility to ensure its security, no? If you open a locker in a bank, they don't ask you to provide a security guard for the safety of your locker,do they? That's their responsibility.

However, the bank does ask you to clearly specify the owners and expect you to remember your details every time you wish to visit your locker. If you end up revealing your details to a thief, who can then access your locker, it's not really the bank's fault, is it? Just like that, AWS does have general firewalls and gatekeepers that are usually meant to keep 'unauthorized' requests out. However, if you end up authorizing a client IP, the firewalls and gatekeepers of AWS will have no choice but to let it through. Thus, it's what AWS calls a 'shared responsibility model', with clearly defined areas which AWS secures, and the others, that the user does.

In this tutorial, we'll be looking at some of the options AWS provides us to ensure security of the servers we rent.

1. User's responsibilities#

The facets of security that the user is responsible for, include :

A. Control network access : Control what requests can come to the server, from what sets of IPs can make requests, and how they can do it(ports). The concept of security groups, which we'll also be doing practically, falls here.

B. Credential management : Who all have the credentials to connect to your server, such as the ..pem file, the access to the private IP of the server, and so on.

C. Server OS updates : What security and critical software updates should be allowed onto the server, how frequently, and what trusted sources.

D. IAM roles : IAM stands for Identity and Access Management, and is mainly useful when different people are responsible for different sets of services on AWS. For instance, the you want to restrict EC2 connection to only a select few, but wish to allow RDS access to some other members of the database team - you can configure that with IAM.

We'll be getting a hands on understanding of A and D, as well as understanding the concepts behind some of the other security practices AWS encourages.

2. Security groups#

We start with this topic, as it's the most commonly configured when working with AWS EC2 for new cloud users.

A security group is a firewall for incoming and outgoing traffic for the server. You can configure it to specify the protocols(SSH, TCP, HTTP, HTTPS etc) and corresponding ports you wish to allow traffic on, and what all IPs you wish to allow traffic from. These are called inbound rules. Similarly, there are outbound rules - that define what all you want your server to be able to access. This is usually kept open, since you're mainly bothered with what comes into the server, and not what goes out.

To understand the role of a security group better, we'll be provisioning an EC2 instance, launching an application on it and customizing the security group to allow access to it. If you already have an EC2 instance running, you may skip the next section.

Setting up an AWS EC2 instance#

As said before, you’ll need an AWS account for the same. If you don’t already have one, you’d need to create it. Remember, it’ll ask you for debit/credit card credentials, but as long as you follow the steps in this tutorial, you will not get charged for it.

To set up an AWS account, go to https://aws.amazon.com and follow the steps to set up an account. You’ll get a confirmatory mail once your account is set up and ready.

Once you login to the account, you should see a screen similar to this

Click on the blue ‘Launch a virtual machine’ line, and you’ll be taken to the EC2 setup screen, wherein you’d have to select an AMI, an Amazon Machine Image.

The next step is choosing an instance type. This describes the server configuration, including CPU, memory, storage, and so on.

Here, we’ll pick the t2.micro instance types, which is the only one available in the free tier.

Once this is done, click on Next: Configure Instance Details

Next, click on Add storage

Next comes the security group option. Do not edit anything in there for now. We'll edit it later.

The content of the file is RSA encrypted, which uniquely determines your access to the instance. Click on create new, give it a name(that you must remember), and download it.

It’s recommended that you download the .pem key file to C:/Users/Home directory on Windows( /home/usr or similar for Linux and Mac), to avoid any access issues.

Next, select the instance, and click on Connect on the top bar. It’ll open this page :

It means that we’re ssh-ing into the instance defined by the DNS(the .amazonaws.com thing), and proof that we’re authorized to do it, is in the pem file.

It’ll ask a confirmation prompt that you have to type yes to, and if all works well, you should see a welcome to Ubuntu text as shown above, which means that you’re now logged into the instance.

Great going.

Setting up React application on EC2#

The next step is to set up a sample application on the instance. We'll be using a simple React todolist app for the same. We just have to clone it on the instance, like we'd have done for our local laptop/PC.

git clone https://github.com/gagangaur/React-TODO-App.git

You need not know any React for this, since we're only focusing on the security aspects.

Once it's cloned, we'll have to install npm, and then set up the dependencies for the project

The commands are


sudo apt get update

sudo apt get npm

cd React-TODO-App

npm install

Finally, once the dependencies are installed, we run the application using npm run start, and if you'd followed all the steps perfecty, you should see the app running on port 5000

So now, ideally, you should be able to see the app running on the server, right? We access the instance using the public IPv4 address. Copy it from the EC2 console home and paste it on the address bar, and add a :5000 at the end to indicate the port number. Did the application load?

Unfortunately not.

The reason is - security group. Remember, we hadn't made any change to the default security group settings when setting up the instance. And by default, inbound rules restrict everything but SSH access into port 22 - which we used to connect to the instance using the ssh -i.. command. To be able to access the running application from our browser, we need to allow access to the port the application is running on, 5000, to the outside world.

To do that, go to AWS. n the left navigation pane, scroll down to find the “Network and Security” section, and within it, Security groups. Open it, and select the new security group we’d created when we were setting up the instance(not the default one).

Below, go to the Inbound rules tab, and hit the edit inbound rules button.

Now, put in a custom TCP connection rule for port 5000, and allow access from 'anywhere'. Note, to avoid issues arising due to DHCP(Read this for more info), we're allowing access from anywhere, but you can restrict only specific IPs to access some ports.

Once that’s done, save the rules, and come back to the public IP page, and refresh. If you didn’t mess up, you should be able to see the application loading on port 5000 now!

This is the one of the most important security measures put in place by AWS to ensure you're in charge of what traffic you allow in and out of the server.

IAM - Identity and Access Management#

In professional settings, you'll be working across a large team, with multiple people with different responsibilities. It's neither required, nor safe to grant each of the users complete access to everything on your instances. For instance, folks in the database team have only to deal with the RDS services, and have little use case for the lambda services.

To manage the permissions for users, we use this service called IAM.

Go to https://console.aws.amazon.com/iamv2/home#/home

You should see a screen similar to this. This is the IAM home page.

Select the Users option from the left navigation tab, and it'll show the the existing list of users - empty initially. Let's create a user

In the AWS access type section, select the password option, and add a custom password of your choice - we're doing this for ease of access. Leave the rest as it is, and click the Next:Permissions button

You'll then see an option to add a user to a group. As the name suggests, a user group is a set of users who'll have similar permissions and accesses. Thus, all members of the database team will have similar set of permissions - to view, edit the database. We won't be bothering with a user group creation in this tutorial.

Click on the attach existing policies directly tab

Here, we have to specify the permissions we wish to grant to this user.

We'll add the PowerUserAccess, since we want this user to have complete control of the EC2 instance.

In the set permissions boundary section, leave it unchanged. Click on the Next:Tags button

Add a tag like so

Click on Next:Review, take a scan at all the options you've chosen, and finally hit Create user to see a screen like this

Woohoo! You have successfully created a new user. You can download the csv containing the user details, or mail the access details to the user you wish to assign it to. The user will then be able to access the information she/he is allotted, without you having to share your root AWS account password. Sweet, no?

Network Isolation#

A virtual private cloud (VPC) is a virtual network in your own logically isolated area in the AWS Cloud. Use separate VPCs to isolate infrastructure by workload or organizational entity.

A subnet is a range of IP addresses in a VPC. When you launch an instance, you launch it into a subnet in your VPC. Use subnets to isolate the tiers of your application (for example, web, application, and database) within a single VPC. Use private subnets for your instances if they should not be accessed directly from the internet.

These were some of the major security features that AWS allows us to leverage and customize, to ensure server security.

Setting up a NodeJS service for production

2021-11-16T00:00:00Z

This post has been written in collaboration with BacktoBackSWE.com, a portal for interview preparation.

Table of contents#

Overview
Prerequisites
Introduction to Node Express and MongoDB
Introducing the application we’ll be using
Introduction to AWS and EC2 hosting services
Setting up an AWS EC2 instance
Cloning Node Express app on server
Setting up MongoDB
Testing the application so far.
Setting up additional packages
Setting up monitoring using PM2
Conclusion
References

Overview#

NodeJS is a server side programming framework using JavaScript. Using NodeJS frameworks like Express, you can create backend services quickly and wire them up with the frontend, all in Javascript.

We’ll be using a Node-Express application built along the lines of the Zomock application, with a MongoDB database. You’ll get a basic understanding of a Node-Express application, and some things you need to consider while building for production. You’ll than setup a remote server using AWS EC2, similar to how you’d done in the React tutorial. You’ll than set up MongoDB using MongoDB’s cloud offering called Atlas, and connect your Node-Express app to MongoDB. Finally you’ll run your service using PM2 to keep the application running even after you’ve closed down the SSH connection. We conclude with some additional steps you can yourself choose to add to your project, and finally, leave you with some references for further information.

Let’s jump in

Prerequisites#

You’re expected to have a basic understanding of what Node and Express are and how to write simple NodeJS code to start a server. Here is a sample tutorial in case you’re entirely new. You should have a basic idea of Postman, which we’ll be using to check if our service is working as expected.

Introduction to Node, Express and MongoDB#

NodeJS(or simply Node) is a JavaScript runtime, and a server side framework, meaning that it allows you to create server side applications, and provides you the environment to run them in. Express is a framework based on NodeJS to help you create the endpoints for the application.

MongoDB is a NoSQL database that stores data in the form of documents and collections. It is NoSQL, since it doesn't have tables and doesn't enforce a fixed schema across all documents.

In case you need further brushing up on any of these, take a look at the links in the last section of the tutorial. Note that while we'll not be focusing on the development aspect, and instead will be looking at the deployment, you're still expected to know the basics to be able to understand some of the concepts we'll be using.

Introduction to the application we'll be using#

We'll be using a simple mock Zomato API express application for the tutorial. This API exposes an endpoint to return a list of restaurants with details like rating, cost. You can also add restaurants by making a POST request. The application uses Node and Express for the logic, and MongoDB as a database, which we'll be setting up from scratch in the coming sections.

Introduction to AWS hosting services and EC2#

Setting up an AWS EC2 instance#

To set up an AWS account, go to https://aws.amazon.com and follow the steps to set up an account. You’ll get a confirmatory mail once your account is set up and ready.

Once you login to the account, you should see a screen similar to this

Click on the blue ‘Launch a virtual machine’ line, and you’ll be taken to the EC2 setup screen, wherein you’d have to select an AMI, an Amazon Machine Image.

The next step is choosing an instance type. This describes the server configuration, including CPU, memory, storage, and so on.

Once this is done, click on Next: Configure Instance Details

Next, click on Add storage

The content of the file is RSA encrypted, which uniquely determines your access to the instance. Click on create new, give it a name(that you must remember), and download it.

It’s recommended that you download the .pem key file to C:/Users/Home directory on Windows( /home/usr or similar for Linux and Mac), to avoid any access issues.

Next, select the instance, and click on Connect on the top bar. It’ll open this page :

It means that we’re ssh-ing into the instance defined by the DNS(the .amazonaws.com thing), and proof that we’re authorized to do it, is in the pem file.

It’ll ask a confirmation prompt that you have to type yes to, and if all works well, you should see a welcome to Ubuntu text as shown above, which means that you’re now logged into the instance.

Great going.

Now, our next step is to bring the code into our instance and run it. To do that, we'll clone the repo we're working with, using

git clone https://github.com/dkp1903/zomock.git

Once it's complete, go to the installed folder using

cd zomock

We'll have to create an additional .env file in the repo. What is this file for? Our app has some configurations and credentials that we'd rather keep secret. This includes things like database passwords, connection urls and so on. Thus, we need a file where we can store this, and NOT commit this file to version control. The .env file is the accepted standard.

In our case, we'll be storing two things - one, the PORT number of our application and two, the connection URL to our MongoDB database, which includes a database username and password. For now, we'll start with just the port number, and add the database URL once we set up the database in the next section. To create the env file, type

nano .env

This will open the env file in the Nano text editor.

Add the following line in there :

PORT=5000

To save the file, press Ctrl + X. You'll be prompted if you want to save the changes. Enter Y, and the file will be saved and you'll go back to the CLI.

The next step is to install the dependencies.

npm install

Did you get an error? Ofcourse you did. You need to install NPM on the instance. How do you do that? The answer’s in the error itself :

sudo apt install npm

If you get an error like this, use the command sudo apt get update and then rerun the above command

This will take a few minutes to complete. Once it’s done, try running npm install again, and you’ll see that this time, you’re able to.

In case you see an error like this now, or anytime throughout this project, add a sudo before any command you run(for eg, sudo npm install)

Now, start the application using

npm run start

You should see a line saying Server running on port 5000

Are we done? Not quite. We still haven't set up the database, and thus, we wouldn't be able to do anything at all with the service. Let's resolve that in the next section.

Setting up MongoDB#

We'll be using the MongoDB cloud service called Atlas to create the database that our Node-Express service will be interacting with. One of the great advantages of MongoDB is this cloud service, that you can setup, configure and maintain without having to install anything at all anywhere, something that's not found in existing relational DB systems like Postgres or MySQL.

MongoDB has a free tier option, and that's what we'll be using. Remember, you'll not be prompted to add your billing details anywhere. If you are, that means you did a step wrong.

To get started, go to mongodb.com, and log in/create an account. Follow through the steps to set up your account.

Then, you'll be asked to select a cluster type. Select the free version as shown

Next, you'll be asked to customize your cluster details like hosting zone. Leave everything unchanged, and ensuring that there's no total cost at the bottom, select Create.

It'll take a minute or two for your cluster to get created. Once it's ready, you should see a screen like this.

Carefully take a look at the various details being shown, such as the R W graph - R and W stand for Reads and Writes respectively, which is an important metric for determining the traffic to your DB.

The connections graph shows the number of connections to your DB. A connection is either via an application, as we'll do, or via the command line, and for practical purposes, represents the number of folks modifying/viewing our database.

The in/out graph shows the bytes transferred to/from the database every second.

Data size is the size of the database.

Now, to establish a connection to the database, we need to do a few things first.

Click on connect next to the cluster name, and you'll be prompted to add a connection IP address. This will clarify what all traffic do we want to allow to connect to the database. Remember, in a production application, you dare not give direct database access to anyone and everyone, or you might end losing/leaking thousands of users' data. However, for ease of access, we'll start with the 'Allow access from anywhere' option, since we'll be trying to connect via an EC2 instance, which has a dynamic IP, and thus, you'd have to keep updating the rules every now and then.

Click on Add IP Address

Next, you have to create a database user. You can create any username and password(make sure you remember it).

Next, you'll be asked to choose a connection method - via shell(CLI), Compass(GUI) or via an application, which is the one we'll use. You'll then be asked to pick a driver version, and a connection string. Ensure that the driver is Node.JS and version is 4.0 and later. Copy the connection string.

Now, go to the .env file we'd created on our server instance. Add a line there (no extra spaces, or you might face unexpected errors) :

MONGO_URL=<the-string-you-had-copied>

And replace the username and password with the user's credentials you had created.

Did you see why we did that? We wish to restrict access to the database, and thus, the connection string, which is used to connect to the database, will only be present in a secure local environment and will not be committed with the rest of the code.

With this, you'll finally have added the last requirement to your code. Now, we can run the application using

npm run start

Now, in addition to the 'Server running on port 5000', you should see an additional

'Connected to database' message as well.

If you don't, you need to recheck your connection string.

Testing the application done so far#

Now, we need to test if the application is actually working. Since it's a backend only service without a frontend, we'd need to use an API testing tool. We'll be going with postman.

Go to postman.com. If it's your first time with Postman, there'll be some setup steps.

If we were developing this on our local laptops/PCs, we'd have used a localhost:5000 link. However, since it's on a remote server, we need to find the IP address of the server.

This IP can be found from the AWS instance details - Public IPV4 address.

Paste the IP into the request field on Postman. Add an http:// before the IP and a :5000 after.

Now, if you check the Readme of the repo, hitting the /restaurants endpoint should retrieve a list of restaurants present in the DB. Add a /restaurants after the :5000 and hit send.

If it works well, you should see an empty array [] in the response tab, since there's no data in the database yet. If you get an error like connection refused or request timed out, recheck the IP.

Now, let's try adding some data to the DB. Another look at the readme file will show that making a POST request to the endpoint /restaurants/add will create a restaurant. So update the endpoint, and add the following restaurant data in the body :

 {
        "_id": "6073ccae8bab295faebb5718",
        "name": "Kiran Plaza",
        "rating": "5",
        "image": "https://i.ibb.co/ZTHr2cM/res-sample.jpg",
        "cost": "350",
        "numOfReviews": "4380",
        "discount": "40%",
        "spec": "Chinese",
        "area": "Koramangala"
  }

Now, rerun the get request, and you should see this restaurant being returned.

Setting up additional packages#

Great, so you got it all running on a server. But we’re not done. What happens if you close off the terminal. Try doing just that and see if your get requests still work.

As expected, they won’t. And that doesn’t make sense. For a server to stay up, you need not have to keep a dedicated computer with a terminal on all day - then there’s no point in holding a remote server.

Fortunately, there’s a simple npm package that can keep your service running even when your terminal isn’t. It’s called pm2(most likely short for process monitoring and management). Apart from ensuring that the server remains up, you can use it to check the status of all your node processes running at any time to figure out which of those are causing the issue, logs management, to track the application and see where errors/bugs/incidents if any, occur, and metrics such as memory consumed.

So, we’ll be installing the same on our server and then configuring it to start our node service. Again SSH into the instance using the ssh -i command, go to the project directory, and write

npm i -g pm2

Note the -g flag. It stands for global, meaning that pm2 will be installed as a global package, not just for our project. This is important, because pm2 is expected to handle the restarting of the application even if our project stops, and any project level dependency would not be able to do it.

Once that’s done, we need to start our service using pm2.

The command for that is

pm2 start zomock/index.js 5000 -i max --watch

-i max - allows us to run processes with the max number of threads available. Because NodeJS is single-threaded, using all available cores will maximize the performance of the app.

--watch - allows the app to automatically restart if there are any changes to the directory.

Note that the above command should be run in the root(outside of the zomock directory)

Now, if you close the terminal and make a GET request, you'll see that you're able to still get a response.

Note : Due to an issue with PM2, sometimes the production environment is unable to parse the MongoDB connection string correctly from the .env file. So, in case you get a connection refused issue when making a get request, declare the mongo_url as a const in index.js itself, and use the constant instead of the process.env.MONGO_URL and you should be good to go

Monitoring using pm2#

In production environments, we often need to monitor our deployed code for issues/crashes, so they can be resolved quickly. Fortunately, pm2 can help us with that as well.

Enter the command pm2 monitor on the terminal.

It'll prompt you to sign up for a pm2 account, and once you do, you'll get a URL which holds the metrics dashboard for your application

If you go to that URL in the browser, you'll be able to see metrics of your application like the requests being made, as well as issues and errors. This is extremely advantageous when working with a large number of users

Conclusion#

Thus, in this tutorial, you learnt how to deploy a Node-Express based application onto an EC2 server you'd set up from scratch. You also set up a MongoDB database and connected it to your application. You then ensured that your application continues running even when you close off the terminal running the development process. Finally, you learnt some concepts of monitoring and set up monitoring for your application using PM2

Some of the most major challenges in backend development for production is to track errors and handle them gracefully. You should further research on how to handle exceptions, how to catch errors, log them and ensure that the user has a seamless experience.

References#

Setting up a production ready application with React

2021-11-14T00:00:00Z

This post has been written in collaboration with BacktoBackSWE.com, a portal for interview preparation.

Table of contents :#

Overview
Prerequisite knowledge
Why should you read this tutorial
Introduction to React - building an app for production
Introduction to AWS EC2
Downloading and running the React app source code locally
Creating a build
Setting up and connecting to a remote EC2 instance
Using pm2 to run the app on the instance
Additional pointers on scaling and future references

Overview#

Creating a website on localhost, versus deploying it in a production environment is like comparing a zoo to a forest. There’s way more stuff you need to consider when you’re building for the end user - including scaling, fallbacks, load balancing, security, monitoring, CDNs and so on. In this tutorial, we’ll take our first step into deploying a React application into a production environment and actually seeing it work live, while learning some important concepts that go into ensuring that the app works as expected, in the way. We’ll be using a sample todolist application and deploy it to an AWS EC2 instance. You are free to use the same sample app, or any app of your choice.

Prerequisites#

Since we’re focusing on deploying the application and not creating it, you need not know everything about React. However, you should be aware of the way React works - the concept of virtual dom, how a page is built and populated, and so on. While we’ll be covering some of the concepts in brief in the following section, the react documentation is a good reference point in case you need to refresh on any of the above concepts.

You’ll also need to set up an AWS account. The steps we follow will fall within the free tier offering of AWS, but you’d still need a debit/credit card to sign up. However, as long as you follow all the steps correctly, you won’t be charged.

Why should you read this tutorial#

Developing a web app UI is only the base camp - the rest of the trip to the top of Mt Everest is in deploying it to real users, ensuring that traffic is balanced, that any failure is monitored and handled, and that any security vulnerabilities that might compromise user data are caught and remedied.

This tutorial will focus on deploying a react application on an EC2 instance. Along the way, you’ll learn how a React build gets created and rendered, how we set up and connect to a remote server thousands of miles away via a few terminal commands, the concepts of instances and security groups, and how we can set these up in a few clicks on AWS.

This knowledge will be critical for you to develop applications built for hundreds of users, which is the aim with which most apps are built. Introduction to React - building for production You’ll most likely be aware of what React is and does, it’s a JavaScript library used to create UI components. It uses a Javascript + HTML like syntax called JSX. The HTML bit defines the way the UI looks, and the JS populates data and adds functionality to the application. React is the most popular frontend library these days, given its learning curve is much less steep compared to its competitors like Angular or Vue.

Your first foray into React development would have started with something like npx create-react-app myapp, a command which bootstraps a sample react application and runs it on localhost:3000. However, when you want to let your users use your app, you can’t give them a localhost:3000 link. You need to first ‘build’ the application using npm run build, which creates a directory called build and contains ‘minified’(simplified) CSS, JS and HTML pages and static assets.

If some of the above concepts sound alien to you, do spend some time in understanding how React works under the hood. Some helpful resources are linked in the last section of the tutorial.

Introduction to AWS hosting services and EC2#

Again, AWS isn’t something you’re new to, or you won’t be reading this tutorial, but a one liner for it is that it’s a cloud hosting solutions provider by Amazon that allows you to host, manage and scale applications. For the sake of this tutorial, AWS will provide you the remote server where your React app will eventually run. The server itself will be located in some Amazon Data center, but you’d be able to access it remotely from your PC via a set of commands. We’ll be using the EC2 service of AWS. EC2 stands for Elastic Compute Cloud, and it does what we described above - lets you access a remote server and host applications on it

Downloading and running the React app locally#

The first step is to get hold of the app which we’re going to deploy. As we said earlier, you can perform the deployment steps with any react app of your choice, but if you don’t have one, or are a nerd student and want to follow instructions down to the letter, clone this repo to your local using the following command :

git clone https://github.com/gagangaur/React-TODO-App.git

Next, we install the dependencies and run the application locally. To do that

cd React-TODO-App npm install npm start

This will start the react app on port 3000 - you can check it out by going to http://localhost:3000 on your browser.

Creating a build#

What you saw on localhost:3000 was a development version of the application - which is visible only to you, and as such cannot be displayed to users.

We need to create a build of this - package that we can then use to show the app to the users. Go to the terminal and type

npm run build

Once the command runs, you’ll notice that a build directory has been created in your root folder. Go to the file explorer and open it. You’ll see a list of assets like images, as well as a folder called static. Open it, to further reveal 2 folders - CSS and JS.

What the build command has done, is that it has converted the React code into these CSS and JS files, which’ll now complement the index.html file to load the app.

Now, how do we view this ‘built’ version of our app? We need to ‘serve’ our static files so that they are the ones that open up on localhost:3000, instead of the development version.

To do that, install a package called serve, using

npm i -g serve

Once that’s done, run

serve -s build

Setting up an AWS EC2 instance#

To set up an AWS account, go to https://aws.amazon.com and follow the steps to set up an account. You’ll get a confirmatory mail once your account is set up and ready.

Once you login to the account, you should see a screen similar to this

Click on the blue ‘Launch a virtual machine’ line, and you’ll be taken to the EC2 setup screen, wherein you’d have to select an AMI, an Amazon Machine Image.

The next step is choosing an instance type. This describes the server configuration, including CPU, memory, storage, and so on.

Once this is done, click on Next: Configure Instance Details

Next, click on Add storage

The content of the file is RSA encrypted, which uniquely determines your access to the instance. Click on create new, give it a name(that you must remember), and download it.

It’s recommended that you download the .pem key file to C:/Users/Home directory on Windows( /home/usr or similar for Linux and Mac), to avoid any access issues.

Next, select the instance, and click on Connect on the top bar. It’ll open this page :

It means that we’re ssh-ing into the instance defined by the DNS(the .amazonaws.com thing), and proof that we’re authorized to do it, is in the pem file.

It’ll ask a confirmation prompt that you have to type yes to, and if all works well, you should see a welcome to Ubuntu text as shown above, which means that you’re now logged into the instance.

Great going.

Now, our next step is to bring the code into our instance and run it. To do that, we’ll do a git clone exactly the same way we cloned the repo on our local system, using the git clone command.

Once you’re done cloning the repo, the next step is to install the dependencies and start the application. Navigate to the repo directory and try running

npm install

Did you get an error? Ofcourse you did. You need to install NodeJS on the instance. How do you do that? The answer’s in the error itself :

sudo apt install nodejs

This will take a few minutes to complete. Once it’s done, try running npm install again, and you’ll see that this time, you’re able to.

Finally, the moment of truth - run

npm run start

Once you see the application live on localhost:5000 written on the terminal, you’ll have to navigate to the server IP to check if it works.

This IP can be found from the AWS instance details - Public IPV4 address. Copy that, and paste it onto a browser tab, and add :3000 after it.

If the application did work correctly - you should be able to see the same screen that you were able to see locally on your machine.

As we’d seen above, a simple npm run start gives us the development version. However, this is a production environment we’re running the app on, and we need to ‘build’ the app, using

npm run build

Then, following the same steps as we did above, install the serve package and use the command

serve -s build to serve the build version.

Looks good. Or does it?

Did you notice the port number? 5000. Do you think we’d be able to access it with the security rules we created?

To find out, go to the public IP browser tab and replace the :3000 by :5000.

Oops. Doesn’t work, does it? Wouldn’t it be great if AWS could just ‘guess’ the port number!

Unfortunately, that functionality is still not active, and thus, we need to manually add the port 5000 to be allowed. To do that, go to the instances page. In the left navigation pane, scroll down to find the “Network and Security” section, and within it, Security groups. Open it, and select the new security group we’d created when we were setting up the instance(not the default one).

Below, go to the Inbound rules tab, and hit the edit inbound rules button.

Now, put in a custom TCP connection rule for port 5000, and allow access from? You guessed it - anywhere.

Once that’s done, save the rules, and come back to the public IP page, and refresh. If you didn’t mess up, you should be able to see the application loading on port 5000 now!

Great, so you got it all running on a server. But we’re not done. What happens if you close off the terminal. Try doing just that and see if your website still works.

As expected, it won’t. And that doesn’t make sense. For a server to stay up, you need not have to keep a dedicated computer with a terminal on all day - then there’s no point in holding a remote server.

Fortunately, there’s a simple npm package that can keep your app running even when your terminal isn’t. It’s called pm2(most likely short for process monitoring and management). Apart from ensuring that the server remains up, you can use it to check the status of all your node processes running at any time to figure out which of those are causing the issue, logs management, to track the application and see where errors/bugs/incidents if any, occur, and metrics such as memory consumed.

So, we’ll be installing the same on our server and then configuring it to start our react app. Again SSH into the instance using the ssh -i command, go to the project directory, and write

npm i -g pm2

Once that’s done, we need to start our app using pm2. And remember, we’re looking at the build version,

The command for that is

pm2 serve React-TODO-App/build/ 3000

Note that the above command should be run in the root. If elsewhere, edit the path to the build folder accordingly. And we’ve used the port to be 3000. You may use 5000 as well.

Now, if you close the terminal, you’ll see that the application continues to stay up and running.

Conclusion#

Thus, in this tutorial, we learnt what it means to build a React app for production, how do we create a build locally, and how it works. We then learned how to set up and configure a remote EC2 server, and managed access details. We then set up our repo on the instance, and ran it. Since we wanted the app to continue running even when we closed the terminal, we used the pm2 package for that.

In future blogs, we’ll be looking at how to add load balancers to balance the traffic on our application.

References#

Objectivizing milestones, the Agile way

2021-11-07T00:00:00Z

Determining when a task is 'done', is often harder than doing the task itself, especially when it involves a creative turn of mind like writing a blog post, painting a portrait, and so on. And going by 'satisfaction' also doesn't work - the creators can almost never be perfectly satisfied with their work, and thus, unless you have a specific stop point, your tasks could end up progressing infinitely. When there're other stakeholders involved, like clients, this problem exacerbates - you say you did your best, but the client still doesn't agree.

The makers of Agile anticipated this problem, and incorporated features and principles in their methodology to ensure that milestones, capacities and goals were objectivised. And it's helpful if we can take a leaf out of the agile book to ensure we are able to arrive at tasks' conclusion much faster.

These are 3 steps that we can take to ensure we're more objective :

1. Objectivizing capacity :#

There's a term in the agile methodology for this - velocity estimation. You have a fixed amount of time and mental resources. You estimate the total number of hours in a day you believe you can work productively for your tasks, and only then take up tasks. For instance, let's say you wish to take up a new side project. In a typical 24 hour work day, you have 8 hours of sleeping, 9 hours of work, and 2 hours of chores, which leaves you with 5. That's your capacity for the day, and you know that even operating at your best, you cannot take up tasks that'd take longer than 5 hours.

This objectivization goes beyond the 'I'll find time to do it today' myth, and instead focuses on a real chunk of time you have available, making you feel less overwhelmed and more in control of your time and energy.

2. Objectivizing acceptance criteria/milestones :#

When is a task 'done'? When does it look 'good enough' to go into the 'Done' category? Unless we have a very clear and objective milestone to achieve, we'll never get anything done to satisfaction. In agile, this takes the form of 'acceptance criteria' for features. Before putting a feature into development, the developer and the product team agree to when that feature will be considered complete to avoid back and forth over expectations.

The same can be, and should be, done with our personal tasks, especially those that require a creative turn of mind. When I first wrote this very post, I never knew what I wanted it to look like at the end, even whether I should write 3 steps or 5. After a lot of confusion that lasted days, I created a set of essential points that I wanted the post to cover, and other features like word limit that I wanted it to have. Only then did I start writing it in earnest and sure enough, I was able to hit the acceptance criteria within less than an hour

Objectivizing habits : We all want good habits. But only a fraction of us actually end up keeping up with our aspired habits for long. And the reason isn't always our laziness and lack of consistency. Often times, habits we plan are so subjective that we don't really know of the next course of action that we should take to keep the streak going, and everyday, we first need to think about what we need to do, and when can we consider our habit done for the day.

Instead of this, having a clear set of actionable items that we should do everyday will slowly switch our bodies to autogear and soon enough, the habits will start coming subconsciously.

In Agile, this takes the form of consistent flows in ceremonies - a daily standup is always limited to answering three questions -

What did you do yesterday
What will you do today
Blockers

A retrospective meeting is always defined by what went well in the previous sprint, and what could have been done better. Converting the subjective into these relatively more objective questions ensures that the ceremonies get completed within time and that they actually do what they're meant to do without losing track.

A simple example of an adaptation in our personal lives is our workout routines. Instead of scheduling 'core workout' twice a week, make it '40 situps and 80 leg rotations' with an increase of 4 reps week on week. This objective goal means that your brain doesn't have to worry about thinking what workout it has to do, only the measurable rep count that needs to be met.

These 3 categories of objectivizing our life facets can therefore, greatly alleviate the progress we make on our tasks and goals.

3 free calendar-cum-todolist apps you can use to time-block your day

2021-11-06T00:00:00Z

The market is overflowing today with todo lists, and with calendar applications. With the definition of 'work' being a lot more than just our job, and with multiple things going through our minds at the same time, these apps ensure that we're able to capture tasks and that we see them through when we have enough bandwidth.

Sadly, both the todo list type of apps, and the calendar type of apps, lack completeness individually. You can type in a zillion tasks in your todo list, but unless you have assigned a time to them and follow that calendar religiously, those tasks will stay untouched on the list forever.

On the other hand, if you have a calendar but no way you can check things off or organize them, you wouldn't really be able to track whether an item you'd scheduled is completed or not.

Thus, the optimal solution would be an application that combines these functionalities - a todolist to manage your tasks, and a calendar to block time for doing them. And here we have shortlisted three free solutions that do just this.

1. Routine #

Disclaimer : At the time of writing this, Routine is still in private beta, and thus, it's not accessible to all. However, it holds great potential and is expected to go live very soon, and thus, is part of this list.

Routine allows you to have an inbox of todo items, and a calendar parallely, synced with your Google calendar. You can drag and drop tasks from the inbox into the calendar. Better, you can even move your GCal events and the same will be synced to your calendar - this is a feature not found in many calendar apps. Additionally, Routine also follows a page model like Notion, where every task/event can be opened as a page for notes. It has Intellisense to automatically gauge date/time from the task name, which is another great plus.

Pros :

Drag drop tasks and gcal events and have a two way sync - many other apps don't allow GCal events do be modified by another app
Very accessible - tasks can be added, and the platform can be navigated just via the keyboard
UI and UX are the best among the three apps discussed here - it breathes minimalism and focus.
Easily schedule tasks thanks to Intellisense.

Cons :

Still in private beta, so many features are lacking. Some of these include :

integrations with other apps like Todoist
no Android app
No Reminders
No project system to sort tasks into

2. Kosmotime#

Kosmotime is a time blocking and tracking application. You can add tasks, sort them into projects, and schedule them onto the calendar. This app also syncs with Google calendar to display GCal events onto the Kosmotime calendar, however, you cannot modify the GCal events. One unique feature about Kosmotime is the concept of focus blocks, wherein you can block a time for a set of tasks grouped together - a manifestation of the concept of time blocking. You can also track time across each task using the inbuilt timer, however, you'll have to remember to start and stop the timer for the task. I once remember working 72 hours non-stop on a task :P

Pros :

Can group tasks into projects
Can drag and drop tasks onto calendar
Can create focus blocks
Time tracking for tasks
Good UI/UX

Cons :

Google calendar events cannot be modified from Kosmotime calendar
Only has integrations for Slack and Asana
No Android application
No reminder option
Lacks time, date intellisense - you have to manually set the time and date for each task or drag and drop it
No dark mode(Might not be a con for many, but it is for me)

3. Plan #

Among the three, plan offers the richest set of features, (and if I am not wrong) has been around the longest. It also allows you to create tasks, sort them into projects and drag and drop them onto calendar, just like the other two. Additionally, you can view the tasks in list, kanban or timeline views. My favorite feature, though, is that it allows you to edit and drag and drop Google calendar events as well. Moreover, it has a chrome extension which creates a plan homepage, and therein, you can even check off events - something that both the above apps miss. That 'checking off' is like productivity-adrenaline.

Additionally, it has a documents feature for you to create and store documents. It also has a metrics dashboard wherein you can view the time spent on each task/project.

The downsides are that it has a buggy UI/UX and is often not responsive(some of the content gets cropped out of the screen and there's no scroll)

Pros :

Allows you to edit and drag calendar events from the app as well, and even allows you to check off Gcal events in its chrome extension
Has on browser reminders(notifications)
Has a variety of views to visualize tasks - list, kanban, timeline

Cons :

UI is buggy and content often gets cropped out of the picture with no option for scrolling

Thus, these were three calendar-cum-todolist management applications that you can use to block time for various tasks, with their pros and cons.

5 Agile processes you can use to improve your personal productivity

2021-11-05T00:00:00Z

Intro to Agile#

For the uninitiated, Agile started as a methodology to deliver software, improving upon the flaws of the then popular waterfall model. The idea was simple - to speed up, you must adapt to change, incorporate it quickly, and deliver updates incrementally. For instance, before Agile, teams spent months finalizing the expected features, then developed them, then tested them, and just as they were about to deliver, they realized a client requirement had changed rendering much of their months of effort practically wasted. And anyone who has worked for any length of time in the tech/corporate world knows that if there's anything fickle in the world, that's client requirements. And Agile was a means to not resist change, but instead accept it, and tune our processes to fit the inevitable change, by delivering small updates in short chunks of time, while taking feedback and incorporating it in further iterations.

And as teams and companies realized that this was not at all a bad idea and helped them deliver more and therefore, improve their financials, they adapted the practice religiously, making tweaks to the process that could suit larger teams and projects. This led to several different forms of Agile making their way out in the open, each suited to a particular team or objective or process - Scrum, Kanban, SAFe, to name a few.

Today, Agile is adapted by almost all development teams around the globe.

But interestingly, you don't have to be a development team, or even a developer to utilize the power of Agile.

Agile for the individual#

Most of us are agile in some way - we spend 15 mins at the start of the day planning our tasks and time - that's like a daily standup in agile. We often don't go all in on an idea, but instead, take an incremental step by step approach to see if the outcome's worth the effort - another agile mindset. However, creating a formal structure including some of these instinctive habits, and adapting a few formal processes can make us more productive in doing tasks, personal projects, as well as general mundane tasks.

5 agile processes you can use in your daily life#

1. Daily standup

A daily standup typically includes answering the following three questions :

What did you accomplish yesterday?
What are your goals for today?
Any blockers?

In a team, each developer talks about her/his own content on the above three points. The same format can be taken up by you personally as well. You start by reflecting on everything you accomplished yesterday, which not only gives you motivation, but also pick up on where you could do better today. You then check out your goals for today - work, personal, social, all of it, and put it up on a todo list so that you can check them off. Finally, you think about any blockers you have - any work task that requires an input from a teammate, a social obligation that's gonna eat up a couple hours of your time? This is helpful for you to set your expectations for the day.

If you already have a standup for work, it's recommended that you do a quick personal standup before it, so that the blockers you find for yourself can be discussed in the team standup.

Are there tools you can use to help you in this? The simplest option would be a calendar to place your blockers, and a todo list to keep track of tasks. If you're feeling nerdy, Dailybot is a slack bot that you can use to customize questions sent to you over Slack that you can answer

2. Weekly/bi-weekly retrospectives

As the name suggests, you retrospect on the past week(s). What went well, what could be improved?

Sprints are usually 2 weeks long in development teams, and a retrospective happens at the end of each sprint to improve the next sprint.

How can you use it personally? You have a habit that you started. You need to take a reflection to see if it's working, and make tweaks to improve. For instance, when reflecting, you realize that you're more able to absorb the content of a book you want to read more in the morning, than at night, and you therefore update your calendar to block time in the morning instead of the night. Similarly, you take note of some activities that are stopping you from being at your productive peak - social media scrolling, Netflix, Zomato, and plan to reduce them in the next week.

The core tenet of agile is adapting to change, and retrospecting ensures that you're aware of what needs to be changed, and how.

Tools you can use for retrospectives can be as easy as looking across the past two weeks on your calendar/todo list to try and recall where things can be optimized. However, if you want to delve into detail and have the time and will power to, you can use time tracking tools like Kosmotime to track the time spent across various tasks, that'll allow you to determine if the outcome of a task was worth the amount of time you spent on it.

3. Velocity estimation and prioritization

Velocity refers to 'units of work' that can be allotted to a given task. A relatively simpler task has lower number of units, and a complex or a time consuming one will have higher. Each team has a fixed number of units per day they can manage to complete, and thus, this allows quantitative estimation of how many tasks they can do in a given sprint.

Prioritization is as the name suggests - needs no explanation.

The same can be applied in our daily tasks as well - in each retrospective/planning session, we assign points to each task we have to do, and decide how many we can fit in a day/week. For instance, while writing this blog post, I assigned 4 points to this task, which should take me a couple of hours, and this helped me blocking my calendar accordingly.

Similarly, task prioritization is critical, since we always have way more to do than we're capable of, and some things need to be done way more urgently than others. Thus, giving priorities to some tasks and accomplishing them sooner should be part of retrospectives, plannings and standups.

Todo lists like Todoist have a priority flag, which can be used to prioritize and then filter tasks on priority. While velocity is not a direct feature in many todo applications, it's as simple as a number you assign to a task, as long as you have a clear idea in mind/on paper, as to the relation between the unit of work and the time you should block for it.

This exercise might often seem like an overkill - you might think that it's much easier to just do a task right away, than go through this entire process of setting velocities, priorities and what not, and while that is indeed the case sometimes, not all tasks can be just 'done'. For instance, I had to plan this blog well beforehand and allot time for it, so that I could research on the content, finalize the points, leave enough time for proofreading and publishing. I could not just sit up one fine day and shoot through all of this in lesser time. It requires some thought on what tasks can be just 'done right away', and what need planning.

4. Frequent delivery

While it may sound like this doesn't apply for a lot of our daily tasks, well, it does really help on long term projects, to complete small chunks of work rather than hoping for a grand release. For instance, doing a Rangoli for Diwali - plan the design one day, the chalk outline another, the colors the third day, instead of waiting for a day you could do it all.

If you're working with other people/clients, this is even more important since you can get feedback before you've done a lot of work that could potentially go to 'waste', if the requirements or priorities change.

When I was writing a blog for a firm some months back, I decided to go through it all in one sitting, for which I took a week, and on the day of presentation, realized that I had gotten the topic all wrong.

Continuous feedback is another tenet critical to agile, and while your tasks might not usually have other people involved, your own feedback and expectations are important enough to ensure are at par, and thus, try to keep deliveries short and frequent.

Jira has versioning for delivery of software products - version 1.0.1, version 1.0.2 and so on. This can be customized for other tasks as well, including those that aren't clear enough to be defined via versions.

5. Using metrics for assessing output

One thing almost everyone can agree on - a task can almost never be 'good enough' for all stakeholders. You will always find ways to do it better. The client will always have further feature requests. And when working with personal tasks which are often not deadline bound from the start, it's tough to know when to stop and when to keep improving. A subjective 'does it look good' is a very ambiguous and unhelpful milestone to achieve, and does nothing to consider the effort put into the activity. Instead, there should be more objective metrics that can help us analyze how we performed and as Taylor Swift puts it, 'If the high was worth the pain'.

Agile has numerous metrics to track team performance, such as cycle time, lead time, burndown, throughput, business value, but almost none of these should be taken up blindly as a metric for your own tasks or projects, because focusing on a wrong metric can lead the project into an entirely non productive direction.

For instance, judging the growth of a blog by number of articles per month isn't a great idea, if you do not consider the quality of articles that go in there.

Instead, you should research and finalize a metric you'll aim to optimize as you go through a project, and customize it so that an improvement in the metric score actually translates into the project becoming better. In the above example, the number of minutes an average user stays on the blog gives a good idea of whether the blog is doing good from the 'quality' perspective.

The tools for this process will depend on the metric you aim to use - Google Analytics for traffic tracking, time tracking for amount of time spent, Jira/other project management tools for number of features delivered, and so on.

Thus, these were 5 agile processes that you could implement in your daily life and personal projects as well, to optimize your productivity. But before you do that, remember the very core tenet of agile - adapting to change. Priorities change, circumstances change, requirements change. And it's important that you realize this, expect the change and adapt to it

An introduction to static code analysis using Sonar

2021-10-02T00:00:00Z

Good programmers write code for humans first, and computers next

No idea who said that above line, or if anyone said it at all before I stole it off the internet, but damn right it is.

Code changes more often than I change my mind(which is saying something), and it's almost certain that the next change to the code you're writing right now, will be done by someone other than you. In such a case, ensuring that code is readable, maintainable, follows a set of standard practices becomes critical.

In a large organization with a crazy big codebase worked on by multiple teams and developers, the problem is exacerbated - no one really knows who wrote the code they are having to debug, and thus, it does save a lot of WTFs if the code follows coding practices.

So now the question comes - who ensures developers follow standard practices? You can't give all developers a book of rules, and ask them to refer to it before each variable name they type. There is a need for a tool that checks code as the developer types, and points out the issues and the flaws

And this tool, is called Sonar.

Intro to Async Javascript

2021-10-04T00:00:00Z

Most developers who come to JavaScript from Java, which uses threads for Asynchronicity are often left wondering - why the hell can JS not do the same? Or can it? Let's find out.

Asynchronicity is the ability to break the regular flow of control in a script, in order to not let the program stall on blocking operations - calls that take a long time to complete - talk network requests or such.

Asynchronicity is usually achieved by using multiple threads - this is the way Java does it - all the stuff that you don't want your main thread to bother wasting time on, just spawn off a new thread for.

But that means, that Java has the ability to directly create and manage threads - this makes it decidedly more complex, but that's the way it was built.

JavaScript is higher order than Java, and was initially meant to be a scripting language, and not to manage threads. Now, when the use case for it did come, the powers that be had two choices - add new features to the language to make threading allowable from within JavaScript, or find another way for it.

And the far thinking powers behind the language, decided to try something else, rather than complicate JavaScript.

How could you make a language Asynchronous, without actually calling multiple threads?

The first place they looked for, is where JS ran - in the browser, initially. Java, on the other hand, runs on a server.

JavaScript doesn't do every little thing itself - it takes help of several web browser 'APIs', to do things - for instance, there's a timer API, an XHR API, and so on.

JavaScript uses these APIs to defer some logic to the browser, and expects the output when it's done.

That means, that JavaScript could also use the same logic to defer code that it knows would take a long time.

And that's where comes in setTimeout(), the gateway to the world of AsyncJS.

To the uninitiated, setTimeout() is a function used within JavaScript, which looks something like this -

setTimeout(() => alert('Hello'), 1000)

Don't worry if it looks like Gobbledygook, here's a more readable version of the same snippet :

setTimeout(function a(){
    alert('Hello')
    }, 1000)

Better now?

setTimeout takes two params - a function, and a time in milliseconds. The function is what you'd call a callback function, a function that will be CALLED BACK, when the time as provided as the second param, is up. Meaning, after 1000ms, the function a is called.

That's the more widely accepted, slightly inaccurate version.

What goes behind the scenes, is horrifyingly amazing(Go figure:/)

setTimeout is NOT a JavaScript function, first. There goes your belief system, but I am sorry - it is what it is.

It's instead, a facade, for a web browser API being called behind the scenes - the Timer API, and for once, in the godforsaken world of bad naming conventions, does exactly what it sounds like - it's a TIMER.

The execution is interesting - when the JavaScript compiler runs into setTimeout, it just does two things - it first heaves a sigh of relief, and then, throws the whole thing, the callback, and the time, to the web browser API, and forgets all about it. Literally, yes. Forgets.

For JavaScript, that line has finished execution. It can continue on with the rest of the code.

Did you see it happen? Did you see how JavaScript went asynchronous without us having to mess with threads?

That's the beauty.

So, what happens when the compiler throws the stuff to the browser API? Simple, the browser counts till the time given as param is up, and then adds the function to the call stack.

The call stack is where functions go, to get called.

This isn't entirely accurate, but that's something we'll discuss in a coming probe.

Till then, just remember this - JavaScript is NOT an Asynchronous language. It's synchronous by design, but supports Async functionality by cleverly utilizing the environment it runs in - the browser.

Coming in the next probe, what really happens with the callback function, and why it isn't very good.

Your next side project like a pro

2021-10-04T00:00:00Z

A lot of us often complained, especially during our college years, the wide discrepancy between how we did side projects in college, vs how they work in the industry, and I am not referring just to the scale and complexity of the industry project.

Even if a project of the same level was done personally by us, vs it being done in a formal software industry setting, the process would be entirely different - the latter would involve requirements analysis, PRD making, design creating, feature branches, code reviews, automated tests, linting and more.

And this means that even if we do projects on the side, it doesn't give us enough confidence of us being able to do justice to our roles immediately in a software industry setting.

But tell you the truth - it doesn't have to be so. To quite an extent, we can tweak our side project structure to emulate as much of a 'real' software project as to give us a pretty fair idea of what we're up against when we enter the industry.

Here are a few points you should look at/implement when doing a side project :

Requirement analysis/Software Requirements Specification/Product Requirements Document : This is one thing we often ignore or take for granted, simply because we start our project with an idea or a set of features in mind, and we usually try to keep em verbal and limited to those. More often, we often follow tutorials that walk us through the project's code step by step, and we emulate the same thing. These factors mean that we don't spend enough time doing a requirement analysis to analyze the feasibility and priority of each feature, create user stories and so on.

This means that we are only looking at a project from a constrained point of view - and this isn't the case in most software projects. In the industry, we're given a bird's eye view of the project, its expected functionalities and user base, and it's we who have to formalize and structure it into requirements. This allows us to think of the business/user side of things, teach us to prioritize important features so that we're spending less effort on unimportant features, and have a clear set of iterative goals in mind.

Creating a PRD or an SRS is subjective - product managers spend days making a PRD in an industry setting, but you need not do the same. Just understanding the contents of a typical SRS and PRD should give you enough knowledge to create a simple one for your next project within an hour.

The important part is to stick to it.
Sprint planning : We often binge our projects - we go into frenzy mode and do em all working 10-12 hours a day for 3-4 days, then give them up for a week, and repeat the cycle. Moreover, what we do in each cycle is also dependent on our moods, what the tutorial guy is teaching and so on. A more scalable idea would be to plan things beforehand in terms of sprints. A sprint is a period of software engineering with a definitive goal and clearly defined expected outcomes. It can range anything from a couple of days to multiple weeks, based on complexity.

The advantage of this, is that after each sprint, you have a significant chunk of the project ready, matching the requirements that you set at the beginning of the sprint, as well as regularly make tweaks to your priorities and deadlines based on each sprint's review.

Again, this isn't as hard as it seems. You have the final SRS/PRD of the project - you need to break it down to ACHIEVABLE chunks, with deadlines for each set of features.

This can and I recommend, should go into a project management tool - preferably JIRA(the most commonly used in the industry), Trello, Notion or such like. No, 'verbally speaking and remembering' sprint features doesn't work. Writing it down on paper might sound appealing and might be a preferred way for many of us, but it doesn't work that way in the industry.

Design (Frontend AND Backend)

In most of our side projects, especially those we follow tutorials for, we immediately start writing teh code - if it's a backend thing, we start making the DB queries and Schema. If it's frontend, we directly start writing React. This, however, doesn't work in the industry, because first, unlike in side projects, you don't have tutorials you can blatantly emulate, and second, there's a lot of open questions that you need to take a call on - the design system of the project, the theme, the schema design, the flow of the website, and a zillion others. A design is created first, then that design is iterated and improved upon, and the design is then implemented in code.

Frontend design, creating UI mockups of the end product is usually done using tools like Figma or Sketch, but you need not do this if you don't want to spend a lot of time learning these. Instead, you can use an tools like Whimsical or Diagrams.net to create a similar but vastly less complex version of the design - something like a Wireframe, so that you know what components go where, the style guides(color palette, typography, transitions etc) and so on. Note that the design is not set in stone, not even in the software industry, it is iterated upon by the designers, the product managers and the developers based on UX, priorities and complexity of implementation, so do not worry if you think you're gonna tweak your design later on - just make sure that changes go from design to code, not vice versa.

In case of a backend/full stack application, you need to create a rough architecture of what are the different entities involved - the DB, the backend server, and so on so that it forms a coherent pathway for data to flow. This architecture can be as simple as a bunch of boxes for simple projects like Todo applications, but increase in complexity when things like Microservices, messaging queues are involved. Another important aspect you should think of beforehand, is schema design - you should not change your database schema whenever you feel like it - in a production environment, it'll cause disaster if you remove one small field and it breaks the app for a zillion users. Schema design is carefully analyzed and planned based on the required fields, while ensuring ACID compliance(in case of SQL DBs), and creating a pathway for schema changes to be made without affecting existing users.

Project structure

As beginners, a lot of us often tend to not be aware, or care a lot, about file structure in projects. Even the tutorials we follow usually do not enforce this. We might have one single file making API calls, creating the UI across multiple tables and so on. This is extremely unscalable - the second you have to add an extra feature, you will get overwhelmed because different code snips, stuck in the same file will confuse you.

Separation of concerns and modularity are extremely critical when doing a software project, especially if we want it to scale seamlessly. For instance, consider a React project - you could have easily dumped your entire code into the single App.js file. But we don't/shouldn't do that. Instead, a separate folder for each component - each folder contains an index.js file to hold the JS file and a styles.scss file, if required. Additionally, Child components are created as subfolders inside the parent component.

Similarly in case of a Node.js backend project, it is recommended to follow the MVC architectural style. You'll have a separate folder for models, which represents the database schema, another one for controllers, which coordinate between the requests we get from frontend, and the business logic, and the services folder, which holds the business logic and makes the API/Database calls.

This kind of structure ensures that if we want to add a service, we very easily are able to do it by adding a file in the services folder, without touching the logic for the models or controller.

This might seem like an unnecessary exercise when there're a few components, but as applications scale to hundreds, or thousands of features, this organization is what keeps the project manageable.

Static Code Analysis - Linting

Linting is the process of checking for basic structural and syntactical correctness in your code, statically, that is, without actually running it. This includes checking formatting of your code, checking redeclaration of variables, poor error handling and lots more.

This is an automated way of improving what was once done manually - a developer would write some code, another senior developer would review it and suggest these style changes and syntactical error rectifications. However, this hurts developer productivity. Linters are scripts that run through the code, check for these issues, and in most cases, even create a commit after fixing the issues, so that you don't waste time on doing this 'menial' stuff.

We have ESLint for JavaScript, SonarQube for Java, Pylint for Python and so on.

These are commonly used in teams, and can be implemented in pet projects to focus on code quality without spending time on it.

Testing

This is, by far, the most important, and most unheeded piece of the Software Development life cycle. I am yet to hear of a case in my lifetime, where I saw a pet project that was tested. It's a whole different ball game in industry projects, however - every project depends on it NOT failing for something, anything the user might do. Mistaken email formats, hitting the back button anytime, and so on. Not to mention the stuff that can go wrong due to network issues - especially in case of critical applications like payment apps.

Imagine if a user is carrying out a large transaction, the bank's servers catch up and the transaction fails, but the user is anyway shown the money deducted popup - imagine the frustration. Testing ensures that such cases are minimized.

Testing in bigger companies is a process as complex as the development itself, however, you need not follow all stages in your pet projects. You can start with unit testing your code, that is, testing separate modules, files, components, to ensure that that particular component works well in isolation. Frameworks like Angular already provide the test files inbuilt, and you only need to tweak them slightly and run them. In React, you can use a library like React-Testing or Jest. Similarly for Java you have JUnit. This type of testing ensures that there's nothing wrong with your component logic - a mistaken API call, an incorrect query and so on. This however, isn't it.

You need to make sure that your entire application works in a good flow. You have to emulate a user's journey through your app as closely as possible, and account for each case - what if the user makes a mistake with the email, what if he/she presses the back button before the transaction is done, and so on. End to end tests ensure that this flow goes on - using libraries like Cypress or Protractor for React, Angular or Selenium etc for Java.

Regular commits

We usually develop our projects on our local system, and once we're done, we push it to Github, to ensure that we can put up the links in our resume. That however, isn't the right way to use version control, and definitely not the way it works in the industry.

In the industry, version control is used for collaboration between multiple developers who work on various branches/forks, and in case of bugs, track what change was the bug introduced in. There's also a concept of Continuous Integration, which means that code is released in increments, tests are automatically run on it, and that code is integrated into the main central codebase.

All of this is pretty easy to implement in our projects too.

First, set milestones in your project that will determine when an important feature has been implemented or done. For instance, adding the css for a login form, creating the core logic for checkout, and so on. Every time you hit a milestone, you create a new branch, make a commit, and instead of pushing directly to the main branch, you raise a pull request.

Now, you go to Github, and merge it into your main branch. Here an additional step that can be taken up is writing automated tests. Tests that run on each deploy to ensure that it builds well and doesn't break anything. This can be configured using the deployment tools like Netlify. Further details in the deployment section.

The advantage of this process is that this is exactly how things work in the industry - different developers are responsible for different features, so instead of pushing everything on the main branch, they create separate branches and raise pull requests.

Code review

This is a process which is norm, necessary and critical in software development teams. There'll be at least 2 other developers who'll look at the code you wrote, to check if it's structured well, logically sound, follows all requirements. This ensures that more bugs can be caught out at the early stage, before moving to the testing environment.

Code reviews are usually done by experienced developers who have already seen lots of code, and thus, are very clear on the common pitfalls and checkpoints they have to ensure are ticked.

Now, in your pet project, in most cases, you'll be working individually, and that means that you don't have anyone to review your code. Anyone but you, that is. And that's what you have to do. Read through your own code. Try to figure out if there are some pieces that can be optimized, refactor it, add helpful comments.

Reading code is a monotonous exercise, especially when you'd written it yourself, but it's critical to ensuring quality, and something you're gonna spend a lot of time doing in the industry, so do develop this habit.

Once you think your code is as optimal and clean as you can make it, mark it reviewed(Github has a feature for that), and only then merge the PR.

Deployment

So you did a project, which rn, runs on localhost:8080. You know how it works and looks like, but how're you gonna show it to others. You can't expect every potential interviewer to download your source code, install the dependencies and run it to check. Moreover, no software project is ever made to live on localhost - it has to go to what is called a 'production environment' sometime, where actual users use it.

There's various levels at which you can deploy your project, and they differ based on the tech stack of your project. If it's just HTML, CSS, JS, you can directly activate Github pages from your repository settings and the project will immediately be live on .github.io .

In case of frontend projects involving a JS framework/library like React, Angular etc, you have to use a platform like Netlify, Vercel or Heroku. Netlify is the easiest of the lot. All you have to do is connect your github repo to your netlify account, specify the 'build' command, and that's it. You'll get a deployed link within minutes. Vercel is similar.

Note that backend projects like NodeJS have to have their server on all the time, unlike projects like React, which only build once, and then serve an index.html file and run JS on the browser. Thus, Netlify/Vercel won't work for backend projects, where you need your server to remain on to accept requests and send responses. Heroku is a good option to start with in this case. It works similar to Netlify in terms of setting up the project.

These platforms, however, abstract away several complexities of deployment, and these are almost never used in software industry settings. In the industry, we use cloud solutions like AWS EC2, GCP or Azure hosting. These provide us servers where we can store and run our app, and they're guaranteed to stay up more than 99.5% of time. These cloud solutions have a zillion other features such as setting up load balancers, domain mapping and more, all of which are common in the industry.

The only concern is that these platforms have a very limited free tier, and if you're not very careful, you could end up getting an extravagant bill, so when you use these, make sure you follow a decent tutorial, and do not do something without understanding its implications. My record is getting a 4 lakh bill from AWS.

Thus, following these steps will set you off on a journey to do your pet projects in a much more professional, industry-oriented fashion, so that you do not face a lot of trouble when entering the industry.

Optional#

Team/group projects

Almost no project in the industry is done by a single person. Even if there's just one developer, there'll be one designer/product manager/tester alongside. And working in a team is world apart from working individually. You have to understand others' code, designs, priorities and tweak your code and thinking accordingly.

This point is still optional because group projects aren't possible or feasible for everyone. However, if you can, find a group of 2-3 like minded friends and do a project together. Assign features to different members, have regular meetings, review each others' code, and it'll literally be like working in the industry.

Asking questions for a software engineer

2021-09-19T00:00:00Z

Asking questions at the right time, in the right way, to the right person

Almost all freshers will agree that they've been encouraged by their teammates and managers to ask questions. However, this has a fine print : it actually goes, 'ask questions, if you can't find out the answer yourself'

And not because they don't wanna answer, but because

If you keep asking questions you can find out with a few google searches, you're no good as a developer - you need to learn the art of googling.
The managers/teammates usually have tasks of their own and are helping you on the side, which means that more often than not, they won't find enough time to answer your questions.
Answering questions is wayyyy tougher than asking them, especially if you're answering to a noob. Suppose you ask a question that you think is innocuous, and has a one line answer - "What does this imported package do". Now, your teammate's mind races back to what the package is, why it was brought in, why it's used, how it's used, and a few zillion other things, most of which would make no sense to you. So, she/he has to filter these out in a way so that you're able to grasp the essentials without feeling dumb or overwhelmed. That's tricky business.

So, what should you do? How should you 'ask questions'?

First, the 'right way'

Any query you find, first google it. Right away, as it is. Maybe you find a blank google search result - very very rare. You'll find something that can complement your understanding in some way, even if it doesn't give you the complete answer. But you'll at least have some more idea and can ask the question to your teammate in a more refined way so that the tough choices your teammate would face, mentioned in point 3 above can be minimized.
Instead of asking a teammate to explain it all - tell her/him what you've understood and ask her/him to validate/correct you. If you've got it 70% right, the teammate only need explain 30%, saving both of your times. If you've got it entirely wrong, the teammate would know that there's something lacking in your fundamental understanding and correct that first. If you've got it entirely right, you're getting a promotion sooner.
Try asking the teammate for a resource where you can learn more about the question you're asking. That way, the teammate will not be under pressure to explain 'everything' to you, and instead, guide you to a resource, which can help you better.
Make a habit of taking notes of what you ask and their answers. We often overestimate our memories and underestimate all the crap that's gonna take a chunk off our memories, so you best have it in written somewhere so that you can save yourself from your teammates' irritation by asking the same question 20 times.
Next, the right time. If you're an overexcited sorta person, you wanna know the entire architecture and each and every package right on the first bloody day of the job, because you then wanna go and be Napoleon. Or if you're the shy sort, you keep stalling, waiting for the 'right time' until it's too late. Figuring out the right time to ask comes mainly with experience, but a thumb rule is that if it's something that's blocking your progress, ask it right away. If you think the teammate is going to come to this question, give her/him an opportunity to address it. If they skip, then ask. Finally, ensure that the teammate is in the right frame of mind when you ask a question, not when they're debugging a critical prod issue.

Finally, the right person. You could ask the same question to an immediate senior, your manager, and your team lead, and get different answers. You need to figure out which of these would work best in the context in which you're seeking an answer.

For instance, if you're struggling with a syntactical issue, you should most likely reach out to an immediate senior, someone who has the closest interface with the code, since they can give you the quickest answer. If you're looking at understanding a big picture of a project or a feature, someone who's been around longer can help better.