Monday, May 14, 2018

Why Blockchain Technology is the Answer to the World’s Banking System Woes

The banking system is inefficient in its current state as it requires the use of multiple third-party verifications and transferring services in order to complete a transaction.  Blockchain can alleviate the need for these organizations and provide the world with a viable solution to the inherent problems facing the banking community.  Blockchain is transforming the way in which we conduct business globally by offering us the ability to perform transactions securely in a peer-to-peer manner without the need of any middleman.


The current state of the global banking system is shabby at best.  We’ve already witnessed multiple government bailouts to date and it is exactly this type of pompous behavior that spawned the birth of cryptocurrencies nine years prior.  Satoshi, the unknown creator of Bitcoin, was even kind enough to let us know that this was his motivation by leaving a reference to the bailout headlines from the London Times embedded in BTC’s genesis block – The Times 03/Jan/2009 Chancellor on brink of the second bailout for banks.


A Brief History of How Banks Came About
To understand the advantages that blockchain has over the current banking system, you need to understand the history of banking.  Banks evolved out of the need to securely store gold.  The first “banker” was nothing more than a gold depository for wealthy people.

Individuals would drop off their gold and the banker would then issue them a receipt that could be used to purchase items around town.   Eventually, the banker realized that the people never all came for their gold at the same time and so he decided to start lending out other people’s gold at a slight mark up or interest rate.

The people eventually became suspicious of the banker’s quickly expanding wealth and one night the banker was cornered by a furious crowd who accused him of spending their gold.   They forced the banker to take them to his vault and show them that he had everyone’s gold.  He gladly obliged as he not only had all of their gold but he now had the interest he made in profit as well.

After realizing what the banker had done, the wealthy individuals demanded in on the action.  This is why banks now have to pay you a small interest on your holdings as well.  Not much has changed since then in regards to the purpose of banks;  It’s still a third-party organization looking to make a profit off of holding your funds.

A New Day
Today, none of these steps are necessary thanks to blockchain technology.  Technology has advanced and it is time for our banking systems to do the same.  The world cannot continue to bail out fraudulent bankers or run on a fiat-based financial system.  Let’s take a moment to see how blockchain technology could eliminate much of these banking problems in the coming years.

Peer-to-Peer
The words peer-to-peer has no use in the current central banking system that relies on a combination of verifications and monitoring platforms to ensure your funds are sent.  Every time you swipe your debit card there are around thirty separate third-party organizations that must coordinate to complete your transaction.



Your bank must first check your balance, verify that it is you actually spending the funds, notify the account where the funds are going to ensure it is correct, interact with the merchant processing firms involved, interact with VISA or MC, and the list goes on.  This is why you can spend your debit card funds quickly but any refunds can take a week or longer to be processed.

Transparency
Cryptocurrencies eliminate this need for third-party verification.  The transparent nature of blockchain technology makes it perfectly suited for financial services.  An individual can check any wallet on the blockchain to ensure funds are present and once they are sent, they go directly to the other individual involved in the transaction.

Send Any Amount Instantly
Have you ever tried to send a large amount of money internationally?  It is a nightmare that can take days to complete depending on the amount you decide to send.  During this layover, your funds are inaccessible to both you and the other party involved in the transaction.  It is common to wait over a week for a large transaction to complete.

Blockchain technology eliminates the need to wait and regardless of the amount of crypto you are sending, the transfer time is the same.  This is a huge advantage that crypto has over the current banking system.

Imagine that you are a large international company that must send millions of dollars in capital internationally.  Blockchain technology would allow you to accomplish this task with unmatched simplicity.  There would be no additional delays and the funds would be more secure than any other form of money transference currently available.  You also avoid any losses from converting your funds between countries.

Record Keeping
Records stored on a blockchain are immutable and easily traceable.  Nobody can hack this data and it is easy to search for a single transaction.  Integrating a blockchain-based system would allow banks to quickly handle common issues such as identity theft or disputed transactions.     Some banks are starting to explore this option and it can be expected that many more will follow suit in the coming months as the technology continues to see adoption.

Bye-Bye Credit Bureaus
The transparent nature of the blockchain eliminates the need for credit bureaus.  A blockchain-based system would allow for anyone to quickly see their financial history and thereby prove their ability to repay a debt.


Currently, credit companies are only obligated to provide you with a single credit report yearly for free.  These are billion dollar organizations that thrive off of a technology that is no longer needed.  Significant time and capital could be realized by eliminating these wasteful institutions from the banking system.

Increased Security
Blockchain technology is light years ahead of the current banking system in terms of security.  The redundant nature of the protocol ensures that your funds are safe and allows you the ability to store your funds personally.

A Blockchain Future
The advantages of blockchain technology are unmistakable and for the first time in history, a global decentralized economy is a reality.  This is exactly why the central banking system has been so vocal on their opposition towards cryptocurrencies such as Bitcoin.  You can expect to see this aggressive rhetoric continue as the banking system continues to go through their identity crisis but in the end, its just math and efficiency will always win out.

This article by David Hamilton was originally published at CoinCentral.com.

Friday, May 4, 2018

In-depth Look at Verge Cryptocurrency & Platform

Verge is a privacy-focused cryptocurrency that aims to keep transactions anonymous and untraceable while allowing for high throughput and fast confirmation times.

The project is entirely open source and community led. There is no company or foundation behind Verge. In fact, the core team signed Verge’s black paper with only their usernames. The community is committed to privacy, anonymity, and decentralization.


The coin originally began as DogeCoinDark in 2014. In February 2016, wanting to distance themselves from both the Doge meme and the “dark” connotation, DogeCoinDark rebranded to Verge. Over the past two years, the project has set a trajectory toward legitimacy for mass market adoption.

Verge is entering the crowded the race to be the top privacy coin. In this article, we’ll take a look at what privacy measures Verge implements. We’ll also do a deep dive on the technology behind Verge and whether this is a project with potential to rise to the top.

Making User Connections Anonymous
Verge attacks the issue of privacy from the vantage of how a user connects to the network.

The internet we all recognize is fairly straightforward. To send information between computers, you use an Internet Service Provider (ISP) or other middleman to facilitate the message. When you send a message, your ISP can see your unique identifier on the internet – your IP address. Your ISP also needs to know the IP address of the destination computer, so it can route the message.

This is okay for normal internet traffic, but it’s not anonymous. Over time, an ISP learns a lot about the IP addresses you’re contacting. They also know where you’re sending messages from. In many cases, signing up with your ISP associates your identity with your IP address, causing multiple anonymity and privacy issues. Verge uses two approaches – Tor and I2P – to address connection anonymization.

Enter Tor
Tor is a well-known anonymization scheme for IP addresses. The name is an acronym that stands for The Onion Router, because the Tor network wraps your message in multiple layers of encryption. Instead of routing your internet connection through one ISP, Tor bounces the connection between many relay computers on the Tor peer-to-peer network.


This changes the message’s IP address many times, making it difficult to trace back to the original sender. With TOR, no one node knows the whole route a message will take. The message quickly becomes anonymous and untraceable. A directory service identifies the path for connections.

Tor is a peer-to-peer network. As you use Tor, you’re also acting as a relay node for other messages getting bounced around the Tor network.

Verge implements Tor as standard for its transactions to anonymize user connections to the blockchain. Making interactions more difficult to link to an IP address.

I2P
The next generation solution to connection anonymization is I2P. While Tor provides directory-based circuit routing, I2P allows for dynamic routing of information packets. There’s no directory on I2P, so the responsive routing of the network can avoid congestion and interruptions.


I2P also divides the routing into two separate tunnels, one outgoing and another incoming. That means that the messages you send to another computer or website follow a different path from the messages you receive in response. Anyone listening in would only see half of the message history, like listening in on only half of a phone call where you don’t know who is speaking or who they’re speaking to.

Tor was intended as a portal for anonymously accessing the ordinary internet. I2P provides a much more robust experience, leading to the creation of a private network within the internet. I2P is a true darknet, with applications written specifically for I2P.

Verge leverages I2P technology for its network as well. You have the option to route your transactions through Tor or I2P but IP anonymization is standard on Verge. Since the entire Verge blockchain is anonymous, the entire community becomes much more difficult to track.

Wraith Protocol
The Wraith Protocol allows users to choose between public and private blockchain transactions. Public transactions would provide transparency and speed. Private blockchain transactions wouldn’t be publicly reviewable at all.

They plan to accomplish these private transactions using stealth addresses routed through Tor. Stealth addresses send funds to one-time use addresses. Only the recipient can identify and redeem funds sent to a stealth address. Stealth addresses are an important component of how Monero, a leading privacy coin, operates. However, Monero also provides more complex cryptography and other features that guarantee its privacy more effectively.

Verge’s cryptography is based on elliptic curves. Elliptic curve cryptography is well-established and very cool. It’s a key part of Bitcoin, and Verge uses a slight variant of the Bitcoin known as Elliptic-Curve Diffie Hellman. It allows parties to share and agree on transaction keys and signatures without an observer learning anything.

Wallets
Verge utilizes the Electrum wallet, originally designed for Bitcoin. Electrum supports Tor and I2P integration. It also allows for secure offline storage of tokens. When you need to send XVG, you can sign the transaction with your private key offline. Once signed, you can broadcast the transaction from an online computer that doesn’t have access to your private keys.

Electrum also supports passphrase key recovery and multisignature, meaning you could require multiple confirmations to send a transaction, increasing security. Finally, the Electrum wallet connects to decentralized servers that index the blockchain. There’s no need to operate a full node or download the entire blockchain transaction history.

Android Wallets
Verge will also support two Android wallets. One for Tor and another for I2P. These mobile wallets include security measures like PIN codes and biometric locking. They also support QR codes to pull balances from paper wallets.

Messaging
Verge has implemented options for messaging transactions, as well. You can send XVG via Telegram, Discord, Twitter, or IRC. It’s simple to send tokens using only a person’s username. A bot will process the transaction and place the funds in a holding address. It’ll then send a message to the recipient with instructions on how to claim the funds. Verge is not the only cryptocurrency to implement messaging payments, but it represents a big leap forward in user experience from an ease of use standpoint.

Messaging payments on Slack and Steem are coming to XVG later this year.

Mining
Verge is among a small handful of projects that are testing out multi-algorithm consensus. This means miners can mine XVG in five different ways. All of the algorithms are proof of work based. However, some favor ASIC hardware while others are GPU compatible or lighter.

The five algorithms are Scrypt, X17, Lyra2rev2, myr-groestl and blake2s. Digibyte pioneered this multialgorithm approach. The benefit is greater decentralization as multiple algorithms mean many different types of mining rigs can participate in XVG mining.

Verge has a target 30-second block time, split between the five algorithms. In total, there will be 16.5 billion XVG, with 9 billion mined in the first year (2014) and 1 billion every year thereafter.

XVG Coin
XVG, originally DogeCoinDark, launched without an ICO or premine. The developers bought Verge coins just like anyone else.

Verge is currently in the top 30 cryptocurrencies worldwide. It is listed on many major exchanges including Binance and Bittrex.

Future Plans
Verge has several future plans that could make the project more compelling as a complete privacy solution.

Atomic Swaps
Starting in 2018, Verge hopes to implement support for atomic swaps with most major cryptocurrencies. Atomic swaps use hash-locks and time-locks to freeze tokens on one blockchain in exchange for the release of tokens on another chain. Verge hopes interoperability with other chains will make it a go-to privacy provider.

Smart contracts
The Rootstock project plans to add a sidechain to Verge that processes smart contracts. It will be Turing complete and comparable to Ethereum. It hasn’t yet launched, so those claims are unverified as yet.

RSK tokens on Rootstock can be pegged to Verge tokens so they’ll have the same value. You can deposit XVG on Verge and spend corresponding RSK on the Rootstock side chain.

Rootstock claims they’ve made a breakthrough in smart contract scalability. Their goal is 2,000 tx/s using off-chain settlement solutions similar to Lightning Network.

Conclusion
Verge is interesting insofar as it’s a decentralized, open source project. However, its lack of formal structure could also be a drawback. Most serious crypto projects these days have a foundation behind them leading development and setting a roadmap.

The project also needs outside review. While many of the technologies they’re implementing have been tested elsewhere, Verge could use a dose of legitimacy from an independent source.

They also don’t have the same kind of resources as their competitors in the privacy space. Monero, Z Cash, and Dash have hundreds of collaborators on their Githubs. Verge only has 12.

That said, hiding IP addresses is an important frontier for blockchain anonymity. If they can solve anonymous smart contracts, that would be a unique breakthrough for the space.

This article by Bennett Garner was originally published at CoinCentral.com.

Stream a file to AWS S3 using Akka Streams (via Alpakka) in Play Framework

In this blog post we’ll see how a file can be streamed from a client (eg: browser) to Amazon S3 (AWS S3) using Alpakka’s AWS S3 connector. Aplakka provides various Akka Stream connectors, integration patterns and data transformations for integration use cases.
The example in this blog post uses Play Framework to provide a user interface to submit a file from a web page directly to AWS S3 without creating any temporary files (on the storage space) during the process. The file will be streamed to AWS S3 using S3’s multipart upload API.

(To understand this blog post basic knowledge of Play Framework and Akka Streams is required. Also, check out What can Reactive Streams offer EE4J by James Roper and also check its Servlet IO section to fully understand the extent to which the example mentioned in this blog post can be helpful)
Let’s begin by looking at the artifacts used for achieving the task at hand
  1. Scala 2.11.11
  2. Play Framework 2.6.10
  3. Alpakka S3 0.18
Now moving on to the fun part, let’s see what the code base will look like. We’ll first create a class for interacting with AWS S3 using the Alpakka S3 connector, let’s name the class as AwsS3Client.
@Singleton
class AwsS3Client @Inject()(system: ActorSystem, materializer: Materializer) {

  private val awsCredentials = new BasicAWSCredentials("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY")
  private val awsCredentialsProvider = new AWSStaticCredentialsProvider(awsCredentials)
  private val regionProvider =
    new AwsRegionProvider {
      def getRegion: String = "us-west-2"
    }
  private val settings = new S3Settings(MemoryBufferType, None, awsCredentialsProvider, regionProvider, false, None, ListBucketVersion2)
  private val s3Client = new S3Client(settings)(system, materializer)

  def s3Sink(bucketName: String, bucketKey: String): Sink[ByteString, Future[MultipartUploadResult]] =
    s3Client.multipartUpload(bucketName, bucketKey)
}
From the first line it can be seen the class is marked as a Singleton, this is because we do not want multiple instances of this class to be created. From the next line it can be seen that ActorSystem and Materializer is injected which is required for configuring the Alpakka’s AWS S3 client. The next few lines are for configuring an instance of Alpakka’s AWS S3 client which will be used for interfacing with your AWS S3 bucket. Also, in the last section of the class there’s a behavior which returns a Akka Streams Sink, of type Sink[ByteSring, Future[MultipartUploadResult]], this Sink does the job of sending the file stream to AWS S3 bucket using AWS multipart upload API.
In order to make this class workable replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your AWS S3 access key and secret key respectively. And replace us-west-2 with your respective AWS region.
Next, let’s look at how the s3Sink behavior of this call can be used to connect our Play Framework’s controller with AWS S3 multipart upload API. But before doing that and slightly digressing from the example [bear with me, it’s going to build up the example further :)], if you followed my previous blog post — Streaming data from PostgreSQL using Akka Streams and Slick in Play Framework [containing Customer Management example] — you might have seen how a CustomerController was used to build a functionality wherein a Play Framework’s route was available to stream the customer data directly from PostgreSQL into a downloadable CSV file (without the need to buffering data as file on storage space). This blog post builds an example on top of the Customer Management example highlighted in the previous blog post. So, we’re going to use the same CustomerController but modify it a bit in terms of adding a new Play Framework’s Action for accepting the file from the web page.
For simplicity, let’s name the controller Action as upload, this Action is used for accepting a file from a web page via one of the reverse route. Let’s first look at the controller code base and then we’ll discuss about the reverse route.
@Singleton
class CustomerController @Inject()(cc: ControllerComponents, awsS3Client: AwsS3Client)
                                  (implicit ec: ExecutionContext) extends AbstractController(cc) {

  def upload: Action[MultipartFormData[MultipartUploadResult]] =
    Action(parse.multipartFormData(handleFilePartAwsUploadResult)) { request =>
      val maybeUploadResult =
        request.body.file("customers").map {
          case FilePart(key, filename, contentType, multipartUploadResult) =>
            multipartUploadResult
          }
 
      maybeUploadResult.fold(
        InternalServerError("Something went wrong!")
      )(uploadResult =>
        Ok(s"File ${uploadResult.key} upload to bucket ${uploadResult.bucket}")
      )
    }

   private def handleFilePartAwsUploadResult: Multipart.FilePartHandler[MultipartUploadResult] = {
     case FileInfo(partName, filename, contentType) =>
       val accumulator = Accumulator(awsS3Client.s3Sink("test-ocr", filename))

       accumulator map { multipartUploadResult =>
         FilePart(partName, filename, contentType, multipartUploadResult)
       }
   }
}

Dissecting the controller code base, it can be seen that the controller is a singleton and the AwsS3Client class that was created earlier is injected in the controller along with the Play ControllerComponents and ExecutionContext.
Let’s look at the private behavior of the CustomerController first, i.e handleFilePartAwsUploadResult. It can be seen that the return type of this behavior is
Multipart.FilePartHandler[MultipartUploadResult]
which is nothing but a Scala type defined inside Play’s Multipart object:
type FilePartHandler[A] = FileInfo => Accumulator[ByteString, FilePart[A]]
It should be noted here that the example uses multipart/form-data encoding for file upload, so the default multipartFormData parser is used by providing a FilePartHandler of type FilePartHandler[MultipartUploadResult]. The type of FilePartHandler is MultipartUploadResult because Alpakka AWS S3 Sink is of type Sink[ByteString, Future[MultipartUploadResult]] to which the file will be finally sent to.
Looking at this private behavior and understanding what it does, it accepts a case class of type FileInfo, creates an Accumulator from s3Sink and then finally maps the result of the Accumulator to a result of type FilePart.
NOTE: Accumulator is essentially a lightweight wrapper around Akka Sink that gets materialized to a Future. It provides convenient methods for working directly with Future as well as transforming the inputs.
Moving ahead and understanding the upload Action, it looks like any other normal Play Framework Action with the only difference that the request body is being parsed to MultipartFormData and then handled via our custom FilePartHandler, i.e handleFilePartAwsUploadResult, which was discussed earlier.
For connecting everything together, we need to enable an endpoint to facilitate this file upload and a view to be able to submit a file. Let’s add a new reverse route to the Play’s route file:
POST /upload controllers.CustomerController.upload
and a view to enable file upload from the user interface
@import helper._

@()(implicit request: RequestHeader)

@main("Customer Management Portal") {
  <h1><b>Upload Customers to AWS S3</b></h1>
  @helper.form(CSRF(routes.CustomerController.upload()), 'enctype -> "multipart/form-data") {
    <input type="file" name="customers">
    <br>
    <input type="submit">
  }
}
Note the CSRF which is required for the form as it is enabled by default in Play Framework.
The entire code base is available at the following repository playakkastreams.
Hope this helps, shout out your queries in the comment section :)
This article was first published on the Knoldus blog.

Tuesday, May 1, 2018

Using Microsoft SQL Server with Scala Slick

This blog post shows simple CRUD operations on Microsoft SQL Server using Scala Slick version 3.2.3. You might be thinking what’s really great about it? Duh! But until Scala Slick 3.2.x was released, using commercial databases was within the horizon of an additional closed source package know as Slick Extensions which supported Slick drivers for following databases
  1. Oracle
  2. IBM DB2
  3. Microsoft SQL Server


Library dependency used for Slick Extensions
libraryDependencies += "com.typesafe.slick" %% "slick-extensions" % "3.0.0"
But with the newer version of Slick, i.e 3.2.x these drivers are now available within the Slick core package as open source release which can also be seen from the change log as well.
If you find yourself struggling with a setup to make Microsoft SQL Server work with Scala Slick in your project, maybe because of the lack of resources available on the web, then read up further :)

TL;DR

SQL Server database configurations
sqlserver = {
 driver = "slick.jdbc.SQLServerProfile$"
 db {
 host = ${?SQLSERVER_HOST}
 port = ${?SQLSERVER_PORT}
 databaseName = ${?SQLSERVER_DB_NAME}

 url = "jdbc:sqlserver://"${sqlserver.db.host}":"${sqlserver.db.port}";databaseName="${sqlserver.db.databaseName}
 user = ${?SQLSERVER_USERNAME}
 password = ${?SQLSERVER_PASSWORD}
 }
}
Database instance
val dbConfig: DatabaseConfig[JdbcProfile] = DatabaseConfig.forConfig("sqlserver")
val db: JdbcProfile#Backend#Database = dbConfig.db

SBT project setup

For the example used in this blog post following dependencies and versions of respective artifacts are used
  1. Scala 2.11.11
  2. SBT 0.13.17
  3. Slick 3.2.3
  4. HikariCP 3.2.3
  5. Mssql JDBC 6.2.1.jre8
which inside our build.sbt file will look like the following set of instructions
name := "mssql-example"

version := "1.0"

scalaVersion := "2.11.11"

libraryDependencies ++= Seq(
 "com.typesafe.slick" %% "slick" % "3.2.3",
 "com.typesafe.slick" %% "slick-hikaricp" % "3.2.3",
 "com.microsoft.sqlserver" % "mssql-jdbc" % "6.2.1.jre8"
)
and the instructions of build.properties file will be
sbt.version = 0.13.17
The settings required to configure Microsoft SQL Server should go inside application.conffile, whose instructions would be to specify the details of our database

sqlserver = {
 driver = "slick.jdbc.SQLServerProfile$"
 db {
  host = ${?SQLSERVER_HOST}
  port = ${?SQLSERVER_PORT}
  databaseName = ${?SQLSERVER_DB_NAME}

  url = "jdbc:sqlserver://"${sqlserver.db.host}":"${sqlserver.db.port}";databaseName="${sqlserver.db.databaseName}
  user = ${?SQLSERVER_USERNAME}
  password = ${?SQLSERVER_PASSWORD}
 }
}
where it can be seen that SQLSERVER_HOST, SQLSERVER_PORT, SQLSERVER_DB_NAME, SQLSERVER_USERNAME and SQLSERVER_PASSWORD are to be provided as environment variables.
Now moving onto our FRM (Functional Relational Mapping) and repository setup, the following import will be used for MS SQL Server Slick driver’s API
import slick.jdbc.SQLServerProfile.api._
And thereafter the FRM will look same as the rest of the FRM’s delineated on the official Slick documentation. For the example on this blog let’s use the following table structure
CREATE TABLE user_profiles (
 id         INT IDENTITY (1, 1) PRIMARY KEY,
 first_name VARCHAR(100) NOT NULL,
 last_name  VARCHAR(100) NOT NULL
)
whose functional relational mapping will look like this:
class UserProfiles(tag: Tag) extends Table[UserProfile](tag, "user_profiles") {

 def id: Rep[Int] = column[Int]("id", O.PrimaryKey, O.AutoInc)

 def firstName: Rep[String] = column[String]("first_name")

 def lastName: Rep[String] = column[String]("last_name")

 def * : ProvenShape[UserProfile] = (id, firstName, lastName) <>(UserProfile.tupled, UserProfile.unapply) // scalastyle:ignore

}
Moving further up with the CRUD operations, they are fairly straightforward as per the integrated query model provided by Slick, which can be seen from the following UserProfileRepository class
class UserProfileRepository {

 val userProfileQuery: TableQuery[UserProfiles] = TableQuery[UserProfiles]

 def insert(user: UserProfile): Future[Int] =
  db.run(userProfileQuery += user)

 def get(id: Int): Future[Option[UserProfile]] =
  db.run(
   userProfileQuery
    .filter(_.id === id)
    .take(1)
    .result
    .headOption)

 def update(id: Int, firstName: String): Future[Int] =
  db.run(
   userProfileQuery
    .filter(_.id === id)
    .map(_.firstName)
    .update(firstName))

 def delete(id: Int): Future[Int] =
  db.run(userProfileQuery.filter(_.id === id).delete)
Lastly, in order to get the database instance using the configurations provided in application.conf file, the following code snippet can be used
val dbConfig: DatabaseConfig[JdbcProfile] = DatabaseConfig.forConfig("sqlserver")
val db: JdbcProfile#Backend#Database = dbConfig.db
Working codebase of this example is available at the following repository: scala-slick-mssql.
Also, if you’re interested in knowing how data can be directly streamed from PostgreSQL to a client using Akka Stream and Scala Slick then you might find the following article useful: Streaming data from PostgreSQL using Akka Streams and Slick in Play Framework
This blog post has been inspired by an endeavor to make Microsoft SQL Server work with Slick and an answer on StackOverFlow which is the reference of the configurations.
Hope this helps :)
This article was first published on the Knoldus blog.

Streaming data from PostgreSQL using Akka Streams and Slick in Play Framework

In this blog post I’ll try to explain the process wherein you can stream data directly from PostgreSQL database using Scala Slick (which is Scala’s database access/query library) and Akka Streams (which is an implementation of Reactive Streams specification on top of Akka toolkit) in Play Framework. The process is going to be pretty straightforward in terms of implementation where data is read from one of the tables in your SQL database as stream and then it is sent/streamed to one of the REST end point configured to download this data.



For better understanding let’s take an example of an application or service which is used for administering a huge customer base of an organisation/company. The person involved in administering the customer base wants to get the entire data-set of customers for let’s say auditing purpose. Based on requirements it would sometimes make sense to stream this data directly into a downloadable file which is what we are going to do in this blog post.
(For this blog post you should have a basic knowledge of using Play Framework and Slick library)

The example uses following dependencies
  1. Play Framework 2.6.10 (“com.typesafe.play” % “sbt-plugin” % “2.6.10”)
  2. Play-Slick 3.0.1 (“com.typesafe.play” %% “play-slick” % “3.0.1”)
  3. Akka Streams 2.5.8 (“com.typesafe.akka” %% “akka-stream” % “2.5.8”)
  4. PostgreSQL 42.1.4 (“org.postgresql” % “postgresql” % “42.1.4”)
Let’s start by assuming we have a customer table in our PostgreSQL database which has the following structure

CREATE TABLE customers (
  id        BIGSERIAL PRIMARY KEY,
  firstname VARCHAR(255) NOT NULL,
  lastname  VARCHAR(255),
  email     VARCHAR(255)
);
Slick’s functional relational mapping corresponding to this table structure should look like this
case class Customer(id: Long,
                    firstName: String,
                    lastName: String,
                    email: String)

trait CustomerTable extends HasDatabaseConfigProvider[slick.jdbc.JdbcProfile] {
  
  import profile.api._

  val customerQuery: TableQuery[CustomerMapping] = TableQuery[CustomerMapping]

  private[models] class CustomerMapping(tag: Tag) extends Table[Customer](tag, "customers") {

    def id: Rep[Long] = column[Long]("id", O.PrimaryKey, O.AutoInc)

    def firstName: Rep[String] = column[String]("firstname")

    def lastName: Rep[String] = column[String]("lastname")

    def email: Rep[String] = column[String]("email")

    def * : ProvenShape[Customer] = (id, firstName, lastName, email) <>(Customer.tupled, Customer.unapply)

  }

}
Now let’s use the customerQuery to get data from the customers table in the form of DatabasePublisher of type Customer, i.e DatabasePublisher[Customer], which is Slick’s implementation of reactive stream’s Publisher where Publisher is the (potential) unbounded sequence of elements that publishes the elements according to the demand from the Subscriber. We will define this inside CustomerRepository.

def customers: DatabasePublisher[Customer] =
  db.stream(
    customerQuery
      .result
      .withStatementParameters(
         rsType = ResultSetType.ForwardOnly,
         rsConcurrency = ResultSetConcurrency.ReadOnly,
         fetchSize = 10000)
      .transactionally)
Certain things to be noted when using PostgreSQL for streaming data/records, which is also noted in Slick’s Official documentation:
  1. The use of transactionally which enforces the code to run on a single Connection with auto commit set as false [setAutoCommit(false)], by default slick is set to run in auto commit mode.
  2. The use of fetchSize so that the JDBC driver does not fetch all rows to the memory (i.e on client side) at once but instead fetch the specified number of rows at a time.
  3. ResultSetType.ForwardOnly sets the type to allow results to be read sequentially so that the cursor will only move forward.
  4. ResultSetConcurrency.ReadOnly makes sure that the ResultSet may not be updated.
Only if all of the above is done will the streaming work properly for PostgreSQL else it won’t and the actions inside the stream behavior will fetch the entire dataset.
So, the database repository code base is now sorted out. Let’s focus on the controller and how it’ll stream this data to a downloadable file.
We can create a new Play controller for the purpose of managing all APIs related to the customers and this controller has access to the CustomerRepository we created earlier in which the customers method is defined and implemented.
We’ll use Play’s simple Result to stream the data to our client, i.e to the person administering the customers on /customers API (added to Play routes) by providing the customer stream to HttpEntity.Streamed case class like this
Result(
      header = ResponseHeader(OK, Map(CONTENT_DISPOSITION → s"attachment; filename=customers.csv")),
      body = HttpEntity.Streamed(csvSource, None, None))
The entire controller method would look something like this
def customers: Action[AnyContent] = Action { implicit request =>
  val customerDatabasePublisher = customerRepository.customers
  val customerSource = Source.fromPublisher(customerDatabasePublisher)

  val headerCSVSource = Source.single(ByteString(""""First Name","Last Name","Email"""" + "\n"))
  val customerCSVSource =
    customerSource.map(data => ByteString(s""""${data.firstName}","${data.lastName}","${data.email}"""" + "\n"))
  
  val csvSource = Source.combine(headerCSVSource, customerCSVSource)(Concat[ByteString])

  Result(
        header = ResponseHeader(OK, Map(CONTENT_DISPOSITION → s"attachment; filename=customers.csv")),
        body = HttpEntity.Streamed(csvSource, None, None))
}
Note that the DatabasePublisher[Customer] is converted to Source of Customer using the Source.fromPublisher helper method which is used to create a Source from Publisher.
Rest of the manipulations are done on the Source to convert the data into readable CSV file format.
Also, note the use of Source.combine method which is used to combine sources with fan-in strategy which in our case is Concat.
Hope this helps :)
The entire code base is available in the following repository playakkastreams.
This article was first published on the Knoldus blog.

NGINX – Understanding and Setting up a reverse proxy server

Let’s start by understanding what a reverse proxy server means and then I’ll lay down the steps for setting up such a server using our beloved NGINX.

A reverse proxy server is one which helps in directing client requests to other (usually backend) servers or to multiple applications hosted on the same server on different ports. You can think of it as an abstraction layer that re-routes traffic from the proxy server to your respective (aforementioned) setup.
But wait, why would one need such kind of a setup? A reverse proxy server helps in
  1. Loading balancing your application backend
  2. Establishing security by adding an additional (publicly accessible) abstraction layer hiding one or multiple (private) servers inside a local area network
  3. Hosting multiple applications on a single server using multi-port setup instead of resorting to IPv6
  4. Boosting performance by compressing traffic on the abstraction layer instead of your web servers
Moving onto the how part, let’s see how can such a reverse proxy server be setup easily for the use case wherein a single server has multiple web applications hosted on it (to understand the other use cases, i.e 1 and 2, you might want to refer to the following blog post – Loading balancing your application made easy). Let’s say two separate applications are running on your server at port 9000 and 9001. So the location of these applications from the perspective of your server would be http://localhost:9000and http://localhost:9001.
(For this example the NGINX setup used is on Ubuntu 16.04 LTS using NGINX version 1.10.0)
Begin by adding a new virtual server configuration file at /etc/nginx/sites-available where /etc/nginx is usually the location where NGINX is setup. Let’s name this file as example.com and add the following instructions to it
server {
  listen 80;
  server_name example.com www.example.com;

  location / {
    proxy_pass http://localhost:9000;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}
From the instructions we can see that the domain names example.com and http://www.example.com are redirecting the traffic to your application running at http://localhost:9000 on the server.
Now, add another virtual server configurations file at the same location but name that file as example.org and add the following instructions to it
server {
 listen 80;
 server_name example.org www.example.org;

 location / {
   proxy_pass http://localhost:9001;
   proxy_set_header Host $http_host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 }
}
You’ll notice there’s a difference in the instructions of this virtual server wherein domain example.org and http://www.example.org are pointing to a separate application running at http://localhost:9001.
Now create the soft links of the two files added to /etc/nginx/sites-available inside /etc/nginx/sites-enabled and restart NGINX.
Soft links to virtual server configuration files can be created using
sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com
and
sudo ln -s /etc/nginx/sites-available/example.org /etc/nginx/sites-enabled/example.org
With this setup all traffic from http://example.com and http://example.org will be proxied to http://localhost:9000 and http://localhost:9001 respectively using our NGINX virtual server configurations.
This article was first published on the Knoldus blog.

NGINX – Redirecting HTTP to HTTPS

Using HTTPS is highly recommended and I cannot stop when explaining the benefits of using it! Also something which is more important is to make sure when HTTPS is setup it is ensured that all traffic via HTTP is blocked in a way that it is redirected to HTTPS. This can be easily achieved by giving NGINX the following set of instructions in the virtual server configurations.

Begin by setting up the virtual server for your application so that a domain name points to it. A simple virtual server configuration using SSL is as follows
server {
 listen 443 ssl;
 server_name example.com;
 
 ssl_certificate /path/to/your/example.com.crt;
 ssl_certificate_key /path/to/your/example.com.key;

 root /path/to/your/content;
 index index.html;
 include /etc/nginx/mime.types;
}
Now setup the redirect from HTTP to HTTPS using:
server {
 listen 80;
 server_name example.com;
 return 301 https://example.com$request_uri;
}
this redirects http://example.com to https://example.com along with the URI in the request using the return directive and status code 301. For example,
http://example.com/user/1 would be redirected to https://example.com/user/1.
This article was first published on the Knoldus blog.

NGINX – Load Balancing your application made easy

Often there are requirements to load balance your application in order to scale out and make your application more performant and NGINX can be used to do just that! It can distribute your traffic to several application servers.

There are several load balancing methods available in NGINX which are
  1. round robin – wherein the requests are distributed in a round-robin fashion
  2. least connected – request is sent to the server with least number of active connections
  3. ip hash – uses a hash-function to determine a server for sending the request
  4. weighted – distributes requests based on servers’ weight specified
I’ll try to elaborate more on it by describing an architecture wherein there’s a set of static content which is made available via NGINX on a domain name and this static content has certain dynamic aspects which are loaded from your application backend. Application backend is what would be load balanced here.
Let’s start with our static content first and assume it is available at /path/to/your/content. A sample NGINX configuration for this would look something like
server {
 server_name example.com;

 root /path/to/your/content;
 index index.html;
 include /etc/nginx/mime.types;
}
where the domain example.com points to our static content available at /path/to/your/content.
Now, in order to render your dynamic content you’ll need a way in your static files to specifiy where that content should be made available from. For that you’ll need to provide the API details. But let’s try to do it via NGINX configurations. Let’s try to make your API available on the same domain as instructed in the configurations above but via URI which is /api. A way of doing it is by configuring the location directive in your server {} block configurations to offload all traffic from the specified URI to your application backend and the configuration of which could look something like
location /api/ {
 proxy_pass http://api/;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection 'upgrade';
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
proxy_pass – used to pass a request to a proxied server
proxy_http_version – sets the HTTP protocol version to 1.1
proxy_set_header – is used to set the headers from client to proxied server
But how do you instruct NGINX where /api points to? It is done using the upstreamdirective which has to be configured inside the http {} block usually located inside nginx.conf file. A sample configurations looks something like this
http {
 upstream api {
  server address-of-your-application-backend.com;
}
Once the aforementioned is done all traffic from http://example.com/api will be redirected to the upstream server as specified in the upstream configurations.
For example, the information of a user of your application is available on your application backend at /user/:id API. This information can now be easily accessed using http://example.com/api/user/1.
Henceforth, upstream is where things start getting interesting which is where the load balancing instructions are specified. You can specifiy multiple servers inside the upstream block
http {
 upstream api {
  server address-of-your-application-backend-1.com;
  server address-of-your-application-backend-2.com;
  server address-of-your-application-backend-3.com;
}
You can see that the load balancing method is not specified in the configurations above and it defaults to round-robin. All requests to get the dynamic data are now re-routed to servers specified in the upstream block in round-robin fashion.
Load balancing method can be specified inside the upstream block like this
http {
 upstream api {
  least_conn;
  server address-of-your-application-backend-1.com;
  server address-of-your-application-backend-2.com;
  server address-of-your-application-backend-3.com;
}
Hope this helps in better understanding of NGINX load balancing and how it can be achieved as a part of your application architecture.
This article was first published on the Knoldus blog.